Methods and apparatuses for prediction refinement with optical flow, bi-directional optical flow, and decoder-side motion vector refinement

By aligning bit depth and workflow for BDOF and PROF in video encoding, the solution addresses inefficiencies in VVC, enhancing coding efficiency and facilitating hardware implementation for improved video encoding and decoding.

JP2024167438AActive Publication Date: 2024-12-03BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2024158573
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2019-10-09
Filing Date
2024-09-12
Publication Date
2024-12-03
Estimated Expiration
2040-10-09

AI Technical Summary

Technical Problem

Existing video encoding techniques, such as those in the Versatile Video Coding (VVC) standard, face inefficiencies in motion compensation due to limitations in bit depth representation and workflow harmonization between Bidirectional Optical Flow (BDOF) and Predictive Refinement with Optical Flow (PROF), leading to suboptimal hardware implementation and coding efficiency.

Method used

The proposed solution integrates BDOF and PROF designs by aligning bit depth representations, harmonizing workflows for bi-prediction, and optimizing sub-block sizes to facilitate hardware pipeline design, while introducing unified control flags for enabling/disabling these tools.

Benefits of technology

This integration enhances coding efficiency by allowing shared pipeline stages, reduces computational complexity, and improves accuracy in motion refinement, resulting in more efficient video encoding and decoding processes.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024167438000001_ABST
    Figure 2024167438000001_ABST
Patent Text Reader

Abstract

To provide a bit-depth representation method, apparatus and non-transitory computer-readable storage medium for depth control for a bi-directional optical flow.SOLUTION: A decoder is configured to: obtain a reference picture I associated with a video block having a video signal; obtain prediction samples I(i,j) of the video block from a reference block in the reference picture I; control internal prediction refinement (PROF) parameters of a PROF derivation process by applying a right shift to the internal PROF parameters based on a bit-shift value to achieve a preset precision; obtain prediction refinement values for samples in the video block based on the PROF derivation process being applied to the video block based on the prediction samples I(i,j); and obtain prediction samples of the video block based on the combination of the prediction samples and the prediction refinement values.SELECTED DRAWING: Figure 10
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to Provisional Application No. 62 / 913,141, filed October 9, 2019, the entirety of which is incorporated herein by reference for all purposes.

[0002] This disclosure relates to video coding and compression. More specifically, this disclosure relates to methods and apparatus for two inter-prediction tools being investigated in the Versatile Video Coding (VVC) standard: prediction refinement with optical flow (PROF) and bi-directional optical flow (BDOF). [Background technology]

[0003] Various video encoding techniques may be used to compress video data. Video encoding is performed according to one or more video encoding standards. For example, video encoding standards include Versatile Video Coding (VVC), Joint Search Test Model (JEM), High Efficiency Video Coding (H.265 / HEVC), Advanced Video Coding (H.264 / AVC), Moving Picture Experts Group (MPEG) coding, and the like. Video encoding generally utilizes prediction methods (e.g., inter-prediction, intra-prediction, and the like) that exploit redundancy present in a video image or sequence. An important goal of video encoding techniques is to compress video data into a form that uses a lower bit rate while avoiding or minimizing degradation of video quality. Summary of the Invention [Problem to be solved by the invention]

[0004] Examples of this disclosure provide methods and apparatus for bit depth control for bidirectional optical flow. [Means for solving the problem]

[0005] According to a first aspect of the disclosure, a bit depth representation method of PROF is provided. The method may include a decoder obtaining a reference picture I associated with a video block in a video signal. The decoder may also obtain a prediction sample I(i,j) of the video block from a reference block in the reference picture I. i and j represent coordinates of one sample with the video block. The decoder may further control an internal PROF parameter of a PROF derivation process by applying a right shift to the internal PROF parameter based on a bit shift value to achieve a preset accuracy. The internal PROF parameter may include a horizontal gradient value, a vertical gradient value, a horizontal motion differential value, and a vertical motion differential value derived for the prediction sample I(i,j). The decoder may also obtain a prediction refinement value for a sample in the video block based on the PROF derivation process applied to the video block based on the prediction sample I(i,j). The decoder may further obtain a prediction sample of the video block based on a combination of the prediction sample and the prediction refinement value.

[0006] According to a second aspect of the present disclosure, a method for controlling bit depth of a BDOF is provided. The method includes: a decoder detects a first reference picture I associated with a video block; (0) and the second reference picture I (1) In display order, the first reference picture I (0) is before the current picture, and the second reference picture I (1) is after the current picture. The decoder also finds the first reference picture I (0) The first predicted sample of a video block from a reference block in (0) (i,j), where i and j represent the coordinates of one sample with the current picture. The decoder may obtain the second reference picture I (1) The second predicted sample of the video block from the reference block in (1)(i,j). The decoder may also control the internal BDOF parameter of the BDOF derivation process by applying a shift to the internal BDOF parameter. The internal BDOF parameter may be calculated based on the first predicted sample I (0) (i,j), the second prediction sample I (1) (i,j), the first prediction sample I (0) (i,j) and the second prediction sample I (1) (i,j) and horizontal and vertical gradient values ​​derived based on the sample difference between (i,j) and intermediate BDOF derivation parameters. The intermediate BDOF derivation parameters may include sGxdI, sGydI, sGx2, sGxGy, and sGy2 parameters. sGxdI and sGydI may include cross-correlation values ​​between the horizontal gradient values ​​and the sample difference values ​​and between the vertical gradient values ​​and the sample difference values. sGx2 and sGy2 may include auto-correlation values ​​of the horizontal and vertical gradient values. sGxGy may include cross-correlation values ​​between the horizontal and vertical gradient values.

[0007] According to a third aspect of the present disclosure, a method of BDOF, PROF, and DMVR is provided. The method may include a decoder receiving three control flags in a sequence parameter set (SPS). A first control flag indicates whether BDOF is enabled for decoding video blocks in a current video sequence. A second control flag indicates whether PROF is enabled for decoding video blocks in the current video sequence. A third control flag indicates whether DMVR is enabled for decoding video blocks in the current video sequence. The decoder may also receive a first presence flag in the SPS when the first control flag is true, a second presence flag in the SPS when the second control flag is true, and a third presence flag in the SPS when the third control flag is true. The decoder may further receive a first picture control flag in a picture header of each picture when the first presence flag in the SPS indicates that BDOF is disabled for video blocks in the picture. The decoder may also receive a second picture control flag in the picture header of each picture when the second presence flag in the SPS indicates that PROF is disabled for video blocks in the picture. The decoder may further receive a third picture control flag in the picture header of each picture when the third presence flag in the SPS indicates that DMVR is disabled for video blocks in the picture.

[0008] According to a fourth aspect of the present disclosure, a computing device is provided. The computing device may include one or more processors and a non-transitory computer-readable memory storing instructions executable by the one or more processors. The one or more processors may be configured to obtain a reference picture I associated with a video block in a video signal. The one or more processors may also be configured to obtain a prediction sample I(i,j) of the video block from a reference block in the reference picture I, where i and j represent coordinates of one sample in the video block. The one or more processors may be further configured to control an internal PROF parameter of a PROF derivation process by applying a right shift to the internal PROF parameter based on a bit shift value to achieve a preset accuracy. The internal PROF parameter may include a horizontal gradient value, a vertical gradient value, a horizontal motion differential value, and a vertical motion differential value derived for the prediction sample I(i,j). The one or more processors may also be configured to obtain a prediction refinement value for a sample in the video block based on a PROF derivation process applied to the video block based on the prediction sample I(i,j). The one or more processors may be further configured to obtain a prediction sample for the video block based on a combination of the prediction sample and the prediction refinement value.

[0009] According to a fifth aspect of the present disclosure, a computing device is provided. The computing device may include one or more processors and a non-transitory computer-readable memory that stores instructions executable by the one or more processors. The one or more processors may generate a first reference picture I associated with a video block. (0) and the second reference picture I (1) In the display order, the first reference picture I (0) is before the current picture, and the second reference picture I (1) The one or more processors may select the first reference picture I (0)The first predicted sample of a video block from a reference block in (0) The one or more processors may be further configured to obtain a prediction refinement value for a sample in the video block based on the prediction sample I(i,j), where i and j represent coordinates of one sample with the current picture. The one or more processors may also be configured to obtain a prediction refinement value for a sample in the video block based on the PROF derivation process applied to the video block based on the prediction sample I(i,j). The one or more processors may further be configured to obtain a prediction refinement value for a sample in the video block based on the prediction sample I(i,j), where i and j represent coordinates of one sample with the current picture. (1) The second predicted sample of the video block from the reference block in (1) The one or more processors may be further configured to obtain an internal BDOF parameter of the BDOF derivation process by applying a shift to the internal BDOF parameter. The internal BDOF parameter may be obtained by applying a shift to the first predicted sample I (0) (i,j), the second prediction sample I (1) (i,j), the first prediction sample I (0) (i,j) and the second prediction sample I (1) (i,j) and horizontal and vertical gradient values ​​derived based on the intermediate BDOF derivation parameters. The intermediate BDOF derivation parameters may include sGxdI, sGydI, sGx2, sGxGy, and sGy2 parameters. sGxdI and sGydI may include cross-correlation values ​​between the horizontal gradient values ​​and the sample difference values ​​and between the vertical gradient values ​​and the sample difference values. sGx2 and sGy2 may include auto-correlation values ​​of the horizontal and vertical gradient values. sGxGy may include cross-correlation values ​​between the horizontal and vertical gradient values. The one or more processors may generate a first predicted sample I (0) (i,j) and the second prediction sample I (1) The one or more processors may be further configured to obtain a motion refinement for samples within the video block based on the BDOF applied to the video block based on (i,j). The one or more processors may be further configured to obtain bi-predictive samples of the video block based on the motion refinement.

[0010] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium having instructions stored thereon is provided. When the instructions are executed by one or more processors of the device, the instructions may cause the device to receive three control flags in a sequence parameter set (SPS). The first control flag indicates whether BDOF is enabled for decoding video blocks in a current video sequence. The second control flag indicates whether PROF is enabled for decoding video blocks in the current video sequence. The third control flag indicates whether DMVR is enabled for decoding video blocks in the current video sequence. The instructions may also cause the device to receive a first presence flag in the SPS when the first control flag is true, a second presence flag in the SPS when the second control flag is true, and a third presence flag in the SPS when the third control flag is true. The instructions may further cause the device to receive a first picture control flag in a picture header of each picture when the first presence flag in the SPS indicates that BDOF is disabled for video blocks in the picture. The instructions may also cause the device to receive a second picture control flag in a picture header of each picture when the second presence flag in the SPS indicates that PROF is disabled for video blocks in the picture. The instructions may further cause the device to receive a third picture control flag in a picture header of each picture when a third presence flag in the SPS indicates that DMVR is disabled for video blocks in the picture.

[0011] It is to be understood that both the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present disclosure.

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure. [Brief description of the drawings]

[0013] [Figure 1]FIG. 2 is a block diagram of an encoder according to an example of the present disclosure. [Diagram 2] FIG. 2 is a block diagram of a decoder according to an example of the present disclosure. [Figure 3A] FIG. 2 illustrates an example of block partitioning in a multi-type tree structure according to an example of the present disclosure. [Figure 3B] FIG. 2 illustrates an example of block partitioning in a multi-type tree structure according to an example of the present disclosure. [Figure 3C] FIG. 2 illustrates an example of block partitioning in a multi-type tree structure according to an example of the present disclosure. [Figure 3D] FIG. 2 illustrates an example of block partitioning in a multi-type tree structure according to an example of the present disclosure. [Figure 3E] FIG. 2 illustrates an example of block partitioning in a multi-type tree structure according to an example of the present disclosure. [Figure 4] 1 is a diagram illustration of a bidirectional optical flow (BDOF) model according to an example of the present disclosure. [Figure 5A] 1 is an illustration of an affine model according to an example of the present disclosure. [Figure 5B] 1 is an illustration of an affine model according to an example of the present disclosure. [Figure 6] 1 is an illustration of an affine model according to an example of the present disclosure. [Figure 7] 1 is an illustration of prediction refinement by optical flow (PROF) according to an example of the present disclosure. [Figure 8] 1 is a BDOF workflow according to an example of the present disclosure. [Figure 9] 1 is a workflow of PROF according to an example of the present disclosure. [Figure 10] 1 is a BDOF method according to an example of the present disclosure. [Figure 11] 1 is a method of BDOF and PROF according to an example of the present disclosure. [Figure 12] 1 is a method of BDOF, PROF, and DMVR according to an example of the present disclosure. [Figure 13] 1 is an illustration of a workflow of PROF for bi-prediction according to an example of the present disclosure. [Figure 14] 1 is an illustration of pipeline stages of the BDOF and PROF process according to the present disclosure. [Figure 15] 1 is an example of a BDOF gradient derivation method according to the present disclosure. [Figure 16] 1 is an example of a method for deriving the gradient of PROF according to the present disclosure. [Figure 17A] 11 is an illustration of deriving template samples for an affine mode according to an example of the present disclosure. [Figure 17B] 11 is an illustration of deriving template samples for an affine mode according to an example of the present disclosure. [Figure 18A] 13 is an illustration of exclusively enabling PROF and LIC for affine mode according to an example of the present disclosure. [Figure 18B] 13 is an illustration of jointly enabling PROF and LIC for affine mode according to an example of the present disclosure. [Figure 19A] FIG. 1 illustrates a proposed padding method applied to a 16×16 BDOF CU according to an example of the present disclosure. [Figure 19B] FIG. 1 illustrates a proposed padding method applied to a 16×16 BDOF CU according to an example of the present disclosure. [Figure 19C] FIG. 1 illustrates a proposed padding method applied to a 16×16 BDOF CU according to an example of the present disclosure. [Figure 19D] FIG. 1 illustrates a proposed padding method applied to a 16×16 BDOF CU according to an example of the present disclosure. [Figure 20] FIG. 1 illustrates an exemplary computing environment coupled with a user interface according to an example of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings, in which the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following description of the exemplary embodiments do not represent all implementations consistent with the present disclosure. Instead, these implementations are merely examples of apparatus and methods consistent with aspects related to the present disclosure as recited in the appended claims.

[0015] The terms used in this disclosure are intended to describe only certain embodiments and are not intended to limit the disclosure. When used in this disclosure and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein means and includes any or all possible combinations of one or more of the associated listed items.

[0016] While terms such as "first," "second," and "third" may be used herein to describe various information, it should be understood that the information should not be limited by these terms. These terms are used only to distinguish one category of information from another category of information. For example, a first piece of information may be referred to as a second piece of information, and similarly, a second piece of information may be referred to as a first piece of information, without departing from the scope of this disclosure. As used herein, the term "if" may be understood to mean "when" or "upon" or "in response to a judgment," depending on the context.

[0017] The first version of the HEVC standard, which offers approximately 50% bitrate savings or equivalent perceptual quality compared to the previous generation video coding standard H.264 / MPEG AVC, was finalized in October 2013. While the HEVC standard offers significant coding improvements over its predecessors, there is evidence that better coding efficiency than HEVC can be achieved with additional coding tools. Based on that, both VCEG and MPEG have started exploration work on new coding techniques for future video coding standardization. A Joint Video Exploration Team (JVET) was formed in October 2015 by ITU-T VECG and ISO / IEC MPEG to initiate significant research into advanced technologies that can enable considerable improvements in coding efficiency. A reference software called the Joint Exploration Model (JEM) was maintained by JVET by integrating several additional coding tools on top of the HEVC Test Model (HM).

[0018] In October 2017, a joint call for proposals (CfP) for video compression with capabilities beyond HEVC was issued by ITU-T and ISO / IEC[9]. In April 2018, 23 CfP responses were received and evaluated at the 10th JVET meeting, demonstrating approximately 40% compression efficiency gain over HEVC. Based on such evaluation results, JVET launched a new project to develop a new generation video coding standard named Versatile Video Coding (VVC). In the same month, one reference software code base called VVC Test Model (VTM) was established to demonstrate a reference implementation of the VVC standard.

[0019] Like HEVC, VVC is built on a block-based hybrid video coding framework.

[0020] Figure 1 shows a schematic diagram of a block-based video encoder for VVC. Specifically, Figure 1 shows an exemplary encoder 100. The encoder 100 has a video input 110, motion compensation 112, motion estimation 114, intra / inter mode decision 116, block predictor 140, adder 128, transform 130, quantization 132, prediction related information 142, intra prediction 118, picture buffer 120, inverse quantization 134, inverse transform 136, adder 126, memory 124, in-loop filter 122, entropy coding 138, and bitstream 144.

[0021] At encoder 100, a video frame is partitioned into multiple video blocks for processing. For a given video block, a prediction is formed based on either an inter-prediction technique or an intra-prediction technique.

[0022] A prediction residual, which represents the difference between a current video block that is part of the video input 110 and its predictor that is part of the block predictor 140, is sent from the adder 128 to a transform 130. The transform coefficients are then sent from the transform 130 to a quantizer 132 for entropy reduction. The quantized coefficients are then provided to an entropy coder 138 to generate a compressed video bitstream. As shown in FIG. 1, prediction related information 142 from the intra / inter mode decision 116, such as video block partition information, motion vectors (MVs), reference picture indexes, and intra prediction modes, are also provided through the entropy coder 138 and stored in the compressed bitstream 144. The compressed bitstream 144 comprises a video bitstream.

[0023] Decoder-related circuitry is also required in the encoder 100 to reconstruct pixels for prediction purposes. First, a prediction residual is reconstructed through inverse quantization 134 and inverse transform 136. This reconstructed prediction residual is combined with a block predictor 140 to generate an unfiltered reconstructed pixel for the current video block.

[0024] Spatial prediction (or "intra prediction") uses pixels from already coded samples of neighboring blocks (called reference samples) in the same video frame as the current video block to predict the current video block.

[0025] Temporal prediction (also called "inter prediction") uses reconstructed pixels from already coded video pictures to predict a current video block. Temporal prediction reduces the temporal redundancy inherent in video signals. A temporal prediction signal for a given coding unit (CU) or coding block is typically signaled by one or more MVs that indicate the amount and direction of motion between the current CU and its temporal references. Furthermore, if multiple reference pictures are supported, one reference picture index is additionally transmitted that is used to identify which reference picture in the reference picture storage the temporal prediction signal comes from.

[0026] Motion estimation 114 takes in signals from video input 110 and picture buffer 120 and outputs a motion estimation signal to motion compensation 112. Motion compensation 112 takes in signals from video input 110, picture buffer 120 and a motion estimation signal from motion estimation 114 and outputs a motion compensation signal to intra / inter mode decision 116.

[0027] After spatial prediction and / or temporal prediction are performed, an intra / inter mode decision 116 in the encoder 100 chooses the best prediction mode, for example based on a rate-distortion optimization method. The block predictor 140 is then subtracted from the current video block, and the resulting prediction residual is decorrelated using transform 130 and quantization 132. The resulting quantized residual coefficients are inverse quantized by inverse quantization 134 and inverse transformed by inverse transform 136 to form a reconstructed residual, which is then added back to the prediction block to form a reconstructed signal for the CU. Additionally, in-loop filtering 122, such as a deblocking filter, sample adaptive offset (SAO), and / or adaptive in-loop filter (ALF), may be applied on the reconstructed CU before it is placed into reference picture storage in the picture buffer 120 and used to encode future video blocks. To form the output video bitstream 144, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to the entropy coding unit 138 for further compression and packing to form the bitstream.

[0028] Figure 1 gives a block diagram of a general block-based hybrid video coding system. The input video signal is processed block by block (called CU). In VTM-1.0, a CU can be up to 128x128 pixels. However, unlike HEVC, which partitions blocks based on quadtree only, in VVC, one coding tree unit (CTU) is split into CUs based on quadtree / binary tree / ternary tree to fit varying local characteristics. In addition, the concept of multiple partitioning unit types in HEVC is removed, i.e., the distinction between CU, prediction unit (PU), and transform unit (TU) no longer exists in VVC, and instead, each CU is always used as a basic unit for both prediction and transformation without further partitioning. In the multi-type tree structure, one CTU is first partitioned by a quadtree structure. Then, each quadtree leaf node can be further partitioned by binary and ternary tree structures. As shown in Figures 3A, 3B, 3C, 3D, and 3E, there are five split types: quadrant, horizontal bisection, vertical bisection, horizontal thirds, and vertical thirds.

[0029] FIG. 3A shows a diagram illustrating block quadrants in a multi-type tree structure in accordance with the present disclosure.

[0030] FIG. 3B shows a diagram illustrating block vertical bisections in a multi-type tree structure in accordance with the present disclosure.

[0031] FIG. 3C shows a diagram illustrating block horizontal bisection in a multi-type tree structure in accordance with the present disclosure.

[0032] FIG. 3D shows a diagram illustrating block vertical third divisions in a multi-type tree structure in accordance with the present disclosure.

[0033] FIG. 3E shows a diagram illustrating horizontal third division of blocks in a multi-type tree structure in accordance with the present disclosure.

[0034] In FIG. 1, spatial prediction and / or temporal prediction may be performed. Spatial prediction (or "intra prediction") uses pixels from samples of already coded neighboring blocks (called reference samples) in the same video picture / slice to predict the current video block. Spatial prediction reduces spatial redundancy inherent in video signals. Temporal prediction (also called "inter prediction" or "motion compensated prediction") uses reconstructed pixels from already coded video pictures to predict the current video block. Temporal prediction reduces temporal redundancy inherent in video signals. The temporal prediction signal for a given CU is typically signaled by one or more motion vectors (MVs) that indicate the amount and direction of motion between the current CU and its temporal references. Also, if multiple reference pictures are supported, one reference picture index is additionally transmitted that is used to identify which reference picture in the reference picture store the temporal prediction signal comes from. After spatial prediction and / or temporal prediction, a mode decision block in the encoder chooses the best prediction mode, for example based on a rate-distortion optimization method. The prediction block is then subtracted from the current video block, and the prediction residual is decorrelated using transform and quantization. The quantized residual coefficients are inverse quantized and inverse transformed to form a reconstructed residual, which is then added back to the prediction block to form a reconstructed signal of the CU. Furthermore, in-loop filtering, such as a deblocking filter, sample adaptive offset (SAO), and adaptive in-loop filter (ALF), may be applied to the reconstructed CU before it is put into the reference picture store and used to encode future video blocks. To form an output video bitstream, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to an entropy coding unit for further compression and packing to form a bitstream.

[0035] Figure 2 shows a schematic block diagram of a video decoder for VVC. Specifically, Figure 2 shows a block diagram of an exemplary decoder 200. The decoder 200 includes a bitstream 210, an entropy decoding 212, an inverse quantization 214, an inverse transform 216, an adder 218, an intra / inter mode selection 220, an intra prediction 222, a memory 230, an in-loop filter 228, a motion compensation 224, a picture buffer 226, prediction related information 234, and a video output 232.

[0036] The decoder 200 is similar to the reconstruction-related section in the encoder 100 of FIG. 1. In the decoder 200, an incoming video bitstream 210 is first decoded through entropy decoding 212 to derive quantized coefficient levels and prediction-related information. The quantized coefficient levels are then processed through inverse quantization 214 and inverse transform 216 to obtain a reconstructed prediction residual. A block predictor mechanism implemented in an intra / inter mode selector 220 is configured to perform either intra prediction 222 or motion compensation 224 based on the decoded prediction information. A set of unfiltered reconstructed pixels is obtained by summing the reconstructed prediction residual from the inverse transform 216 and the prediction output generated by the block predictor mechanism using a summer 218.

[0037] The reconstructed blocks may further pass through an in-loop filter 228 before being stored in a picture buffer 226, which serves as a reference picture store. The reconstructed video in the picture buffer 226 may be transmitted to drive a display device, as well as used to predict future video blocks. In situations where the in-loop filter 228 is turned on, a filtering operation is performed on these reconstructed pixels to derive the final reconstructed video output 232.

[0038] FIG. 2 gives a schematic block diagram of a block-based video decoder. The video bitstream is first entropy decoded in an entropy decoding unit. The coding mode and prediction information are sent to either a spatial prediction unit (if intra-coded) or a temporal prediction unit (if inter-coded) to form a prediction block. The residual transform coefficients are sent to an inverse quantization unit and an inverse transform unit to reconstruct the residual block. The prediction block and the residual block are then summed. The reconstructed block may further pass through in-loop filtering before being stored in a reference picture storage. The reconstructed video in the reference picture storage is then sent to drive a display device, as well as used to predict future video blocks.

[0039] In general, the basic inter prediction technique applied in VVC is kept the same as that of HEVC, except that some modules are further extended and / or enhanced. In particular, for all previous video standards, one coding block may be associated with only one MV when the coding block is uni-predicted, or only two MVs when the coding block is bi-predicted. Due to such limitations of traditional block-based motion compensation, small motion may still remain in the prediction samples after motion compensation, thus adversely affecting the overall efficiency of motion compensation. In order to improve both the granularity and accuracy of MVs, two sample-wise refinement methods based on optical flow, namely, bidirectional optical flow (BDOF) for affine mode and prediction refinement by optical flow (PROF), are currently being researched for the VVC standard. In the following, the main technical aspects of the two inter coding tools are briefly reviewed.

[0040] Bidirectional Optical Flow In VVC, BDOF is applied to refine the prediction samples of bi-predicted coding blocks. Specifically, as shown in Figure 4, BDOF is a sample-by-sample motion refinement performed on top of block-based motion compensation prediction when bi-prediction is used.

[0041] FIG. 4 illustrates an example of a BDOF model in accordance with the present disclosure.

[0042] Motion refinement for each 4 × 4 subblock (v x ,v y ) is calculated by minimizing the difference between the L0 predicted sample and the L1 predicted sample after BDOF is applied within a 6 × 6 window Ω around the subblock. Specifically, (v x ,v y ) is

number

[0043] The values ​​of S1, S2, S3, S5 and S6 are

number

number

number

[0044] Based on the motion refinement derived in (1), the final bi-predictive sample of the CU is

number

[0045] Affine Mode In HEVC, only a translational motion model is applied for motion compensation prediction. Meanwhile, in the real world, there are many kinds of motions, such as zoom in / out, rotation, perspective motion, and other irregular motions. In VVC, affine motion compensation prediction is applied by signaling one flag per inter-coded block to indicate whether a translational motion model or an affine motion model is applied for inter prediction. In the current VVC design, two affine modes, including a 4-parameter affine mode and a 6-parameter affine mode, are supported for one affine-coded block.

[0046] The four-parameter affine model has the following parameters: two parameters for translational motion in horizontal and vertical directions, respectively, one parameter for zoom motion and one parameter for rotation motion in both directions. The horizontal zoom parameter is equal to the vertical zoom parameter. The horizontal rotation parameter is equal to the vertical rotation parameter. To achieve better adaptation of the motion vectors and affine parameters, in VVC, those affine parameters are transformed into two MVs (also called control point motion vectors (CPMVs)) located at the top-left and top-right corners of the current block. As shown in Figures 5A and 5B, the affine motion field of a block is described by two control point MVs (V0, V1).

[0047] FIG. 5A illustrates an example of a four-parameter affine model in accordance with the present disclosure.

[0048] FIG. 5B illustrates an example of a four-parameter affine model in accordance with the present disclosure.

[0049] Based on the control point motion, the motion field of one affine coded block (v x ,v y )teeth,

number

[0050] The 6-parameter affine mode has the following parameters: two parameters for translational motion in the horizontal and vertical directions, one parameter for zoom motion in the horizontal direction and one parameter for rotational motion, and one parameter for zoom motion in the vertical direction and one parameter for rotational motion. The 6-parameter affine motion model is coded using 3 MVs in 3 CPMVs.

[0051] FIG. 6 illustrates an example of a six-parameter affine model in accordance with the present disclosure.

[0052] As shown in Figure 6, the three control points of one six-parameter affine block are at the top-left, top-right, and bottom-left corners of the block. The motion at the top-left control point is for translation motion, the motion at the top-right control point is for rotation and zoom motion in the horizontal direction, and the motion at the bottom-left control point is for rotation and zoom motion in the vertical direction. Compared with the four-parameter affine motion model, the rotation and zoom motion in the horizontal direction of six parameters may not be the same as those in the vertical direction. Suppose (V0, V1, V2) are the MVs of the top-left, top-right, and bottom-left corners of the current block in Figure 6, the motion vectors (v x ,v y ) is calculated using three MVs at the control points.

number

[0053] Optical flow based prediction refinement for affine modes In order to improve the affine motion compensation accuracy, PROF, which refines sub-block-based affine motion compensation based on the optical flow model, is currently being studied in the current VVC. Specifically, after implementing sub-block-based affine motion compensation, the luma prediction sample of one affine block is modified by one sample refinement value derived based on the optical flow equation. In detail, the operation of PROF can be summarized as the following four things:

[0054] Step 1: Sub-block based affine motion compensation is performed to generate sub-block predictions I(i,j) using the sub-block MVs derived in (6) for the 4-parameter affine model and (7) for the 6-parameter affine model.

[0055] Step 2: The spatial gradients of each prediction sample g x (i,j) and g y(i,j) is

number

[0056] To compute the gradient, it is necessary that one additional row / column of prediction samples is generated on each side of one subblock. To reduce memory bandwidth and complexity, the samples on the extended boundary are copied from the nearest integer pixel location in the reference picture to avoid an additional interpolation process.

[0057] Step 3: The luma prediction refinement value is

number

[0058] 7 illustrates a PROF process for affine mode according to the present disclosure. FIG 7 includes blocks 710, 720, and 730. Block 730 is a rotated version of block 720.

[0059] Since the affine model parameters and pixel locations relative to the subblock center do not change by subblock, Δv(i,j) can be calculated for the first subblock and reused for other subblocks in the same CU. Let Δx and Δy be the horizontal and vertical offsets from the sample location (i,j) to the center of the subblock to which the sample belongs, then Δv(i,j) is

number

[0060] Based on the affine sub-block MV derivation equations (6) and (7), the MV difference Δv(i,j) can be derived. Specifically, for the four-parameter affine model, JPEG2024167438000015.jpg20117 It is.

[0061] For a six-parameter affine model, JPEG2024167438000016.jpg39117, where (v 0x ,v 0y ), (v 1x ,v 1y ), (v 2x ,v 2y ) are the top-left, top-right, and bottom-left control points MV of the current coding block, and w and h are the width and height of the block. In the existing PROF design, the MV difference Δv x and Δv y is always derived to 1 / 32-pel accuracy.

[0062] Local Lighting Compensation Local illumination compensation (LIC) is a coding tool used to address the problem of local illumination changes that exist between temporally adjacent pictures. A pair of weight and offset parameters is applied to a reference sample to obtain a predicted sample of one current block. The general mathematical model is:

number

number

[0063] In addition to being applied to normal inter blocks that contain at most one motion vector per prediction direction (L0 or L1), LIC is also applied to affine mode coded blocks, where one coding block is further split into multiple smaller sub-blocks, and each sub-block may be associated with different motion information. To derive reference samples for LIC of affine mode coded blocks, as shown in Figures 17A and 17B (described below), the reference samples in the top template of one affine coded block are fetched using the motion vectors of each sub-block in the top sub-block row, while the reference samples in the left template are fetched using the motion vectors of the sub-blocks in the left sub-block column. Then, the same LLMSE derivation method as shown in (12) is applied to derive LIC parameters based on the composite template.

[0064] 17A shows an example for deriving a template sample for the affine mode according to the present disclosure. This example includes a Cur Frame 1720 and a Cur CU 1722. Cur Frame 1720 is the current frame. Cur CU 1722 is the current coding unit.

[0065] FIG. 17B shows an example for deriving template samples for affine mode. This example includes Ref Frame 1740, Col CU 1742, A Ref 1743, B Ref 1744, C Ref 1745, D Ref 1746, E Ref 1747, F Ref 1748, and G Ref 1749. Ref Frame 1740 is a reference frame. Col CU 1742 is a collocated coding unit. A Ref 1743, B Ref 1744, C Ref 1745, D Ref 174,6 E Ref 1747, F Ref 1748, and G Ref 1749 are reference samples.

[0066] Inefficiency of prediction refinement by optical flow for affine modes. Although PROF can improve the coding efficiency of the affine mode, its design can still be further improved. In particular, considering the fact that both PROF and BDOF are built on the concept of optical flow, it is highly desirable to harmonize the designs of PROF and BDOF as much as possible so that PROF can make the most of the existing logic of BDOF to facilitate hardware implementation. Based on such consideration, the following inefficiencies regarding the interaction between the current PROF design and BDOF design are identified in this disclosure.

[0067] First, as explained in the section "Optical Flow-Based Prediction Refinement for Affine Modes", in equation (8), the accuracy of the gradient is determined based on the internal bit depth. Meanwhile, the MV difference, i.e., Δv x and Δv y is always derived with an accuracy of 1 / 32 pel. Correspondingly, based on equation (9), the accuracy of the derived PROF refinement depends on the internal bit depth. However, similar to BDOF, in order to keep higher PROF derivation accuracy, PROF is applied on the predicted sample values ​​with an intermediate high bit depth (i.e., 16 bits). Therefore, regardless of the internal coding bit depth, the accuracy of the prediction refinement derived by PROF should match the accuracy of the intermediate predicted samples, i.e., 16 bits. In other words, the representation bit depth of the MV difference and gradient in the existing PROF design is not perfectly suited to derive accurate prediction refinement compared to the predicted sample accuracy (i.e., 16 bits). Meanwhile, based on the comparison of equations (1), (4), and (8), the existing PROF and BDOF use different accuracy to represent sample gradient and MV difference. As pointed out before, such a non-integrated design is not desirable for hardware because the existing BDOF logic cannot be reused.

[0068] Second, as discussed in the section "Optical Flow Prediction Refinement for Affine Mode", when one current affine block is bi-predicted, PROF is applied to the prediction samples in lists L0 and L1 separately, and then the extended L0 and L1 prediction signals are averaged to generate the final bi-prediction signal. On the contrary, instead of deriving PROF refinement separately for each prediction direction, BDOF derives prediction refinement at once, and then the prediction refinement is applied to extend the combined L0 and L1 prediction signals. Figures 8 and 9 (described below) compare the current BDOF and PROF workflows for bi-prediction. In real codec hardware pipeline designs, different main encoding / decoding modules are usually assigned to each pipeline stage so that more coding blocks can be processed in parallel. However, due to the difference between the BDOF workflow and the PROF workflow, this may lead to the difficulty of having one and the same pipeline design that can be shared by BDOF and PROF, which is not favorable for real codec implementation.

[0069] FIG. 8 illustrates a workflow of BDOF according to the present disclosure. The workflow 800 includes L0 motion compensation 810, L1 motion compensation 820, and BDOF 830. The L0 motion compensation 810 may be, for example, a list of motion compensation samples from a previous reference picture. The previous reference picture is a reference picture before the current picture in the video block. The L1 motion compensation 820 may be, for example, a list of motion compensation samples from a next reference picture. The next reference picture is a reference picture after the current picture in the video block. The BDOF 830 takes the motion compensation samples from the L1 motion compensation 810 and the L1 motion compensation 820 and outputs prediction samples, as described above with respect to FIG. 4.

[0070] FIG. 9 illustrates a workflow of an existing PROF according to the present disclosure. The workflow 900 includes L0 motion compensation 910, L1 motion compensation 920, L0 PROF 930, L1 PROF 940, and average 960. The L0 motion compensation 910 may be, for example, a list of motion compensation samples from a previous reference picture. The previous reference picture is a reference picture before a current picture in a video block. The L1 motion compensation 920 may be, for example, a list of motion compensation samples from a next reference picture. The next reference picture is a reference picture after a current picture in a video block. The L0 PROF 930 takes the L0 motion compensation samples from the L0 motion compensation 910 as described above with respect to FIG. 7 and outputs a motion refinement value. The L1 PROF 940 takes the L1 motion compensation samples from the L1 motion compensation 920 as described above with respect to FIG. 7 and outputs a motion refinement value. AVERAGE 960 averages the motion refinement value output of L0 PROF 930 and L1 PROF 940.

[0071] Third, for both BDOF and PROF, gradients need to be calculated for each sample inside the current coding block, which requires generating one additional row / column of predicted samples on each side of the block. To avoid the additional computational complexity of sample interpolation, predicted samples in the extension region around the block are directly copied from reference samples at integer positions (i.e., without interpolation). However, according to existing designs, integer samples at different locations are selected to generate gradient values ​​for BDOF and PROF. Specifically, for gradient calculation, integer reference samples to the left of the predicted sample (for horizontal gradients) and above the predicted sample (for vertical gradients) are used for BDOF, and integer reference samples closest to the predicted sample are used for PROF. Similar to the bit depth representation problem, such a non-integrated gradient calculation method is also undesirable for hardware codec implementation.

[0072] Fourth, as pointed out before, the motivation of PROF is to compensate for small MV differences between the MV of each sample and the sub-block MV derived at the center of the sub-block to which the sample belongs. According to the current PROF design, PROF is always invoked when one coding block is predicted by affine mode. However, as shown in equations (6) and (7), the sub-block MV of one affine block is derived from the control point MV. Therefore, when the difference between the control point MVs is relatively small, the MV at each sample position should be consistent. In such cases, the benefit of applying PROF may be very limited, so considering the performance / complexity tradeoff, it may not be worth doing PROF.

[0073] Improved prediction refinement using optical flow for affine modes In this disclosure, a method is provided to improve and simplify the existing PROF design to facilitate hardware codec implementation. In particular, special attention is paid to harmonizing the BDOF and PROF designs to maximize sharing of existing BDOF logic with PROF. In general, the main aspects of the proposed technology in this disclosure are summarized as follows:

[0074] First, in order to improve the coding efficiency of PROF while achieving one more integrated design, a method is proposed to unify the representation bit depths of the sample gradients and MV differences used by BDOF and PROF.

[0075] Second, to facilitate hardware pipeline design, we propose to harmonize the workflow of PROF with that of BDOF for bi-prediction. Specifically, unlike the existing PROF, which derives prediction refinement separately for L0 and L1, the proposed method derives prediction refinement applied to the combined L0 and L1 prediction signals at once.

[0076] Third, two methods are proposed to reconcile the derivation of integer reference samples to calculate the gradient values ​​used by BDOF and PROF.

[0077] Fourth, to reduce the computational complexity, an early stopping method is proposed to adaptively disable the PROF process for an affine coded block when some conditions are met.

[0078] Improved bit-depth representation design of PROF gradient and MV difference As analyzed in the "Problem Statement" section, the representation bit depths of MV difference and sample gradient in the current PROF are not aligned to derive accurate prediction refinement. Moreover, the representation bit depths of sample gradient and MV difference are inconsistent between BDOF and PROF, which is unfavorable for hardware. In this section, an improved bit depth representation method is proposed by extending the bit depth representation method of BDOF to PROF. Specifically, in the proposed method, the horizontal gradient and vertical gradient at each sample position are

number

[0079] In addition, let Δx and Δy be the horizontal and vertical offsets from one sample location to the center of the subblock to which the sample belongs, expressed with 1 / 4-pel accuracy. The corresponding PROF MV difference Δv(x,y) at the sample location is

number

[0080] For a six-parameter affine model, JPEG2024167438000022.jpg41149, where (v 0x ,v 0y ), (v 1x ,v 1y ), (v 2x ,v 2y ) are the top-left, top-right, and bottom-left control points MV of the current coding block, expressed in 1 / 16 pel precision, and w and h are the width and height of the block.

[0081] In the above discussion, as shown in equations (13) and (14), a pair of fixed right shifts is applied to calculate the gradient and MV difference values. In practice, different bitwise right shifts may be applied to (13) and (14) to achieve various representation accuracy of gradient and MV difference for different tradeoffs between intermediate calculation accuracy and bit width of the internal PROF derivation process. For example, when the input video contains a lot of noise, the derived gradient may not be reliable to represent the true local horizontal gradient value / vertical gradient value at each sample. In such a case, it makes sense to use more bits than the gradient to represent the MV difference. On the other hand, when the input video shows stable motion, the MV difference derived by the affine model should be very small. If so, using high-precision MV difference cannot bring additional benefits of increasing the accuracy of the derived PROF refinement. In other words, in such a case, it is more beneficial to use more bits to represent the gradient value. Based on the above considerations, in one or more embodiments of the present disclosure, one general method for calculating gradient and MV difference for PROF is proposed below. Specifically, the horizontal and vertical gradients at each sample location are n a is calculated by applying right shifts to the difference of adjacent predicted samples, i.e.

number

number

number

[0082] In another embodiment, another PROF bit depth control method is proposed as follows: In this method, the horizontal and vertical gradients at each sample position are still n times the right shift. a The corresponding PROF MV difference Δv(x,y) at a sample position is calculated similarly to (13) by applying the bits to the difference value of the adjacent predicted sample. It should be calculated as JPEG2024167438000026.jpg12118.

[0083] In addition, to keep the overall PROF derivation at the appropriate internal bit depth, clipping is applied to the derived MV difference as follows: JPEG2024167438000027.jpg12119 where limit is Calculated as JPEG2024167438000028.jpg28163.

[0084] In addition, in one or more embodiments of the present disclosure, a PROF bit depth control solution is proposed, in which the horizontal and vertical PROF motion refinement at each sample position (i,j) is Derived as JPEG2024167438000029.jpg13151.

[0085] Furthermore, the derived horizontal and vertical motion refinements Clipped as JPEG2024167438000030.jpg13151.

[0086] Now, taking into account the motion refinement derived above, the final PROF sample refinement at location (i,j) is Calculated as JPEG2024167438000031.jpg6151.

[0087] In another embodiment of the present disclosure, another PROF bit depth control solution is proposed. In the second method, the horizontal and vertical PROF motion refinement at sample position (i,j) is Derived as JPEG2024167438000032.jpg14149.

[0088] The derived motion refinement is then Clipped as JPEG2024167438000033.jpg13149.

[0089] Therefore, taking into account the motion refinement derived above, the final PROF sample refinement at location (i,j) is Calculated as JPEG2024167438000034.jpg6149.

[0090] In one or more embodiments of the present disclosure, it is proposed to combine the motion refinement accuracy control method in the first solution with the PROF sample refinement derivation method in the second solution. Specifically, this method determines that the horizontal and vertical PROF motion refinements at each sample position (i,j) are Derived as JPEG2024167438000035.jpg13149.

[0091] Furthermore, the derived horizontal and vertical motion refinements Clipped as JPEG2024167438000036.jpg13146.

[0092] Now, taking into account the motion refinement derived above, the final PROF sample refinement at location (i,j) is Calculated as JPEG2024167438000037.jpg6147.

[0093] In one or more embodiments, the following PROF sample refinement derivation method is proposed.

[0094] First, Calculate horizontal and vertical motion refinements of PROF to an accuracy of 1 / 32 pel by applying a fixed right shift, as shown as JPEG2024167438000038.jpg13147.

[0095] Second, we clip the computed PROF motion refinement values ​​to a single symmetric range [-31, 31]. JPEG2024167438000039.jpg13147

[0096] Third, sample PROF refinement Calculated as JPEG2024167438000040.jpg8147.

[0097] 10 shows a method for expressing bit depth in PROF, which can be applied to a decoder, for example.

[0098] At step 1010, the decoder may obtain a reference picture I associated with a video block in the video signal.

[0099] In step 1012, the decoder may obtain a predictive sample I(i,j) of the video block from a reference block in reference picture I. i and j may represent coordinates of one sample with the video block.

[0100] In step 1014, the decoder may control the internal PROF parameters of the PROF derivation process by applying a right shift to the internal PROF parameters based on a bit shift value to achieve a preset precision. The internal PROF parameters include a horizontal gradient value, a vertical gradient value, a horizontal motion differential value, and a vertical motion differential value derived for the prediction sample I(i,j).

[0101] In step 1016, the decoder may obtain a prediction refinement value for a sample in the video block based on the PROF derivation process applied to the video block based on the prediction sample I(i,j).

[0102] At step 1018, the decoder may obtain a prediction sample for the video block based on a combination of the prediction sample and the prediction refinement value.

[0103] In addition, the same parameter derivation method is also v x =sGx2>0 ? Clip3(-31,31,-(sGxdI<<2)>>Floor(Log2(sGx2))):0 v y =sGy2>0 ? Clip3(-31,31,((sGydI<<2)-((v x *sGxGy m )<<12+v x *sGxGy s )>>1)>>Floor(Log2(sGy2))):0 It can be applied to the BDOF sample refinement process, as exemplified by: m , sGxGy s , and sGy2 are intermediate BDOF derived parameters.

[0104] 11 shows a bit depth control method for BDOF, which can be applied to a decoder, for example.

[0105] In step 1110, the decoder derives a first reference picture I associated with the video block. (0) and the second reference picture I (1) In display order, the first reference picture I (0) is before the current picture, and the second reference picture I (1) is after the current picture.

[0106] In step 1112, the decoder reads the first reference picture I (0) The first predicted sample of a video block from a reference block in (0) (i,j), where i and j may represent the coordinates of one sample with the current picture.

[0107] In step 1114, the decoder (1) The second predicted sample of the video block from the reference block in (1) (i,j) can be obtained.

[0108] In step 1116, the decoder may control the internal BDOF parameter of the BDOF derivation process by applying a shift to the internal BDOF parameter. (0) (i,j), the second prediction sample I (1) (i,j), the first prediction sample I (0) (i,j) and the second prediction sample I (1) (i,j) and horizontal and vertical gradient values ​​derived based on the sample difference between (i,j) and intermediate BDOF derivation parameters. The intermediate BDOF derivation parameters include sGxdI, sGydI, sGx2, sGxGy, and sGy2 parameters. sGxdI and sGydI include cross-correlation values ​​between the horizontal gradient values ​​and the sample difference values ​​and between the vertical gradient values ​​and the sample difference values. sGx2 and sGy2 include auto-correlation values ​​of the horizontal gradient values ​​and vertical gradient values. sGxGy includes cross-correlation values ​​between the horizontal gradient values ​​and the vertical gradient values.

[0109] In step 1118, the decoder calculates the first predicted sample I (0) (i,j) and the second prediction sample I (1) Based on the BDOF applied to the video block based on (i,j), a motion refinement for samples in the video block may be obtained.

[0110] At step 1120, the decoder may obtain bi-predictive samples for the video block based on the motion refinement.

[0111] A harmonized workflow of BDOF and PROF for bi-prediction As discussed before, when one affine-coded block is bi-predicted, the current PROF is applied unilaterally. More specifically, PROF sample refinements are derived separately and applied to the prediction samples in lists L0 and L1. Then, the refined prediction signals from lists L0 and L1, respectively, are averaged to generate the final bi-predictive signal for the block. This is in contrast to the BDOF design, where sample refinements are derived and applied to the bi-predictive signal. Such differences between the BDOF and PROF bi-predictive workflows may not be favorable for practical codec pipeline designs.

[0112] To facilitate the hardware pipeline design, one simplification method according to the present disclosure is to modify the bi-prediction process of PROF so that the workflow of the two prediction refinement methods is harmonized. Specifically, instead of applying refinement separately for each prediction direction, the proposed PROF method derives prediction refinement once based on the control point MV of lists L0 and L1, and then the derived prediction refinement is applied to the combined L0 and L1 prediction signals to enhance the quality. Specifically, based on the MV difference derived in equation (14), the final bi-prediction samples of one affine coding block are calculated by the proposed method as follows:

number

[0113] FIG. 13 shows the corresponding PROF process when the proposed bi-predictive PROF method is applied. The PROF process 1300 includes L0 motion compensation 1310, L1 motion compensation 1320, and bi-predictive PROF 1330. The L0 motion compensation 1310 may be, for example, a list of motion compensation samples from a previous reference picture. The previous reference picture is a reference picture before a current picture in a video block. The L1 motion compensation 1320 may be, for example, a list of motion compensation samples from a next reference picture. The next reference picture is a reference picture after a current picture in a video block. The bi-predictive PROF 1330 takes motion compensation samples from the L1 motion compensation 1310 and the L1 motion compensation 1320 as described above, and outputs bi-predictive samples.

[0114] To demonstrate the potential benefits of the proposed method for hardware pipeline design, Fig. 14 (described below) shows an example to illustrate the pipeline stages when both BDOF and the proposed PROF are applied. In Fig. 14, the decoding process of one inter block mainly includes three things.

[0115] First, parse / decode the MV of the coding block and fetch the reference samples.

[0116] Second, generate an L0 prediction signal and / or an L1 prediction signal for the coding block.

[0117] Third, perform sample-wise refinement of the generated bi-predictive samples based on BDOF when the coding block is predicted by one non-affine mode or PROF when the coding block is predicted by an affine mode.

[0118] FIG. 14 illustrates an example pipeline stage when both BDOF and the proposed PROF are applied according to the present disclosure. FIG. 14 demonstrates the potential benefits of the proposed method for hardware pipeline design. The pipeline stage 1400 includes MV parsing / decoding and fetching reference samples 1410, motion compensation 1420, and BDOF / PROF 1430. The pipeline stage 1400 encodes video blocks BLK0, BKL1, BKL2, BKL3, and BLK4. Each video block starts at MV parsing / decoding and fetching reference samples 1410, then moves sequentially to motion compensation 1420, then motion compensation 1420, and BDOF / PROF 1430. This means that BLK0 does not start the process in the pipeline stage 1400 until BLK0 moves to motion compensation 1420. As time passes from T0 to T1, T2, T3, and T4, it is the same for all stages and video blocks.

[0119] As shown in Fig. 14, after the proposed harmonization method is applied, both BDOF and PROF are directly applied to bi-prediction samples. Given that BDOF and PROF are applied to different types of coding blocks (i.e., BDOF is applied to non-affine blocks and PROF is applied to affine blocks), the two coding tools cannot be invoked simultaneously. Therefore, their corresponding decoding processes can be implemented by sharing the same pipeline stage. This is more efficient than existing PROF designs, in which it is difficult to allocate the same pipeline stage to both BDOF and PROF due to the different workflows of bi-prediction.

[0120] In the above discussion, the proposed method only considers the harmony of BDOF and PROF workflows. However, according to the existing design, the basic operation units of the two coding tools are also implemented in different sizes. For example, in the case of BDOF, one coding block is W s ×H s where W is the number of sub-blocks of size s =min(W,16) and H s =min(H,16), where W and H are the width and height of the coding block. BDOF operations, such as gradient calculation and sample refinement derivation, are performed independently for each subblock. Meanwhile, as described before, an affine coding block is divided into 4×4 subblocks, and each subblock is assigned one individual MV derived based on either a four-parameter affine model or a six-parameter affine model. Since PROF is applied only to affine blocks, its basic operation unit is a 4×4 subblock. Similar to the bi-prediction workflow problem, using a different basic operation unit size for PROF than BDOF is also unfavorable for hardware implementation, making it difficult for BDOF and PROF to share the same pipeline stage of the overall decoding process. To solve such problems, in one or more embodiments, it is proposed to adjust the subblock size of the affine mode to be the same as the subblock size of the BDOF. For example, according to the proposed method, when one coding block is coded by the affine mode, one coding block is coded by W s ×H s where W s =min(W,16) and H s=min(H,16), where W and H are the width and height of the coding block. Each subblock is assigned one individual MV and considered as one independent PROF operation unit. It is worth mentioning that the independent PROF operation unit ensures that the PROF operation on it is performed without referring to information from neighboring PROF operation units. For example, the PROF MV difference at one sample position is calculated as the difference between the MV at the sample position and the MV at the center of the PROF operation unit where the sample is located, and the gradient used by the PROF derivation is calculated by padding samples along each PROF operation unit. The asserted benefits of the proposed method mainly include the following aspects: 1) simplified pipeline architecture with unified basic operation unit size for both motion compensation and BDOF / PROF refinement, 2) reduced memory bandwidth usage due to enlarged subblock size for affine motion compensation, and 3) reduced per-sample computation complexity of fractional sample interpolation.

[0121] It should also be mentioned that due to the reduced computational complexity of the proposed method (i.e., item 3), the existing 6-tap interpolation filter constraint for affine-coded blocks can be removed. Instead, the default 8-tap interpolation for non-affine-coded blocks is also used for affine-coded blocks. The overall computational complexity in this case is still comparable to the existing PROF design (based on 4×4 sub-blocks with 6-tap interpolation filters).

[0122] Reconciliation of gradient derivations for BDOF and PROF. As explained before, both BDOF and PROF calculate the gradient of each sample inside the current coding block, which accesses one additional row / column of predicted samples on each side of the block. To avoid additional interpolation complexity, the required predicted samples in the extension region around the block boundary are directly copied from the integer reference samples. However, as pointed out in the "Problem Statement" section, integer samples at different locations are used to calculate the gradient values ​​of BDOF and PROF.

[0123] In order to achieve a more unified design, two methods are proposed below to integrate the gradient derivation methods used by BDOF and PROF. In the first method, it is proposed to adjust the gradient derivation method of PROF to be the same as the gradient derivation method of BDOF. For example, by the first method, the integer positions used to generate the predicted samples in the extended region are determined by flooring down the fractional sample positions, i.e., the selected integer sample positions are to the left of the fractional sample positions (for horizontal gradients) and above the fractional sample positions (for vertical gradients).

[0124] In the second method, it is proposed to adjust the gradient derivation method of BDOF to be the same as the gradient derivation method of PROF. More specifically, when the second method is applied, the integer reference sample closest to the predicted sample is used for the gradient calculation.

[0125] An example of using the BDOF gradient derivation method according to the present disclosure is shown in Figure 15. In Figure 15, the empty circles 1510 represent reference samples at integer positions, the triangles 1530 represent fractional predicted samples of the current block, and the black circles 1520 represent integer reference samples used to fill the extension region of the current block.

[0126] An example of using the PROF gradient derivation method according to the present disclosure is shown in Figure 16. In Figure 16, the open circles 1610 represent reference samples at integer positions, the triangles 1630 represent fractional predicted samples of the current block, and the solid circles 1620 represent integer reference samples used to fill the extension region of the current block.

[0127] Figures 15 and 16 illustrate the corresponding integer sample locations used to derive gradients for BDOF and PROF when the first method (Figure 15) and the second method (Figure 16) are applied, respectively. In Figures 15 and 16, the empty circles represent reference samples at integer positions, the triangles represent fractional predicted samples of the current block, and the dotted circles represent integer reference samples used to fill the extension region of the current block for gradient derivation.

[0128] In addition, according to the existing BDOF and PROF designs, the predicted sample padding is implemented at different coding levels. For example, for BDOF, the padding is applied along the boundary of sbWidth×sbHeight subblocks, where sbWidth=min(CUWidth, 16) and sbHeight=min(CUHeight, 16). CUWidth and CUHeight are the width and height of one CU. Meanwhile, the padding of PROF is always applied at the 4×4 subblock level. In the above discussion, only the padding method is integrated between BDOF and PROF, but the padding subblock size is still different. Given that it requires different modules to be implemented for the padding process of BDOF and PROF, this is also not favorable for practical hardware implementation. In order to achieve one more integrated design, it is proposed to integrate the subblock padding size of BDOF and PROF. In one or more embodiments of the present disclosure, it is proposed to apply the predicted sample padding of BDOF at the 4×4 level. For example, with this method, a CU is first divided into multiple 4x4 sub-blocks, and after motion compensation of each 4x4 sub-block, the extension samples along the top / bottom and left / right boundaries are padded by copying the corresponding integer sample positions.

[0129] Figures 19A, 19B, 19C, and 19D illustrate an example in which the proposed padding method is applied to one 16x16 BDOF CU, where the dashed lines represent 4x4 sub-block boundaries and the blue bands represent padded samples of each 4x4 sub-block.

[0130] FIG. 19A illustrates the proposed padding method applied to a 16×16 BDOF CU according to the present disclosure, where the dashed line represents the top-left 4×4 sub-block boundary 1920.

[0131] FIG. 19B illustrates the proposed padding method applied to a 16×16 BDOF CU according to the present disclosure, where the dashed line represents the top right 4×4 sub-block boundary 1940.

[0132] FIG. 19C illustrates the proposed padding method applied to a 16×16 BDOF CU according to the present disclosure, where the dashed line represents the bottom-left 4×4 sub-block boundary 1960.

[0133] FIG. 19D illustrates the proposed padding method applied to a 16×16 BDOF CU, where the dashed line represents the bottom-right 4×4 sub-block boundary 1980, in accordance with the present disclosure.

[0134] High-level signaling syntax for enabling / disabling BDOF, PROF, and DMVR In the existing BDOF and PROF designs, two different flags are signaled in the sequence parameter set (SPS) to separately control the enabling / disabling of the two coding tools. However, due to the similarity between BDOF and PROF, it is more desirable to enable and / or disable BDOF and PROF from a high level by one and the same control flag. Based on such consideration, one new flag called sps_bdof_prof_enabled_flag is introduced in the SPS as shown in Table 1. As shown in Table 1, the enabling and disabling of BDOF depends only on sps_bdof_prof_enabled_flag. When the flag is equal to 1, BDOF is enabled for coding the video content in the sequence. Otherwise, when sps_bdof_prof_enabled_flag is equal to 0, BDOF is not applied. Meanwhile, in addition to sps_bdof_prof_enabled_flag, an SPS-level affine control flag, namely sps_affine_enabled_flag, is also used to conditionally enable and disable PROF. When both flags sps_bdof_prof_enabled_flag and sps_affine_enabled_flag are equal to 1, PROF is enabled for all coding blocks that are coded in affine mode. When the flag sps_bdof_prof_enabled_flag is equal to 1 and sps_affine_enabled_flag is equal to 0, PROF is disabled. [Table 1]

[0135] sps_bdof_prof_enabled_flag specifies whether bidirectional optical flow and optical flow prediction refinement are enabled. When sps_bdof_prof_enabled_flag is equal to 0, both bidirectional optical flow and optical flow prediction refinement are disabled. When sps_bdof_prof_enabled_flag is equal to 1 and sps_affine_enabled_flag is equal to 1, both bidirectional optical flow and optical flow prediction refinement are enabled. Otherwise (sps_bdof_prof_enabled_flag is equal to 1 and sps_affine_enabled_flag is equal to 0), bidirectional optical flow is enabled and optical flow prediction refinement is disabled.

[0136] sps_bdof_prof_dmvr_slice_preset_flag specifies when the flag slice_disable_bdof_prof_dmvr_flag is signaled at the slice level. When the flag is equal to 1, the syntax slice_disable_bdof_prof_dmvr_flag is signaled for each slice that refers to the current sequence parameter set. Otherwise (when sps_bdof_prof_dmvr_slice_present_flag is equal to 0), the syntax slice_disabled_bdof_prof_dmvr_flag is not signaled at the slice level. When this flag is not signaled, it is inferred to be 0.

[0137] In addition, when the proposed SPS level BDOF and PROF control flags are used, the corresponding control flag no_bdof_constraint_flag in the general constraint information syntax should also be modified according to the table below. [Table 2]

[0138] no_bdof_prof_constraint_flag equal to 1 specifies that sps_bdof_prof_enabled_flag is equal to 0. no_bdof_constraint_flag equal to 0 imposes no constraints.

[0139] In addition to the above SPS BDOF / PROF syntax, it is proposed to introduce another control flag at slice level, namely slice_disable_bdof_prof_dmvr_flag is introduced to disable BDOF, PROF and DMVR. The SPS flag sps_bdof_prof_dmvr_slice_present_flag, which is signaled in SPS when either DMVR or BDOF / PROF sps level control flag is true, is used to indicate the presence of slice_disable_bdof_prof_dmvr_flag. If present, slice_disable_bdof_dmvr_flag is signaled. Table 2 illustrates a modified slice header syntax table after the proposed syntax is applied. In another embodiment, it is proposed to still use two control flags in the slice header to separately control the enabling / disabling of BDOF and DMVR, and the enabling / disabling of PROF. For example, this method uses two flags in the slice header. One flag, slice_disable_bdof_dmvr_slice_flag, is used to control on / off of BDOF and DMVR, and the other flag, disable_prof_slice_flag, is used to control on / off of PROF alone. [Table 3]

[0140] In another embodiment, it is proposed to control BDOF and PROF separately by two different SPS flags. For example, two separate SPS flags sps_bdof_enable_flag and sps_prof_enable_flag are introduced to enable / disable the two tools separately. In addition, one high-level control flag no_prof_constraint_flag needs to be added in the general_constrain_info() syntax table to forcefully disable the PROF tool. [Table 4]

[0141] sps_bdof_enabled_flag specifies whether bidirectional optical flow is enabled. When sps_bdof_enabled_flag is equal to 0, bidirectional optical flow is disabled. When sps_bdof_enabled_flag is equal to 1, bidirectional optical flow is enabled.

[0142] sps_prof_enabled_flag specifies whether optical flow prediction refinement is enabled. When sps_prof_enabled_flag is equal to 0, optical flow prediction refinement is disabled. When sps_prof_enabled_flag is equal to 1, optical flow prediction refinement is enabled. [Table 5]

[0143] no_prof_constraint_flag equal to 1 specifies that sps_prof_enabled_flag is equal to 0. no_prof_constraint_flag equal to 0 imposes no constraints.

[0144] At the slice level, in one or more embodiments of the present disclosure, it is proposed to introduce another control flag at the slice level, namely slice_disable_bdof_prof_dmvr_flag, to disable BDOF, PROF, and DMVR together. In another embodiment, it is proposed to add two separate flags at the slice level, namely slice_disable_bdof_dmvr_flag and slice_disable_prof_flag. The first flag (i.e., slice_disable_bdof_dmvr_flag) is used to adaptively switch on / off BDOF and DMVR for one slice, and the second flag (i.e., slice_disable_prof_flag) is used to control enabling and disabling of PROF tool at the slice level. In addition, when the second method is applied, the flag slice_disable_bdof_dmvr_flag needs to be signaled only when either the SPS BDOF or the SPS DMVR flag is enabled, and the flag needs to be signaled only when the SPS PROF flag is enabled.

[0145] At the 16th JVET meeting, the VVC draft adopted the picture header. The picture header is signaled once per slice, and the syntax elements are already present in the slice header and do not change from slice to slice.

[0146] Based on the adopted picture header, one or more embodiments of the present disclosure propose to control BDOF, DMVR, and PROF control flags from the current slice header to the picture header. For example, in the proposed method, three different control flags sps_dmvr_picture_header_present_flag, sps_bdof_picture_header_present_flag, and sps_prof_picture_header_present_flag are signaled in the SPS. When one of the three flags is signaled as true, one additional control flag is signaled in the picture header to indicate that the corresponding tools (i.e., DMVR, BDOF, and PROF) are enabled or disabled for the slices that reference the picture header. The proposed syntax elements are specified as follows: [Table 6]

[0147] sps_dmvr_picture_header_preset_flag specifies whether the flag picture_disable_dmvr_flag is signaled in the picture header. When the flag is equal to 1, the syntax picture_disable_dmvr_flag is signaled for every picture that references the current sequence parameter set. Otherwise, the syntax picture_disable_dmvr_flag is not signaled in the picture header. When this flag is not signaled, it is inferred to be 0.

[0148] sps_bdof_picture_header_preset_flag specifies whether the flag picture_disable_bdof_flag is signaled in the picture header. When the flag is equal to 1, the syntax picture_disable_bdof_flag is signaled for every picture that references the current sequence parameter set. Otherwise, the syntax picture_disable_bdof_flag is not signaled in the picture header. When this flag is not signaled, it is inferred to be 0.

[0149] sps_prof_picture_header_preset_flag specifies whether the flag picture_disable_prof_flag is signaled in the picture header. When the flag is equal to 1, the syntax picture_disable_prof_flag is signaled for every picture that references the current sequence parameter set. Otherwise, the syntax picture_disable_prof_flag is not signaled in the picture header. When this flag is not signaled, it is inferred to be 0. [Table 7]

[0150] picture_disable_dmvr_flag specifies whether the dmvr tool is enabled for slices that reference the current picture header. When the flag is equal to 1, the dmvr tool is enabled for slices that reference the current picture header. Otherwise, the dmvr tool is disabled for slices that reference the current picture header. When this flag is not present, it is inferred to be 0.

[0151] picture_disable_bdof_flag specifies whether the bdof tools are enabled for slices that reference the current picture header. When the flag is equal to 1, the bdof tools are enabled for slices that reference the current picture header. Otherwise, the bdof tools are disabled for slices that reference the current picture header.

[0152] picture_disable_prof_flag specifies whether the prof tool is enabled for slices that reference the current picture header. When the flag is equal to 1, the prof tool is enabled for slices that reference the current picture header. Otherwise, the prof tool is disabled for slices that reference the current picture header.

[0153] 12 shows the BDOF, PROF and DMVR methods, which can be applied, for example, to a decoder.

[0154] At step 1210, the decoder may receive three control flags in a sequence parameter set (SPS): a first control flag indicates whether BDOF is enabled for decoding the video block in the current video sequence; a second control flag indicates whether PROF is enabled for decoding the video block in the current video sequence; and a third control flag indicates whether DMVR is enabled for decoding the video block in the current video sequence.

[0155] In step 1212, the decoder may receive a first presence flag in the SPS when the first control flag is true, a second presence flag in the SPS when the second control flag is true, and a third presence flag in the SPS when the third control flag is true.

[0156] At step 1214, the decoder may receive the first picture control flag in the picture header of each picture when the first presence flag in the SPS indicates that BDOF is disabled for a video block in the picture.

[0157] At step 1216, the decoder may receive a second picture control flag in the picture header of each picture when the second presence flag in the SPS indicates that PROF is disabled for video blocks in the picture.

[0158] At step 1218, the decoder may receive a third picture control flag in the picture header of each picture when the third presence flag in the SPS indicates that DMVR is disabled for video blocks in the picture.

[0159] Early termination of PROF based on control point MV difference According to the current PROF design, PROF is always invoked when one coding block is predicted by affine mode. However, as shown in equations (6) and (7), the sub-block MVs of one affine block are derived from the control point MVs. Therefore, when the difference between the control point MVs is relatively small, the MVs at each sample position should be consistent. In such a case, the benefit of applying PROF may be very limited. Therefore, to further reduce the average computation complexity of PROF, it is proposed to adaptively skip PROF-based sample refinement based on the maximum MV difference between the sample-wise MVs and the sub-block-wise MVs in one 4 × 4 sub-block. Since the values ​​of the PROF MV difference of samples inside one 4 × 4 sub-block are symmetric with respect to the sub-block center, the maximum horizontal and vertical PROF MV difference can be calculated based on equation (10):

number

[0160] According to this disclosure, different metrics may be used in determining whether the MV difference is small enough to skip the PROF process.

[0161] In one example, based on equation (19), the sum of the absolute maximum horizontal MV difference and the absolute maximum vertical MV difference is less than one predefined threshold, i.e.,

number

[0162] In another example, If the maximum value of JPEG2024167438000051.jpg2563 is less than or equal to the threshold, the PROF process may be skipped.

number

[0163] In addition to the above two examples, the teachings of this disclosure are also applicable when other metrics are used in determining whether the MV difference is small enough to skip the PROF process.

[0164] In the above method, PROF is skipped based on the magnitude of MV difference. Meanwhile, in addition to MV difference, PROF sample refinement is also calculated based on local gradient information at each sample location in one motion compensation block. For prediction blocks that contain less frequent details (e.g., flat areas), the gradient value tends to be small, and as a result, the derived sample refinement value should be small. Taking this into account, according to another embodiment, it is proposed to apply PROF only to prediction samples of blocks that contain sufficient frequent information.

[0165] Different metrics may be used in determining whether a block contains enough high-frequency information for it to be worth invoking the PROF process for that block. In one example, the decision is made based on the average magnitude (i.e., absolute value) of the gradient of the samples in the predicted block. If the average magnitude is less than a threshold, the predicted block is classified as a flat area and PROF should not be applied. Otherwise, the predicted block is deemed to contain enough high-frequency detail that PROF is still applicable. In another example, the maximum magnitude of the gradient of the samples in the predicted block may be used. If the maximum magnitude is less than a threshold, PROF should be skipped for the block. In yet another example, the difference I between the maximum and minimum sample values ​​of the predicted block is used. max -I min may be used to determine whether PROF should be applied to the block. If such difference value is less than a threshold, PROF should be skipped for the block. It is worth noting that the teachings of this disclosure are also applicable when some other metric is used in determining whether a given block contains sufficient frequent information.

[0166] Handles the interaction between PROF and LIC for affine modes Since the neighboring reconstructed samples (i.e., templates) of the current block are used by LIC to derive the linear model parameters, the decoding of one LIC coded block depends on the full reconstruction of its neighboring samples. Due to such interdependence, for practical hardware implementation, LIC needs to be performed in the reconstruction stage where the neighboring reconstructed samples become available for LIC parameter derivation. Since the block reconstruction must be performed sequentially (i.e., one by one), the throughput (i.e., the amount of work per unit time that can be done in parallel) is one important issue to consider when applying other coding methods together to a LIC coded block. In this section, two methods are proposed to handle the interaction when both PROF and LIC are enabled for the affine mode.

[0167] In the first embodiment of the present disclosure, it is proposed to apply the PROF mode and the LIC mode exclusively to one affine coding block. As previously discussed, in the existing design, PROF is implicitly applied to all affine blocks without signaling, while one LIC flag is signaled or inherited at the coding block level to indicate whether the LIC mode is applied to one affine block. According to the method in the present disclosure, it is proposed to conditionally apply PROF based on the value of the LIC flag of one affine block. When the flag is equal to 1, only LIC is applied by adjusting the prediction sample of the whole coding block based on the LIC weight and offset. Otherwise (i.e., the LIC flag is equal to 0), PROF is applied to the affine coding block to refine the prediction sample of each sub-block based on the optical flow model.

[0168] FIG. 18A illustrates one exemplary flow chart of a decoding process based on the proposed method in which PROF and LIC are not allowed to be applied simultaneously.

[0169] 18A shows an example of a decoding process based on the proposed method where PROF and LIC are not allowed according to the present disclosure. The decoding process 1820 includes decision steps 1822, LIC 1824, and PROF 1826. Decision step 1822 is to determine whether the LIC flag is on, and according to the decision, the following is taken: LIC 1824 is the application of LIC if the LIC flag is set; PROF 1826 is the application of PROF if the LIC flag is not set.

[0170] In the second embodiment of the present disclosure, it is proposed to apply LIC after PROF to generate a prediction sample of one affine block. For example, after sub-block-based affine motion compensation is performed, the prediction sample is refined based on PROF sample refinement, and then LIC is

number

[0171] 18B illustrates an example of a decoding process in which PROF and LIC are applied according to the present disclosure. The decoding process 1860 includes affine motion compensation 1862, LIC parameter derivation 1864, PROF 1866, and LIC sample adjustment 1868. Affine motion compensation 1862 applies affine motion and is input to LIC parameter derivation 1864 and PROF 1866. LIC parameter derivation 1864 is applied to derive LIC parameters. PROF 1866 is the PROF being applied. LIC sample adjustment 1868 is the LIC weight and offset parameters combined with PROF.

[0172] Figure 18B illustrates an exemplary decoding workflow when the second method is applied. As shown in Figure 18B, since LIC uses templates (i.e., adjacent reconstructed samples) to calculate a LIC linear model, the LIC parameters can be derived as soon as the adjacent reconstructed samples are available. This means that PROF refinement and LIC parameter derivation can be performed simultaneously.

[0173] The LIC weights and offsets (i.e., α and β) and the PROF refinements (i.e., ΔI[x]) are generally floating point. In the preferred hardware implementation, those floating point operations are usually implemented as a multiplication with an integer value followed by a right shift operation by a number of bits. In existing LIC and PROF designs, the two tools are designed separately, so each has N LIC Bit and N PROF Two different right shifts by bits are applied in two stages.

[0174] According to a third embodiment, to improve the coding gain when PROF and LIC are jointly applied to an affine coded block, it is proposed to apply the LIC-based sample adjustment and the PROF-based sample adjustment with high accuracy by combining those two right-shift operations into one and applying it at the end to derive the final predicted samples (as shown in (12)) of the current block.

[0175] Addressing the multiplication overflow issue when combining PROF with weighted prediction and bi-prediction with CU-level weights (BCW) According to the current PROF design in the VVC working draft, PROF can be applied together with weighted prediction (WP). For example, when they are combined, a prediction signal for one affine CU is generated by the following procedure:

[0176] First, for each sample at location (x,y), we calculate the L0 prediction refinement ΔI0(x,y) based on PROF to add the refinement to the original L0 prediction sample I0(x,y), i.e.

number

[0177] Second, for each sample at location (x,y), we calculate an L1 prediction refinement ΔI1(x,y) based on PROF to add the refinement to the original L1 predicted sample I1(x,y), i.e.

number

[0178] Third, combine the refined L0 and L1 prediction samples, i.e.,

number

[0179] As can be seen from the above equations, due to the sample-by-sample refinement, i.e., ΔI0(x,y) and ΔI1(x,y), the predicted samples after PROF (i.e., I0'(x,y) and I1'(x,y)) have one increased dynamic range than the original predicted samples (i.e., I0(x,y) and I1(x,y)). Considering that the refined predicted samples are multiplied by the WP and BCW weighting factors, this increases the length of the multiplier required. For example, based on the current design, when the intra-coding bit depth is in the range of 8-12 bits, the dynamic range of the predicted signals I0(x,y) and I1(x,y) is 16 bits. However, after PROF, the dynamic range of the predicted signals I0'(x,y) and I1'(x,y) becomes 17 bits. Therefore, when PROF is applied, a 16-bit multiplication overflow problem may occur in some cases. In order to fix such overflow issues, several methods are proposed below.

[0180] First, in the first method, it is proposed to disable WP and BCW when PROF is applied to one affine CU.

[0181] Second, in the second method, it is proposed to apply one clip operation to the derived sample refinements before adding them to the original prediction samples, so that the dynamic range of the refined prediction samples I0'(x,y) and I1'(x,y) has the same dynamic bit depth as that of the original prediction samples I0(x,y) and I1(x,y). For example, such a method can be used to reduce the sample refinements ΔI0(x,y) and ΔI1(x,y) in (23) and (24) to It is modified by introducing one clip operation, represented as JPEG2024167438000057.jpg19134, where dI=dI base +max(0,BD-12), where BD is the intra-coding bit depth and dI base is the base bit depth value. In one or more embodiments, dI base It is proposed to set the value of dI to 14. In another embodiment, it is proposed to set this value to 13. In another embodiment, it is proposed to set the value of dI directly so that it is fixed. In one example, it is proposed to set the value of dI to 13, i.e. the sample refinement is clipped to the range [-4096, 4095]. In another example, it is proposed to set the value of dI to 14, i.e. the sample refinement is clipped to the range [-8192, 8191].

[0182] First, in the third method, instead of clipping the sample refinement, it is proposed to directly clip the refined prediction samples so that the refined samples have the same dynamic range as those of the original prediction samples. For example, by the third method, the refined L0 and L1 samples are JPEG2024167438000058.jpg21134, where dR=16+max(0,BD-12) (or equivalently max(16,BD+4)), where BD is the intra coding bit depth.

[0183] Secondly, in the fourth method, it is proposed to apply a certain right shift to the refined L0 and L1 prediction samples before WP and BCW, and then an additional left shift adjusts the final prediction samples to the original accuracy. For example, the final prediction samples are JPEG2024167438000059.jpg9147, where nb is the number of additional bit shifts to be applied, which can be determined based on the corresponding dynamic range of the PROF sample refinement.

[0184] Third, in the fifth method, it is proposed to split each multiplication of the L0 / L1 predicted samples into two multiplications by the corresponding WP / BCW weights in (25), and both of the two multiplications are No more than 16 bits, as written as JPEG2024167438000060.jpg13147.

[0185] 20 illustrates a computing environment 2010 coupled with a user interface 2060. The computing environment 2010 may be part of a data processing server. The computing environment 2010 includes a processor 2020, a memory 2040, and an I / O interface 2050.

[0186] The processor 2020 typically controls the overall operation of the computing environment 2010, such as operations associated with display, data acquisition, data communication, and image processing. The processor 2020 may include one or more processors for executing instructions for performing all or part of the methods described above. Additionally, the processor 2020 may include one or more modules that facilitate interconnection between the processor 2020 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single chip machine, a GPU, or the like.

[0187] The memory 2040 is configured to store various types of data to support the operation of the computing environment 2010. The memory 2040 may include predefined software 2042. Examples of such data include instructions for any application or method operating on the computing environment 2010, video data sets, image data, etc. The memory 2040 may be implemented by using any type of volatile or non-volatile memory device, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk, or a combination thereof.

[0188] The I / O interface 2050 provides an interface between the processor 2020 and a peripheral interface module, such as a keyboard, a click wheel, buttons, etc. The buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. The I / O interface 2050 may be coupled to an encoder and a decoder.

[0189] In some embodiments, a non-transitory computer readable storage medium is also provided that includes a number of programs, such as contained in memory 2040, executable by the processor 2020 in the computing environment 2010 to perform the methods described above. For example, the non-transitory computer readable storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0190] A non-transitory computer-readable storage medium stores a plurality of programs for execution by a computing device having one or more processors, where the plurality of programs, when executed by the one or more processors, cause the computing device to perform the above-described method for motion prediction.

[0191] In some embodiments, the computing environment 2010 may be implemented with one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), graphical processing units (GPUs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0192] The description of the present disclosure has been presented for purposes of illustration and is not intended to be exhaustive or limiting of the present disclosure. Many modifications, variations and alternative implementations will be apparent to one skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings.

[0193] Examples have been chosen and described to explain the principles of the disclosure and to enable those skilled in the art to understand the disclosure in its various implementations and to best utilize the underlying principles and various implementations with various modifications suited to the particular applications contemplated. It is therefore to be understood that the scope of the disclosure is not limited to the specific examples of implementations disclosed, and that modifications and other implementations are intended to be included within the scope of the disclosure.

Claims

1. A bit depth representation method for prediction refinement by optical flow (PROF) for decoding a video signal, comprising: obtaining, at a decoder, a reference picture I associated with a video block in the video signal; At the decoder, obtaining a predicted sample I(i,j) of the video block from a reference block in the reference picture I, where i and j represent coordinates of a sample comprising the video block; controlling, in the decoder, an internal PROF parameter of a PROF derivation process by applying a right shift to the internal PROF parameter based on a bit shift value to achieve a preset precision, the internal PROF parameter including a horizontal gradient value, a vertical gradient value, a horizontal motion differential value, and a vertical motion differential value derived for the prediction sample I(i,j); At the decoder, obtaining a prediction refinement value for a sample in the video block based on the prediction sample I(i,j) having been applied to the video block by the PROF derivation process; and At the decoder, obtaining a prediction sample for the video block based on a combination of the prediction sample and the prediction refinement value.

2. The method of claim 1 , wherein the preset precision is equal to 1 / 32 pel.

3. controlling the internal PROF parameter of the PROF derivation process by applying a right shift to the internal PROF parameter; In the decoder, obtaining a horizontal gradient value of a first predicted sample I(i,j) based on a difference between the first predicted sample I(i+1,j) and the first predicted sample I(i-1,j); In the decoder, obtaining a vertical gradient value of a first predicted sample I(i,j) based on a difference between a first predicted sample I(i,j+1) and a first predicted sample I(i,j−1); In the decoder, obtaining control point motion vectors (MVs) of the first predicted sample I(i,j), the control point MVs including MVs of a top left corner, a top right corner, and a bottom left corner block of a block including the video block; obtaining, in the decoder, affine model parameters derived based on the control points MV; At the decoder, a horizontal MV difference Δv for the first predicted sample I(i,j) based on the affine model parameters. x (i, j) and the vertical MV difference Δv y Obtaining (i, j); In the decoder, the horizontal MV difference Δv x (i, j) and the vertical MV difference Δv y 2. The method of claim 1 , comprising: right shifting (i,j) by a bit shift value, wherein the bit shift value is equal to eight.

4. In the decoder, the horizontal MV difference Δv x Clipping (i,j) to the symmetric range of [-31,31]; In the decoder, the vertical MV difference Δv y clipping (i,j) to the symmetric range of [-31,31]; The method of claim 3 further comprising:

5. Obtaining the prediction refinement values ​​for samples in the video block, In the decoder, the horizontal gradient value, the horizontal MV difference Δv x (i,j), the vertical gradient value, and the vertical MV difference Δv y 4. The method of claim 3, comprising obtaining the prediction refinement value based on (i,j).

6. A bidirectional optical flow (BDOF) bit depth control method for decoding a video signal, comprising: At the decoder, the first reference picture I associated with the video block (0) and the second reference picture I (1) and obtaining, in display order, the first reference picture I (0) is before the current picture, and the second reference picture I (1) is after the current picture; and In the decoder, the first reference picture I (0) A first predicted sample I of the video block from a reference block in (0) (i,j), where i and j represent the coordinates of one sample comprising the current picture; In the decoder, the second reference picture I (1) A second predicted sample I of the video block from a reference block in (1) Obtaining (i, j); In the decoder, controlling an internal BDOF parameter of a BDOF derivation process by applying a shift to the internal BDOF parameter, the internal BDOF parameter being adjusted based on the first prediction sample I (0) (i, j), the second predicted sample I (1) (i, j), the first predicted sample I (0) (i, j) and the second predicted sample I (1) (i,j) and horizontal and vertical gradient values ​​derived based on intermediate BDOF-derived parameters, the intermediate BDOF-derived parameters including sGxdI, sGydI, sGx2, sGxGy, and sGy2 parameters, sGxdI and sGydI including cross-correlation values ​​between the horizontal gradient values ​​and sample difference values ​​and between the vertical gradient values ​​and the sample difference values, sGx2 and sGy2 including auto-correlation values ​​of the horizontal gradient values ​​and the vertical gradient values, and sGxGy including the cross-correlation value between the horizontal gradient values ​​and the vertical gradient values; In the decoder, the first predicted sample I (0) (i, j) and the second predicted sample I (1) obtaining a motion refinement for samples within the video block based on the BDOF applied to the video block based on (i,j); At the decoder, obtaining bi-predictive samples for the video block based on the motion refinement.

7. controlling an internal bit depth of the BDOF derivation process for representation precision of internal BDOF derived parameters in the decoder; In the decoder, the first predicted sample I (0) (i, j), the second predicted sample I (1) obtaining the intermediate BDOF-derived parameters based on (i, j); At the decoder, obtaining a horizontal motion refinement value based on the sGx2 and sGxdI parameters; At the decoder, obtaining a vertical motion refinement value based on the sGy2, sGydI, and sGxGy parameters; and at the decoder clipping the horizontal and vertical motion refinement values ​​to a symmetric range of [-31, 31].

8. A method for Bidirectional Optical Flow (BDOF), Prediction Refinement by Optical Flow (PROF), and Decoder-Side Motion Vector Refinement (DMVR), comprising: receiving, by a decoder, three control flags in a sequence parameter set (SPS), where a first control flag indicates whether the BDOF is enabled for decoding the video block in a current video sequence, a second control flag indicates whether the PROF is enabled for decoding the video block in the current video sequence, and a third control flag indicates whether the DMVR is enabled for decoding the video block in the current video sequence; receiving, by the decoder, a first presence flag in the SPS when the first control flag is true, a second presence flag in the SPS when the second control flag is true, and a third presence flag in the SPS when the third control flag is true; receiving, by the decoder, a first picture control flag in a picture header of each picture when the first presence flag in the SPS indicates that the BDOF is disabled for the video block in the picture; receiving, by the decoder, a second picture control flag in the picture header of each picture when the second presence flag in the SPS indicates that the PROF is disabled for the video block in the picture; receiving, by the decoder, a third picture control flag in the picture header of each picture when the third presence flag in the SPS indicates that the DMVR is disabled for the video block in the picture.

9. receiving, by the decoder, the three control flags in the SPS; receiving, by the decoder, a sps_bdof_enabled_flag flag, the sps_bdof_enabled_flag flag indicating whether the BDOF is enabled for decoding the video blocks in the sequence; receiving, by the decoder, a sps_prof_enabled_flag flag, the sps_prof_enabled_flag flag indicating whether the PROF is enabled for decoding the video blocks in the sequence; 9. The method of claim 8, comprising: receiving, by the decoder, a sps_dmvr_enabled_flag flag, the sps_dmvr_enabled_flag flag indicating whether the DMVR is enabled to decode the video blocks in the sequence.

10. receiving, by the decoder, an sps_dmvr_picture_header_present_flag flag when an sps_dmvr_enabled_flag flag is true, the sps_dmvr_picture_header_present_flag flag signaling whether a picture_disable_dmvr_flag flag is signaled in the picture header of each picture that references the current SPS; receiving, by the decoder, a sps_bdof_picture_header_present_flag flag when a sps_bdof_enabled_flag flag is true, the sps_bdof_picture_header_present_flag flag signaling whether a picture_disable_bdof_flag flag is signaled in the picture header of each picture that references the current SPS; receiving, by the decoder, a sps_prof_picture_header_present_flag flag when a sps_prof_enabled_flag flag is true, the sps_prof_picture_header_present_flag flag signaling whether a picture_disable_prof_flag flag is signaled in the picture header of each picture that references the current SPS; The method of claim 8 further comprising:

11. applying the BDOF at a decoder to generate, by the decoder, the predicted samples of inter-coded blocks that are not coded in affine mode when the value of the sps_bdof_picture_header_present_flag flag is false; applying the PROF at a decoder to generate, by the decoder, the predicted samples of the inter-coded block coded in the affine mode when the value of the sps_prof_picture_header_present_flag flag is false; applying the DMVR at a decoder to generate, by the decoder, the predicted samples of the inter-coded blocks that are not coded in the affine mode when the value of the sps_dmvr_picture_header_present_flag flag is false; The method of claim 10 further comprising:

12. receiving, by the decoder, a picture header control flag when the sps_dmvr_picture_header_present_flag flag is true, the picture header control flag being the picture_disable_dmvr_flag flag signaling that the DMVR is disabled for a slice that references the picture header. The method of claim 10.

13. receiving, by the decoder, a picture header control flag when the sps_bdof_picture_header_present_flag flag is true, the picture header control flag being the picture_disable_bdof_flag flag signaling that the BDOF is disabled for a slice that references the picture header. The method of claim 10.

14. receiving, by the decoder, a picture header control flag when the sps_prof_picture_header_present_flag flag is true, the picture header control flag being the picture_disable_prof_flag flag signaling that the PROF is disabled for a slice that references the picture header. The method of claim 10.

15. 1. A computing device comprising: one or more processors; and a non-transitory computer-readable storage medium storing instructions executable by the one or more processors, the one or more processors comprising: obtaining a reference picture I associated with a video block in the video signal; obtaining a prediction sample I(i,j) of the video block from a reference block in the reference picture I, where i and j represent coordinates of a sample comprising the video block; controlling intra prediction refinement by optical flow (PROF) parameters of a PROF derivation process by applying a right shift to the internal PROF parameters based on a bit shift value to achieve a preset accuracy, the internal PROF parameters including horizontal gradient value, vertical gradient value, horizontal motion differential value, and vertical motion differential value derived for the prediction sample I(i,j); obtaining a prediction refinement value for a sample in the video block based on the prediction sample I(i,j) having been applied to the video block by the PROF derivation process; and obtaining a prediction sample for the video block based on a combination of the prediction sample and the prediction refinement value.

16. 16. The computing device of claim 15, wherein the preset precision is equal to 1 / 32 pel.

17. the one or more processors configured to control the internal PROF parameter of the PROF derivation process by applying a right shift to the internal PROF parameter; obtaining a horizontal gradient value of the first predicted sample I(i,j) based on a difference between the first predicted sample I(i+1,j) and the first predicted sample I(i-1,j); obtaining a vertical gradient value of a first predicted sample I(i,j) based on a difference between the first predicted sample I(i,j+1) and the first predicted sample I(i,j−1); Obtaining control point motion vectors (MVs) of the first predicted sample I(i,j), the control point MVs including MVs of a top left corner, a top right corner, and a bottom left corner block of a block including the video block; obtaining affine model parameters derived based on the control points MV; A horizontal MV difference Δv for the first predicted sample I(i,j) based on the affine model parameters x (i, j) and the vertical MV difference Δv y Obtaining (i, j); The horizontal MV difference Δv x (i, j) and the vertical MV difference Δv y 16. The computing device of claim 15, further configured to: right-shift (i,j) by the bit-shift value, wherein the bit-shift value is equal to eight.

18. the one or more processors: The horizontal MV difference Δv x Clipping (i,j) to the symmetric range of [-31,31]; The vertical MV difference Δv y20. The computing device of claim 18, further configured to: clip (i,j) to the symmetric range of [-31,31].

19. The one or more processors configured to obtain the prediction refinement values ​​for samples in the video block, The horizontal gradient value, the horizontal MV difference Δv x (i,j), the vertical gradient value, and the vertical MV difference Δv y The computing device of claim 18 , further configured to obtain the prediction refinement value based on (i, j).

20. 1. A computing device comprising: one or more processors; and a non-transitory computer-readable storage medium storing instructions executable by the one or more processors, the one or more processors comprising: The first reference picture I associated with the video block (0) and the second reference picture I (1) and obtaining, in display order, the first reference picture I (0) is before the current picture, and the second reference picture I (1) is after the current picture; and The first reference picture I (0) A first predicted sample I of the video block from a reference block in (0) (i,j), where i and j represent the coordinates of one sample comprising the current picture; The second reference picture I (1) A second predicted sample I of the video block from a reference block in (1) Obtaining (i, j); Controlling an internal BDOF parameter of a BDOF derivation process by applying a shift to the internal BDOF parameter, wherein the internal BDOF parameter is a shift of the first prediction sample I (0)(i, j), the second predicted sample I (1) (i, j), the first predicted sample I (0) (i, j) and the second predicted sample I (1) (i,j) and horizontal and vertical gradient values ​​derived based on intermediate BDOF-derived parameters, the intermediate BDOF-derived parameters including sGxdI, sGydI, sGx2, sGxGy, and sGy2 parameters, sGxdI and sGydI including cross-correlation values ​​between the horizontal gradient values ​​and sample difference values ​​and between the vertical gradient values ​​and the sample difference values, sGx2 and sGy2 including auto-correlation values ​​of the horizontal gradient values ​​and the vertical gradient values, and sGxGy including the cross-correlation value between the horizontal gradient values ​​and the vertical gradient values; The first prediction sample I (0) (i, j) and the second predicted sample I (1) obtaining a motion refinement for samples within the video block based on the BDOF applied to the video block based on (i,j); obtaining bi-predictive samples of the video block based on the motion refinement.

21. the one or more processors configured to control an internal bit depth of the BDOF derivation process for representation precision of internal BDOF-derived parameters, The first prediction sample I (0) (i, j), the second predicted sample I (1) obtaining the intermediate BDOF-derived parameters based on (i, j); obtaining a horizontal motion refinement value based on the sGx2 and sGxdI parameters; obtaining a vertical motion refinement value based on the sGy2, sGydI, and sGxGy parameters; The computing device of claim 20 , further configured to: clip the horizontal motion refinement value and the vertical motion refinement value to a symmetric range of [−31, 31].

22. 1. A non-transitory computer-readable storage medium storing a plurality of programs for execution by a computing device having one or more processors, the plurality of programs, when executed by the one or more processors, causing the computing device to: receiving, by a decoder, three control flags in a sequence parameter set (SPS), where a first control flag indicates whether bidirectional optical flow (BDOF) is enabled for decoding a video block in a current video sequence, a second control flag indicates whether prediction refinement with optical flow (PROF) is enabled for decoding the video block in the current video sequence, and a third control flag indicates whether decoder-side motion vector refinement (DMVR) is enabled for decoding the video block in the current video sequence; receiving, by the decoder, a first presence flag in the SPS when the first control flag is true, a second presence flag in the SPS when the second control flag is true, and a third presence flag in the SPS when the third control flag is true; receiving, by the decoder, a first picture control flag in a picture header of each picture when the first presence flag in the SPS indicates that the BDOF is disabled for the video block in the picture; receiving, by the decoder, a second picture control flag in the picture header of each picture when the second presence flag in the SPS indicates that the PROF is disabled for the video block in the picture; and receiving, by the decoder, a third picture control flag in the picture header of each picture when the third presence flag in the SPS indicates that the DMVR is disabled for the video block in the picture.

23. The plurality of programs are written to the computing device, receiving, by the decoder, when a sps_dmvr_enabled_flag flag is true, a sps_dmvr_picture_header_present_flag flag, the sps_dmvr_picture_header_present_flag flag signaling whether a picture_disable_dmvr_flag flag is signaled in the picture header of each picture that references the current SPS; receiving, by the decoder, when a sps_bdof_enabled_flag flag is true, a sps_bdof_picture_header_present_flag flag, the sps_bdof_picture_header_present_flag flag signaling whether a picture_disable_bdof_flag flag is signaled in the picture header of each picture that references the current SPS; 23. The non-transitory computer-readable storage medium of claim 22, further comprising: receiving, by the decoder, a sps_prof_picture_header_present_flag flag when a sps_prof_enabled_flag flag is true, the sps_prof_picture_header_present_flag flag signaling whether a picture_disable_prof_flag flag is signaled in the picture header of each picture that references a current SPS.

24. The plurality of programs are written to the computing device, applying the BDOF at a decoder to generate the predicted samples for inter-coded blocks that are not coded in affine mode when the value of the sps_bdof_picture_header_present_flag flag is false; applying the PROF at a decoder to generate the predicted samples of the inter-coded block coded in the affine mode when the value of the sps_prof_picture_header_present_flag flag is false; 23. The non-transitory computer-readable storage medium of claim 22, further comprising: applying the DMVR at a decoder to generate the predicted samples for the inter-coded blocks that are not coded in the affine mode when a value of the sps_dmvr_picture_header_present_flag flag is false.

25. The plurality of programs are written to the computing device, 25. The non-transitory computer-readable storage medium of claim 24, further comprising: receiving, by the decoder, a picture header control flag when the sps_dmvr_picture_header_present_flag flag is true, the picture header control flag being the picture_disable_dmvr_flag flag signaling that the DMVR is disabled for a slice that references the picture header.

26. The plurality of programs are written to the computing device, 25. The non-transitory computer-readable storage medium of claim 24, further comprising: receiving, by the decoder, a picture header control flag when the sps_bdof_picture_header_present_flag flag is true, the picture header control flag being the picture_disable_bdof_flag flag signaling that the BDOF is disabled for a slice that references the picture header.

27. The plurality of programs are written to the computing device, 25. The non-transitory computer-readable storage medium of claim 24, further comprising: receiving, by the decoder, a picture header control flag when the sps_prof_picture_header_present_flag flag is true, the picture header control flag being the picture_disable_prof_flag flag signaling that the PROF is disabled for a slice that references the picture header.

28. The plurality of programs are written to the computing device, 25. The non-transitory computer-readable storage medium of claim 24, further comprising: receiving, by the decoder, a picture header control flag when the sps_prof_picture_header_present_flag flag is true, the picture header control flag being the picture_disable_prof_flag flag signaling that the PROF is disabled for a slice that references the picture header.

Citation Information

Patent Citations

  • Method and apparatus for signaling video coding information

    JP2022547599A

  • Method and apparatus for optical flow prediction refinement, bidirectional optical flow and decoder-side motion vector refinement

    JP7281602B2

  • Method and apparatus for optical flow prediction refinement, bidirectional optical flow and decoder-side motion vector refinement - Patents.com

    JP7559136B2

  • Method and apparatus for encoding or decoding video

    US20210084315A1