Methods and apparatuses for prediction refinement with optical flow, bi-directional optical flow, and decoder-side motion vector refinement

Abstract

A bit depth representation method, device, and non-transient computer-readable storage medium are provided. A decoder acquires a reference picture (I) associated with a video block within a video signal. The decoder acquires predicted samples (I(i,j)) of the video block from the reference block within the reference picture (I). The decoder controls the internal PROF parameters by applying a right shift to the internal PROF parameters of the PROF derivation process based on bit shift values ​​to achieve a preset precision. The decoder acquires predicted improvement values ​​for samples within the video block based on the application of the PROF derivation process to the video block based on the predicted samples (I(i,j)). The decoder acquires the predicted samples of the video block based on a combination of the predicted samples and the predicted improvement values.

Description

Technology Field

[0001] This application is based on and claims priority to Provisional Application No. 62 / 913,141 filed on October 9, 2019, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to video coding and compression. More specifically, the present disclosure relates to methods and apparatus for two inter-prediction tools investigated in versatile video coding (VVC) standards, namely prediction refinement with optical flow (PROF) and bi-directional optical flow (BDOF). Background Technology

[0003] Various video coding techniques can be used to compress video data. Video coding is performed according to one or more video coding standards. For example, video coding standards include Universal Video Coding (VVC), Joint Exploration Test Model (JEM), High-Efficiency Video Coding (H.265 / HEVC), Advanced Video Coding (H.264 / AVC), and Moving Picture Expert Group (MPEG) coding. Video coding generally utilizes prediction methods (e.g., inter-prediction, intra-prediction, etc.) that take advantage of redundancy present in video images or sequences. An important goal of video coding techniques is to compress video data by using lower bit rates while avoiding or minimizing degradation in video quality.

[0004] Examples of the present disclosure provide methods and apparatus for bit-depth control for bidirectional optical flow.

[0005] According to a first aspect of the present disclosure, a method for representing the bit depth of a PROF is provided. This method involves a decoder having a reference picture ( I It may include a step of acquiring ). The decoder also, a reference picture ( I Predicted samples of the video block from the reference block within ) I ( i , j )) can be obtained. i and j represent the coordinates of a single sample containing a video block. The decoder can further control the internal PROF parameters by applying a right shift to the internal PROF parameters of the PROF derivation process based on bit shift values ​​to achieve preset precision. The internal PROF parameters are the prediction samples ( I ( i , j It may include horizontal slope values, vertical slope values, horizontal motion difference values, and vertical motion difference values ​​derived for )). The decoder also includes prediction samples ( I ( i , j Predicted improvement values ​​for samples within a video block can be obtained based on the application of a PROF derivation process to the video block based on )). The decoder can additionally obtain predicted samples of the video block based on a combination of predicted samples and predicted improvement values.

[0006] According to a second aspect of the present disclosure, a bit depth control method for BDOF is provided. The method involves a decoder having a first reference picture associated with a video block ( I (0) ) and second reference picture( I (1)It may include a step of obtaining ). In the display order, the first reference picture ( I (0) ) is in front of the current picture and the second reference picture( I (1) ) is behind the current picture. The decoder also, the first reference picture ( I (0) First prediction samples of the video block from the reference block within ) I (0) ( i , j )) can be obtained. i and j represent the coordinates of one sample having the current picture. The decoder additionally, a second reference picture ( I (1) Second prediction samples of the video block from the reference block within ) I (1) ( i , j )) can be obtained. The decoder can also control the internal BDOF parameters by applying a shift to the internal BDOF parameters of the BDOF derivation process. The internal BDOF parameters are the first prediction samples ( I (0) ( i , j )), second prediction samples( I (1) ( i , j )), first prediction samples( I (0) ( i , j )) and second prediction samples( I (1) ( i , jIt includes sample differences between )) and horizontal slope values ​​and vertical slope values ​​derived based on intermediate BDOF derivation parameters. The intermediate BDOF derivation parameters may include sGxdI, sGydI, sGx2, sGxGy, and sGy2 parameters. sGxdI and sGydI may include cross-correlation values ​​between horizontal slope values ​​and sample difference values, and between vertical slope values ​​and sample difference values. sGx2 and sGy2 may include auto-correlation values ​​of horizontal slope values ​​and vertical slope values. sGxGy may include cross-correlation values ​​between horizontal slope values ​​and vertical slope values.

[0007] According to a third aspect of the present disclosure, a BDOF, PROF, and DMVR method is provided. The method may include the step of a decoder receiving three control flags in a sequence parameter set (SPS). A first control flag indicates whether BDOF is enabled to decode video blocks in the current video sequence. A second control flag indicates whether PROF is enabled to decode video blocks in the current video sequence. A third control flag indicates whether DMVR is enabled to decode video blocks in the current video sequence. The decoder may also receive a first presence flag in the SPS when the first control flag is true, a second presence flag in the SPS when the second control flag is true, and a third presence flag in the SPS when the third control flag is true. The decoder may additionally receive a first picture control flag in the picture header of each picture when a first presence flag in the SPS indicates that BDOF is disabled for video blocks in the picture. The decoder may also receive a second picture control flag in the picture header of each picture when a second presence flag in the SPS indicates that PROF is disabled for video blocks in the picture. The decoder may additionally receive a third picture control flag in the picture header of each picture when a third presence flag in the SPS indicates that DMVR is disabled for video blocks in the picture.

[0008] According to a fourth aspect of the present disclosure, a computing device is provided. The computing device may include one or more processors and non-transient computer-readable memory for storing instructions executable by one or more processors. One or more processors have a reference picture ( IIt can be configured to acquire ). One or more processors can also, reference picture ( I Predicted samples of the video block from the reference block within ) I ( i , j It can be configured to acquire )). i and j represent the coordinates of a single sample having a video block. One or more processors may be additionally configured to control internal PROF parameters by applying a right shift to the internal PROF parameters of the PROF derivation process based on a bit shift value to achieve preset precision. The internal PROF parameters are prediction samples ( I ( i , j It may include horizontal slope values, vertical slope values, horizontal motion difference values, and vertical motion difference values ​​derived for )). One or more processors may also include prediction samples ( I ( i , j It can be configured to obtain prediction improvement values ​​for samples within a video block based on the application of a PROF derivation process to the video block based on )). One or more processors may be additionally configured to obtain prediction samples of the video block based on a combination of prediction samples and prediction improvement values.

[0009] According to a fifth aspect of the present disclosure, a computing device is provided. The computing device may include one or more processors and non-transient computer-readable memory for storing instructions executable by one or more processors. One or more processors have a first reference picture ( I (0) ) and second reference picture( I (1) It can be configured to acquire ). In the display order, the first reference picture ( I (0)) is in front of the current picture and the second reference picture( I (1) ) is behind the current picture. One or more processors are the first reference picture ( I (0) First prediction samples of the video block from the reference block within ) I (0) ( i , j It can be further configured to acquire )). i and j represent the coordinates of a single sample having the current picture. One or more processors also, prediction samples ( I ( i , j It can be configured to obtain prediction improvement values ​​for samples within a video block based on the application of a PROF derivation process to the video block based on )). One or more processors are configured to obtain a second reference picture ( I (1) Second prediction samples of the video block from the reference block within ) I (1) ( i , j It may be further configured to obtain )). One or more processors may be further configured to control internal BDOF parameters by applying shifts to the internal BDOF parameters of the BDOF derivation process. The internal BDOF parameters are the first prediction samples ( I (0) ( i , j )), second prediction samples( I (1) ( i , j )), first prediction samples( I (0) ( i , j )) and second prediction samples( I (1) ( i , jIt includes sample differences between )), and horizontal slope values ​​and vertical slope values ​​derived based on intermediate BDOF derivation parameters. The intermediate BDOF derivation parameters may include sGxdI, sGydI, sGx2, sGxGy, and sGy2 parameters. sGxdI and sGydI may include cross-correlation values ​​between horizontal slope values ​​and sample difference values, and between vertical slope values ​​and sample difference values. sGx2 and sGy2 may include auto-correlation values ​​of horizontal slope values ​​and vertical slope values. sGxGy may include cross-correlation values ​​between horizontal slope values ​​and vertical slope values. One or more processors ( I (0) ( i , j )) and second prediction samples( I (1) ( i , j It may be further configured to obtain motion enhancements for samples within a video block based on the application of BDOF to the video block based on )). One or more processors may be further configured to obtain bidirectional prediction samples of the video block based on the motion enhancements.

[0010] According to a sixth aspect of the present disclosure, a non-transient computer-readable storage medium is provided in which instructions are stored. When instructions are executed by one or more processors of the device, the instructions may cause the device to receive three control flags in a sequence parameter set (SPS). A first control flag indicates whether BDOF is enabled to decode video blocks in the current video sequence. A second control flag indicates whether PROF is enabled to decode video blocks in the current video sequence. A third control flag indicates whether DMVR is enabled to decode video blocks in the current video sequence. The instructions may also cause the device to receive a first presence flag in the SPS when the first control flag is true, a second presence flag in the SPS when the second control flag is true, and a third presence flag in the SPS when the third control flag is true. The commands may additionally cause the device to receive a first picture control flag in the picture header of each picture when a first presence flag in the SPS indicates that BDOF is disabled for video blocks in the picture. The commands may also cause the device to receive a second picture control flag in the picture header of each picture when a second presence flag in the SPS indicates that PROF is disabled for video blocks in the picture. The commands may additionally cause the device to receive a third picture control flag in the picture header of each picture when a third presence flag in the SPS indicates that DMVR is disabled for video blocks in the picture.

[0011] It should be understood that the general description above and the following detailed description are merely examples and are not intended to limit the content of this disclosure. Brief explanation of the drawing

[0012] The accompanying drawings included in part of this specification and constituting therein illustrate examples consistent with the present disclosure and serve to explain the principles of the present disclosure together with the description. FIG. 1 is a block diagram of an encoder according to an example of the present disclosure. FIG. 2 is a block diagram of a decoder according to an example of the present disclosure. FIG. 3a is a drawing illustrating block partitions in a multi-type tree structure according to an example of the present disclosure. FIG. 3b is a drawing illustrating block partitions in a multi-type tree structure according to an example of the present disclosure. FIG. 3c is a drawing illustrating block partitions in a multi-type tree structure according to an example of the present disclosure. FIG. 3d is a drawing illustrating block partitions in a multi-type tree structure according to an example of the present disclosure. FIG. 3e is a drawing illustrating block partitions in a multi-type tree structure according to an example of the present disclosure. FIG. 4 is a drawing example of a bidirectional light flow (BDOF) model according to an example of the present disclosure. FIG. 5a is an example of an affine model according to one example of the present disclosure. FIG. 5b is an example of an affine model according to one example of the present disclosure. FIG. 6 is an example of an affine model according to one example of the present disclosure. FIG. 7 is an example of a prediction improvement (PROF) by light flow according to an example of the present disclosure. FIG. 8 is a workflow of a BDOF according to an example of the present disclosure. FIG. 9 is a workflow of PROF according to an example of the present disclosure. FIG. 10 is a BDOF method according to an example of the present disclosure. FIG. 11 is a BDOF and PROF method according to an example of the present disclosure. FIG. 12 is a BDOF, PROF, and DMVR method according to an example of the present disclosure. FIG. 13 is an example of a workflow of PROF for bidirectional prediction according to an example of the present disclosure. FIG. 14 is an example of pipeline stages of BDOF and PROF processes according to the present disclosure. FIG. 15 is an example of a method for deriving the slope of BDOF according to the present disclosure. FIG. 16 is an example of a method for deriving the slope of PROF according to the present disclosure. FIG. 17a is an example of deriving template samples for an affine mode according to an example of the present disclosure. FIG. 17b is an example of deriving template samples for an affine mode according to an example of the present disclosure. FIG. 18a is an example of exclusively enabling PROF and LIC for an affine mode according to one example of the present disclosure. FIG. 18b is an example of jointly enabling PROF and LIC for an affine mode according to an example of the present disclosure. FIG. 19a is a drawing illustrating a proposed padding method applied to a 16×16 BDOF CU according to one example of the present disclosure. FIG. 19b is a drawing illustrating a proposed padding method applied to a 16×16 BDOF CU according to one example of the present disclosure. FIG. 19c is a drawing illustrating a proposed padding method applied to a 16×16 BDOF CU according to one example of the present disclosure. FIG. 19d is a drawing illustrating a proposed padding method applied to a 16×16 BDOF CU according to one example of the present disclosure. FIG. 20 is a drawing illustrating a computing environment combined with a user interface according to an example of the present disclosure. Specific details for implementing the invention

[0013] Exemplary embodiments will now be mentioned, and such examples are illustrated in the accompanying drawings. Unless otherwise indicated, the following description relates to the accompanying drawings in which the same numbers in different drawings represent the same or similar elements. The implementations presented in the following description of exemplary embodiments do not represent all implementations consistent with the present disclosure. Instead, the implementations are merely examples of devices and methods consistent with aspects related to the present disclosure, such as those mentioned in the appended claims.

[0014] The terms used in this disclosure are intended to describe specific embodiments only and are not intended to limit this disclosure. As used in this disclosure and the appended claims, singular forms are intended to include plural forms unless the context clearly indicates otherwise. It will also be understood that the term “and / or” as used herein is intended to mean and include any or all possible combinations of one or more of the associated enumerated items.

[0015] Terms such as “first,” “second,” “third,” etc., may be used to describe various information in this specification, but it will be understood that such information should not be limited to these terms. These terms are used solely to distinguish one category of information from another. For example, without departing from the scope of this disclosure, the first information may be referred to as the second information; similarly, the second information may also be referred to as the first information. As used in this specification, the term “if” may be understood to mean “when,” “at,” or “in response to a judgment” depending on the context.

[0016] The first version of the HEVC standard was finalized in October 2013, offering approximately 50% bitrate savings or equivalent perceptual quality compared to the previous generation video coding standard, H.264 / MPEG AVC. While the HEVC standard provides significant coding improvements over its predecessor, there is evidence that superior coding efficiency can be achieved with additional coding tools compared to HEVC. Based on this, both VCEG and MPEG began exploring new coding technologies for future video coding standardization. A Joint Video Exploration Team (JVET) was formed by ITU-T VECG and ISO / IEC MPEG in October 2015 to begin significant research on advanced technologies capable of enabling substantial improvements in coding efficiency. A reference software called the Joint Exploration Model (JEM) was maintained by the JVET by integrating several additional coding tools on top of the HEVC Test Model (HM).

[0017] In October 2017, a call for proposals (CfP) for video compression with capabilities beyond HEVC was announced by ITU-T and ISO / IEC [9]. In April 2018, 23 CfP responses were received and evaluated at the 10th JVET meeting, demonstrating a compression efficiency gain of approximately 40% compared to HEVC. Based on these evaluation results, JVET launched a new project to develop a new generation of video coding standards named Universal Video Coding (VVC). In the same month, a reference software codebase called the VVC test model (VTM) was established to demonstrate a reference implementation of the VVC standard.

[0018] Like HEVC, VVC is built on a block-based hybrid video coding framework.

[0019] FIG. 1 illustrates a general diagram of a block-based video encoder for VVC. Specifically, FIG. 1 illustrates a conventional encoder (100). The encoder (100) has a video input (110), motion compensation (112), motion estimation (114), intra / inter mode determination (116), a block predictor (140), an adder (128), a transform (130), a quantization (132), prediction-related information (142), an intra prediction (118), a picture buffer (120), a de-quantization (134), a de-transform (136), an adder (126), a memory (124), an in-loop filter (122), an entropy coding (138), and a bitstream (144).

[0020] In the encoder (100), video frames are partitioned into multiple video blocks for processing. For each given video block, a prediction is formed based on an inter-prediction approach or an intra-prediction approach.

[0021] A prediction residual representing the difference between the current video block, which is part of the video input (110), and the predictor, which is part of the block predictor (140), is transmitted from the adder (128) to the transform (130). Then, transform coefficients are transmitted from the transform (130) to the quantization (132) for entropy reduction. Subsequently, the quantized coefficients are supplied to the entropy (138) to generate a compressed video bitstream. As illustrated in FIG. 1, prediction-related information (142) from the intra / inter mode determination (116), such as video block partition information, motion vectors (MV), reference picture index, and intra prediction mode, is also supplied via entropy coding (138) and stored in the compressed bitstream (144). The compressed bitstream (144) contains the video bitstream.

[0022] In the encoder (100), decoder-related circuits are also required to reconstruct pixels for prediction purposes. First, the prediction residual is reconstructed through inverse quantization (134) and inverse transformation (136). This reconstructed prediction residual is combined with a block predictor (140) to generate unfiltered reconstructed pixels for the current video block.

[0023] Spatial prediction (or "intra-prediction") predicts the current video block using pixels from samples of already coded neighboring blocks (referred to as reference samples) in the same video frame as the current video block.

[0024] Time prediction (also referred to as "inter prediction") predicts the current video block using reconstructed pixels from already coded video pictures. Time prediction reduces the temporal redundancy inherent in the video signal. The time prediction signal for a given coding unit (CU) or coding block is typically signaled by one or more MVs that indicate the amount and direction of movement between the current CU and its time reference. Additionally, if multiple reference pictures are supported, an additional reference picture index is transmitted, which is used to identify from which reference picture in the reference picture store the time prediction signal comes.

[0025] Motion estimation (114) takes the video input (110) and signal from the picture buffer (120) and outputs the motion estimation signal to the motion compensation (112). The motion compensation (112) takes the video input (110), the signal from the picture buffer (120), and the motion estimation signal from the motion estimation (114) and outputs the motion compensation signal to the intra / inter mode determination (116).

[0026] After spatial and / or temporal prediction is performed, the intra / inter mode determination (116) of the encoder (100) selects the best prediction mode based, for example, on a rate distortion optimization method. Then, a block predictor (140) is subtracted from the current video block; the resulting prediction residual is de-correlated using a transform (130) and quantization (132). The resulting quantized residual coefficients are de-quantized by de-quantization (134) and de-quantized by inverse transform (136) to form a reconstructed residual, and the reconstructed residual is then added back to the prediction block to form the reconstructed signal of the CU. Before the reconstructed CU enters the reference picture storage of the picture buffer (120) and is used to code subsequent video blocks, additional in-loop filtering (122), such as a deblocking filter, a sample adaptive offset (SAO), and / or an adaptive in-loop filter (ALF), may be applied to the reconstructed CU. To form the output video bitstream (144), the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all transmitted to the entropy coding unit (138) to be further compressed and packed to form the bitstream.

[0027] Figure 1 provides a block diagram of a typical block-based hybrid video encoding system. Input video signals are processed in blocks (referred to as CUs). In VTM-1.0, a CU can be up to 128×128 pixels. However, unlike HEVC, which partitions blocks based solely on quad trees, in VVC, a single coding tree unit (CTU) is divided into CUs based on quad / binary / ternary trees to adapt to various local characteristics. Additionally, the concept of multiple partition unit types in HEVC is eliminated; that is, the separation of CUs, prediction units (PUs), and transform units (TUs) no longer exists in VVC; instead, each CU is always used as the base unit for both prediction and transformation without additional partitions. In a multi-type tree structure, a CTU is first partitioned by a quad tree structure. Subsequently, each quad tree leaf node can be further partitioned by binary and ternary tree structures. As shown in FIGS. 3a, 3b, 3c, 3d, and 3e, there are five partitioning types, namely quaternary partitioning, horizontal binary partitioning, vertical binary partitioning, horizontal ternary partitioning, and vertical ternary partitioning.

[0028] FIG. 3a illustrates a diagram illustrating a block quaternary partition in a multi-type tree structure according to the present disclosure.

[0029] FIG. 3b illustrates a block vertical binary partition in a multi-type tree structure according to the present disclosure.

[0030] FIG. 3c illustrates a diagram illustrating a block horizontal binary partition in a multi-type tree structure according to the present disclosure.

[0031] FIG. 3d illustrates a block vertical ternary partition in a multi-type tree structure according to the present disclosure.

[0032] FIG. 3e illustrates a diagram illustrating a block horizontal ternary partition in a multi-type tree structure according to the present disclosure.

[0033] In FIG. 1, spatial prediction and / or temporal prediction may be performed. Spatial prediction (or "intra prediction") predicts the current video block using pixels from samples of already coded neighboring blocks (referred to as reference samples) within the same video picture / slice. Spatial prediction reduces spatial redundancy inherent in the video signal. Temporal prediction (also referred to as "inter prediction" or "motion compensation prediction") predicts the current video block using reconstructed pixels from already coded video pictures. Temporal prediction reduces temporal redundancy inherent in the video signal. A temporal prediction signal for a given CU is typically signaled by one or more motion vectors (MVs) indicating the amount and direction of motion between the current CU and its temporal reference. Additionally, if multiple reference pictures are supported, a reference picture index is additionally transmitted, which is used to identify from which reference picture in the reference picture store the temporal prediction signal originates. After spatial and / or temporal prediction, the encoder's mode determination block selects the best prediction mode based, for example, on a rate distortion optimization method. Then, the prediction block is subtracted from the current video block; the prediction residuals are inversely correlated and quantized using a transform. The quantized residual coefficients are inversely quantized and inversely transformed to form the reconstructed residuals, which are then added back to the prediction block to form the reconstructed signal of the CU. Additionally, before the reconstructed CU enters the reference picture store and is used to code subsequent video blocks, in-loop filtering, such as a block separation filter, a sample adaptive offset (SAO), and an adaptive in-loop filter (ALF), may be applied to the reconstructed CU. To form the output video bitstream, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all transmitted to the entropy coding unit, where they are further compressed and packed to form the bitstream.

[0034] FIG. 2 illustrates a general block diagram of a video decoder for VVC. Specifically, FIG. 2 illustrates a block diagram of a conventional decoder (200). The decoder (200) has a bitstream (210), entropy decoding (212), inverse quantization (214), inverse transform (216), an adder (218), intra / inter mode selection (220), intra prediction (222), memory (230), an in-loop filter (228), motion compensation (224), a picture buffer (226), prediction-related information (234), and a video output (232).

[0035] The decoder (200) is similar to the reconstruction-related section residing in the encoder (100) of FIG. 1. In the decoder (200), the incoming video bitstream (210) is first decoded through entropy decoding (212) to derive quantized coefficient levels and prediction-related information. Subsequently, the quantized coefficient levels are processed through inverse quantization (214) and inverse transformation (216) to obtain a reconstructed prediction residual. A block predictor mechanism implemented in an intra / inter mode selector (220) is configured to perform intra prediction (222) or motion compensation (224) based on the decoded prediction information. A set of unfiltered reconstructed pixels is obtained by using a summer (218) to sum the reconstructed prediction residual from the inverse transformation (216) and the prediction output generated by the block predictor mechanism.

[0036] The reconstructed block may additionally pass through a loop filter (228) before being stored in a picture buffer (226) which functions as a reference picture storage. The reconstructed video in the picture buffer (226) can be transmitted to drive a display device and can also be used to predict future video blocks. In situations where the loop filter (228) is switched on, a filtering operation is performed on these reconstructed pixels to yield a final reconstructed video output (232).

[0037] Figure 2 provides a general block diagram of a block-based video decoder. The video bitstream is first entropy decoded in an entropy decoding unit. Coding mode and prediction information are transmitted to a spatial prediction unit (if intra-coded) or a temporal prediction unit (if inter-coded) to form a prediction block. Residual transformation coefficients are transmitted to an inverse quantization unit and an inverse transformation unit to reconstruct the residual block. Subsequently, the prediction block and the residual block are added together. The reconstructed block may additionally undergo in-loop filtering before being stored in a reference picture repository. Then, the reconstructed video in the reference picture repository is not only transmitted to drive a display device but is also used to predict subsequent video blocks.

[0038] In general, the basic inter-prediction techniques applied to VVC remain identical to those of HEVC, except that some modules are further extended and / or enhanced. In particular, for all preceding video standards, when a coding block is single-predicted, a single coding block can be associated with only a single MV, or when a coding block is bidirectionally predicted, a single coding block can be associated with two MVs. Due to these limitations of conventional block-based motion compensation, small motions may still remain within the predicted samples after motion compensation, thus negatively affecting the overall efficiency of motion compensation. To improve both the granularity and precision of MVs, two sample-wise enhancement methods based on optical flow—namely, Bidirectional Optical Flow for Affine Modes (BDOF) and Prediction Enhancement by Optical Flow (PROF)—are investigated for the current VVC standard. Next, the key technical aspects of the two inter-coding tools are briefly reviewed.

[0039] Bidirectional light flow

[0040] In VVC, BDOF is applied to improve the prediction samples of bidirectionally predicted coding blocks. Specifically, as illustrated in FIG. 4, BDOF is a sample-by-sample motion improvement performed on block-based motion compensation predictions when bidirectional prediction is used.

[0041] FIG. 4 illustrates an example of a BDOF model according to the present disclosure.

[0042] Movement refinement of each 4×4 subblock v x , v y ) is calculated by minimizing the difference between L0 and L1 prediction samples after BDOF is applied within a single 6×6 window (Ω) around the subblock. Specifically, ( v x , v y The value of ) is derived as follows:

[0043]

[0044] Here is a floor function; clip3(min, max, x) is a function that clips a given value (x) within the range [min, max]; the symbol >> represents a bitwise right shift operation; and the symbol << represents a bitwise left shift operation; th BDOF is a motion improvement threshold to prevent errors propagated due to irregular local movements, and this is 1< <max(5, bit-depth -7) and, bit-depth is the internal bit depth. In (1), And, am.

[0045] S 1, S 2, S 3, S 5 and S The values ​​of 6 are calculated as follows:

[0046]

[0047]

[0048] Here

[0049]

[0050] I ( k ) ( i , j ) is a list generated with medium high precision (i.e., 16 bits). k ( k Coordinates of the predicted signal within = 0, 1) i , j It is a sample value from ) and; and ε₀ is the horizontal and vertical slope of a sample, obtained by directly calculating the difference between two neighboring samples, i.e., as follows:

[0051]

[0052] Based on the motion improvement derived in (1), the final bidirectional prediction samples of the CU are calculated by interpolating the L0 / L1 prediction samples along the motion trajectory based on the light flow model, as indicated by the following equation:

[0053]

[0054]

[0055] Here shift and o offset are the right shift and offset values ​​applied to combine the L0 and L1 prediction signals for bidirectional prediction, and these are 15 - bit-depth and 1 << (14 - bit-depthIt is equal to ) + 2·(1 << 13). Based on the bit depth control method above, it is guaranteed that the maximum bit depth of the intermediate parameters of the entire BDOF process does not exceed 32 bits, and that the largest input for multiplication is within 15 bits, that is, that a single 15-bit multiplier is sufficient for BDOF implementations.

[0056] Affin mode

[0057] In HEVC, only translational motion models are applied for motion compensation prediction. In the real world, there are many types of motion, such as zooming in / out, rotation, perspective motions, and other irregular motions. In VVC, affine motion compensation prediction is applied by signaling a flag for each intercoding block to indicate whether translational motion or an affine motion model is applied for inter prediction. In the current VVC design, two affine modes, including a 4-parameter affine mode and a 6-parameter affine mode, are supported for a single affine coding block.

[0058] The 4-parameter affine model has the following parameters: two parameters for translational movement in the horizontal and vertical directions, one parameter for zoom movement, and one parameter for rotational movement in both directions. The horizontal zoom parameter is the same as the vertical zoom parameter. The horizontal rotation parameter is the same as the vertical rotation parameter. To achieve better accommodation of motion vectors and affine parameters, in VVC, such affine parameters are converted into two MVs (also referred to as control point motion vectors (CPMV)) located at the top left corner and top right corner of the current block. As illustrated in FIGS. 5a and 5b, the affine motion field of the block is described by two control point MVs (V0, V1).

[0059] FIG. 5a illustrates an example of a 4-parameter affine model according to the present disclosure.

[0060] FIG. 5b illustrates an example of a 4-parameter affine model according to the present disclosure.

[0061] Based on the movement of control points, the movement field of a single affine-coded block ( v x , v y ) is explained as follows:

[0062]

[0063] The 6-parameter affine mode has the following parameters: two parameters for translation in the horizontal and vertical directions, one parameter for zoom movement in the horizontal direction and one parameter for rotational movement, one parameter for zoom movement in the vertical direction and one parameter for rotational movement, respectively. The 6-parameter affine movement model is coded as three MVs in three CPMVs.

[0064] FIG. 6 illustrates an example of a 6-parameter affine model according to the present disclosure.

[0065] As illustrated in FIG. 6, three control points of a single 6-parameter affine block are located at the top left, top right, and bottom left corners of the block. Movement at the top left control point is related to translational motion, movement at the top right control point is related to rotation and zoom motion in the horizontal direction, and movement at the bottom left control point is related to rotation and zoom motion in the vertical direction. Compared to a 4-parameter affine motion model, the rotation and zoom motion in the horizontal direction of the 6-parameter model may not be the same as these movements in the vertical direction. Assuming that (V0, V1, V2) are the MVs of the top left, top right, and bottom left corners of the current block in FIG. 6, each sub-block ( v x , v y The motion vector of ) is derived as follows using the three MVs at the control points:

[0066]

[0067] Prediction improvement by optical flow for affine modes

[0068] To improve the precision of affine motion compensation, a PROF that enhances subblock-based affine motion compensation based on an optical flow model is currently being investigated at VVC. Specifically, after performing subblock-based affine motion compensation, a luma prediction sample of a single affine block is modified by a single sample enhancement value derived based on the optical flow equation. In detail, the operations of the PROF can be summarized in the following four steps:

[0069] Step 1: Subblock prediction using the subblock MVs derived from the 4-parameter affine model (6) and the 6-parameter affine model (7) (I ( i , j Subblock-based affine motion compensation is performed to generate )).

[0070] Step 2: The spatial gradients of each predicted sample ( g x ( i , j ), g y ( i , j )) is calculated as follows:

[0071]

[0072] To calculate the gradients, one additional row / column of prediction samples needs to be generated for each face of a subblock. To reduce memory bandwidth and complexity, samples on the extended boundaries are copied from the nearest integer pixel position of the reference picture to avoid additional interpolation processes.

[0073] Step 3: The Luma prediction improvement value is calculated by the following:

[0074]

[0075] Here, Δ v ( i , j )Is v ( i , j Sample location ( denoted by ) i , j Pixel MV and pixel( calculated for ) i , j It is the difference between the subblock MVs of the subblock where ) is located. Additionally, in the current PROF design, after adding prediction improvement to the original prediction sample, a single clip operation is performed to clip the value of the improved prediction sample to within 15 bits, i.e., as follows:

[0076]

[0077] Here I (i , j ) and I r ( i , j ) are each positions ( i , j These are the original prediction samples and improved prediction samples in ).

[0078] FIG. 7 illustrates a PROF process for an affine mode according to the present disclosure. FIG. 7 includes block (710), block (720), and block (730). Block (730) is a rotated block of block (720).

[0079] Since the pixel positions and affine model parameters relative to the subblock center do not vary for each subblock, Δ v ( i , j ) is calculated for the first subblock and can be reused for other subblocks within the same CU. Δ x and Δ y is the sample location ( i , j If Δ is defined as the horizontal and vertical offset from ) to the center of the subblock to which the sample belongs, then v ( i , j ) can be derived as follows:

[0080]

[0081] Based on the affine subblock MV derivation formulas (6) and (7), the MV difference Δ v ( i , j ) can be derived. Specifically, for a 4-parameter affine model, it is as follows:

[0082]

[0083] For the 6-parameter affine model, it is as follows:

[0084]

[0085] Here ( v 0 x , v 0 y ), ( v 1 x , v 1 y ), ( v 2 x , v 2 y ) are the top left, top right, and bottom left control points MV of the current coding block, and w and h is the width and height of the block. In the existing PROF design, the MV difference (Δ v x , Δ v y ) is always derived with a precision of 1 / 32-pel.

[0086] Local lighting compensation

[0087] Local illumination compensation (LIC) is a coding tool used to address the problem of local illumination variations existing between temporally adjacent pictures. To acquire predicted samples for a single current block, pairs of weight and offset parameters are applied to reference samples. A general mathematical model is given as follows:

[0088]

[0089] Here P r [ x + v ] is a motion vector( v It is a reference block indicated by ), and [α, β] are corresponding pairs of weight and offset parameters for the reference block, and P [ x] is the final prediction block. Pairs of weight and offset parameters are estimated using the least linear mean square error (LLMSE) algorithm based on the template of the current block (i.e., neighboring reconstructed samples) and the template's reference block (derived using the motion vector of the current block). By minimizing the mean square difference between the template samples and the template's reference samples, the mathematical expressions for α and β can be derived as follows:

[0090]

[0091] Here I represents the number of samples within the template. P c [ x i ] is the i-th sample of the current block's template, and P r [ x i ] is a motion vector( v This is a reference sample of the i-th template sample based on ).

[0092] In addition to being applied to regular interblocks containing at most one motion vector for each prediction direction (L0 or L1), LIC is also applied to affine mode coded blocks, where one coding block is divided into a number of smaller subblocks and each subblock can be associated with different motion information. As illustrated in FIGS. 17a and 17b (described below), to derive reference samples for the LIC of the affine mode coded blocks, reference samples within the top-level template of one affine coding block are fetched using the motion vector of each subblock in the top-level subblock row, while reference samples in the left-side template are fetched using the motion vectors of the subblocks in the left-side subblock column. Then, the same LLMSE derivation method as illustrated in (12) is applied to (12) to derive LIC parameters based on the composite template.

[0093] FIG. 17a illustrates an example for deriving template samples for an affine mode according to the present disclosure. The example includes a Cur frame (1720) and a Cur CU (1722). The Cur frame (1720) is the current frame. The Cur CU (1722) is the current coding unit.

[0094] FIG. 17b illustrates an example for deriving template samples for an affine mode. The example includes Ref frame (1740), Col CU (1742), A Ref (1743), B Ref (1744), C Ref (1745), D Ref (1746), E Ref (1747), F Ref (1748), and G Ref (1749). Ref frame (1740) is a reference frame. Col CU (1742) is a collocated coding unit. A Ref (1743), B Ref (1744), C Ref (1745), D Ref (1746), E Ref (1747), F Ref (1748), and G Ref (1749) are reference samples.

[0095] Inefficiencies of Prediction Improvement by Optical Flow for Affine Modes

[0096] While PROF can improve the coding efficiency of affine modes, the design can still be further improved. In particular, given that both PROF and BDOF are built upon the optical flow concept, it is highly desirable to harmonize the designs of PROF and BDOF as much as possible so that PROF can maximize the utilization of BDOF's existing logic to enable hardware implementations. Based on these considerations, the following inefficiencies regarding the interaction between current PROF and BDOF designs are identified in this disclosure.

[0097] First, as described in the section "Prediction improvement by optical flow for affine modes," in Equation (8), the precision of the slopes is determined based on the internal bit depth. On the other hand, the MV difference, i.e., Δ v x and Δ v y It is always derived with a precision of 1 / 32-pel. Correspondingly, based on Equation (9), the precision of the derived PROF improvement depends on the internal bit depth. However, to maintain a higher PROF derivation precision, the PROF is applied over the predicted sample values ​​at an intermediate high bit depth (i.e., 16-bit). Therefore, regardless of the internal coding bit depth, the precision of the predicted improvements derived by the PROF must match the precision of the intermediate predicted samples, i.e., 16-bit. In other words, the bit depths representing the MV difference and slopes in the existing PROF design are not perfectly matched to derive accurate predicted improvements for the predicted sample precision (i.e., 16-bit). Meanwhile, based on a comparison of Equations (1), (4), and (8), the existing PROF and BDOF use different precisions to represent the sample slopes and MV difference. As previously noted, this non-integrated design is undesirable for hardware because the existing BDOF logic cannot be reused.

[0098] Secondly, as discussed in the section "Prediction Enhancement by Optical Flow for Affine Modes," when a current affine block is bidirectionally predicted, PROF is applied individually to the prediction samples of lists L0 and L1; then, the enhanced L0 and L1 prediction signals are averaged to generate the final bidirectional prediction signal. Conversely, instead of deriving PROF enhancements for each prediction direction individually, BDOF derives a prediction enhancement once, which is then applied to enhance the combined L0 and L1 prediction signals. Figures 8 and 9 (described below) compare the current BDOF and PROF workflows for bidirectional prediction. In actual codec hardware pipeline designs, different major encoding / decoding modules are typically assigned to each pipeline stage to allow more coding blocks to be processed in parallel. However, due to the differences between BDOF and PROF workflows, this can lead to the difficulty of having a single pipeline design that can be shared by BDOF and PROF, which is unfavorable to actual codec implementations.

[0099] FIG. 8 illustrates the workflow of a BDOF according to the present disclosure. The workflow (800) includes L0 motion compensation (810), L1 motion compensation (820), and BDOF (830). L0 motion compensation (810) may be, for example, a list of motion compensation samples from a previous reference picture. The previous reference picture is a reference picture prior to the current picture within the video block. L1 motion compensation (820) may be, for example, a list of motion compensation samples from a next reference picture. The next reference picture is a reference picture following the current picture within the video block. As described above in relation to FIG. 4, the BDOF (830) takes motion compensation samples from the L0 motion compensation (810) and the L1 motion compensation (820) and outputs prediction samples.

[0100] FIG. 9 illustrates the workflow of an existing PROF according to the present disclosure. The workflow (900) includes L0 motion compensation (910), L1 motion compensation (920), L0 PROF (930), L1 PROF (940), and average (960). L0 motion compensation (910) may be, for example, a list of motion compensation samples from a previous reference picture. The previous reference picture is a reference picture prior to the current picture within the video block. L1 motion compensation (920) may be, for example, a list of motion compensation samples from a next reference picture. The next reference picture is a reference picture following the current picture within the video block. L0 PROF (930) takes L0 motion compensation samples from L0 motion compensation (910) as described above in relation to FIG. 7 and outputs motion improvement values. L1 PROF (940) takes L1 motion compensation samples from L1 motion compensation (920) as described above in relation to FIG. 7 and outputs motion improvement values. Average (960) averages the motion improvement value outputs of L0 PROF (930) and L1 PROF (940).

[0101] Thirdly, for both BDOF and PROF, gradients need to be calculated for each sample within the current coding block, which requires generating an additional row / column of predicted samples on each face of the block. To avoid the additional computational complexity of sample interpolation, predicted samples in the extended area around the block are copied directly from reference samples at integer positions (i.e., without interpolation). However, according to the existing design, integer samples at different locations are selected to generate the gradient values ​​for BDOF and PROF. Specifically, for BDOF, integer reference samples located to the left of the predicted sample (for horizontal gradients) and above the predicted sample (for vertical gradients) are used; for PROF, the integer reference sample closest to the predicted sample is used for gradient calculations. Similar to the bit depth representation problem, this non-integrated gradient calculation method is also undesirable for hardware codec implementations.

[0102] Fourth, as previously noted, the motivation for PROF is to compensate for small differences in MV between the subblock MV derived from the center of the subblock to which the sample belongs and the MV of each sample. According to the current PROF design, PROF is always called when a coding block is predicted by an affine mode. However, as indicated in Equations (6) and (7), the subblock MVs of a single affine block are derived from control point MVs. Therefore, when the difference between control point MVs is relatively small, the MVs at each sample position must be consistent. In such cases, since the benefit of applying PROF may be very limited, it may not be worthwhile to perform PROF when considering the performance / complexity trade-off.

[0103] Improvements to prediction enhancements based on optical flow for affine modes

[0104] In the present disclosure, methods are provided to improve and simplify existing PROF designs to enable hardware codec implementations. In particular, special care is taken to harmonize the designs of BDOF and PROF in order to share existing BDOF logic with PROF as much as possible. In general, the main aspects of the techniques proposed in the present disclosure are summarized as follows.

[0105] First, to improve the coding efficiency of PROF while achieving one or more integrated designs, a method is proposed to integrate the representation bit depth of sample gradients and the MV difference used by BDOF and PROF.

[0106] Second, to enable hardware pipeline design, it is proposed to harmonize the workflow of PROF and BDOF for bidirectional prediction. Specifically, unlike the existing PROF, which derives prediction improvements for L0 and L1 individually, the proposed method derives a prediction improvement applied to the combined L0 and L1 prediction signals once.

[0107] Third, two methods are proposed to harmonize the derivation of integer reference samples to calculate the gradient values ​​used by BDOF and PROF.

[0108] Fourth, to reduce computational complexity, early termination methods are proposed to adaptively disable the PROF process for affine coding blocks when specific conditions are met.

[0109] Design of improved bit depth representation of PROF gradients and MV differences

[0110] As analyzed in the "Problem Statement" section, the representation bit depths of the MV difference and the sample gradients of the current PROF are not aligned to derive accurate prediction improvements. Furthermore, the representation bit depths of the sample gradients and the MV difference are inconsistent between BDOF and PROF, which is unfavorable to hardware. In this section, an improved bit depth representation method is proposed by extending the bit depth representation method of BDOF to PROF. Specifically, in the proposed method, the horizontal and vertical gradients at each sample position are calculated as follows:

[0111]

[0112] Additionally, Δ x and Δ y Assuming that is the horizontal and vertical offset expressed with ¼-pel accuracy from a single sample position to the center of the subblock to which the sample belongs, then Δ is the corresponding PROF MV difference at the sample position. v ( x , y ) is derived as follows:

[0113]

[0114] Here is the bit depth of the gradient values ​​used by the BDOF process, i.e. = max (5, (bitdepth - 7)) + 1. In Equations (13) and (14), c , d , e and f These are affine parameters derived based on affine control point MVs. Specifically, for a 4-parameter affine model, they are as follows:

[0115]

[0116] For the 6-parameter affine model, it is as follows:

[0117]

[0118] Here ( v 0 x , v 0 y ), ( v 1 x , v 1 y ), ( v 2 x , v 2 y ) are the top-left, top-right, and bottom-left control points MV of the current coding block expressed in 1 / 16-pel precision, and w and h is the width and height of the block.

[0119] In the discussion above, as illustrated in Equations (13) and (14), a pair of fixed right shifts are applied to calculate the values ​​of the slopes and MV differences. In practice, different bitwise right shifts are applied to (13) and (14) to achieve varying representational precisions of the MV differences and slopes, representing different trade-offs between the bit width of the internal PROF derivation process and intermediate computational precision. For example, when the input video contains a lot of noise, the derived slopes may not be reliable for representing accurate local horizontal / vertical slope values ​​in each sample. In such cases, it is more reasonable to use more bits than slopes to represent the MV differences. On the other hand, when the input video exhibits normal motion, the MV differences derived by the affine model must be very small. If they are very small, using high-precision MV differences cannot provide additional benefit in increasing the precision of the derived PROF improvement. In other words, in such cases, it is more advantageous to use more bits to represent the slope values. Based on the above considerations, in one or more embodiments of the present disclosure, a general method is proposed below for calculating the slopes and MV differences for PROF. Specifically,n a Assuming that a right shift is applied to the difference between neighboring predicted samples, that is, that the horizontal and vertical slopes at each sample position are calculated by the following equation:

[0120]

[0121] Δ, the difference between the corresponding PROF MV at the sample position v ( x , y ) must be calculated as follows:

[0122]

[0123] Here, Δ x and Δ y is the horizontal and vertical offset expressed with ¼-pel accuracy from a single sample location to the center of the subblock to which the sample belongs, and c , d , e and f are affine parameters derived based on 1 / 16-pel affine control point MVs. Finally, the final PROF improvement of the sample is calculated as follows:

[0124]

[0125] In another embodiment, a different PROF bit depth control method is proposed as follows. In this method, n a By applying right-shifts of bits to the difference values ​​of neighboring predicted samples, the horizontal and vertical slopes at each sample position are still calculated. Δ, which is the difference between the corresponding PROF MV at the sample position, is used. v ( x , y ) must be calculated as follows:

[0126]

[0127] Additionally, to maintain the overall PROF derivation at an appropriate internal bit depth, clipping is applied to the derived MV difference as follows:

[0128]

[0129] The limit here is It is the same threshold as, and clip3(min, max, x) is a function that clips a given value (x) within the range [min, max]. For example, n b The value of is 2 max(5, bitdepth -7) It is set to. Finally, the sample's PROF improvement is calculated as follows:

[0130]

[0131] Additionally, in one or more embodiments of the present disclosure, a PROF bit depth control solution is proposed. In this method, horizontal and vertical PROF movement improvements at each sample position (i, j) are derived as follows:

[0132]

[0133] Additionally, the derived horizontal and vertical movement improvements are clipped as follows:

[0134]

[0135] Given the motion improvements derived above, the final PROF sample improvement at position (i, j) is calculated as follows:

[0136]

[0137] In another embodiment of the present disclosure, a different PROF bit depth control solution is proposed. In the second method, horizontal and vertical PROF movement improvements at sample positions (i, j) are derived as follows:

[0138]

[0139] Next, the derived movement improvements are clipped as follows:

[0140]

[0141] Therefore, given the motion improvements derived above, the final PROF sample improvement at position (i, j) is calculated as follows:

[0142]

[0143] In one or more embodiments of the present disclosure, a method for controlling the precision of motion improvement in a solution and a method for deriving PROF sample improvement in a second solution are proposed. Specifically, by this method, horizontal and vertical PROF motion improvements at each sample position (i, j) are derived as follows:

[0144]

[0145] Additionally, the derived horizontal and vertical movement improvements are clipped as follows:

[0146]

[0147] Given the motion improvements derived above, the final PROF sample improvement at position (i, j) is calculated as follows:

[0148]

[0149] In one or more embodiments, the following method for deriving PROF sample improvements is proposed:

[0150] First, calculate the PROF horizontal and vertical motion improvements to achieve 1 / 32-pel precision by applying the following fixed right shifts as instructed:

[0151]

[0152] Second, clip the calculated PROF motion improvement values ​​to a single symmetric range [-31, 31].

[0153]

[0154] Third, the PROF improvement of the sample is calculated as follows:

[0155]

[0156] Figure 10 illustrates a bit depth representation method of PROF. This method can be applied to a decoder, for example.

[0157] In step (1010), the decoder is associated with a reference picture ( I You can obtain ).

[0158] In step (1012), the decoder is a reference picture ( I Predicted samples of the video block from the reference block within ) I ( i , j )) can be obtained. i and j can represent the coordinates of a single sample containing a video block.

[0159] In step (1014), the decoder can control the internal PROF parameters by applying a right shift to the internal PROF parameters of the PROF derivation process based on a bit shift value to achieve a preset precision. The internal PROF parameters are prediction samples ( I ( i , j Includes horizontal slope values, vertical slope values, horizontal movement difference values ​​and vertical movement difference values ​​derived for )).

[0160] In step (1016), the decoder predicts samples ( I ( i , j Predicted improvement values ​​for samples within a video block can be obtained based on the application of the PROF derivation process to the video block based on ))

[0161] In step (1018), the decoder can obtain predicted samples of the video block based on a combination of predicted samples and predicted improvement values.

[0162] Additionally, as exemplified below, the same parameter derivation method can also be applied to the BDOF sample improvement process:

[0163]

[0164] Here, sGxdI, sGx2, sGxGym, sGxGys, and sGy2 are intermediate BDOF derivation parameters.

[0165] Figure 11 illustrates a bit depth control method for BDOF. This method can be applied to a decoder, for example.

[0166] In step (1110), the decoder is associated with a first reference picture ( I (0) ) and second reference picture( I (1) ) can be obtained. In the display order, the first reference picture ( I (0) ) is in front of the current picture and the second reference picture( I (1) ) is currently behind the picture.

[0167] In step (1112), the decoder is a first reference picture ( I (0) First prediction samples of the video block from the reference block within ) I (0) ( i , j )) can be obtained. i and j can represent the coordinates of one sample having the current picture.

[0168] In step (1114), the decoder is a second reference picture ( I (1) Second prediction samples of the video block from the reference block within ) I (1) ( i , j You can obtain ))

[0169] In step (1116), the decoder can control the internal BDOF parameters by applying a shift to the internal BDOF parameters of the BDOF derivation process. The internal BDOF parameters are the first prediction samples ( I (0) ( i , j )), second prediction samples( I (1) ( i , j )), first prediction samples( I (0) ( i , j )) and second prediction samples( I (1) ( i , j It includes sample differences between )), and horizontal slope values ​​and vertical slope values ​​derived based on intermediate BDOF derivation parameters. The intermediate BDOF derivation parameters include the sGxdI, sGydI, sGx2, sGxGy, and sGy2 parameters. sGxdI and sGydI include cross-correlation values ​​between horizontal slope values ​​and sample difference values, and between vertical slope values ​​and sample difference values. sGx2 and sGy2 include auto-correlation values ​​of horizontal slope values ​​and vertical slope values. sGxGy includes cross-correlation values ​​between horizontal slope values ​​and vertical slope values.

[0170] In step (1118), the decoder uses first prediction samples ( I (0) ( i , j )) and second prediction samples( I (1) ( i , j Motion enhancements for samples within a video block can be obtained based on the application of BDOF to the video block based on ))

[0171] In step (1120), the decoder can obtain bidirectional prediction samples of the video block based on motion improvements.

[0172] Harmonized workflows of BDOF and PROF for bidirectional forecasting

[0173] As previously discussed, when a single affine coding block is bidirectionally predicted, the current PROF is applied unidirectionally. More specifically, PROF sample improvements are derived individually and applied to the prediction samples in lists L0 and L1. Subsequently, the respective improved prediction signals from lists L0 and L1 are averaged to generate the block's final bidirectional prediction signal. This contrasts with the BDOF design, where sample improvements are derived and applied to the bidirectional prediction signal. This difference between the bidirectional prediction workflows of BDOF and PROF can be unfavorable to actual codec pipeline design.

[0174] To enable hardware pipeline design, one simplification method according to the present disclosure is to modify the bidirectional prediction process of PROF so that the workflows of two prediction improvement methods are harmonized. Specifically, instead of applying improvements for each prediction direction individually, the proposed PROF method derives prediction improvements once based on the control point MVs of lists L0 and L1; then, the derived prediction improvements are applied to the combined L0 and L1 prediction signals to improve quality. Specifically, based on the MV difference derived in Equation (14), the final bidirectional prediction samples of one affine coding block are calculated by the following proposed method:

[0175]

[0176] Here shift and o offset is a right shift value and an offset value applied to combine L0 and L1 prediction signals for bidirectional prediction, and these are equal to (15 - bit depth) and 1 << (14 - bit depth) + (2 << 13), respectively. Furthermore, as shown in (18), the clip operation in the conventional PROF design (as shown in (9)) is eliminated in the proposed method.

[0177] FIG. 13 illustrates a corresponding PROF process when the proposed bidirectional prediction PROF method is applied. The PROF process (1300) includes L0 motion compensation (1310), L1 motion compensation (1320), and bidirectional prediction PROF (1330). L0 motion compensation (1310) may be, for example, a list of motion compensation samples from a previous reference picture. The previous reference picture is a reference picture prior to the current picture within the video block. L1 motion compensation (1320) may be, for example, a list of motion compensation samples from a next reference picture. The next reference picture is a reference picture following the current picture within the video block. As described above, the bidirectional prediction PROF (1330) takes motion compensation samples from the L0 motion compensation (1310) and the L1 motion compensation (1320) and outputs bidirectional prediction samples.

[0178] To demonstrate the potential benefits of the proposed method for hardware pipeline design, Fig. 14 (described below) illustrates an example for illustrating pipeline stages when both BDOF and the proposed PROF are applied. In Fig. 14, the decoding process of a single inter-block mainly comprises three steps:

[0179] First, parse / decode the MVs of the coding block and fetch the reference samples.

[0180] Second, generate L0 and / or L1 prediction signals of the coding block.

[0181] Third, sample-by-sample refinement of the generated bidirectional prediction samples is performed based on BDOF when the coding block is predicted by a single non-affine mode, or based on PROF when the coding block is predicted by an affine mode.

[0182] FIG. 14 illustrates an exemplary pipeline stage when both BDOF and the proposed PROF are applied in accordance with the present disclosure. FIG. 14 demonstrates the potential benefits of the proposed method for hardware pipeline design. The pipeline stage (1400) includes parsing / decoding of MV and fetching of reference samples (1410), motion compensation (1420), and BDOF / PROF (1430). The pipeline stage (1400) will encode video blocks (BLK0, BKL1, BKL2, BKL3, BLK4). Each video block will start at the parsing / decoding of MV and fetching of reference samples (1410), move to motion compensation (1420), and then sequentially move to motion compensation (1420) and BDOF / PROF (1430). This means that BLK0 will not start in the pipeline stage (1400) until BLK0 moves to motion compensation (1420). This is the same for all stages and video blocks as time moves from T0 to T1, T2, T3, and T4.

[0183] As illustrated in Fig. 14, after the proposed harmony method is applied, both BDOF and PROF are directly applied to the bidirectional prediction samples. Considering that BDOF and PROF are applied to different types of coding blocks (i.e., BDOF is applied to non-affine blocks and PROF is applied to affine blocks), the two coding tools cannot be called simultaneously. Therefore, their corresponding decoding processes can be performed by sharing the same pipeline stage. This is more efficient than existing PROF designs, where it is difficult to allocate the same pipeline stage to both BDOF and PROF due to the different bidirectional prediction workflows of BDOF and PROF.

[0184] In the discussion above, the proposed method considers only the coordination of workflows between BDOF and PROF. However, according to existing designs, the basic operation units for the two coding tools are also performed in different sizes. For example, in the case of BDOF, a single coding block is W s ×H s It is divided into multiple subblocks having a size of , where W s = min(W, 16) and H s= min(H, 16), where W and H are the width and height of the coding block. BDOF operations, such as gradient calculation and sample improvement derivation, are performed independently for each subblock. On the other hand, as previously described, the affine coding block is divided into 4×4 subblocks, and each subblock is assigned a single individual MV derived based on 4-parameter or 6-parameter affine models. Since PROF applies only to affine blocks, its basic operation unit is a 4×4 subblock. Similar to the bidirectional prediction workflow problem, using a different basic operation unit size for PROF than for BDOF is also unfriendly to hardware implementations and makes it difficult for BDOF and PROF to share the same pipeline stage of the entire decoding process. To address this problem, in one or more embodiments, it is proposed to align the subblock size of the affine mode with the subblock size of the BDOF. For example, according to the proposed method, if a coding block is coded by the affine mode, that coding block is W s ×H s It will be divided into subblocks having the size of, where W s = min(W, 16) and H s= min(H, 16), where W and H are the width and height of the coding block. A single individual MV is assigned to each subblock, and each subblock is considered as a single independent PROF operation unit. It is worth noting that an independent PROF operation unit ensures that the PROF operation for it is performed without referencing information from neighboring PROF operation units. For example, the difference in PROF MV at a single sample position is calculated as the difference between the MV at the sample position and the MV at the center of the PROF operation unit where the sample is located; the gradients used by PROF derivation are calculated by padding the samples along each PROF operation unit. The claimed benefits of the proposed method include the following aspects: 1) a simplified pipeline architecture with an integrated base operation unit size for both motion compensation and BDOF / PROF enhancement; 2) reduced memory bandwidth usage due to the enlarged subblock size for affine motion compensation; and 3) reduced per-sample computational complexity of fractional sample interpolation.

[0185] It should also be noted that due to the reduced computational complexity by the proposed method (i.e., Item 3), the existing 6-tab interpolation filter constraint for affine coding blocks can be removed. Instead, the default 8-tab interpolation for non-affine coding blocks is also used for affine coding blocks. In this case, the overall computational complexity can still be favorably compared to the existing PROF design (based on 4×4 subblocks with a 6-tab interpolation filter).

[0186] Harmonization of gradient derivation for BDOF and PROF

[0187] As previously explained, both BDOF and PROF calculate the gradient of each sample within the current coding block, accessing one additional row / column of predicted samples for each face of the block. To avoid additional interpolation complexity, the necessary predicted samples within the extended area around the block boundary are copied directly from the integer reference samples. However, as noted in the "Problem Description" section, integer samples at different locations are used to calculate the gradient values ​​for BDOF and PROF.

[0188] To achieve one or more uniform designs, two methods are proposed to integrate the gradient derivation methods used by BDOF and PROF. In the first method, it is proposed to align the gradient derivation method of PROF with the gradient derivation method of BDOF. For example, by the first method, the integer position used to generate prediction samples in the extended region is determined by flooring down the fractional sample position; that is, the selected integer sample position is located to the left of the fractional sample position (for horizontal gradients) and above the fractional sample position (for vertical gradients). In the second method, it is proposed to align the gradient derivation method of BDOF with the gradient derivation method of PROF. More specifically, when the second method is applied, the integer reference sample closest to the prediction sample is used in the gradient calculations.

[0189] FIG. 15 illustrates an example of using a slope derivation method of BDOF according to the present disclosure. In FIG. 15, empty circles (1510) represent reference samples at integer positions, triangles (1530) represent fractional prediction samples of the current block, and black circles (1520) represent integer reference samples used to fill the extended area of ​​the current block.

[0190] FIG. 16 illustrates an example of using a method for deriving the slope of PROF according to the present disclosure. In FIG. 16, empty circles (1610) represent reference samples at integer positions, triangles (1630) represent fractional prediction samples of the current block, and black circles (1620) represent integer reference samples used to fill the extended area of ​​the current block.

[0191] FIGS. 15 and 16 illustrate corresponding integer sample positions used to derive gradients for BDOF and PROF when the first method (Fig. 15) and the second method (Fig. 16) are applied, respectively. In FIGS. 15 and 16, empty circles represent reference samples at integer positions, triangles represent fractional prediction samples of the current block, and patterned circles represent integer reference samples used to fill the extended area of ​​the current block for gradient derivation.

[0192] Additionally, according to existing BDOF and PROF designs, predictive sample padding is performed at different coding levels. For example, for BDOF, padding is applied along the boundaries of each sbWidth × sbHeight subblock, where sbWidth = min(CUWidth, 16) and sbHeight = min(CUHeight, 16). CUWidth and CUHeight are the width and height of a single CU. On the other hand, padding in PROF is always applied at the 4×4 subblock level. In the discussion above, while the padding subblock sizes are still different, only the padding method is integrated between BDOF and PROF. This is also not favorable for actual hardware implementation, considering that different modules need to be implemented for the padding processes of BDOF and PROF. To achieve one or more integrated designs, it is proposed to integrate the subblock padding sizes of BDOF and PROF. In one or more embodiments of the present disclosure, it is proposed to apply predicted sample padding of a BDOF at a 4×4 level. For example, by this method, a CU is first divided into a plurality of 4x4 subblocks; and after motion compensation of each 4×4 subblock, samples extended along the top / bottom and left / right boundaries are padded by copying corresponding integer sample positions. FIGS. 19a, 19b, 19c, and 19d illustrate an example in which the proposed padding method is applied to a single 16×16 BDOF CU, wherein dashed lines represent the 4x4 subblock boundaries and blue bands represent the padded samples of each 4x4 subblock.

[0193] FIG. 19a illustrates a proposed padding method applied to a 16×16 BDOF CU according to the present disclosure, wherein dashed lines indicate the uppermost left 4×4 subblock boundary (1920).

[0194] FIG. 19b illustrates a proposed padding method applied to a 16×16 BDOF CU according to the present disclosure, wherein dashed lines indicate the uppermost right 4×4 subblock boundary (1940).

[0195] FIG. 19c illustrates a proposed padding method applied to a 16×16 BDOF CU according to the present disclosure, wherein dashed lines indicate the bottom left 4×4 subblock boundary (1960).

[0196] FIG. 19d illustrates a proposed padding method applied to a 16×16 BDOF CU according to the present disclosure, wherein dashed lines indicate the bottom right 4×4 subblock boundary (1980).

[0197] High-level signaling syntax for enabling / disabling BDOF, PROF, and DMVR

[0198] In existing BDOF and PROF designs, two different flags are signaled in the Sequence Parameter Set (SPS) to individually control the enable / disable of the two coding tools. However, due to the similarity between BDOF and PROF, it is more desirable to enable and / or disable BDOF and PROF from a high level using a single, identical control flag. Based on these considerations, a new flag called sps_bdof_prof_enabled_flag is introduced into the SPS, as shown in Table 1. As shown in Table 1, the enable and disable of BDOF depend solely on sps_bdof_prof_enabled_flag. When the flag is equal to 1, BDOF is enabled to code video content in the sequence. Otherwise, when sps_bdof_prof_enabled_flag is equal to 0, BDOF will not be applied. On the other hand, in addition to sps_bdof_prof_enabled_flag, an SPS level affine control flag, namely sps_affine_enabled_flag, is also used to conditionally enable and disable PROF. When both of the two flags, sps_bdof_prof_enabled_flag and sps_affine_enabled_flag, are equal to 1, PROF is enabled for all coding blocks coded in affine mode. When the flag sps_bdof_prof_enabled_flag is equal to 1 and sps_affine_enabled_flag is equal to 0, PROF is disabled.

[0199] Table 1 Modified SPS Syntax Table with Proposed BDOF / PROF Enable / Disable Flags

[0200]

[0201] sps_bdof_prof_enabled_flagSpecifies whether bidirectional optical flow and prediction enhancement by optical flow are enabled. When sps_bdof_prof_enabled_flag is equal to 0, both bidirectional optical flow and prediction enhancement by optical flow are disabled. When sps_bdof_prof_enabled_flag is equal to 1 and sps_affine_enabled_flag is equal to 1, both bidirectional optical flow and prediction enhancement by optical flow are enabled. Otherwise (when sps_bdof_prof_enabled_flag is equal to 1 and sps_affine_enabled_flag is equal to 0), bidirectional optical flow is enabled and prediction enhancement by optical flow is disabled.

[0202] sps_bdof_prof_dmvr_slice_preset_flag Specifies when the flag slice_disable_bdof_prof_dmvr_flag is signaled at the slice level. When the flag is equal to 1, the syntax slice_disable_bdof_prof_dmvr_flag is signaled for each slice referencing the current set of sequence parameters. Otherwise (when sps_bdof_prof_dmvr_slice_present_flag is equal to 0), the syntax slice_disabled_bdof_prof_dmvr_flag will not be signaled at the slice level. When the flag is not signaled, it is inferred to be 0.

[0203] Additionally, when the proposed SPS level BDOF and PROF control flags are used, the corresponding control flag no_bdof_constraint_flag within the general constraint information syntax must also be modified according to the table below:

[0204]

[0205] Same as 1 no_bdof_prof_constraint_flag Specifies that is equal to 0. no_bdof_constraint_flag equal to 0 does not impose a constraint.

[0206] In addition to the SPS BDOF / PROF syntax above, it is proposed to introduce another control flag at the slice level, namely, slice_disable_bdof_prof_dmvr_flag to disable BDOF, PROF, and DMVR. The SPS flag sps_bdof_prof_dmvr_slice_present_flag, which is signaled at the SPS when either of the DMVR or BDOF / PROF sps level control flags is true, is used to indicate the presence of slice_disable_bdof_prof_dmvr_flag. If present, slice_disable_bdof_dmvr_flag is signaled. Table 2 illustrates the modified slice header syntax table after the proposed syntax is applied. In another embodiment, it is proposed to use two control flags in the slice header to individually control the enable / disable of BDOF and DMVR and the enable / disable of PROF. For example, two flags are used in the slice header by this method: one flag slice_disable_bdof_dmvr_slice_flag is used to control the on / off of BDOF and DMVR, and the other flag disable_prof_slice_flag is used to control the on / off of PROF alone.

[0207] Table 2 Modified SPS Syntax Table with Proposed BDOF / PROF Enable / Disable Flags

[0208]

[0209] In another embodiment, it is proposed to control BDOF and PROF individually by two different SPS flags. For example, to enable / disable the two tools individually, two separate SPS flags sps_bdof_enable_flag and sps_prof_enable flag are introduced. Additionally, to forcibly disable the PROF tool, one high-level control flag no_prof_constraint_flag needs to be added to the general_constrain_info() syntax table.

[0210]

[0211] sps_bdof_enabled_flag Specifies whether bidirectional optical flow is enabled. When sps_bdof_enabled_flag is equal to 0, bidirectional optical flow is disabled. When sps_bdof_enabled_flag is equal to 1, bidirectional optical flow is enabled.

[0212] sps_prof_enabled_flag Specifies whether prediction enhancement by light flow is enabled. When sps_prof_enabled_flag is equal to 0, prediction enhancement by light flow is disabled. When sps_prof_enabled_flag is equal to 1, prediction enhancement by light flow is enabled.

[0213]

[0214] Same as 1 no_prof_constraint_flag Specifies that sps_prof_enabled_flag is equal to 0. no_prof_constraint_flag equal to 0 does not impose constraints.

[0215] At the slice level, in one or more embodiments of the present disclosure, it is proposed to introduce another control flag at the slice level, namely, slice_disable_bdof_prof_dmvr_flag to disable BDOF, PROF, and DMVR together. In another embodiment, it is proposed to add two separate flags at the slice level, namely slice_disable_bdof_dmvr_flag and slice_disable_prof_flag. The first flag (i.e., slice_disable_bdof_dmvr_flag) is used to adaptively switch BDOF and DMVR on / off for a single slice, while the second flag (i.e., slice_disable_prof_flag) is used to control the enable and disable of the PROF tool at the slice level. Additionally, when the second method is applied, the flag slice_disable_bdof_dmvr_flag needs to be signaled only when the SPS BDOF or SPS DMVR flag is enabled, and the flag needs to be signaled only when the SPS PROF flag is enabled.

[0216] At the 16th JVET meeting, the Picture Header was adopted in the VVC draft. The Picture Header signals once per slice, while the syntax elements previously in the Slice Header remain unchanged per slice.

[0217] Based on the adopted picture header, in one or more embodiments of the present disclosure, it is proposed to control BDOF, DMVR, and PROF control flags from the current slice header to the picture header. For example, in the proposed method, three different control flags sps_dmvr_picture_header_present_flag, sps_bdof_picture_header_present_flag, and sps_prof_picture_header_present_flag are signaled in the SPS. When one of the three flags is signaled as true, one additional control flag will be signaled in the picture header to indicate that the corresponding tools (i.e., DMVR, BDOF, and PROF) are enabled or disabled for the slices referencing the picture header. The proposed syntax elements are specified as follows.

[0218]

[0219] sps_dmvr_picture_header_preset_flag Specifies whether the flag picture_disable_dmvr_flag is signaled in the picture header. When the flag is equal to 1, the syntax picture_disable_dmvr_flag is signaled for each picture referencing the current set of sequence parameters. Otherwise, the syntax picture_disable_dmvr_flag will not be signaled in the picture header. When the flag is not signaled, it is inferred to be 0.

[0220] sps_bdof_picture_header_preset_flagSpecifies whether the flag picture_disable_bdof_flag is signaled in the picture header. When the flag is equal to 1, the syntax picture_disable_bdof_flag is signaled for each picture referencing the current set of sequence parameters. Otherwise, the syntax picture_disable_bdof_flag will not be signaled in the picture header. When the flag is not signaled, it is inferred to be 0.

[0221] sps_prof_picture_header_preset_flag Specifies whether the flag picture_disable_prof_flag is signaled in the picture header. When the flag is equal to 1, the syntax picture_disable_prof_flag is signaled for each picture referencing the current set of sequence parameters. Otherwise, the syntax picture_disable_prof_flag will not be signaled in the picture header. When the flag is not signaled, it is inferred to be 0.

[0222]

[0223] picture_disable_dmvr_flag Specifies whether the dmvr tool is enabled for slices referencing the current picture header. When the flag is equal to 1, the dmvr tool is enabled for slices referencing the current picture header. Otherwise, the dmvr tool is disabled for slices referencing the current picture header. When the flag does not exist, the flag is inferred to be 0.

[0224] picture_disable_bdof_flag Specifies whether the bdof tool is enabled for slices referencing the current picture header. When the flag is equal to 1, the bdof tool is enabled for slices referencing the current picture header. Otherwise, the bdof tool is disabled for slices referencing the current picture header.

[0225] picture_disable_prof_flag Specifies whether the prof tool is enabled for slices referencing the current picture header. When the flag is equal to 1, the prof tool is enabled for slices referencing the current picture header. Otherwise, the prof tool is disabled for slices referencing the current picture header.

[0226] Figure 12 illustrates the BDOF, PROF, and DMVR methods. These methods can be applied to a decoder, for example.

[0227] In step (1210), the decoder may receive three control flags from the sequence parameter set (SPS). The first control flag indicates whether BDOF is enabled to decode video blocks in the current video sequence. The second control flag indicates whether PROF is enabled to decode video blocks in the current video sequence. The third control flag indicates whether DMVR is enabled to decode video blocks in the current video sequence.

[0228] In step (1212), the decoder can receive a first existence flag from the SPS when the first control flag is true, a second existence flag from the SPS when the second control flag is true, and a third existence flag from the SPS when the third control flag is true.

[0229] In step (1214), the decoder can receive a first picture control flag in the picture header of each picture when the first presence flag in the SPS indicates that BDOF is disabled for video blocks in the picture.

[0230] In step (1216), the decoder can receive a second picture control flag in the picture header of each picture when the second presence flag in the SPS indicates that the PROF is disabled for the video blocks in the picture.

[0231] In step (1218), the decoder can receive a third picture control flag in the picture header of each picture when the third presence flag in the SPS indicates that the DMVR is disabled for the video blocks in the picture.

[0232] Early termination of PROF based on control point MV difference

[0233] According to the current PROF design, PROF is always invoked when a coding block is predicted by an affine mode. However, as indicated in Equations (6) and (7), the subblock MVs of a single affine block are derived from control point MVs. Therefore, when the difference between control point MVs is relatively small, the MVs at each sample position must be consistent. In such cases, the benefit of applying PROF can be very limited. Therefore, to further reduce the average computational complexity of PROF, it is proposed to adaptively skip PROF-based sample improvement based on the maximum MV difference between the subblock-specific MV and the sample-specific MV within a single 4x4 subblock. Since the values ​​of the PROF MV differences of samples within a single 4x4 subblock are symmetric with respect to the subblock center, the maximum horizontal and vertical PROF MV differences can be calculated based on Equation (10) as follows:

[0234]

[0235] According to the present disclosure, different metrics may be used to determine whether the MV difference is small enough to skip the PROF process.

[0236] For example, based on Equation (19), the PROF process may be omitted when the sum of the absolute maximum horizontal MV difference and the absolute maximum vertical MV difference is less than a predefined threshold, i.e., as follows:

[0237]

[0238] In another example, and If the maximum value of is not greater than the threshold, the PROF process may be skipped.

[0239]

[0240] Here, MAX(a, b) is a function that returns the larger value among the input values ​​(a, b).

[0241] In addition to the two examples above, the concept of the present disclosure is also applicable to cases where other metrics are used to determine whether the MV difference is small enough to skip the PROF process.

[0242] In the above method, PROF is skipped based on the magnitude of the MV difference. On the other hand, in addition to the MV difference, PROF sample improvement is also calculated based on local gradient information at each sample location within a motion-compensated block. For prediction blocks containing less high-frequency details (e.g., flat areas), the gradient values ​​tend to be small, so the values ​​of the derived sample improvements should be small. Taking this into consideration, according to another embodiment, it is proposed to apply PROF only to prediction samples of blocks containing sufficiently high-frequency information.

[0243] Different metrics may be used to determine whether a block contains sufficient high-frequency information to warrant a call to the PROF process. For example, this decision is based on the average magnitude (i.e., absolute value) of the slopes of the samples within the prediction block. If the average magnitude is smaller than a threshold, the prediction block is classified as a flat region, and PROF should not be applied; otherwise, the prediction block is considered to contain sufficient high-frequency details, provided that PROF is still applicable. In another example, the maximum magnitude of the slopes of the samples within the prediction block may be used. If the maximum magnitude is smaller than a threshold, PROF should be skipped for the block. In yet another example, to determine whether PROF should be applied to the block, the difference between the maximum and minimum sample values ​​of the prediction block ( I max - I min ,) may be used. If this difference value is smaller than a threshold, PROF for the block should be skipped. It is also worth noting that the concept of the present disclosure is applicable to cases where other metrics are used to determine whether a given block contains sufficiently high-frequency information.

[0244] Handling of interactions between PROF and LIC for affine modes

[0245] Since neighboring reconstructed samples (i.e., templates) of the current block are used by LIC to derive linear model parameters, the decoding of a single LIC coding block depends on the entire reconstruction of its neighboring samples. Due to this interdependence, for actual hardware implementations, LIC needs to be performed at the reconstruction stage where neighboring reconstructed samples become available for deriving LIC parameters. Because block reconstruction must be performed sequentially (i.e., one by one), throughput (i.e., the amount of work that can be performed in parallel per unit of time) is a critical issue to consider when applying other coding methods to LIC coding blocks. In this section, two methods are proposed to handle the interaction when both PROF and LIC are enabled for affine mode.

[0246] In the first embodiment of the present disclosure, it is proposed to exclusively apply a PROF mode and a LIC mode to a single affine coding block. As previously discussed, in conventional designs, PROF is implicitly applied to all affine blocks without signaling, while a single LIC flag is signaled or followed at the coding block level to indicate whether a LIC mode is applied to a single affine block. According to the method of the present disclosure, it is proposed to conditionally apply PROF based on the value of the LIC flag of a single affine block. When the flag is equal to 1, only LIC is applied by adjusting the prediction samples of the entire coding block based on the LIC weight and offset. Otherwise (i.e., if the LIC flag is equal to 0), PROF is applied to the affine coding block to improve the prediction samples of each subblock based on the optical flow model.

[0247] FIG. 18a illustrates an exemplary flowchart of a decoding process based on the proposed method, wherein PROF and LIC are not allowed to be applied simultaneously.

[0248] FIG. 18a illustrates an example of a decoding process based on a proposed method in which PROF and LIC are not allowed, in accordance with the present disclosure. The decoding process (1820) includes a determination step (1822), a LIC (1824), and a PROF (1826). The determination step (1822) determines whether the LIC flag is on, and the next step is taken according to the determination. LIC (1824) is the application of LIC when the LIC flag is set. PROF (1826) is the application of PROF when the LIC flag is not set.

[0249] In a second embodiment of the present disclosure, it is proposed to apply LIC after PROF to generate prediction samples of a single affine block. For example, after subblock-based affine motion compensation is completed, the prediction samples are enhanced based on PROF sample enhancement; then, LIC is performed by applying a pair of weights and offsets (derived from the template and its reference samples) to the PROF-adjusted prediction samples to obtain the final prediction samples of the block, as exemplified below:

[0250]

[0251] Here P r [ x + v ] is a motion vector( v It is the reference block of the current block indicated by ); α and β are the LIC weight and offset; P [ x ] is the final prediction block; Δ I [ x ] is a PROF improvement as derived from (17).

[0252] FIG. 18b illustrates an example of a decoding process to which PROF and LIC are applied according to the present disclosure. The decoding process (1860) includes affine motion compensation (1862), LIC parameter derivation (1864), PROF (1866), and LIC sample adjustment (1868). Affine motion compensation (1862) applies affine motion and is an input to LIC parameter derivation (1864) and PROF (1866). LIC parameter derivation (1864) is applied to derive LIC parameters. PROF (1866) is the application of PROF. LIC sample adjustment (1868) is the combination of LIC weights and offset parameters with PROF.

[0253] FIG. 18b illustrates an exemplary decoding workflow when the second method is applied. As shown in FIG. 18b, since LIC computes a linear model of LIC using a template (i.e., neighboring reconstructed samples), LIC parameters can be derived as soon as neighboring reconstructed samples become available. This means that PROF improvement and LIC parameter derivation can be performed simultaneously.

[0254] LIC weights and offsets (i.e., α and β) and PROF improvement (i.e., Δ I [ x ]) are generally floating-point numbers. In favorable hardware implementations, such floating-point operations are typically implemented by multiplying a single integer value followed by a right shift operation by the number of bits. In existing LIC and PROF designs, because the two tools are designed separately, N LIC Bits and N PROF Two different right shifts are applied in two stages, each corresponding to a bit.

[0255] According to the third embodiment, to improve coding gain when PROF and LIC are jointly applied to an affine coding block, it is proposed to apply LIC-based and PROF-based sample adjustments with high precision. This is achieved by combining their two right-shift operations into one and applying it to the end to derive the final predicted samples of the current block (as shown in (12)).

[0256] Solution to the multiplication overflow problem when combining PROF by weighted prediction and bi-prediction with CU-level weights (BCW).

[0257] According to the current PROF design of the VVC working draft, PROF can be applied in conjunction with weighted prediction (WP). For example, when combined, the prediction signal of a single affine CU will be generated by the following procedures:

[0258] First, position( x , y For each sample in ), L0 prediction improvement based on PROF (Δ I 0( x , y Calculate )) and the original L0 prediction sample( I 0( x , y Adding improvements to )) that is:

[0259]

[0260] Here I 0 ' ( x , y ) is an improved sample and; g h 0( x , y ) and g v 0( x , y ) and Δ v x 0( x , y ) and Δ v y 0( x , y) is a position( x , y These are the L0 horizontal / vertical tilts and L0 horizontal / vertical movement improvements in ).

[0261] Second, position ( x , y For each sample in ), L1 prediction improvement based on PROF (Δ I 1( x , y Calculate )) and the original L1 prediction samples( I 1( x , y Adding improvements to )) that is:

[0262]

[0263] Here I 1 ' ( x , y ) is an improved sample and; g h 1( x , y ) and g v 1( x , y ) and Δ v x 1( x , y ) and Δ v y 1( x , y ) is a position( x , y These are improvements to L1 horizontal / vertical inclinations and L1 horizontal / vertical movements in ).

[0264] Third, the improved L0 and L1 prediction samples are combined, i.e.:

[0265]

[0266] Here W 0 and W 1 is the WP and BCW weight; shift and offsetis the offset and right shift applied to the weighted average of the L0 and L1 prediction signals for bidirectional prediction for WP and BCW. Here, the parameters for WP are W 0 and W 1 and offset Meanwhile, the parameters for BCW are W 0 and W 1 and shift Includes

[0267] As can be seen from the equations above, the sample-specific improvement, i.e., Δ I 0( x , y ) and Δ I 1( x , y Due to ), the predicted samples after PROF (i.e., I 0 ' ( x , y ) and I 1 ' ( x , y )) are the original prediction samples (i.e., I 0( x , y ) and I 1( x , y It will have an increased dynamic range compared to )). Considering that the improved prediction samples will be multiplied by WP and BCW weighting factors, this will increase the length of the required multiplier. For example, based on the current design, when the internal coding bit depth is in the range of 8 to 12 bits, the prediction signal ( I 0( x , y ), I 1( x , y The dynamic range of )) is 16-bit. However, after PROF, the prediction signal ( I 0( x , y ), I 1( x , yThe dynamic range of )) is 17-bit. Therefore, when PROF is applied, this can potentially cause 16-bit multiplication overflow issues. To address these overflow problems, several methods are proposed below:

[0268] First, in the first method, it is proposed that WP and BCW be disabled when PROF is applied to a single affine CU.

[0269] Second, in the second method, improved prediction samples ( I 0 ' ( x , y ), I 1 ' ( x , y The dynamic range of )) is the original prediction samples( I 0( x , y ), I 1( x , y It is proposed to apply a clip operation to the derived sample enhancements before adding them to the original prediction samples to have the same dynamic bit depth as )). For example, by this method, the sample enhancements (Δ in (23) and (24) I 0( x , y ), Δ I 1( x , y )) will be modified by introducing a single clip operation as shown below:

[0270]

[0271] Here dI = dI base + max(0, BD - 12) And, BD is the internal coding bit depth; dI base is the base bit depth value. In one or more embodiments, dI base It is suggested to set the value of dI to 14. In another embodiment, it is suggested to set this value to 13. In another embodiment, it is suggested to directly set the value of dI to a fixed value. For example, it is suggested to set the value of dI to 13, that is, the sample improvement will be clipped to the range [-4096, 4095]. In another example, it is suggested to set the value of dI to 14, that is, the sample improvement will be clipped to the range [-8192, 8191].

[0272] First, in the third method, instead of clipping sample improvements so that the improved samples have the same dynamic range as the original predicted samples, it is proposed to directly clip the improved predicted samples. For example, by the third method, the improved L0 and L1 samples will be as follows:

[0273]

[0274] Here dR = 16 + max(0, BD-12) (or equivalently max(16, BD+4) ) and BD is the internal coding bit depth.

[0275] Second, in the fourth method, it is proposed to apply specific right shifts to the L0 and L1 prediction samples normalized prior to WP and BCW; subsequently, the final prediction samples are adjusted to their original precisions by additional left shifts. For example, the final prediction samples are derived as follows:

[0276]

[0277] Here, nb is the number of additional bit shifts applied, which can be determined based on the corresponding dynamic range of the improvements of the PROF samples.

[0278] Third, in the fifth method, as described below, it is proposed to divide each multiplication of L0 / L1 prediction samples with the corresponding WP / BCW weights of (25) into two multiplications, both of which do not exceed 16 bits:

[0279]

[0280] FIG. 20 illustrates a computing environment (2010) combined with a user interface (2060). The computing environment (2010) may be part of a data processing server. The computing environment (2010) includes a processor (2020), memory (2040), and an I / O interface (2050).

[0281] The processor (2020) typically controls the overall operations of the computing environment (2010), such as operations related to display, data acquisition, data communication, and image processing. The processor (2020) may include one or more processors that execute instructions to perform all or part of the steps of the methods described above. Furthermore, the processor (2020) may include one or more modules that enable interaction between the processor (2020) and other components. The processor may be a Central Processing Unit (CPU), a microprocessor, a single-chip machine, a GPU, etc.

[0282] Memory (2040) is configured to store various types of data to support the operation of the computing environment (2010). Memory (2040) may include predetermined software (2042). Examples of such data include instructions for any applications or methods operating on the computing environment (2010), video data sets, image data, etc. Memory (2040) may be implemented by using any type of volatile or non-volatile memory devices, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disc.

[0283] The I / O interface (2050) provides an interface between the processor (2020) and peripheral interface modules such as a keyboard, a click wheel, and buttons. The buttons may include, but are not limited to, a home button, a scan start button, and a scan stop button. The I / O interface (2050) may be combined with an encoder and a decoder.

[0284] In some embodiments, a non-transient computer-readable storage medium is also provided, comprising a plurality of programs contained in memory (2040), which are executable by a processor (2020) in a computing environment (2010) to perform the methods described above. For example, the non-transient computer-readable storage medium may be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0285] A non-transient computer-readable storage medium stores internally a plurality of programs to be executed by a computing device having one or more processors, wherein when executed by one or more processors, the plurality of programs cause the computing device to perform the previously described method for motion prediction.

[0286] In some embodiments, the computing environment (2010) may be implemented with one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), graphical processing units (GPUs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0287] The description of the present disclosure is provided for illustrative purposes only, but is not intended to be exhaustive or limited to the present disclosure. Many modifications, variations, and alternative embodiments having the advantage of the teachings set forth in the foregoing descriptions and the associated drawings will be apparent to those skilled in the art.

[0288] These examples have been selected and described to explain the principles of the present disclosure and to enable others skilled in the art to understand the present disclosure in relation to various implementations and to make the best use of the underlying principles and various implementations with various modifications suited to the specific use being considered. Accordingly, the scope of the present disclosure is not limited to the specific examples of the disclosed implementations, and it should be understood that modifications and other implementations are intended to be included within the scope of the present disclosure.

Claims

Claim 1 As a video decoding method, a decoder receives three control flags of a sequence parameter set (SPS) — wherein a first control flag indicates whether Bidirectional Light Flow (BDOF) is enabled to decode a video block in the current video sequence, a second control flag indicates whether Predictive Improvement by Light Flow (PROF) is enabled to decode a video block in the current video sequence, and a third control flag indicates whether Decoder Side Motion Vector Improvement (DMVR) is enabled to decode a video block in the current video sequence — the decoder receives a first presence flag in the SPS when the first control flag is true, a second presence flag in the SPS when the second control flag is true, and a third presence flag in the SPS when the third control flag is true; the decoder receives a first picture in the picture header of each picture referencing the SPS when the first presence flag in the SPS is true A video decoding method comprising: a step of receiving a control flag - the first picture control flag indicates whether the BDOF is disabled for a video block in the picture -; a step in which the decoder receives a second picture control flag in the picture header of each picture referencing the SPS when the second presence flag in the SPS is true - the second picture control flag indicates whether the PROF is disabled for a video block in the picture -; and a step in which the decoder receives a third picture control flag in the picture header of each picture referencing the SPS when the third presence flag in the SPS is true - the third picture control flag indicates whether the DMVR is disabled for a video block in the picture -. Claim 2 A video decoding method according to claim 1, wherein the decoder receives three control flags of a sequence parameter set (SPS), the step of the decoder receiving a sps_bdof_enabled_flag flag - the sps_bdof_enabled_flag flag indicates whether the BDOF is enabled to decode a video block in the current video sequence -; the step of the decoder receiving a sps_prof_enabled_flag flag - the sps_prof_enabled_flag flag indicates whether the PROF is enabled to decode a video block in the current video sequence -; and the step of the decoder receiving a sps_dmvr_enabled_flag flag - the sps_dmvr_enabled_flag flag indicates whether the DMVR is enabled to decode a video block in the current video sequence - Claim 3 In paragraph 2, the step of the decoder receiving a first existence flag in the SPS when the first control flag is true, a second existence flag in the SPS when the second control flag is true, and a third existence flag in the SPS when the third control flag is true; the step of the decoder receiving a sps_dmvr_picture_header_present_flag flag when the sps_dmvr_enabled_flag flag is true - the sps_dmvr_picture_header_present_flag flag signals whether the picture_disable_dmvr_flag flag is signaled in the picture header of each picture referencing the SPS -; the step of the decoder receiving a sps_bdof_picture_header_present_flag flag when the sps_bdof_enabled_flag flag is true - the sps_bdof_picture_header_present_flag flag is the A video decoding method comprising: signaling whether the picture_disable_bdof_flag flag is signaled in the picture header of each picture referencing the SPS; and the step of the decoder receiving the sps_prof_picture_header_present_flag flag when the sps_prof_enabled_flag flag is true, wherein the sps_prof_picture_header_present_flag flag signals whether the picture_disable_prof_flag flag is signaled in the picture header of each picture referencing the SPS. Claim 4 A video decoding method according to claim 3, further comprising: a step in which, when the value of the sps_bdof_picture_header_present_flag flag is false, the decoder applies the BDOF to generate a prediction sample of an intercoding block not coded in affine mode; a step in which, when the value of the sps_prof_picture_header_present_flag flag is false, the decoder applies the PROF to generate a prediction sample of an intercoding block coded in affine mode; and a step in which, when the value of the sps_dmvr_picture_header_present_flag flag is false, the decoder applies the DMVR to generate a prediction sample of an intercoding block not coded in affine mode. Claim 5 A video decoding method according to claim 3, wherein the decoder receives a third picture control flag in the picture header of each picture referencing the SPS when the third presence flag in the SPS is true, and the decoder receives a picture header control flag when the sps_dmvr_picture_header_present_flag flag is true—the picture header control flag is a picture_disable_dmvr_flag flag that signals whether the DMVR is disabled for a slice referencing the picture header. Claim 6 A video decoding method according to claim 3, wherein the decoder receives a first picture control flag in the picture header of each picture referencing the SPS when the first presence flag in the SPS is true, and the decoder receives a picture header control flag when the sps_bdof_picture_header_present_flag flag is true—the picture header control flag is a picture_disable_bdof_flag flag that signals whether the BDOF is disabled for the slice referencing the picture header. Claim 7 A video decoding method according to claim 3, wherein the decoder receives a second picture control flag in the picture header of each picture referencing the SPS when the second presence flag in the SPS is true, and the decoder receives a picture header control flag when the sps_prof_picture_header_present_flag flag is true—the picture header control flag is a picture_disable_prof_flag flag that signals whether the PROF is disabled for the slice referencing the picture header. Claim 8 A video decoding method according to claim 1, wherein the bit depth of the video data is greater than 12. Claim 9 A video decoding method according to claim 1, wherein the bit depth of the video data is the same as 12. Claim 10 A computer device comprising: one or more processors as a computing device; a non-transient computer-readable storage medium for storing instructions executable by said one or more processors, wherein said one or more processors are configured to perform a prediction improvement method according to any one of claims 1 to 7. Claim 11 In paragraph 10, a computer device in which the bit depth of the video data is greater than 12. Claim 12 In paragraph 10, a computer device in which the bit depth of the video data is the same as 12. Claim 13 A non-transient computer-readable storage medium storing a bitstream to be decoded by any one of the methods of paragraphs 1 through 7. Claim 14 In paragraph 13, a non-transient computer-readable storage medium in which the bit depth of the video data is greater than 12. Claim 15 In paragraph 13, a non-transient computer-readable storage medium having a bit depth of video data equal to 12. Claim 16 A computer program stored in a computer-readable storage medium, comprising a plurality of instructions executed by one or more processors, wherein the plurality of instructions cause the one or more processors to perform a prediction improvement method described in any one of claims 1 to 7. Claim 17 In paragraph 16, a computer program stored on a computer-readable storage medium, wherein the bit depth of the video data is greater than 12. Claim 18 In paragraph 16, a computer program stored on a computer-readable storage medium, wherein the bit depth of the video data is the same as 12.

Images