Method and device for prediction refinement using optical flow
By coordinating the design of BDOF and PROF tools, unifying bit depth and simplifying pipeline design, the problem of insufficient motion vector granularity and accuracy in inter-frame prediction in the prior art is solved, and the efficiency of video encoding and decoding and the friendliness of hardware implementation are improved.
Patent Information
- Application Number
- CN202210578295.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2019-07-12
- Filing Date
- 2020-07-10
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2040-07-10
AI Technical Summary
Existing video encoding and decoding technologies have insufficient granularity and accuracy of motion vectors in inter-frame prediction, resulting in low encoding and decoding efficiency. Especially when using affine mode, there is an inconsistent bit depth in the design of optical flow prediction refinement tools. and computing complexity issues, it is difficult to share pipeline design in hardware implementation.
A unified approach is proposed to improve the workflow of PROF with BDOF by coordinating the design of bidirectional optical flow (BDOF) and using the predictive refinement (PROF) tool to unify the bit depth representation of sample gradient and motion vector difference. Coordinate, simplify sub-block size and gradient export methods for affine mode, reduce computational complexity, and enable/disable the tool through high-level signaling syntax control.
It improves the encoding efficiency of video encoding and decoding, simplifies hardware pipeline design, reduces memory bandwidth and computing complexity, and realizes more unified and efficient inter prediction refinement, which is suitable for different types of coding blocks.
Smart Images

Figure CN115002454B_ABST
Abstract
Description
[0001] This application is a divisional application of the patent application with the same name with application number 202080059396.0 filed on July 10, 2020.
[0002] CROSS-REFERENCE TO RELATED APPLICATIONS
[0003] This application is based on and claims the benefit of priority of Provisional Application No. 62 / 872,700 filed on July 10, 2019 and Provisional Application No. 62 / 873,837 filed on July 12, 2019, the entire contents of which are incorporated herein by reference for all purposes. Technical Field
[0004] The present disclosure relates to video coding and compression. More specifically, the present disclosure relates to methods and apparatuses for two inter-frame prediction tools studied in the Versatile Video Codec (VVC) standard, namely, Prediction Refinement with Optical Flow (PROF) and Bidirectional Optical Flow (BDOF). Background Art
[0005] Various video codec techniques can be used to compress video data. Video coding is performed according to one or more video codec standards. For example, video codec standards include Versatile Video Codec (VVC), Joint Exploration Test Model (JEM), High Efficiency Video Codec (H.265 / HEVC), Advanced Video Codec (H.264 / AVC), Moving Picture Experts Group (MPEG) codec, etc. Video codecs typically use prediction methods (e.g., inter-frame prediction, intra-frame prediction, etc.) that exploit redundancy present in video images or sequences. An important goal of video coding techniques is to compress video data into a form that uses a lower bit rate while avoiding or minimizing the degradation of video quality. Summary of the invention
[0006] Examples of the present disclosure provide methods and apparatus for motion vector prediction in video coding.
[0007] According to a first aspect of the present disclosure, a unified method for decoding bidirectional optical flow (BDOF) of a video signal and prediction refinement using optical flow (PROF) is provided. The decoder may divide a video block into a plurality of non-overlapping video sub-blocks, wherein at least one non-overlapping video sub-block of the plurality of non-overlapping video sub-blocks may be associated with two motion vectors. The decoder may obtain a first reference picture I associated with the two motion vectors of at least one non-overlapping video sub-block of the plurality of non-overlapping video sub-blocks. (0) and the second reference picture I (1) In the display order, the first reference picture I (0) It can be before the current picture and the second reference picture I (1)The decoder can be from the first reference picture I (0) The reference block in obtains the first prediction sample I of the video sub-block (0) (i, j). i and j can represent the coordinates of a sample based on the current picture. The decoder can obtain the coordinates of the sample from the second reference picture I (1) The reference block in obtains the second prediction sample I of the video sub-block (1) (i, j). The decoder can obtain the first prediction sample I (0) (i, j) and the second prediction sample I (1) The horizontal gradient value and vertical gradient value of (i, j) are obtained by the decoder based on BDOF when the video block is not encoded in affine mode. When the video block is encoded in affine mode, the decoder can obtain the motion refinement of the samples in the video sub-block based on PROF. The decoder can then obtain the predicted sample of the video block based on the motion refinement.
[0008] According to a second aspect of the present disclosure, a method for decoding a BDOF and a PROF of a video signal is provided. The method may include obtaining, at a decoder, a first reference picture I associated with a video block. (0) and the second reference picture I (1) In the display order, the first reference picture I (0) It can be before the current picture and the second reference picture I (1) The method may also include, at the decoder, obtaining a first reference picture I (0) The reference block in obtains the first prediction sample I of the video block (0) (i, j). i and j represent the coordinates of a sample based on the current picture. The method may include, at the decoder, obtaining a sample from the second reference picture I (1) The reference block in obtains the second prediction sample I of the video block (1) (i, j). The method may also include receiving at least one flag by the decoder. The at least one flag may be signaled by the encoder in a sequence parameter set (SPS) and signals whether BDOF and PROF are enabled for the current video block. The method may include, at the decoder and when the at least one flag is enabled, applying BDOF to the first prediction sample I based on the first prediction sample I when the video block is not encoded in affine mode. (0) (i, j) and the second prediction sample I (1) (i, j) derives a motion refinement of the video block. The method may further include, at the decoder and when at least one flag is enabled, applying PROF to, when the video block is encoded in affine mode, based on the first prediction sample I (0) (i, j) and the second prediction sample I (1)(i, j) derive a motion refinement for the video block. The method may also include obtaining, at a decoder, a prediction sample for the video block based on the motion refinement.
[0009] According to a third aspect of the present disclosure, a computing device for decoding a video signal is provided. The computing device may include one or more processors and a non-temporary computer-readable memory storing instructions that can be executed by the one or more processors. The one or more processors may be configured to divide a video block into a plurality of non-overlapping video sub-blocks. At least one non-overlapping video sub-block of the plurality of non-overlapping video sub-blocks may be associated with two motion vectors. The one or more processors may also be configured to obtain a first reference picture I associated with two motion vectors of at least one non-overlapping video sub-block of the plurality of non-overlapping video sub-blocks. (0) and the second reference picture I (1) In the display order, the first reference picture I (0) It can be before the current picture and the second reference picture I (1) The one or more processors may also be configured to start from the first reference picture I (0) The reference block in obtains the first prediction sample I of the video sub-block (0) (i, j). i and j represent the coordinates of a sample based on the current picture. The one or more processors may also be configured to obtain a sample from the second reference picture I (1) The reference block in obtains the second prediction sample I of the video sub-block (1) (i, j). The one or more processors may also be configured to obtain a first prediction sample I (0) (i, j) and the second prediction sample I (1) The one or more processors may also be configured to obtain a motion refinement of a sample in a video sub-block based on BDOF when the video block is not encoded in an affine mode. The one or more processors may also be configured to obtain a motion refinement of a sample in a video sub-block based on PROF when the video block is encoded in an affine mode. The one or more processors may also be configured to obtain a predicted sample of a video block based on the motion refinement.
[0010] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium having instructions stored therein is provided. When the instructions are executed by one or more processors of a device, the instructions may cause the device to execute, at a decoder, obtaining a first reference picture I associated with a video block. (0) and the second reference picture I (1) In the display order, the first reference picture I (0) It can be before the current picture and the second reference picture I (1)The instructions may also cause the device to perform at the decoder from the first reference picture I (0) The reference block in obtains the first prediction sample I of the video block (0) (i, j). i and j represent the coordinates of a sample based on the current picture. The instructions may also cause the device to perform at the decoder from the second reference picture I (1) The reference block in obtains the second prediction sample I of the video block (1) (i, j). The instructions may also cause the device to perform receiving at least one flag through a decoder. The at least one flag may be signaled by the encoder in the SPS and signal whether BDOF and PROF are enabled for the current video block. The instructions may also cause the device to perform at the decoder and when the at least one flag is enabled, when the video block is not encoded in affine mode, applying BDOF to predict the video block based on the first prediction sample I (0) (i, j) and the second prediction sample I (1) (i, j) derives a motion refinement of the video block. The instructions may also cause the device to execute at a decoder and when at least one flag is enabled, when the video block is encoded in an affine mode, applying PROF to derive a motion refinement of the video block based on the first prediction sample I (0) (i, j) and the second prediction sample I (1) (i, j) deriving motion refinements for the video block. The instructions may also cause the apparatus to perform, at a decoder, obtaining prediction samples for the video block based on the motion refinements. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0012] Figure 1 is a block diagram of an encoder according to an example of the present disclosure.
[0013] Figure 2 is a block diagram of a decoder according to an example of the present disclosure.
[0014] Figure 3A is a diagram illustrating block partitioning in a multi-type tree structure according to an example of the present disclosure.
[0015] Figure 3B is a diagram illustrating block partitioning in a multi-type tree structure according to an example of the present disclosure.
[0016] Figure 3C is a diagram illustrating block partitioning in a multi-type tree structure according to an example of the present disclosure.
[0017] Figure 3D is a diagram illustrating block partitioning in a multi-type tree structure according to an example of the present disclosure.
[0018] Figure 3E is a diagram illustrating block partitioning in a multi-type tree structure according to an example of the present disclosure.
[0019] Figure 4 is a schematic diagram of a BDOF model according to an example of the present disclosure.
[0020] Figure 5A is an illustration of an affine model according to an example of the present disclosure.
[0021] Figure 5B is an illustration of an affine model according to an example of the present disclosure.
[0022] Figure 6 is an illustration of an affine model according to an example of the present disclosure.
[0023] Figure 7 is an illustration of a PROF according to an example of the present disclosure.
[0024] Figure 8 is a workflow of a BDOF according to an example of the present disclosure.
[0025] Fig. 9 is a workflow of a PROF according to an example of the present disclosure.
[0026] Fig.10 A unified method for BDOF and PROF for decoding a video signal according to an example of the present disclosure.
[0027] Fig.11 A method for decoding BDOF and PROF of a video signal according to an example of the present disclosure.
[0028] Fig.12 is an illustration of a workflow of PROF for bidirectional prediction according to an example of the present disclosure.
[0029] Fig.13 is a diagram of the pipeline stages of the BDOF and PROF processes according to the present disclosure.
[0030] Fig.14 is a diagram of a gradient derivation method of BDOF according to the present disclosure.
[0031] Fig.15 is a diagram of the gradient derivation method of PROF according to the present disclosure.
[0032] Fig.16A is a diagram of a sample template derived for an affine mode according to an example of the present disclosure.
[0033] Fig. 16B is a diagram of a sample template derived for an affine mode according to an example of the present disclosure.
[0034] Fig.17A is an illustration of exclusively enabling PROF and LIC for affine mode according to an example of the present disclosure.
[0035] Fig. 17B is an illustration of jointly enabling PROF and LIC for affine mode according to an example of the present disclosure.
[0036] Fig.18A is a diagram illustrating a proposed padding method applied to a 16×16 BDOF CU according to an example of the present disclosure.
[0037] Fig.18B is a diagram illustrating a proposed padding method applied to a 16×16 BDOF CU according to an example of the present disclosure.
[0038] Fig.18C is a diagram illustrating a proposed padding method applied to a 16×16 BDOF CU according to an example of the present disclosure.
[0039] Fig.18D is a diagram illustrating a proposed padding method applied to a 16×16 BDOF CU according to an example of the present disclosure.
[0040] Fig.19 is a diagram illustrating a computing environment coupled with a user interface according to examples of the present disclosure. DETAILED DESCRIPTION
[0041] Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings, wherein, unless otherwise specified, the same numerals in different drawings represent the same or similar elements. The implementations set forth in the following description of exemplary embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with aspects of the present disclosure as described in the appended claims.
[0042] The terms used in this disclosure are only used for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms as well. It should also be understood that the term "and / or" used herein is intended to represent and include any or all possible combinations of one or more of the associated listed items.
[0043] It should be understood that although the terms "first", "second", "third", etc. may be used herein to describe various information, the information should not be limited by these terms. These terms are only used to distinguish one type of information from another type of information. For example, without departing from the scope of the present disclosure, the first information may be referred to as the second information; similarly, the second information may also be referred to as the first information. As used herein, the term "if" may be understood to mean "when", "upon", or "in response to a judgment", depending on the context.
[0044] The first version of the HEVC standard was completed in October 2013, and it provides about 50% bitrate savings or equivalent perceptual quality compared to the previous generation video codec standard H.264 / MPEG AVC. Although the HEVC standard provides significant codec improvements over its predecessor, there is evidence that higher codec efficiency can be achieved compared to HEVC using additional codec tools. On this basis, both VCEG and MPEG began exploring new codec technologies for future video codec standardization. ITU-TVECG and ISO / IEC MPEG established the Joint Video Exploration Team (JVET) in October 2015 to begin major research on advanced technologies that can achieve significant improvements in codec efficiency. JVET maintains reference software called the Joint Exploration Model (JEM) by integrating several additional codec tools on top of the HEVC Test Model (HM).
[0045] In October 2017, ITU-T and ISO / IEC issued a joint call for proposals (CfP) on video compression with capabilities beyond HEVC. In April 2018, 23 CfPfP responses were received and evaluated at the 10th JVET meeting, indicating that the compression efficiency was improved by about 40% over HEVC. Based on such evaluation results, JVET launched a new project to develop a new generation of video codec standards, which is called the Versatile Video Codec (VVC). In the same month, a reference software code base called the VVC Test Model (VTM) was established to demonstrate the reference implementation of the VVC standard.
[0046] Like HEVC, VVC is built on a block-based hybrid video codec framework.
[0047] Figure 1 An overall diagram of a block-based video encoder for VVC is shown. Specifically, Figure 1A typical encoder 100 is shown. The encoder 100 has a video input 110, motion compensation 112, motion estimation 114, intra / inter mode decision 116, block prediction value 140, adder 128, transform 130, quantization 132, prediction related information 142, intra prediction 118, picture buffer 120, inverse quantization 134, inverse transform 136, adder 126, memory 124, loop filter 122, entropy coding 138, and bitstream 144.
[0048] In encoder 100, a video frame is partitioned into multiple video blocks for processing. For each given video block, a prediction is formed based on either an inter-prediction method or an intra-prediction method.
[0049] The prediction residual is sent from adder 128 to transform 130, which represents the difference between the current video block (part of video input 110) and its prediction value (part of block prediction value 140). The transform coefficients are then sent from transform 130 to quantization 132 for entropy reduction. The quantized coefficients are then fed to entropy coding 138 to generate a compressed video bitstream. Figure 1 As shown, prediction related information 142 (e.g., video block partition information, motion vector (MV), reference picture index, and intra prediction mode) from intra / inter mode decision 116 is also fed through entropy coding 138 and saved into a compressed bitstream 144. The compressed bitstream 144 comprises a video bitstream.
[0050] In encoder 100, decoder-related circuitry is also required to reconstruct pixels for prediction. First, the prediction residual is reconstructed by inverse quantization 134 and inverse transformation 136. The reconstructed prediction residual is combined with the block prediction value 140 to generate the unfiltered reconstructed pixels of the current video block.
[0051] Spatial prediction (or "intra-prediction") uses pixels of samples from already encoded neighboring blocks (called reference samples) in the same video frame as the current video block to predict the current video block.
[0052] Temporal prediction (also called "inter prediction") uses reconstructed pixels from an already coded video picture to predict the current video block. Temporal prediction reduces temporal redundancy inherent in video signals. The temporal prediction signal for a given coding unit (CU) or coding block is typically signaled by one or more MVs that indicate the amount and direction of motion between the current CU and its temporal reference. In addition, if multiple reference pictures are supported, a reference picture index is additionally sent that identifies which reference picture in the reference picture storage the temporal prediction signal comes from.
[0053] The motion estimation 114 receives the video input 110 and the signal from the picture buffer 120, and outputs the motion estimation signal to the motion compensation 112. The motion compensation 112 receives the video input 110, the signal from the picture buffer 120, and the motion estimation signal from the motion estimation 114, and outputs the motion compensation signal to the intra / inter mode decision 116.
[0054] After performing spatial and / or temporal prediction, an intra / inter mode decision 116 in the encoder 100 selects the best prediction mode, for example based on a rate-distortion optimization method. The block prediction value 140 is then subtracted from the current video block, and the resulting prediction residual is decorrelated using a transform 130 and a quantization 132. The resulting quantized residual coefficients are inverse quantized by an inverse quantization 134 and inverse transformed by an inverse transform 136 to form a reconstructed residual, which is then added back to the prediction block to form a reconstructed signal for the CU. In addition, before the reconstructed CU is placed in the reference picture storage device of the picture buffer 120 and used to encode future video blocks, a loop filter 122 (e.g., a deblocking filter, a sample adaptive offset (SAO), and / or an adaptive loop filter (ALF)) may be applied to the reconstructed CU. To form an output video bitstream 144, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to an entropy coding unit 138 to be further compressed and packaged to form a bitstream.
[0055] Figure 1 shows a block diagram of a general block-based hybrid video coding system. The input video signal is processed block by block (called coding unit (CU)). In VTM-1.0, a CU can be up to 128×128 pixels. However, unlike HEVC, which partitions blocks based on quadtree only, in VVC, a coding tree unit (CTU) is split into CUs to accommodate different local characteristics based on quadtree / binary tree / ternary tree. In addition, the concept of multiple partition unit types in HEVC is removed, that is, there is no separation of CU, prediction unit (PU) and transform unit (TU) in VVC; instead, each CU is always used as a basic unit for both prediction and transform without further partitioning. In the multi-type tree structure, a CTU is first partitioned according to the quadtree structure. Then, each quadtree leaf node can be further partitioned according to the binary tree and ternary tree structures.
[0056] like Figure 3A , Figure 3B , Figure 3C , Figure 3D and Figure 3E As shown in , there are five types of partitioning, quadruple partitioning, horizontal binary partitioning, vertical binary partitioning, horizontal ternary partitioning, and vertical ternary partitioning.
[0057] Figure 3AA diagram illustrating block quad partitioning in a multi-type tree structure according to the present disclosure is shown.
[0058] Figure 3B A diagram illustrating vertical binary partitioning of blocks in a multi-type tree structure according to the present disclosure is shown.
[0059] Figure 3C A diagram illustrating block-level binary partitioning in a multi-type tree structure according to the present disclosure is shown.
[0060] Figure 3D A diagram illustrating vertical ternary partitioning of blocks in a multi-type tree structure according to the present disclosure is shown.
[0061] Figure 3E A diagram illustrating block-level ternary partitioning in a multi-type tree structure according to the present disclosure is shown.
[0062] exist Figure 1 In the video code, spatial prediction and / or temporal prediction can be performed. Spatial prediction (or "intra-frame prediction") uses pixels from samples of already coded neighboring blocks (called reference samples) in the same video picture / slice to predict the current video block. Spatial prediction reduces the spatial redundancy inherent in the video signal. Temporal prediction (also known as "inter-frame prediction" or "motion compensated prediction") uses reconstructed pixels from already coded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in the video signal. The temporal prediction signal of a given CU is usually represented by one or more motion vectors (MVs), which indicate the amount and direction of motion between the current CU and its temporal reference. In addition, if multiple reference pictures are supported, a reference picture index is additionally sent, which is used to identify which reference picture in the reference picture storage device the temporal prediction signal comes from. After spatial and / or temporal prediction, the mode decision block in the encoder selects the best prediction mode, for example, based on a rate-distortion optimization method. The prediction block is then subtracted from the current video block; and the prediction residual is decorrelated and quantized using a transform. The quantized residual coefficients are inverse quantized and inverse transformed to form a reconstructed residual, which is then added back to the prediction block to form a reconstructed signal for the CU. In addition, loop filtering (e.g., deblocking filter, sample adaptive offset (SAO), and adaptive loop filter (ALF)) may be applied to the reconstructed CU before the reconstructed CU is placed in a reference picture repository and used to encode future video blocks. To form an output video bitstream, the coding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to the entropy coding unit to be further compressed and packaged to form a bitstream.
[0063] Figure 2 The overall block diagram of the video decoder for VVC is shown. Specifically, Figure 2A block diagram of a typical decoder 200 is shown. The decoder 200 has a bitstream 210, entropy decoding 212, inverse quantization 214, inverse transform 216, adder 218, intra / inter mode selection 220, intra prediction 222, memory 230, loop filter 228, motion compensation 224, picture buffer 226, prediction related information 234 and video output 232.
[0064] Decoder 200 is similar to Figure 1 The decoder 200 is a decoder that decodes the video bitstream 210. In the decoder 200, the incoming video bitstream 210 is first decoded by entropy decoding 212 to derive the quantization coefficient levels and prediction related information. The quantization coefficient levels are then processed by inverse quantization 214 and inverse transform 216 to obtain reconstructed prediction residuals. The block prediction value mechanism implemented in the intra / inter mode selector 220 is configured to perform intra prediction 222 or motion compensation 224 based on the decoded prediction information. A set of unfiltered reconstructed pixels is obtained by adding the reconstructed prediction residuals from the inverse transform 216 to the prediction output generated by the block prediction value mechanism using a summer 218.
[0065] The reconstructed block may further pass through a loop filter 228 before being stored in a picture buffer 226 used as a reference picture repository. The reconstructed video in the picture buffer 226 may be sent to drive a display device, as well as used to predict future video blocks. With the loop filter 228 turned on, a filtering operation is performed on these reconstructed pixels to derive a final reconstructed video output 232.
[0066] Figure 2 The overall block diagram of a block-based video decoder is given. The video bitstream is first entropy decoded at the entropy decoding unit. The coding mode and prediction information are sent to the spatial prediction unit (if intra-frame coding) or the temporal prediction unit (if inter-frame coding) to form a prediction block. The residual transform coefficients are sent to the inverse quantization unit and the inverse transform unit to reconstruct the residual block. The prediction block and the residual block are then added. The reconstructed block can be further loop filtered before being stored in the reference picture repository. The reconstructed video in the reference picture repository is then sent out to drive the display device and is used to predict future video blocks.
[0067] In general, the basic inter prediction techniques applied in VVC remain the same as those in HEVC, except that several modules are further extended and / or enhanced. In particular, for all previous video standards, a coding block can only be associated with a single MV when the coding block is unidirectionally predicted, or with two MVs when the coding block is bidirectionally predicted. Due to this limitation of traditional block-based motion compensation, small motions can still remain within the prediction samples after motion compensation, which has a negative impact on the overall efficiency of motion compensation. In order to improve both the granularity and accuracy of MVs, two optical flow-based sample-by-sample refinement methods are currently studied for the VVC standard, namely, bidirectional optical flow (BDOF) for affine modes and prediction refinement (PROF) using optical flow. The main technical aspects of these two inter-frame codec tools are briefly reviewed below.
[0068] Bidirectional optical flow
[0069] In VVC, BDOF is applied to refine the prediction samples of bidirectional prediction coding blocks. Specifically, Figure 4 As shown, BDOF is a sample-by-sample motion refinement performed on top of block-based motion compensated prediction when using bidirectional prediction.
[0070] Figure 4 A diagram of a BDOF model according to the present disclosure is shown.
[0071] The motion refinement (v x , v y ) is calculated by minimizing the difference between L0 and L1 prediction samples after applying BDOF within a 6×6 window Ω around the subblock. Specifically, (v x , v y ) is exported as
[0072]
[0073] in is a floor function; clip3(min, max, x) is a function that clips a given value x within the range [min, max]; the symbol >> represents a bitwise right shift operation; the symbol << represents a bitwise left shift operation; th BDOF is the motion refinement threshold used to prevent propagation errors due to irregular local motion, which is equal to 1<<max(5, bit-depth-7)., where bit-depth is the internal bit depth. In (1),
[0074] The values of S1, S2, S3, S5, and S6 are calculated as
[0075]
[0076] in
[0077]
[0078] Among them I (k) (i, j) is the sample value at coordinate (i, j) of the prediction signal in list k, k = 0, 1, which is generated with medium-high precision (i.e., 16 bits); and are the horizontal and vertical gradients of a sample, which are obtained by directly calculating the difference between two adjacent samples of the sample, i.e.
[0079]
[0080] Based on the motion refinement derived in (1), the final bidirectional prediction samples of the CU are calculated by interpolating the L0 / L1 prediction samples along the motion trajectory based on the optical flow model, as shown below
[0081]
[0082]
[0083] Among them, shift and o offset are the right shift value and offset value applied to the combined L0 and L1 prediction signals for bidirectional prediction, which are equal to 15-bit-depth and 1<<(14-bit-depth)+2·(1<<13), respectively. Based on the above bit depth control method, it is ensured that the maximum bit depth of the intermediate parameters of the entire BDOF process does not exceed 32 bits, and the maximum input of the multiplication is within 15 bits, that is, for BDOF implementation, a 15-bit multiplier is sufficient.
[0084] Affine Mode
[0085] In HEVC, only the translational motion model is applied to motion compensated prediction. In the real world, there are many kinds of motion, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, affine motion compensated prediction is applied by signaling a flag for each inter-frame coding block to indicate whether the translational motion or the affine motion model is applied to the inter-frame prediction. In the current VVC design, two affine modes are supported for an affine coding block, including a 4-parameter affine mode and a 6-parameter affine mode.
[0086] The 4-parameter affine model has the following parameters: two parameters for horizontal and vertical translation motion, one parameter for scaling motion, and one parameter for rotation motion in both directions. The horizontal scaling parameter is equal to the vertical scaling parameter. The horizontal rotation parameter is equal to the vertical rotation parameter. In order to better adapt the motion vector and affine parameters, in VVC, these affine parameters are converted into two MVs (also called control point motion vectors (CPMVs)) located at the top left and top right corners of the current block. Figure 5A and Figure 5B As shown in , the affine motion field of a block is described by two control points MV(V0, V1).
[0087] Figure 5A A diagram of a 4-parameter affine model according to the present disclosure is shown.
[0088] Figure 5B A diagram of a 4-parameter affine model according to the present disclosure is shown.
[0089] Based on the control point motion, an affine encoding block’s motion field (v x , v y ) is described as
[0090]
[0091] The 6-parameter affine mode has the following parameters: two parameters for translation motion in the horizontal and vertical directions respectively, one parameter for scaling motion and one parameter for rotation motion in the horizontal direction, one parameter for scaling motion and one parameter for rotation motion in the vertical direction. The 6-parameter affine motion model is encoded using three MVs at three CPMVs.
[0092] Figure 6 A diagram of a 6-parameter affine model according to the present disclosure is shown.
[0093] like Figure 6 As shown, the three control points of a 6-parameter affine block are located at the top left, top right, and bottom left corners of the block. The motion at the top left control point is related to translation, the motion at the top right control point is related to horizontal rotation and scaling, and the motion at the bottom left control point is related to vertical rotation and scaling. Compared with the 4-parameter affine motion model, the 6-parameter horizontal rotation and scaling motions may be different from those in the vertical direction. Assume that (V0, V1, V2) is Figure 6 The MVs of the upper left, upper right, and lower left corners of the current block, and the motion vectors of each sub-block (v x , v y ) is derived using the three MVs at the control points as:
[0094]
[0095] Prediction refinement using optical flow for affine models
[0096] In order to improve the accuracy of affine motion compensation, PROF is being studied in the current VVC, which refines the sub-block based affine motion compensation based on the optical flow model. Specifically, after performing sub-block based affine motion compensation, the brightness prediction sample of an affine block is modified by a sample refinement value derived based on the optical flow equation. In detail, the operation of PROF can be summarized as the following four steps:
[0097] Step 1: Perform sub-block based affine motion compensation to generate sub-block predictions I(i, j) using the sub-block MVs derived in (6) for the 4-parameter affine model and in (7) for the 6-parameter affine model.
[0098] Step 2: The spatial gradient g of each predicted sample x (i, j) and g y (i, j) is calculated as
[0099]
[0100] To compute the gradient, an additional row / column of prediction samples needs to be generated on each side of a subblock. To reduce memory bandwidth and complexity, samples on the extended boundary are copied from the nearest integer pixel position in the reference picture to avoid additional interpolation.
[0101] Step 3: Calculate the brightness prediction refinement value using the following formula
[0102] ΔI(i, j) = g x (i, j)*Δv x (i, j)+g y (i, j)*Δv y (i,j) (9)
[0103] Where Δv(i, j) is the difference between the pixel MV calculated for the sample position (i, j) (denoted by v(i, j)) and the sub-block MV of the sub-block where the pixel (i, j) is located. In addition, in the current PROF design, after adding prediction refinement to the original prediction sample, a clipping operation is performed to clip the value of the refined prediction sample to within 15 bits, that is,
[0104] I r (i, j) = I (i, j) + ΔI (i, j)
[0105] I r (i, j) = clip3(-2 14 , 2 14 -1,I r(i, j))
[0106] Where I(i, j) and Ir(i, j) are the original prediction sample and the refined prediction sample at position (i, j), respectively.
[0107] Figure 7 The PROF process for affine mode according to the present disclosure is shown.
[0108] Because the affine model parameters and the pixel positions relative to the sub-block center do not change between sub-blocks, Δv(i, j) can be calculated for the first sub-block and reused for other sub-blocks in the same CU. Let Δx and Δy be the horizontal and vertical offsets from the sample position (i, j) to the center of the sub-block to which the sample belongs, Δv(i, j) can be derived as
[0109]
[0110] Based on the affine sub-block MV derivation equations (6) and (7), the MV difference Δv(i, j) can be derived. Specifically, for the 4-parameter affine model,
[0111]
[0112] For the 6-parameter affine model,
[0113]
[0114] Among them, (v 0x , v 0y )、(v 1x , v 1y )、(v 2x , v 2y ) are the upper left, upper right and lower left control points MV of the current coding block, w and h are the width and height of the block. In the existing PROF design, the MV difference Δv x and Δv y Always export with 1 / 32 pixel precision.
[0115] Local illumination compensation
[0116] Local illumination compensation (LIC) is a codec tool used to address the problem of local illumination variations between temporally adjacent pictures. A pair of weights and offset parameters are applied to a reference sample to obtain a predicted sample for the current block. The general mathematical model is as follows
[0117] P[x]=α*P r [x+v]+β (11)
[0118] Among them, P r[x+v] is the reference block indicated by the motion vector v, [α, β] is the corresponding pair of weight and offset parameters of the reference block, and P[x] is the final prediction block. The pair of weight and offset parameters is estimated based on the template of the current block (i.e., the adjacent reconstructed samples) and the reference block of the template (derived using the motion vector of the current block) using the minimum linear mean square error (LLMSE) algorithm. By minimizing the mean square error between the template samples and the reference samples of the template, the mathematical representation of α and β can be derived as follows
[0119]
[0120] Where I represents the number of samples in the template. c [x i ] is the i-th sample of the template of the current block, and P r [x i ] is the reference sample of the i-th template sample based on the motion vector v.
[0121] In addition to being applied to regular inter blocks that contain at most one motion vector for each prediction direction (L0 or L1), LIC is also applied to affine-coded blocks, where a coded block is further split into multiple smaller sub-blocks and each sub-block can be associated with different motion information. In order to derive reference samples for LIC of affine-coded blocks, as Fig.16A and Fig. 16B As shown, the reference samples in the top template of an affine coded block are obtained using the motion vectors of each sub-block in the top sub-block row, while the reference samples in the left template are obtained using the motion vectors of the sub-blocks in the left sub-block column. Afterwards, the same LLMSE derivation method as shown in (12) is applied to derive LIC parameters based on the composite template.
[0122] Fig.16A A diagram for deriving template samples for affine mode according to the present disclosure is shown. The diagram includes CurFrame 1620 and Cur CU 1622. Cur Frame 1620 is the current frame. Cur CU 1622 is the current coding unit.
[0123] Fig. 16B1644, C Ref 1645, D Ref 1646, E Ref 1647, F Ref 1648, and G Ref 1649. RefFrame 1640 is a reference frame. Col CU 1642 is a co-located coding unit. A Ref 1643, B Ref 1644, C Ref 1645, D Ref 1646, E Ref 1647, F Ref 1648, and G Ref 1649 are reference samples.
[0124] The terms used in this disclosure are only used for the purpose of describing illustrative examples and are not intended to limit the disclosure. As used in this disclosure and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an", and "the" are intended to include the plural forms as well. It should also be understood that, unless the context clearly indicates otherwise, the terms "or" and "and / or" used herein are intended to represent and include any or all possible combinations of one or more of the associated listed items.
[0125] It should be understood that although the terms "first", "second", "third", etc. may be included herein to describe various information, the information should not be limited by these terms. These terms are only used to distinguish one type of information from another type of information. For example, without departing from the scope of the present disclosure, the first information may include what is referred to as the second information; similarly, the second information may also be referred to as the first information. As used herein, the term "if" may be understood to mean "when" or "upon" or "in response to", depending on the context.
[0126] References throughout this specification to "one example," "an example," "an exemplary example," etc. in the singular or plural form mean that one or more specific features, structures, or characteristics described in conjunction with the example are included in at least one example of the present disclosure. Thus, phrases such as "in one example" or "in an example," "in an exemplary example," etc. appearing in the singular or plural form throughout this specification do not necessarily all refer to the same example. Furthermore, specific features, structures, or characteristics in one or more examples may be included and combined in any suitable manner.
[0127] Current BDOF, PROF, and LIC designs
[0128] Although PROF can improve the coding efficiency of affine patterns, its design can still be further improved. In particular, considering that both PROF and BDOF are built on the concept of optical flow, it is highly desirable to coordinate the designs of PROF and BDOF as much as possible so that PROF can maximize the use of BDOF's existing logic to facilitate hardware implementation. Based on this consideration, the following inefficiencies in the interaction between the current PROF and BDOF designs are identified in this disclosure.
[0129] As described in the “Prediction Refinement with Optical Flow for Affine Mode” section, in Equation (8), the accuracy of the gradient is determined based on the internal bit depth. On the other hand, the MV difference (i.e., Δv x and Δv y ) is always derived with an accuracy of 1 / 32 pixel. Accordingly, based on formula (9), the accuracy of the derived PROF refinement depends on the internal bit depth. However, similar to BDOF, PROF is applied to prediction sample values at medium and high bit depths (i.e., 16 bits) in order to maintain higher PROF derived accuracy. Therefore, regardless of the internal coding bit depth, the accuracy of the prediction refinement derived by PROF should match the accuracy of medium prediction samples (i.e., 16 bits). In other words, the representation bit depth of MV differences and gradients in the existing PROF design is not completely matched to derive accurate prediction refinement relative to the prediction sample accuracy (i.e., 16 bits). At the same time, based on the comparison of formulas (1), (4) and (8), the existing PROF and BDOF use different accuracy to represent sample gradients and MV differences. As pointed out earlier, this non-uniform design is undesirable for hardware because the existing BDOF logic cannot be reused.
[0130] As discussed in the "Prediction Refinement with Optical Flow for Affine Mode" section, when a current affine block is bidirectionally predicted, PROF is applied to the prediction samples in lists L0 and L1 separately; then, the enhanced L0 and L1 prediction signals are averaged to generate the final bidirectional prediction signal. In contrast, instead of deriving PROF refinements separately for each prediction direction, BDOF derives prediction refinements once and then applies them to enhance the combined L0 and L1 prediction signals. Figure 8 and Fig. 9 (described below) compares the current BDOF and PROF workflows for bidirectional prediction. In actual codec hardware pipeline design, different main encoding / decoding modules are usually assigned to each pipeline stage so that more encoding blocks can be processed in parallel. However, due to the differences between the BDOF and PROF workflows, this may make it difficult to have an identical pipeline design that can be shared by BDOF and PROF, which is not friendly to actual codec implementations.
[0131] Figure 8 The workflow of BDOF according to the present disclosure is shown. The workflow 800 includes L0 motion compensation 810, L1 motion compensation 820, and BDOF 830. L0 motion compensation 810 can be, for example, a list of motion compensated samples from a previous reference picture. The previous reference picture is a reference picture before the current picture in the video block. L1 motion compensation 820 can be, for example, a list of motion compensated samples from a next reference picture. The next reference picture is a reference picture after the current picture in the video block. BDOF 830 obtains motion compensated samples from L0 motion compensation 810 and L1 motion compensation 820 and outputs predicted samples, as described above with respect to Figure 4 As described.
[0132] Fig. 9 The workflow of the prior art PROF according to the present disclosure is shown. The workflow 900 includes L0 motion compensation 910, L1 motion compensation 920, L0 PROF 930, L1 PROF 940, and averaging 960. L0 motion compensation 910 can be, for example, a list of motion compensated samples from a previous reference picture. The previous reference picture is a reference picture before the current picture in the video block. For example, L1 motion compensation 920 can be a list of motion compensated samples from a next reference picture. The next reference picture is a reference picture after the current picture in the video block. L0 PROF 930 obtains the L0 motion compensated samples from L0 motion compensation 910 and outputs motion refinement values, as described above with respect to Figure 7 L1 PROF 940 obtains L1 motion compensation samples from L1 motion compensation 920 and outputs motion refinement values, as described above with respect to Figure 7 Averaging 960 averages the motion refinement value outputs of L0 PROF 930 and L1 PROF 940.
[0133] 1) For both BDOF and PROF, gradients need to be calculated for each sample within the current coding block, which requires generating an additional row / column of prediction samples on each side of the block. In order to avoid the additional computational complexity of sample interpolation, the prediction samples in the extended area around the block are copied directly from the reference samples at integer positions (i.e., no interpolation). However, according to the existing design, integer samples at different positions are selected to generate gradient values for BDOF and PROF. Specifically, for BDOF, integer reference samples located to the left of the prediction sample (for horizontal gradients) and above the prediction sample (for vertical gradients) are used; for PROF, the integer reference samples closest to the prediction sample are used for gradient calculation. Similar to the bit depth representation problem, this non-uniform gradient calculation method is also undesirable for hardware codec implementation.
[0134] 2) As pointed out earlier, the motivation of PROF is to compensate for small MV differences between the MV of each sample and the sub-block MV derived at the center of the sub-block to which the sample belongs. According to the current PROF design, PROF is always called when a coding block is predicted by an affine mode. However, as indicated by equations (6) and (7), the sub-block MVs of an affine block are derived from the control point MVs. Therefore, when the difference between the control point MVs is relatively small, the MV at each sample position should be consistent. In this case, since the benefit of applying PROF may be very limited, it may not be worthwhile to perform PROF when considering the performance / complexity trade-off.
[0135] Improvements to BDOF, PROF, and LIC
[0136] In the present disclosure, methods are provided to improve and simplify the existing PROF design to facilitate hardware codec implementation. In particular, special attention is paid to coordinating the design of BDOF and PROF to maximize the sharing of existing BDOF logic with PROF. In general, the main aspects of the technology proposed in the present disclosure are summarized as follows.
[0137] 1) To improve the coding efficiency of PROF while achieving a more unified design, a method is proposed to unify the representation bit depth of sample gradients and MV differences used by BDOF and PROF.
[0138] 2) To facilitate hardware pipeline design, we propose to coordinate the workflow of PROF with that of BDOF for bidirectional prediction. Specifically, unlike the existing PROF that derives prediction refinements for L0 and L1 separately, the proposed method derives prediction refinements once, which are applied to the combined L0 and L1 prediction signals.
[0139] 3) Two methods are proposed to coordinate the derivation of integer reference samples to compute the gradient values used by BDOF and PROF.
[0140] 4) To reduce the computational complexity, an early termination method is proposed to adaptively disable the PROF process for affine coded blocks when certain conditions are met.
[0141] Improved bit-depth representation design for PROF gradient and MV difference
[0142] As analyzed in the Problem Statement section, the representation bit depths of MV differences and sample gradients in the current PROF are not aligned to derive accurate prediction refinement. In addition, the representation bit depths of sample gradients and MV differences between BDOF and PROF are inconsistent, which is hardware-unfriendly. In this section, an improved bit depth representation method is proposed by extending the bit depth representation method of BDOF to PROF. Specifically, in the proposed method, the horizontal gradient and vertical gradient at each sample position are calculated as
[0143]
[0144] In addition, assuming that Δx and Δy are the horizontal and vertical offsets from a sample position to the center of the sub-block to which the sample belongs, expressed with 1 / 4 pixel accuracy, the corresponding PROF MV difference at the sample position is derived as
[0145]
[0146] Where dMvBits is the bit depth of the gradient value used by the BDOF process, that is, dMvBits = max(5, (bit-depth-7)) + 1. In formulas (13) and (14), c, d, e, and f are affine parameters derived based on the affine control point MV. Specifically, for a 4-parameter affine model,
[0147]
[0148] For the 6-parameter affine model,
[0149]
[0150] Among them, (v 0x , v 0y )、(v 1x , v 1y )、(v 2x , v 2y ) are the upper left, upper right and lower left control point MVs of the current coding block, expressed with 1 / 16 pixel precision, and w and h are the width and height of the block.
[0151] In the above discussion, as shown in formulas (13) and (14), a pair of fixed right shifts are applied to calculate the values of gradient and MV difference. In practice, different bit-by-bit right shifts can be applied to (13) and (14) to achieve various representation accuracy of gradient and MV difference to make different trade-offs between medium calculation accuracy and bit width of internal PROF derivation process. For example, when the input video contains a lot of noise, the derived gradient may not reliably represent the true local horizontal / vertical gradient value at each sample. In this case, it makes more sense to use more bits to represent MV difference compared to gradient. On the other hand, when the input video shows stable motion, the MV difference derived by the affine model should be very small. If so, using high-precision MV difference cannot provide additional benefit to improve the accuracy of derived PROF refinement. In other words, in this case, it is more advantageous to use more bits to represent gradient values. Based on the above considerations, in one embodiment of the present disclosure, a general method for calculating gradient and MV difference for PROF is proposed below. Specifically, it is assumed that the horizontal gradient and vertical gradient at each sample position are obtained by applying n to the difference of adjacent prediction samples. a It is calculated by shifting right, that is,
[0152]
[0153] The corresponding PROF MV difference Δv(x, y) at the sample location should be calculated as
[0154]
[0155] where Δx and Δy are the horizontal and vertical offsets from a sample position to the center of the sub-block to which the sample belongs, expressed with 1 / 4 pixel accuracy, and c, d, e, and f are affine parameters derived based on the 1 / 16 pixel affine control point MV. Finally, the final PROF refinement of the sample is calculated as
[0156] ΔI(i, j)=(g x (i, j)*Δv x (i, j)+g y (i, j)*Δv y (i, j)+1)>>1 (17)
[0157] In another embodiment of the present disclosure, another PROF bit depth control method is proposed as follows. In this method, the horizontal gradient and vertical gradient at each sample position are still calculated as in (18) by applying n to the difference of adjacent prediction samples. a The corresponding PROF MV difference at the sample position should be calculated as:
[0158] Δv x(i,j)=(c*Δx+d*Δy)>>(14-n a )
[0159] Δv y (i,j)=(e*Δx+f*Δy)>>(14-n a )
[0160] Additionally, to keep the entire PROF export at the appropriate internal bit depth, cropping is applied to the exported MV differences, as follows:
[0161] Δv x (i, j) = Clip3 (-limit, limit, Δv x (i, j))
[0162] Δv y (i, j) = Clip3 (-limit, limit, Δv y (i, j))
[0163] Where limit is equal to is a threshold, and clip3(min, max, x) is a function that clips a given value x within the range [min, max]. In an example, n b Set the value to 2 max(5,bit-depth-7) Finally, the PROF refinement of the sample is calculated as
[0164] ΔI(i, j) = g x (i, j)*Δv x (i, j)+g y (i, j)*Δv y (i, j)
[0165] Coordinated workflow of BDOF and PROF for bidirectional prediction
[0166] As discussed earlier, when an affine coded block is bidirectionally predicted, currently PROF is applied in a unilateral manner. More specifically, PROF sample refinements are derived separately and applied to the prediction samples in lists L0 and L1. Afterwards, the refined prediction signals from lists L0 and L1, respectively, are averaged to generate the final bidirectional prediction signal for the block. This is in contrast to the BDOF design, where sample refinements are derived and applied to the bidirectional prediction signals. This difference between the bidirectional prediction workflows of BDOF and PROF may not be friendly to practical codec pipeline design.
[0167] To facilitate hardware pipeline design, a simplified method according to the present disclosure is used to modify the bidirectional prediction process of PROF so that the workflow of the two prediction refinement methods is coordinated. Specifically, instead of applying refinement separately for each prediction direction, the proposed PROF method derives prediction refinements once based on the control point MVs of lists L0 and L1; the derived prediction refinements are then applied to the combined L0 and L1 prediction signals to improve quality. Specifically, based on the MV differences derived in formula (14), the final bidirectional prediction samples of an affine coded block are calculated by the proposed method as
[0168]
[0169] Among them, shift and o offset are the right shift value and the offset value applied to combine the L0 and L1 prediction signals for bidirectional prediction, which are equal to (15-bit-depth) and 1<<(14-bit-depth)+(2<<13), respectively. In addition, as shown in (18), the clipping operation in the existing PROF design (as shown in (9)) is removed in the proposed method.
[0170] Fig.12 A diagram of a PROF process when the proposed bidirectional prediction PROF method is applied according to the present disclosure is shown. The PROF process 1200 includes L0 motion compensation 1210, L1 motion compensation 1220, and bidirectional prediction PROF 1230. L0 motion compensation 1210 can be, for example, a list of motion compensation samples from a previous reference picture. The previous reference picture is a reference picture before the current picture in the video block. L1 motion compensation 1220 can be, for example, a list of motion compensation samples from a next reference picture. The next reference picture is a reference picture after the current picture in the video block. As described above, the bidirectional prediction PROF 1230 obtains motion compensation samples from L0 motion compensation 1210 and L1 motion compensation 1220 and outputs bidirectional prediction samples.
[0171] Fig.12 The corresponding PROF process when the proposed bidirectional prediction PROF method is applied is shown. The PROF process 1200 includes L0 motion compensation 1210, L1 motion compensation 1220 and bidirectional prediction PROF 1230. L0 motion compensation 1210 can be, for example, a list of motion compensation samples from a previous reference picture. The previous reference picture is a reference picture before the current picture in the video block. L1 motion compensation 1220 can be, for example, a list of motion compensation samples from the next reference picture. The next reference picture is a reference picture after the current picture in the video block. As described above, the bidirectional prediction PROF 1230 obtains motion compensation samples from L0 motion compensation 1210 and L1 motion compensation 1220 and outputs bidirectional prediction samples.
[0172] To demonstrate the potential benefits of the proposed approach for hardware pipeline design, Fig.13 An example is shown to illustrate the pipeline stages when both BDOF and the proposed PROF are applied. Fig.13 In , the decoding process of an inter-frame block mainly includes three steps:
[0173] 1) Parse / decode the MV of the coded block and obtain the reference sample.
[0174] 2) Generate L0 and / or L1 prediction signals for the coding block.
[0175] 3) When the coding block is predicted by a non-affine mode, sample-by-sample refinement is performed on the generated bidirectional prediction samples based on BDOF; or when the coding block is predicted by an affine mode, sample-by-sample refinement is performed on the generated bidirectional prediction samples based on PROF.
[0176] Fig.13 A diagram showing example pipeline stages when both BDOF and the proposed PROF are applied according to the present disclosure is shown. Fig.13 The potential benefits of the proposed hardware pipeline design method are demonstrated. Pipeline stage 1300 includes parsing / decoding MV and obtaining reference samples 1310, motion compensation 1320, BDOF / PROF 1330. Pipeline stage 1300 will encode video blocks BLK0, BKL1, BKL2, BKL3 and BLK4. Each video block will start with parsing / decoding MV and obtaining reference samples 1310, and move to motion compensation 1320, then motion compensation 1320, BDOF / PROF 1330 in sequence. This means that BLK0 will not start in the pipeline stage 1300 process until BLK0 moves to motion compensation 1320. This is true for all stages and video blocks as time goes from T0 to T1, T2, T3 and T4.
[0177] exist Fig.13 In , the decoding process of an inter-frame block mainly includes three steps:
[0178] First, parse / decode the MV of the coded block and obtain the reference sample.
[0179] Second, generate L0 and / or L1 prediction signals for the coding block.
[0180] Third, when the coding block is predicted by a non-affine mode, sample-by-sample refinement is performed on the generated bidirectional prediction samples based on BDOF; or when the coding block is predicted by an affine mode, sample-by-sample refinement is performed on the generated bidirectional prediction samples based on PROF.
[0181] like Fig.13 As shown, after applying the proposed coordination method, both BDOF and PROF are directly applied to the bidirectional prediction samples. Considering that BDOF and PROF are applied to different types of coding blocks (i.e., BDOF is applied to non-affine blocks, while PROF is applied to affine blocks), these two codecs cannot be called at the same time. Therefore, their corresponding decoding processes can be performed by sharing the same pipeline stages. This is more efficient than the existing PROF design, in which it is difficult to assign the same pipeline stages to both BDOF and PROF due to their different bidirectional prediction workflows.
[0182] In the above discussion, the proposed method only considers the coordination of the workflow of BDOF and PROF. However, according to the existing design, the basic operation units of the two codec tools are performed in different sizes. Specifically, for BDOF, a coding block is divided into multiple blocks of size W. s ×H s The sub-blocks, where W s =min(W,16) and H s =min(H, 16), where W and H are the width and height of the coding block. BODF operations (e.g., gradient calculation and sample refinement derivation) are performed independently for each sub-block. On the other hand, as described above, the affine coding block is divided into 4×4 sub-blocks, each of which is assigned a separate MV derived based on a 4-parameter or 6-parameter affine model. Because PROF only applies to affine blocks, its basic operation unit is a 4×4 sub-block. Similar to the bidirectional prediction workflow problem, using a different basic operation unit size for PROF than for BDOF is also unfriendly to hardware implementation, and makes it difficult for BDOF and PROF to share the same pipeline stage of the entire decoding process. In order to solve such a problem, in one embodiment, it is proposed to align the sub-block size of the affine mode to be the same as the sub-block size of BDOF. Specifically, according to the proposed method, when a coding block is encoded by an affine mode, it will be divided into sub-blocks of size W. s ×H s The sub-blocks, where W s =min(W,16) and H s=min(H, 16), where W and H are the width and height of the coding block. Each sub-block is assigned a separate MV and is regarded as an independent PROF operation unit. It is worth mentioning that the independent PROF operation unit ensures that the PROF operation is performed on it without referring to information from adjacent PROF operation units. Specifically, the PROF MV difference at a sample position is calculated as the difference between the MV at the sample position and the MV at the center of the PROF operation unit where the sample is located; the gradient used for PROF export is calculated by filling samples along each PROF operation unit. The benefits achieved by the proposed method mainly include the following aspects: 1) a simplified pipeline architecture with a unified basic operation unit size for both motion compensation and BDOF / PROF refinement; 2) reduced memory bandwidth usage, which is due to the expansion of the sub-block size of affine motion compensation; 3) reduced per-sample computational complexity of fractional sample interpolation.
[0183] It should also be mentioned that due to the reduced computational complexity using the proposed method (i.e., item 3), the existing 6-tap interpolation filter constraint for affine coded blocks can be removed. Instead, the default 8-tap interpolation used for non-affine coded blocks is also used for affine coded blocks. In this case, the overall computational complexity can still be comparable to the existing PROF design (i.e., based on 4×4 sub-blocks with 6-tap interpolation filters).
[0184] Harmonization of gradient exports for BDOF and PROF
[0185] As described previously, both BDOF and PROF compute the gradient for each sample within the current coding block, accessing one additional row / column of prediction samples on each side of the block. To avoid additional interpolation complexity, the required prediction samples in the extended region around the block boundaries are copied directly from the integer reference samples. However, as noted in the Problem Statement section, integer samples at different locations are used to compute the gradient values for BDOF and PROF.
[0186] To achieve a more unified design, two methods are proposed below to unify the gradient derivation methods used by BDOF and PROF. In the first method, it is proposed to align the gradient derivation method of PROF to be the same as the gradient derivation method of BDOF. Specifically, with the first method, the integer positions used to generate the prediction samples in the extended area are determined by rounding down the fractional sample positions, i.e., the selected integer sample positions are located to the left of the fractional sample positions (for horizontal gradients) and above the fractional sample positions (for vertical gradients).
[0187] In the second method, it is proposed to align the gradient derivation method of BDOF to be the same as that of PROF. In more detail, when the second method is applied, the integer reference samples closest to the predicted samples are used for gradient calculation.
[0188] Fig.14 An example of a gradient derivation method using BDOF according to the present disclosure is shown. Fig.14 In , the open circles represent reference samples at integer positions, the triangles represent fractional prediction samples of the current block, and the grey circles represent integer reference samples used to fill the extended area of the current block.
[0189] Fig.15 An example of a gradient derivation method using PROF according to the present disclosure is shown. Fig.15 In , the open circles represent reference samples at integer positions, the triangles represent fractional prediction samples of the current block, and the grey circles represent integer reference samples used to fill the extended area of the current block.
[0190] Fig.14 and Fig.15 The results are shown in Figure 2. Fig.12 ) and the second method ( Fig.13 ) is used to derive the corresponding integer sample positions of the gradients for BDOF and PROF. Fig.14 and Fig.15 In , the open circles represent reference samples at integer positions, the triangles represent fractional prediction samples of the current block, and the patterned circles represent integer reference samples used to fill the extended area of the current block for gradient derivation.
[0191] In addition, according to the existing BDOF and PROF designs, prediction sample padding is performed at different coding levels. Specifically, for BDOF, padding is applied along the boundaries of each sbWidth×sbHeight sub-block, where sbWidth=min(CUWidth, 16) and sbHeight=min(CUHeight, 16). CUWidth and CUHeight are the width and height of a CU. On the other hand, PROF padding is always applied at the 4×4 sub-block level. In the above discussion, only the padding method is unified between BDOF and PROF, while the padding sub-block sizes are still different. This is also not friendly to actual hardware implementation, considering that different modules need to be implemented for the padding process of BDOF and PROF. In order to achieve a more unified design, it is proposed to unify the sub-block padding size of BDOF and PROF. In one embodiment of the present disclosure, it is proposed to apply prediction sample padding of BDOF at the 4×4 level. Specifically, with this method, the CU is first divided into multiple 4×4 sub-blocks; after motion compensation is performed on each 4×4 sub-block, the extended samples along the top / bottom and left / right boundaries are filled by copying the corresponding integer sample positions.
[0192] Fig.18A , Fig.18B , Fig.18C and Fig.18D An example of applying the proposed padding method to a 16×16 BDOFCU is shown, where the dashed lines represent the 4×4 sub-block boundaries and the grey bands represent the padding samples for each 4×4 sub-block.
[0193] Fig.18A The proposed padding method applied to a 16×16 BDOF CU according to the present disclosure is shown, where the dashed line represents the top left 4×4 sub-block boundary 1820 .
[0194] Fig.18B The proposed padding method applied to a 16×16 BDOF CU according to the present disclosure is shown, where the dashed line represents the top right 4×4 sub-block boundary 1840 .
[0195] Fig.18C The proposed padding method applied to a 16×16 BDOF CU according to the present disclosure is shown, where the dashed line represents the lower left 4×4 sub-block boundary 1860 .
[0196] Fig.18D The proposed padding method applied to a 16×16 BDOF CU according to the present disclosure is shown, where the dashed line represents the bottom right 4×4 sub-block boundary 1880 .
[0197] Fig.10A unified method of BDOF and PROF for decoding a video signal according to the present disclosure is shown. For example, the method can be applied to a decoder.
[0198] In step 1010, the decoder may divide the video block into a plurality of non-overlapping video sub-blocks. At least one non-overlapping video sub-block of the plurality of non-overlapping video sub-blocks may be associated with two motion vectors.
[0199] In step 1012, the decoder may obtain a first reference picture I associated with two motion vectors of at least one non-overlapping video sub-block of the plurality of non-overlapping video sub-blocks. (0) and the second reference picture I (1) In the display order, the first reference picture I (0) Before the current picture and the second reference picture I (1) After the current image.
[0200] In step 1014, the decoder can obtain the first reference picture I (0) The reference block in obtains the first prediction sample I of the video sub-block (0) (i, j). i and j can represent the coordinates of a sample based on the current image.
[0201] In step 1016, the decoder can obtain the second reference picture I (1) The reference block in obtains the second prediction sample I of the video sub-block (1) (i, j).
[0202] In step 1018, the decoder may obtain the first prediction sample I (0) (i, j) and the second prediction sample I (1) The horizontal and vertical gradient values of (i, j).
[0203] In step 1020 , when the video block is not encoded in affine mode, the decoder may obtain motion refinement of samples in the video sub-block based on the BDOF.
[0204] In step 1022, when the video block is encoded in affine mode, the decoder may obtain motion refinement of samples in the video sub-block based on PROF.
[0205] In step 1024, the decoder may obtain prediction samples for the video block based on the motion refinement.
[0206] Enable / disable high-level signaling syntax for BDOF, PROF, and DMVR
[0207] In the existing BDOF and PROF design, two different flags are signaled in the SPS to separately control the enabling / disabling of the two codec tools. However, due to the similarities between BDOF and PROF, it is more desirable to enable and / or disable BDOF and PROF from a high level through a same control flag. Based on such considerations, a new flag is introduced at the SPS, called sps_bdof_prof_enabled_flag, as shown in Table 1. As shown in Table 1, the enabling and disabling of BDOF depends only on sps_bdof_prof_enabled_flag. When the flag is equal to 1, BDOF is enabled to encode and decode the video content in the sequence. Otherwise, when sps_bdof_prof_enabled_flag is equal to 0, BDOF will not be applied. On the other hand, in addition to sps_bdof_prof_enabled_flag, the SPS level affine control flag (i.e., sps_affine_enabled_flag) is also used to conditionally enable and disable PROF. When the flags sps_bdof_prof_enabled_flag and sps_affine_enabled_flag are both equal to 1, PROF is enabled for all coding blocks coded in affine mode. When the flags sps_bdof_prof_enabled_flag are equal to 1 and sps_affine_enabled_flag are equal to 0, PROF is disabled.
[0208] Fig.11 A method for decoding BDOF and PROF of a video signal according to the present disclosure is shown. For example, the method can be applied to a decoder.
[0209] In step 1110, the decoder may obtain a first reference picture I associated with the video block. (0) and the second reference picture I (1) In the display order, the first reference picture I (0) Before the current picture and the second reference picture I (1) After the current image.
[0210] In step 1112, the decoder can obtain the first reference picture I (0) The reference block in obtains the first prediction sample I of the video block (0) (i, j). i and j can represent the coordinates of a sample based on the current image.
[0211] In step 1114, the decoder can obtain the second reference picture I (1) The reference block in obtains the second prediction sample I of the video block (1) (i, j).
[0212] In step 1116, the decoder may receive at least one flag. The at least one flag is signaled by the encoder in the SPS and signals whether BDOF and PROF are enabled for the current video block.
[0213] In step 1118, when at least one flag is enabled, the decoder may apply BDOF to generate a prediction image based on the first prediction sample I when the video block is not encoded in affine mode. (0) (i, j) and the second prediction sample I (1) (i, j) derives the motion refinement of the video block.
[0214] In step 1120, when at least one flag is enabled, the decoder may apply PROF to generate a prediction image based on the first prediction sample I when the video block is encoded in affine mode. (0) (i, j) and the second prediction sample I (1) (i, j) derives the motion refinement of the video block.
[0215] In step 1122 , the decoder may obtain prediction samples of the video block based on the motion refinement.
[0216] Table 1 Modified SPS syntax table with proposed BDOF / PROF enable / disable flags
[0217] sps_bdof_prof_enabled_flag specifies whether bidirectional optical flow and prediction refinement with optical flow are enabled. When sps_bdof_prof_enabled_flag is equal to 0, both bidirectional optical flow and prediction refinement with optical flow are disabled. When sps_bdof_prof_enabled_flag is equal to 1 and sps_affine_enabled_flag is equal to 1, both bidirectional optical flow and prediction refinement with optical flow are enabled. Otherwise (sps_bdof_prof_enabled_flag is equal to 1 and sps_affine_enabled_flag is equal to 0), bidirectional optical flow is enabled, while prediction refinement with optical flow is disabled.
[0218] sps_bdof_prof_dmvr_slice_preset_flag specifies when the flag slice_disable_bdof_prof_dmvr_flag is signaled at slice level. When the flag is equal to 1, the syntax slice_disable_bdof_prof_dmvr_flag is signaled for each slice that references the current sequence parameter set. Otherwise (when sps_bdof_prof_dmvr_slice_present_flag is equal to 0), the syntax slice_disabled_bdof_prof_dmvr_flag will not be signaled at slice level. When the flag is not signaled, it is inferred to be 0.
[0219] In addition to the above-mentioned SPS BDOF / PROF syntax, it is proposed to introduce another control flag at the slice level, namely, slice_disable_bdof_prof_dmvr_flag for disabling BDOF, PROF and DMVR. The SPS flag sps_bdof_prof_dmvr_slice_present_flag (which is signaled in the SPS when either DMVR or BDOF / PROF sps level control flag is true) is used to indicate the presence of slice_disable_bdof_prof_dmvr_flag. If present, slice_disable_bdof_dmvr_flag is signaled. Table 2 shows the modified slice header syntax table after applying the proposed syntax.
[0220] Table 2 Modified SPS syntax table with proposed BDOF / PROF enable / disable flags
[0221]
[0222]
[0223] Early termination of PROF based on control point MV difference
[0224] According to the current PROF design, PROF is always called when predicting a coding block through an affine mode. However, as shown in formulas (6) and (7), the sub-block MVs of an affine block are derived from the control point MVs. Therefore, when the difference between the control point MVs is relatively small, the MVs at each sample position should be consistent. In this case, the benefit of applying PROF may be very limited. Therefore, in order to further reduce the average computational complexity of PROF, it is proposed to adaptively skip PROF-based sample refinement based on the maximum MV difference between the sample-by-sample MV and the sub-block-by-sub-block MV within a 4×4 sub-block. Because the values of the PROF MV differences of samples within a 4×4 sub-block are symmetric about the sub-block center, the maximum horizontal and vertical PROF MV differences can be calculated based on formula (10) as
[0225]
[0226] According to the present disclosure, different metrics may be used when determining that the MV difference is small enough to skip the PROF process.
[0227] In one example, based on formula (19), when the sum of the absolute maximum horizontal MV difference and the absolute maximum vertical MV difference is not greater than a predefined threshold, the PROF process can be skipped, that is,
[0228]
[0229] In another example, when and When the maximum value of is not greater than the threshold, the PROF process can be skipped.
[0230]
[0231] Where MAX(a, b) is a function that returns the larger value between the input values a and b.
[0232] In addition to the above two examples, the spirit of the current disclosure is also applicable to the case where other metrics are used to determine whether the MV difference is small enough to skip the PROF process.
[0233] In the above method, PROF is skipped based on the magnitude of the MV difference. On the other hand, PROF sample refinement is calculated based on the local gradient information at each sample position in a motion compensated block in addition to the MV difference. For prediction blocks containing less high-frequency details (e.g., flat areas), the gradient values tend to be small, so the value of the derived sample refinement should be small. Taking this into account, according to another embodiment of the current disclosure, it is proposed to apply PROF only to the prediction samples of the blocks containing sufficient high-frequency information.
[0234] Different metrics can be used to determine whether a block contains enough high-frequency information to make it worth calling the PROF process for the block. In one example, a decision is made based on the average magnitude (i.e., absolute value) of the gradients of samples within the prediction block. When the average magnitude is less than a threshold, the prediction block is classified as a flat area and PROF should not be applied; otherwise, the prediction block is considered to contain enough high-frequency details where PROF is still applicable. In another example, the maximum magnitude of the gradients of samples within the prediction block can be used. If the maximum magnitude is less than a threshold, PROF is skipped for the block. In yet another example, the difference Imax-Imin between the maximum sample value and the minimum sample value of the prediction block can be used to determine whether PROF is applicable to the block. If such a difference is less than a threshold, PROF is skipped for the block. It is worth noting that the spirit of the present disclosure is also applicable to the case where some other metrics are used to determine whether a given block contains enough high-frequency information.
[0235] Handling the interaction between PROF and LIC for affine mode
[0236] Because the neighboring reconstructed samples (i.e., templates) of the current block are used by LIC to derive the linear model parameters, the decoding of a LIC-coded block depends on the complete reconstruction of its neighboring samples. Due to this interdependence, for practical hardware implementation, LIC needs to be performed in the reconstruction phase where the neighboring reconstructed samples become available for LIC parameter derivation. Because block reconstruction must be performed sequentially (i.e., one after another), throughput (i.e., the amount of work that can be done in parallel per unit time) is an important issue to consider when applying other codec methods jointly to LIC-coded blocks. In this section, two methods are proposed to handle the interaction when both PROF and LIC are enabled for affine mode.
[0237] In a first embodiment of the present disclosure, it is proposed to exclusively apply PROF mode and LIC mode for an affine coding block. As discussed above, in the existing design, PROF is implicitly applied to all affine blocks without signaling, while a LIC flag is signaled or inherited at the coding block level to indicate whether the LIC mode is applied to an affine block. According to the method of the present disclosure, it is proposed to conditionally apply PROF based on the value of the LIC flag of an affine block. When the flag is equal to 1, only LIC is applied by adjusting the prediction samples of the entire coding block based on the LIC weights and offsets. Otherwise (i.e., the LIC flag is equal to 0), PROF is applied to the affine coding block to refine the prediction samples of each sub-block based on the optical flow model.
[0238] Fig.17A An example flow chart of the decoding process based on the proposed method is shown, where simultaneous application of PROF and LIC is not allowed.
[0239] Fig.17A A diagram of a decoding process based on the proposed method according to the present disclosure is shown, in which PROF and LIC are not allowed. The decoding process 1720 includes a LIC flag enabled 1722 step, LIC 1724, and PROF 1726. LIC flag enabled 1722 is a step of determining whether the LIC flag is set and proceeding to the next step based on the determination. If the LIC flag is set, LIC 1724 is the application of LIC. If the LIC flag is not set, PROF 1726 is the application of PROF.
[0240] In a second embodiment of the present disclosure, it is proposed to apply LIC after PROF to generate prediction samples of an affine block. Specifically, after completing the sub-block based affine motion compensation, the prediction samples are refined based on PROF sample refinement; then, LIC is performed by applying a pair of weights and offsets (such as derived from the template and its reference samples) to the prediction samples adjusted by PROF to obtain the final prediction samples of the block, as shown in
[0241] P[x]=α*(P r [x+v]+ΔI[x])+β (22)
[0242] Among them, P r [x+v] is the reference block of the current block indicated by motion vector v; α and β are the LIC weights and offsets; P[x] is the final prediction block; ΔI[x] is the PROF refinement derived in (17).
[0243] Fig. 17B A diagram of a decoding process according to the present disclosure is shown, where PROF and LIC are applied. The decoding process 1760 includes affine motion compensation 1762, LIC parameter derivation 1764, PROF 1766, and LIC sample adjustment 1768. Affine motion compensation 1762 applies affine motion and is an input to LIC parameter derivation 1764 and PROF 1766. LIC parameter derivation 1764 is applied to derive LIC parameters. PROF 1766 is the PROF being applied. LIC sample adjustment 1768 is the LIC weight and offset parameters being combined with PROF.
[0244] Fig. 17B An exemplary decoding workflow when the second method is applied is shown. Fig. 17B As shown, since LIC uses templates (i.e., neighboring reconstructed samples) to compute the LIC linear model, LIC parameters can be derived immediately as soon as neighboring reconstructed samples become available. This means that PROF refinement and LIC parameter derivation can be performed simultaneously.
[0245] LIC weights and biases (i.e., α and β) and PROF refinements (i.e., ΔI[x]) are usually floating point numbers. For hardware-friendly implementations, these floating point operations are usually implemented as multiplications with an integer value followed by a right shift of multiple bits. In existing LIC and PROF designs, since these two tools are designed separately, N is applied in two stages. LIC Bits and N PROF Two different right shifts of bits.
[0246] According to the third embodiment of the present disclosure, in order to improve the coding gain when PROF and LIC are jointly applied to affine coded blocks, it is proposed to apply LIC-based sample adjustment and PROF-based sample adjustment with high precision. This is done by combining their two right shift operations into one and applying it at the end to derive the final prediction sample of the current block (as shown in (12)).
[0247] The above method can be implemented using an apparatus including one or more circuits, including an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components. The apparatus can use circuits in combination with other hardware or software components to perform the above method. Each module, submodule, unit or subunit disclosed above can be implemented at least in part using one or more circuits.
[0248] Fig.19 A computing environment 1910 is shown coupled to a user interface 1960. The computing environment 1910 may be part of a data processing server. The computing environment 1910 includes a processor 1920, a memory 1940, and an I / O interface 1950.
[0249] The processor 1920 typically controls the overall operation of the computing environment 1910, for example, operations associated with display, data acquisition, data communication, and image processing. The processor 1920 may include one or more processors to execute instructions to perform all or some of the steps in the above method. In addition, the processor 1920 may include one or more modules that facilitate the interaction between the processor 1920 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single-chip microcomputer, a GPU, etc.
[0250] The memory 1940 is configured to store various types of data to support the operation of the computing environment 1910. The memory 1940 may include predetermined software 1942. Examples of such data include instructions for any application or method operating on the computing environment 1910, video data sets, image data, etc. The memory 1940 may be implemented by using any type of volatile or non-volatile memory device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.
[0251] I / O interface 1950 provides an interface between processor 1920 and peripheral interface modules (e.g., keyboard, click wheel, buttons, etc.). Buttons may include, but are not limited to, a home button, a start scan button, and a stop scan button. I / O interface 1950 may be coupled to an encoder and a decoder.
[0252] In some embodiments, a non-transitory computer-readable storage medium is also provided, which includes multiple programs, for example, included in the memory 1940, and can be executed by the processor 1920 in the computing environment 1910 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.
[0253] The non-transitory computer-readable storage medium has stored therein a plurality of programs for execution by a computing device having one or more processors, wherein the plurality of programs, when executed by the one or more processors, causes the computing device to perform the above-mentioned motion prediction method.
[0254] Here, the computing environment 1910 can be implemented using one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), graphics processing units (GPUs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.
[0255] The description of the present disclosure is presented for purposes of illustration and is not intended to be exhaustive or to limit the present disclosure. Many modifications, variations, and alternative implementations will be apparent to those of ordinary skill in the art having the benefit of the teachings presented in the foregoing description and the associated drawings.
[0256] These examples are selected and described in order to explain the principles of the present disclosure and enable others skilled in the art to understand the various implementations of the present disclosure and to best utilize the basic principles and various implementations with various modifications suitable for the specific use contemplated. Therefore, it should be understood that the scope of the present disclosure is not limited to the specific examples of the disclosed implementations, and modifications and other implementations are intended to be included within the scope of the present disclosure.
Claims
1. A method for video encoding, comprising: Dividing a video block of a current picture into a plurality of video sub-blocks, wherein at least one video sub-block of the plurality of video sub-blocks is associated with two motion vectors; Obtaining a first reference picture and a second reference picture associated with two motion vectors of the at least one video sub-block of the plurality of video sub-blocks, wherein the first reference picture precedes the current picture and the second reference picture follows the current picture in display order; Obtaining a first prediction sample of the video sub-block from the first reference picture; Obtain a second prediction sample of the video sub-block from the second reference picture; Obtaining horizontal gradient values and vertical gradient values of the first prediction sample and the second prediction sample for bidirectional optical flow BDOF and prediction refinement PROF using optical flow; Obtaining motion refinement of samples in the video sub-block based on the BDOF or the PROF; and Obtaining a prediction sample of the video block based on the motion refinement; Wherein, obtaining the horizontal gradient value and the vertical gradient value of the first prediction sample and the second prediction sample includes: Extended samples are derived along top, bottom, left and right boundaries of the first prediction block and the second prediction block of the video sub-block by copying corresponding integer sample positions.
2. The method of claim 1, wherein: The export extension sample includes: padding extended samples along the left boundary and the right boundary by copying from left integer reference samples to fractional sample positions; and Extended samples are padded along the top and bottom boundaries by copying from the upper integer reference samples to fractional sample locations.
3. The method of claim 1, wherein: The export extension sample includes: deriving extended samples along the left boundary and the right boundary by copying in the horizontal direction from integer reference samples closest to the corresponding fractional sample positions; and Extended samples are derived along the top boundary and the bottom boundary by copying in the vertical direction from the integer reference sample closest to the corresponding fractional sample position.
4. The method of claim 1, wherein: Each video sub-block includes 4 samples in width and 4 samples in height.
5. A computing device comprising: one or more processors; A non-transitory computer-readable storage medium storing instructions executable by the one or more processors, wherein the one or more processors are configured to: Dividing a video block of a current picture into a plurality of video sub-blocks, wherein at least one video sub-block of the plurality of video sub-blocks is associated with two motion vectors; Obtaining a first reference picture and a second reference picture associated with two motion vectors of the at least one video sub-block of the plurality of video sub-blocks, wherein the first reference picture precedes the current picture and the second reference picture follows the current picture in display order; Obtaining a first prediction sample of the video sub-block from the first reference picture; Obtain a second prediction sample of the video sub-block from the second reference picture; Obtaining horizontal gradient values and vertical gradient values of the first prediction sample and the second prediction sample for bidirectional optical flow BDOF and prediction refinement PROF using optical flow; Obtaining motion refinement of samples in the video sub-block based on the BDOF or the PROF; and Obtaining a prediction sample of the video block based on the motion refinement; The one or more processors configured to obtain the horizontal gradient value and the vertical gradient value of the first prediction sample and the second prediction sample are further configured to: Extended samples are derived along top, bottom, left and right boundaries of the first prediction block and the second prediction block of the video sub-block by copying corresponding integer sample positions.
6. The computing device of claim 5, wherein: The one or more processors configured to derive the extended sample are further configured to: padding extended samples along the left boundary and the right boundary by copying from left integer reference samples to fractional sample positions; and Extended samples are padded along the top and bottom boundaries by copying from the upper integer reference samples to fractional sample locations.
7. The computing device of claim 5, wherein: The one or more processors configured to derive the extended sample are further configured to: deriving extended samples along the left boundary and the right boundary by copying in a horizontal direction from integer reference samples closest to corresponding fractional sample positions; as well as Extended samples are derived along the top boundary and the bottom boundary by copying in the vertical direction from the integer reference sample closest to the corresponding fractional sample position.
8. The computing device of claim 5, wherein: Each video sub-block includes 4 samples in width and 4 samples in height.
9. A method for transmitting a bit stream, wherein the bit stream is generated by the method according to any one of claims 1-4.
10. A computer program product comprising a plurality of instructions, which when executed by one or more processors cause the one or more processors to perform the method according to any one of claims 1 to 4.
