Method and apparatus for video encoding and decoding using bi-directional optical flow adapted to weighted prediction

Abstract

Different implementations are described, in particular implementations for video encoding and decoding are presented. According to an implementation, in a method for encoding or decoding a portion of an image, at least the equation of the bilateral optical flow is modified for unequal weights in a weighted prediction of the portion of the image. According to a non-limiting example, modifying the bilateral optical flow equation comprises bit-depth shifting the unequal weights used in the weighted prediction, quantizing the unequal weights used in the weighted prediction or deriving modified weights from the unequal weights used in the weighted prediction to apply to a prediction of the portion of the image.

Description

TECHNICAL FIELD

[0001] At least one embodiment of the present application relates generally to a method or apparatus, for example, for encoding or decoding a portion of an image, and more particularly, to a method or apparatus including modifying bilateral optical flow in inter prediction for weighted prediction. BACKGROUND

[0002] One or more implementations relate generally to video compression. At least some embodiments relate to improving compression efficiency compared to existing video compression systems such as HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265, and MPEG-H Part 2 described in “ITU-T H.265 Telecommunication standardization sector of ITU, Series H: Audiovisual and multimedia systems, Infrastructure of audiovisual services - Coding of moving video, High efficiency video coding, Recommendation ITU-T H.265,” or compared to video compression systems under development such as VVC (Versatile Video Coding, a new standard developed by JVET (Joint Video Exploration Team)).

[0003] To achieve high compression efficiency, image and video coding schemes typically employ partitioning of pictures, prediction including motion vector prediction, and transform exploiting spatial and temporal redundancy in video content. Generally, intra or inter prediction is used to exploit intra or inter frame correlation followed by transform of the difference between original picture and predicted picture, typically denoted as prediction error or prediction residual, into frequency domain coefficients, quantization and entropy coding of the coefficients. To reconstruct the video, the compressed data is decoded by inverse processes corresponding to entropy decoding, inverse quantization, inverse transform and prediction.

[0004] Figure 1 An exemplary flowchart of an inter prediction process in a codec such as VVC is shown. For example, the Joint Exploration Model (JEM) and VVC reference software developed by the JVET (Joint Video Exploration Team) group describe an inter prediction process pipeline at Figure 1 Some tools associated with each stage of the inter prediction process pipeline at

[0005] - Generalized bi-prediction (GBI) or bi-prediction with CU-level weights (BCW), where the weighting during the blending of the 2 predictions uses unequal weights.

[0006] - BIO: Bi-directional optical flow (also known as BDOF).

[0007] Compared to existing video compression systems such as HEVC or VVC, there is a need to improve the bi-prediction syntax and process in inter-coded blocks. SUMMARY

[0008] It is an object of the present application to overcome at least one of the drawbacks of the prior art.

[0009] To this end, according to a general aspect of at least one embodiment, a method for encoding a portion of an image is presented, comprising using a sample-wise refinement of bi-directional optical flow, wherein at least the equation of the bi-directional optical flow is modified for unequal weights in weighted prediction of the portion of the image.

[0010] According to another general aspect of at least one embodiment, a method for decoding a portion of an image is presented, comprising using a sample-wise refinement of bi-directional optical flow, wherein at least the equation of the bi-directional optical flow is modified for unequal weights in weighted prediction of the portion of the image.

[0011] According to another general aspect of at least one embodiment, an apparatus for video encoding is presented, comprising means for implementing any of the embodiments of the encoding method.

[0012] According to another general aspect of at least one embodiment, an apparatus for video decoding is presented, comprising means for implementing any of the embodiments of the decoding method.

[0013] According to another general aspect of at least one embodiment, a device for video encoding is provided, comprising one or more processors and at least one memory. The one or more processors are configured to implement any of the embodiments of the encoding method.

[0014] According to another general aspect of at least one embodiment, a device for video decoding is provided, comprising one or more processors and at least one memory. The one or more processors are configured to implement any of the embodiments of the decoding method.

[0015] According to another general aspect of at least one embodiment, the weighted prediction is one of generalized bi-prediction, bi-prediction with coding unit level weights, weighted prediction with image level weights, weighted bi-prediction based on image distance.

[0016] According to another general aspect of at least one embodiment, modifying at least one bi-directional optical flow equation comprises bit-depth shifting of unequal weights used in weighted prediction.

[0017] According to another general aspect of at least one embodiment, modifying at least one bi-directional optical flow equation comprises quantizing unequal weights used in weighted prediction.

[0018] According to another general aspect of at least one of the embodiments, modifying the at least one bi-directional optical flow equation includes deriving modified weights from unequal weights used in the weighted prediction to apply to the prediction of the portion of the picture.

[0019] According to another general aspect of at least one of the embodiments, a non-transitory computer readable medium containing data content generated according to the method or apparatus of any of the foregoing descriptions is presented.

[0020] According to another general aspect of at least one of the embodiments, a signal or bitstream including video data generated according to the method or apparatus of any of the foregoing descriptions is presented.

[0021] One or more of the embodiments of the disclosure also provide a computer readable storage medium having stored thereon instructions for performing bi-directional optical flow, encoding or decoding video data according to any of the above-described methods. The embodiments also provide a computer readable storage medium having stored thereon a bitstream generated according to the above-described methods. The embodiments also provide a method and apparatus for transmitting the bitstream generated according to the above-described methods. The embodiments of the disclosure also provide a computer program product comprising instructions for performing any of the described methods. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 An exemplary flowchart of an inter-prediction process in an encoder architecture is shown.

[0023] Figure 2 An exemplary embodiment of a bi-prediction process in HEVC is shown.

[0024] Figure 3 An exemplary extended CU region used in BDOF according to certain embodiments is shown.

[0025] Figure 4 An exemplary flowchart of determination of a prediction mode compatibility tree according to certain embodiments is shown.

[0026] Figure 5 An example of an optical flow trajectory according to certain embodiments is shown.

[0027] Figure 6 A block diagram of an embodiment of a video encoder in which various aspects of the embodiments can be implemented is shown.

[0028] Figure 7 A block diagram of an embodiment of a video decoder in which various aspects of the embodiments can be implemented is shown.

[0029] Figure 8A block diagram showing an example apparatus in which various aspects of embodiments can be implemented is shown.

[0030] Figure 9 An exemplary flowchart of a bi-directional optical flow process in inter prediction for weighted prediction is shown in accordance with certain embodiments. DETAILED DESCRIPTION

[0031] It should be understood that the figures and descriptions have been simplified to illustrate elements that are relevant for a clear understanding of the present principles, while

[0032] Various embodiments are described in relation to encoding / decoding of images. They can be applied to encoding / decoding of parts of images, such as slices or tiles, tile groups or entire image sequences.

[0033] Various methods are described above, and each of the methods includes one or more steps or actions for achieving the described method. The order in which

[0034] At least some embodiments relate to introducing weighting in inter prediction methods and more specifically in bi-directional optical flow (BIO).

[0035] In the following, some limitations regarding generalized bi-prediction (GBI or BCW) and bi-directional optical flow (BIO or BDOF) are disclosed, general embodiments of an improved BIO process are disclosed, several embodiments of an improved BIO process incorporating GBI weighting are disclosed, several embodiments of a unified weighting of GBI and asymmetric bi-prediction are disclosed, and additional information and general embodiments are disclosed.

[0036] Generalized bi-prediction (GBI)

[0037] In HEVC, the average of the 2 uni-prediction signals used for bi-prediction is done with higher precision than the input or internal bit-depth. Figure 2 An exemplary embodiment of a bi-prediction process in HEVC is shown. The bi-prediction formula is shown in the following equation 1, where offset and shift are parameters used to normalize the final predictor to the input bit-depth.

[0038] P bidir = (P L0 + P L1 + offset) » shift Equation 1

[0039] and P L0 with P L1 are prediction samples constructed using a motion compensation (MC) process with reference samples from list L0 and list L1 in bi-prediction mode. In addition, to increase the accuracy of the sample-based calculations, the sample values are stored with increased bit-depth precision.

[0040] In general, the prediction samples P L0 with P L1 are modified by an internal offset (OFFSETINTERNAL) and expressed using an internal bit-depth (BITDEPTHINTERNAL). The offset is derived as:

[0041] OFFSETINTERNAL == (1 « (BITDEPTHINTERNAL - 1))

[0042] In general, BITDEPTHINTERNAL = 14.

[0043] This means that for the final prediction P' L0 we have the following relation:

[0044] P' L0 = (P L0 + OFFSETINTERNAL) » (BITDEPTHINTERNAL - bitdepth)

[0045] It gives the final value:

[0046] shift = (BITDEPTHINTERNAL + 1 - bitdepth)

[0047] and

[0048] offset = (1 « (shift - 1)) + OFFSETINTERNAL

[0049] Since there is no rounding in the intermediate stage, the HEVC interpolation filter allows certain implementation optimizations.

[0050] In VVC, multiple weights are used to average 2 uni-predictions (GBI) to obtain bi-prediction. In general, the weights used are {-1 / 4, 5 / 4}, {3 / 8, 5 / 8} or {1 / 2, 1 / 2} (where {1 / 2, 1 / 2} are the weights used in HEVC) and the bi-prediction formula is modified as shown in equation 2. Only one weight is used for the whole block.

[0051] P bidir= ((1 - w1) * P L0 + w1 * P L1 + offset) » shift Equation 2

[0052] Thus, the new value is:

[0053] shift = BITDEPTHINTERNAL - bitdepth + GbiBd

[0054] and

[0055] offset = (1 « (shift - 1)) + OFFSETINTERNAL « GbiBd

[0056] where GbiBd is the bit-depth of the weight of the GBI, typically GbiBd = 3 as in the example in Table 1. Note that the sum of the Gbi weights (w0 + w1) is 1.

[0057] GBI weight values Gbi molecules Binarization of Gbi indices -1 / 4 -1 0000 3 / 8 3 001 1 / 2 1 1 5 / 8 5 01 5 / 4 5 0001

[0058] Table 1 : Examples of GBI weights.

[0059] Bi-directional optical flow

[0060] Recent developments of VVC (test model VTM-3) include a bi-directional optical flow (BDOF) tool. BDOF (previously known as BIO) is a less complex version that requires much less computation, especially in terms of the number of multiplications and the size of the multipliers, compared to previous versions of BIO.

[0061] BDOF is used to refine the bi-directional prediction signal of a CU at the 4x4 subblock level. BDOF is applied to a CU if the following conditions are met:

[0062] 1) the height of the CU is not 4, and the size of the CU is not 4x8,

[0063] 2) the CU is not coded using affine mode or ATMVP merge mode;

[0064] 3) the CU is coded using the "true" bi-prediction mode, i.e., one of the two reference pictures is before the current picture in display order and the other is after the current picture in display order.

[0065] BDOF is only applied to the luma component.

[0066] As its name indicates, the BDOF mode is based on the concept of optical flow, which assumes that the motion of an object is smooth and that its luminance is constant along the considered time interval. For each 4x4 subblock, a motion refinement (v) is computed by minimizing the difference between the L0 and L1 prediction samples. The refined prediction signal is then computed asx , v y ). Then, sample values of bi-prediction in the 4x4 sub-block are adjusted using motion refinement. The following steps are applied in the BDOF process.

[0067] First, the horizontal and vertical gradients of the two prediction signals are calculated by directly computing the difference between two neighboring samples, and k = 0, 1, i.e.,

[0068]

[0069] where I (k) (i, j) is the sample value at the coordinates (i, j) of the prediction signal in the list k, k = 0, 1.

[0070] Then, the auto-correlation and cross-correlation of the gradients S1, S2, S3, S5 and S6 are computed as

[0071]

[0072] where:

[0073]

[0074] where Ω is a 6x6 window around the 4x4 sub-block.

[0075] Then using the cross-correlation and auto-correlation terms, the motion refinement (v x , v y ) is derived using the following equation:

[0076]

[0077] where th′ BIO = 2 13-BD . and is the floor function.

[0078] Based on the motion refinement and the gradients, the following adjustment is computed for each sample in the 4x4 sub-block:

[0079]

[0080] Finally, the BDOF samples of the CU are computed by adjusting the bi-prediction samples as follows:

[0081] pred BDOF (x, y) = (I (0) (x, y) + I (1) (x, y) + b(x, y) + o offset ) » shift Equation 3-6

[0082] In the above, n a 、n b and The values ​​of are respectively equal to 3, 6 and 12. These values ​​are selected so that the multiplier in the BDOF process does not exceed 15 bits and the maximum bit width of the intermediate parameters in the BDOF process remains within 32 bits.

[0083] In order to derive the gradient value, it is necessary to generate some prediction samples I in the list k (k=0, 1) outside the current CU boundary (k) (i, j). Figure 3 As shown, BDOF in VTM-3.0 uses an extended row / column around the CU boundary. To control the computational complexity of generating prediction samples outside the boundary, a bilinear filter is used to generate prediction samples in the extended region (white locations), and a normal 8-tap motion-compensated interpolation filter is used to generate prediction samples within the CU (gray locations). These extended sample values ​​are used only for gradient calculations. For the remaining steps in the BDOF process, if any samples and gradient values ​​outside the CU boundary are needed, they are padded (i.e., repeated) from its nearest neighbors. Figure 3 An exemplary extended CU region used in BDOF is shown.

[0084] Bidirectional Optical Flow (BIO) involves per-sample motion refinement performed on top of block-by-block motion compensation for bidirectional prediction. Sample-level motion refinement uses no signaling. In the case of bidirectional prediction, BIO aims to refine the motion of each sample based on Hermite interpolation of optical flow, assuming linear displacement between the two reference pictures.

[0085] Some limitations of BDOF and GBI

[0086] In the latest development of VVC (Test Model VTM-3), the three modes GBI, BDOF and Asymmetric Bidirectional Prediction are currently exclusive and not always available in all possible coding modes. Figure 4 An exemplary flow chart for determining a prediction mode compatibility tree according to the latest development of VVC is shown. Figure 4 In the figure, the dashed line shows the case when the mode is off, and the solid line shows the case when the mode is active. Advantageously, the prediction mode compatibility tree makes decoding less complex (reducing the number of mode selection combinations to explore and easing the decoding pipeline). In the latest version of VVC, BDOF and GBI are exclusive, and BDOF is only activated for symmetric bidirectional prediction (i.e., when the reference picture is in the past and future of the current frame).

[0087] In the previous prior art method (previous test model version of VVC), the test model software includes weighting the gradient images according to the POC of the reference picture and the current picture to handle the case where the current POC is not centered between POC0 and POC1 (as depicted in Figure 5 the asymmetric bi-directional prediction). However, the weighting is not fully done in the BDOF process: only the spatial gradient is weighted by the temporal distance (equation 3-1).

[0088] In the previous prior art method, a method for asymmetric bi-directional prediction with conformant weighting is disclosed, especially the method used in LDB (low delay bi-directional prediction profile). Therefore, it is desirable to implement the conforming of the BDOF process of the weights of the asymmetric bi-directional prediction. Moreover, the test model software of the previous prior art method includes the possible GBI and BDOF homogenization by simply considering the GBI weights in the final BDOF equation 3-6. It is also desirable to refine the GBI and BDOF homogenization method.

[0089] At least one embodiment of a method for adapting the BDOF process to weighted prediction

[0090] To overcome the above limitations, at least one embodiment is disclosed:

[0091] - Incorporation of GBI weighting in the BDOF process, considering the bit-depth constraints.

[0092] - Unification of the weighting of GBI and asymmetric bi-directional prediction constraints

[0093] Therefore, at least one embodiment of a method for adapting BDOF to At GBI weighting

[0094] - Improved BIO+GBI weighting performance due to the disclosed offset / shift

[0095] - Inclusion of weighted prediction in the BIO process without loss of accuracy of non-weighted prediction

[0096] - Inclusion of weighted prediction (instead of prediction sample weighting) in the BIO process to maintain maximum accuracy

[0097] - Allow approximation (by quantization as described in the following embodiments) of the weighting during the BIO process to keep the stored bit-depth low.

[0098] Figure 9An exemplary flow chart of a bidirectional optical flow process in inter-frame prediction for weighted prediction according to a specific embodiment is shown. Advantageously, the bidirectional optical flow process is applied to any inter-frame prediction of an encoding method, decoding method, encoder or decoder. As previously described, the bidirectional optical flow process includes a sample-by-sample refinement of the inter-frame prediction, wherein a correction term b is used to adjust the sample values ​​of the bidirectional prediction. According to the present principles, the bidirectional optical flow is adapted to weighted prediction and at least the equations of the bidirectional optical flow are modified for unequal weights in the weighted prediction of portions of an image. According to non-limiting examples, the weighted prediction is one of generalized bidirectional prediction, bidirectional prediction with decoding unit level weights, weighted prediction with image level weights, and weighted bidirectional prediction based on image distance. As described in detail below, depending on different characteristics, the bidirectional optical flow equations are modified to include bit depth shift weights used in the weighted prediction, the weights used in the weighted prediction are quantized, or modified weights are derived from the weights used in the weighted prediction to be applied to the prediction of the portion of the image.

[0099] like Figure 9 As shown and explained by BDOF equations 4-x and 5-x, bidirectional optical flow consists of accessing a first prediction of the portion of the image I (0) and the second prediction I (1) The preliminary step S110 is to access the weights of the weighted prediction and use the first prediction I (0) , Second Prediction I (1) and weights ω0 and ω1 as input to apply the modified bidirectional optical flow to obtain the refined prediction pred BDOF Another preliminary step S120 of (x, y).

[0100] At least one first embodiment of a method for adapting the BDOF process to GBI weighting

[0101] According to a specific variant of the first embodiment, the weighting of the GBI is simplified. In this variant of the first embodiment, we consider the weighting on the image from an arbitrary weighting of the image, where usually the GBI weights are applied on the image.

[0102] According to a specific variation of the first embodiment, only the final equations 3-6 are modified, rather than applying weights to the input predictions of the entire BIO process.

[0103] pred BDOF (x, y) = (2(ω0I (0) (x, y)+ω1I (1) (x, y)) / (ω0+w1)+b(x, y)+o offset )>>shift

[0104] where ω0 and ω0 are GBI weights, typically with 3-bit weights.

[0105] Advantageously, to improve precision, the equations are rewritten where ω0and ω0are the GBI numerators (see Table 1):

[0106] pred BDOF (x, y) = ((ω0I (0) (x, y) + ω1I (1) (x, y)) + (b(x, y) « shift2) + o offset ) » shift where the new values are:

[0107] shift = BITDEPTHINTERNAL - bitdepth + GbiBd

[0108] shift2 = GbiBd

[0109] o offset = ((1 « (shift - 1)) + OFFSETINTERNAL « GbiBd) + (1 « (shift2 - 1))

[0110] Note that the new offset now integrates the correction factor used for rounding to account for the final shift of the GBI correction term.

[0111] Advantageously, to avoid shifting shift2

[0112] pred BDOF (x, y) = ((ω0I (0) (x, y) + ω1I (1) (x, y)) + (b(x, y) « shift2) + o offset ) » shift

[0113] This process can be further simplified by changing the calculation of the b correction factor. Indeed, the b factor is shifted by shift2 (typically shifi2 = 3).

[0114] From equations 3-5:

[0115]

[0116] Divide by 2 and remove rounding, we get:

[0117]

[0118] When v x and v y factors can be further shifted (shift2 - 1) (as the division in 3-5 has already removed 1 bit). Then further adapt the shift operation in 3-4:

[0119]

[0120]

[0121] with

[0122]

[0123] and th' = th + 2 BIO can be adapted accordingly compared to the original threshold.

[0124] This embodiment advantageously avoids the division (or shift) in equations 3-5 and increases the precision of the factor v x and v y . In summary, the complexity is reduced compared to the full GBI and BIO combination and the performance is improved compared to the nascent GBI and BIO combination with only the last step of BIO.

[0125] At least one second embodiment of a method for adapting the BIO process to asymmetric bi-prediction

[0126] In this at least one second embodiment, we consider that the weighting on the image generally comes from the (asymmetric) temporal distance between images.

[0127] In the original BIO equation, taking into account the weight on each predicted image leads to the following change in equation (3-x), the other equations remaining identical to the one disclosed by A. Alshin with E. Alshin in “Bi- directional optical flow for future codecs” in DCC 2016.

[0128] Spatial gradient:

[0129]

[0130] BIO correction:

[0131]

[0132] Final weighting according to Hermite interpolation

[0133]

[0134] In a variant of the at least one second embodiment, a constant bit-depth is maintained for the BIO weighting. The BIO process is adapted while keeping the same bit-depth stored internally (corresponding to the gradient in equation 4-1). We show a process of a non-limiting example where 1 bit is weighted (the weight can take two values: 1 or 2).

[0135] In this example, we assume that the weights can only take values 1 or 2 (optionally with a sign). We define M = (1 « n) = 2.

[0136] Spatial gradient:

[0137] The equation is changed as follows:

[0138]

[0139]

[0140] The value of m is calculated as follows:

[0141] p = min(|ω0|, |ω1|)

[0142]

[0143]

[0144] where the clip function clip(x, a, b) clips the x value to be between a and b, a / b represents integer division.

[0145] (a) m0 = m1 = m = 0, if |ω0'| == 1 && |ω1'| == 1

[0146] (b) Otherwise

[0147] In the first case (a), the BIO process is not changed compared to the regular process.

[0148] In the second case, the spatial gradients are weighted such that the spatial gradient with the smallest weight will be half of the original spatial gradient, while the spatial gradient with the largest weight will be unchanged.

[0149] For example, for n = 1, (M = 2), ω0 = 2 ω1 = 6:

[0150] ω'0 = 1

[0151] ω'1 = 2

[0152] m0 = 1, m1 = 0, m = 1

[0153] Variation for weight calculation

[0154] It is possible to quantify the scale factor between the maximum and minimum weight. For example, with the maximum weight ω c and a quantization of the scale factor by 2, we obtain:

[0155]

[0156] For example, for n = 1, (M = 2), ω0= 2 ω1= 4: Without quantization of the scale factors, the result is:

[0157] ω'0= 1

[0158] ω'1= 2

[0159] m0= 1, m1= 0, m = 1

[0160] With scale factors of 2, the result is:

[0161] ω'0= 1

[0162] ω'1= 1

[0163] m0= 0, m1= 0, m = 0

[0164] Bio correction

[0165] The bio correction term is modified as follows:

[0166]

[0167] Since the weights are coded on n = 1 bits, At most 4, and the division can be done using a shift operation.

[0168] The term (ω'0+ ω'1) 2 Can be:

[0169] - 4, when the weights are equal (equal to 1), then the division is done using a bit shift.

[0170] - or 9 (when one weight is equal to 1 and the other to 2). In this case, the division is approximated by dividing by 8 (using a bit shift) and the square of the largest weight is approximated by 3 (instead of 4).

[0171] Example: ω'0= 1 ω'1= 2

[0172]

[0173] Final weighting:

[0174]

[0175] In the final weighting, the weights are changed to use a bit shift operation instead of a division. In the case where the weights are equal, the process is unchanged. We describe the process for one case (the other is symmetrical).

[0176] Example: ω'0= 1 ω'1= 2

[0177] pred BDOF (x, y) = ((24I(0) (x, y) + 8I (1) (x, y)) / 32 + b(x, y) + o offset ) » shift

[0178] Additional embodiments and information

[0179] This application describes a number of directions, including tools, features, embodiments, models, methods, etc. Many of these aspects are described as having specificity, and are often described in a way that can sound limiting at least to show individual characteristics. However, this is for purposes of clarity of description, and does not limit the application or scope of those aspects. In fact, all of the different aspects can be combined and interchanged to provide additional aspects. Moreover, these aspects can also be combined and interchanged with aspects described in earlier documents.

[0180] The aspects described and contemplated in this application can be implemented in many different forms. The following Figure 6 , 7 and 8 provide some embodiments, but other embodiments are contemplated and the discussion of Figure 6 , 7 and 8 does not limit the breadth of implementation. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to communicating generated or encoded bitstreams. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media having bitstreams generated according to any of the described methods stored thereon.

[0181] In this application, the terms “reconstruct” and “decode” can be used interchangeably, the terms “pixel” and “sample” can be used interchangeably, and the terms “image,” “picture,” and “frame” can be used interchangeably. Generally, but not necessarily, the term “reconstruct” is used on the encoder side, and “decode” is used on the decoder side.

[0182] Various methods are described herein, and each method includes one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, an order of the specific steps and / or actions can be modified or combined and / or used.

[0183] The various methods and other aspects described in this application can be used to modify modules, such as Figure 6 and Figure 7The motion compensation (170) and motion estimation modules (175, 275) of the illustrated video encoder 100 and decoder 200, in addition, aspects of the disclosure are not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations, whether preexisting or future-developed, as well as extensions of any such standards and recommendations, including VVC and HEVC. The aspects described in this application can be used individually or in combination, unless otherwise indicated or technically precluded.

[0184] Various numerical values are used in this application. The specific values are for example purposes, and the described aspects are not limited to these specific values.

[0185] Figure 6 An encoder 100 is shown. Variations of this encoder 100 are contemplated, but for clarity the encoder 100 is described below without describing all contemplated variations.

[0186] Before being encoded, a video sequence can undergo pre-encoding processing (101), e.g., applying a color transform to the input color pictures (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a re-mapping of the input picture components in order to get a signal distribution more resilient to compression (e.g., using a histogram equalization of one of the color components). Metadata can be associated with the pre-processing and be attached to the bitstream.

[0187] In the encoder 100, pictures are encoded by the encoder elements as follows. A picture to be encoded is partitioned (102) and processed, e.g., in units of CUs. Each unit is encoded using, e.g., intra or inter mode. When a unit is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) which of the intra or inter mode to use to encode a unit, and indicates the intra / inter decision by, e.g., a prediction mode flag. A prediction residual is computed, e.g., by subtracting (110) the prediction block from the original image block.

[0188] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, as well as the motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder can skip the transform and directly apply quantization on the untransformed residual signal. The encoder can bypass the transform and quantization, i.e., directly code the residual without applying the transform or quantization processes.

[0189] The encoder decodes the coded blocks to provide references for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. Loop filters (165) are applied to the reconstructed image to perform, e.g., deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in the reference picture buffer (180).

[0190] Figure 7 A block diagram of a video decoder 200 is illustrated. In the decoder 200, the bitstream is decoded by decoder elements as described below. The video decoder 200 generally performs a decoding pass that is inverse to the encoding pass described in Figure 6 The encoder 100 generally also performs video decoding as part of encoding video data.

[0191] In particular, the input of the decoder includes a video bitstream, which can be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other coded information. Picture partitioning information indicates how the pictures are partitioned. The decoder can therefore split (235) the pictures according to the decoded picture partitioning information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual is combined (255) with a prediction block to reconstruct the image block. The prediction block can be obtained (270) from intra prediction (260) or motion-compensated prediction (i.e., inter prediction) (275). Loop filters (265) are applied to the reconstructed image. The filtered image is stored in the reference picture buffer (280).

[0192] The decoded pictures can further undergo post-decoding processing (285), e.g., inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping that performs the inverse of the remapping process performed in the pre-encoding processing (101). The post-decoding processing can use metadata derived in the pre-encoding processing and signaled in the bitstream.

[0193] Figure 7A block diagram illustrating an example of a system in which various aspects and embodiments can be implemented is shown. The system 1000 can be implemented as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 1000 can be implemented in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, the system 1000 is communicatively coupled to one or more other systems, or other electronic devices, via, for example, a communications bus or through dedicated input and / or output ports. In various embodiments, the system 1000 is configured to implement one or more of the aspects described herein.

[0194] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing, for example, the various aspects described herein. The processor 1010 can include embedded memory, input output interface, and various other circuitries known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which can include non-volatile memory and / or volatile memory, including, but not limited to, Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), flash, magnetic disk drive, and / or optical disk drive. The storage device 1040 can include an internal storage device, an attached storage device (including a removable and non-removable storage device) and / or a network accessible storage device, as non-limiting examples.

[0195] The system 1000 includes an encoder / decoder module 1030 configured, for example, to process data to provide encoded video or decoded video, and the encoder / decoder module 1030 can include its own processor and memory. The encoder / decoder module 1030 represents module(s) that can be included in a device to perform the encoding and / or decoding functions. As is known, a device can include one or both of the encoding and decoding modules. Additionally, the encoder / decoder module 1030 can be implemented as a separate element in the system 1000 or can be incorporated in the processor 1010 as a combination of hardware and software as known to those skilled in the art.

[0196] Program code to be loaded onto the processor 1010 or the encoder / decoder 1030 to perform the various aspects described in this document can be stored in the storage device 1040 and then loaded onto the memory 1020 for execution by the processor 1010. In accordance with various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 can store one or more of various items during the performance of processes described herein. These stored items can include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from processing of equations, formulas, operations, and operational logic.

[0197] In some embodiments, the memory within the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing needed during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device can be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory can be the memory 1020 and / or the storage device 1040, e.g., dynamic volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store, for example, the operating system of a television. In at least one embodiment, fast external dynamic volatile memory, such as RAM, is used as working memory for video coding and decoding operations, such as working memory for MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by the JVET (Joint Video Experts Team)).

[0198] Input to elements of the system 1000 can be provided through various input devices as shown in block 1130. Such input devices include, but are not limited to, (i) a radio frequency (RF) portion that receives RF signals transmitted, for example, over the air by a broadcaster, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Other examples not shown in FIG. 10 include composite video.

[0199] In various embodiments, the input devices of block 1130 have associated respective input processing elements as known in the art. For example, an RF portion can be associated with elements appropriate to the items (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a frequency band), (ii) downconverting the selected signal, (iii) band-limiting again to a narrower frequency band to select (for example) a signal frequency band which can be referred to as a channel in certain embodiments, (iv) demodulating the downconverted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets. The RF portion of various embodiments includes one or more elements to perform these functions, for example, frequency selectors, signal selectors, band-limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF portion can include a tuner that performs various of these functions, including for example, downconverting the received signal (s) to lower frequency (s) (for example, an intermediate frequency or a near-baseband frequency) or to baseband. In one set-top box embodiment, the RF portion and its associated input processing element (s) receive an RF signal transmitted through a wired (for example, cable) medium, and perform frequency selection by filtering, downconverting, and filtering again to a desired frequency band. The order of the above-described (and other) elements is rearranged in various embodiments, some of these elements are removed, and / or other elements performing similar or different functions are added. Adding elements can include inserting elements between existing elements, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF portion includes an antenna.

[0200] Additionally, the USB and / or HDMI terminals can include respective interface processors for connecting the system 1000 to other electronic devices through USB and / or HDMI connections. It will be appreciated that various aspects of input processing, for example, Reed-Solomon error correction, can be implemented as desired within, for example, a separate input processing IC or within the processor 1010. Similarly, aspects of USB or HDMI interface processing can be implemented within a separate interface IC or within the processor 1010 as desired. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements (including, for example, the processor 1010 and the encoder / decoder 1030) which operate in conjunction with memory and storage elements to process the data stream as desired for presentation on an output device.

[0201] The various elements of the system 1000 can be disposed within an integrated housing, and the various elements can be interconnected and transmit data therebetween using suitable connection means 1140, for example, internal buses (including Inter-IC (I2C) buses, wiring, and printed circuit boards) as known in the art.

[0202] The system 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 can include, but is not limited to, a transceiver configured to transmit and to receive data over the communication channel 1060. The communication interface 1050 can include, but is not limited to, a modem or network card, and the communication channel 1060 can be implemented, for example, within a wired and / or a wireless medium.

[0203] In various embodiments, data is streamed or otherwise provided to the system 1000 using a wireless network, such as a Wi-Fi network, for example, IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). Wi-Fi signals of these embodiments are received through the communication channel 1060 and the communication interface 1050 adapted for Wi-Fi communication. The communication channel 1060 of these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other over-the-top communication. Other embodiments provide streamed data to the system 1000 using a set-top box that delivers data through an HDMI connection of the input block 1130. Still other embodiments provide streamed data to the system 1000 using an RF connection of the input block 1130. As noted above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use wireless networks other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0204] The system 1000 can provide output signals to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. The display 1100 of various embodiments incorporates one or more of the following: a touchscreen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display, for example. The display 1100 can be used in a television, a tablet, a laptop computer, a cellular telephone (mobile phone), or other device. The display 1100 can also be integrated with other components (e.g., as in a smartphone), or separate (e.g., an external monitor for a laptop computer). The other peripheral devices 1120 include one or more of the following in various examples of embodiments: a standalone digital video disc (or digital versatile disc) (either can be referred to as a DVR), a disc player, a stereo system, and / or a lighting system, for example. Various embodiments use one or more of the peripheral devices 1120 to provide functionality based on the output of the system 1000. For example, a disc player performs the functionality of playing the output of the system 1000.

[0205] In various embodiments, control signals are transmitted between the system 1000 and the display 1100, speaker 1110, or other peripheral device 1120 using signaling such as AV Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices can be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, output devices can be connected to the system 1000 via communication interface 1050 using communication channel 1060. The display 1100 and speaker 1110 can be integrated into a single unit in an electronic device (e.g., a television) along with other components of the system 1000. In various embodiments, the display interface 1070 includes a display driver, such as a timing controller chip.

[0206] For example, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 may alternatively be separate from one or more of the other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0207] These embodiments may be implemented by computer software implemented by the processor 1010 or a combination of hardware and software. As a non-limiting example, embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as, as non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 1010 may be of any type suitable for the technical environment and may include, as non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0208] Various implementations involve decoding. As used in this application, "decoding" may include, for example, all or part of a process performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by the decoder of the various implementations described in this application, such as determining inter-frame prediction including BIO processes and GBI weighting in asymmetric bidirectional prediction.

[0209] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to difference decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and difference decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to a more general decoding process will be clear based on the specific context of description, and will be believed by those of skill in the art to be fully appreciated.

[0210] Various implementations relate to encoding. In a similar manner as discussed above with respect to "decoding," "encoding" as used in this application can include all or a portion of a process performed, e.g., on an input video sequence, in order to produce an encoded bitstream. In various embodiments, such a process includes one or more processes typically performed by an encoder, e.g., partitioning, difference encoding, transform, quantization, and entropy encoding. In various embodiments, such a process also or alternatively includes processes performed by an encoder of various implementations described in this application, e.g., determining inter prediction including BIO processes and GBI weighting in asymmetric bi-prediction.

[0211] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to difference encoding, and in another embodiment, "encoding" refers to a combination of difference encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to a more general encoding process will become clear based on the specific context of description, and will be believed by those of skill in the art to be fully appreciated.

[0212] Note that syntax elements used herein are descriptive terms. Thus, they do not exclude other syntax element names.

[0213] When an apparatus is presented in flow diagram form, it is to be understood that same also provides a block diagram for a corresponding method / process. Similarly, where an apparatus is presented in block diagram form, it is to be understood that same also provides a flow diagram for a corresponding method / process.

[0214] The implementations and aspects described herein can be implemented in, for example, a method or a process, an apparatus, a software program, a data stream, or a signal. Even if only discussed in the context of a single form of implementation (for example, discussed only as a method), the implementation discussed can also be implemented in other forms (for example, an apparatus or program). For example, an apparatus can be implemented in, for example, appropriate hardware, software, and firmware. The methods can be implemented in, for example, an apparatus such as, for example, a processor, which includes, for example, a processing device, such as a computer, a

[0215] References to “one embodiment,” “an embodiment,” “one implementation,” “an implementation,” and the like, mean that a particular feature, structure, characteristic, and so forth being described can be included in at least one embodiment. Thus, the appearance of the phrases in various places in the specification is not necessarily referring to the same embodiment.

[0216] In addition, the application can relate to “determining” various pieces of information. Determining the information can include one or more of, for example, estimating the information, calculating the information, predicting the information, or retrieving the information from memory.

[0217] Furthermore, the application can relate to “accessing” various pieces of information. Accessing the information can include one or more of, for example, receiving the information, retrieving the information (e.g., from memory), storing the information, moving the information, copying the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0218] In addition, the application can refer to “receiving” various pieces of information. As with “accessing,” receiving is intended to be a broad term. Receiving the information can include one or more of, for example, accessing the information, or retrieving the information (e.g., from memory). Furthermore, “receiving” is typically involved in one way or another during operations such as storing the information, processing the information, transmitting the information, moving the information, copying the information, erasing the information, calculating the information, determining the information, predicting the information, or estimating the information.

[0219] It will be understood that, in cases for example of “A / B,” “A and / or B,” and “at least one of A and B,” any of the following is also intended: selection only of the first listed option (A) or only of the second listed option (B) or of both options (A and B). As a further example, in cases of “A, B, and / or C” or “at least one of A, B, and C,” such phrasing is intended to cover the selection of the first listed option (A) only, or the selection of the second listed option (B) only, or the selection of the third listed option (C) only, or the selection of the first and the second listed options (A and B) only, or the selection of the first and third listed options (A and C) only, or the selection of the second and third listed options (B and C) only, or the selection of all three options (A and B and C). This can be extended, as is clear to one of ordinary skill in this and related arts, for as many items as are listed.

[0220] Furthermore, as used herein, the word "signal" refers to, among other things, indicating something to a corresponding decoder. For example, in certain embodiments, an encoder signals a particular one of a plurality of parameters used to implement weighting in an inter prediction process, such as a bit depth considered in a shift. As such, in one embodiment, the same parameter is used at the encoder side and the decoder side. Thus, for example, the encoder can communicate (explicit signaling) the particular parameter to the decoder so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter as well as other parameters, signaling can be used without communication (implicit signaling) to simply allow the decoder to know and select the particular parameter. By avoiding any actual functionality of the transmission, bit savings are achieved in various embodiments. It should be appreciated that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder. While the foregoing relates to the verb form of the word "signal", the word "signal" can also be used as a noun herein.

[0221] As will be evident to one of ordinary skill in the art, implementations can produce a variety of signals formatted to carry information that can be, for example, stored or transmitted. The information can include, for example, instructions for performing a method, or data produced by one of the described implementations. For example, a signal can be formatted to carry the bitstream of an described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of spectrum) or a baseband signal. The formatting can include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information that the signal carries can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0222] We describe a number of embodiments. Features of the embodiments can be provided individually or in any combination across various claim classes and types. Further, embodiments can include one or more of the following features, devices or aspects across various claim classes and types, individually or in any combination:

[0223] • modifying the inter prediction process to enable application of weighting parameters in the decoder and / or encoder.

[0224] • modifying the bi-directional optical flow in the inter prediction process to accommodate generalized bi-directional prediction weighting applied in the decoder and / or encoder.

[0225] • enabling bit depth shifting in the modified bi-directional optical flow in the decoder and / or encoder.

[0226] • Enabling approximation / quantization of the weighting of the decoder and / or encoder in the modified bi-directional optical flow.

[0227] • Using in the decoder and / or encoder in the modified bi-directional optical flow a prediction weighting instead of a prediction sample weighting.

[0228] • Modifying and / or unifying the weighting for both GBI and asymmetric bi-prediction in the inter-prediction process applied in the decoder and / or encoder.

[0229] • Inserting in the signaling a syntax element enabling the decoder to identify the weighting in the modified inter-prediction method to use.

[0230] • Selecting the modified inter-prediction method to apply at the decoder based on these syntax elements.

[0231] • A bitstream or signal comprising one or more of the described syntax elements or variants thereof.

[0232] • A bitstream or signal comprising syntax conveying information generated according to any of the described embodiments.

[0233] • Inserting in the signaling a syntax element enabling the decoder to modify the inter-prediction method in a manner corresponding to that used by an encoder.

[0234] • Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal comprising one or more of the described syntax elements or variants thereof.

[0235] • Creating and / or transmitting and / or receiving and / or decoding according to any of the described embodiments.

[0236] • A method, process, apparatus, medium storing instructions, medium storing data or signal according to any of the described embodiments.

[0237] • A TV, set-top box, cell phone, tablet or other electronic device performing an adaptation of the inter-prediction method according to any of the described embodiments.

[0238] • A TV, set-top box, cell phone, tablet or other electronic device performing an adaptation of the inter-prediction method according to any of the described embodiments and displaying (e.g. using a monitor, screen or other type of display) the resulting image.

[0239] • A TV, set-top box, cell phone, tablet or other electronic device that selects (e.g., using a tuner) a channel to receive a signal comprising an encoded image, and performs an adaptation of the inter prediction method according to any of the described embodiments.

[0240] • A TV, set-top box, cell phone, tablet or other electronic device that receives (e.g., using an antenna) a signal comprising an encoded image over the air, and performs an adaptation of the inter prediction method according to any of the described embodiments.

Claims

1. A method comprising: performing sample-by-sample motion refinement using bidirectional optical flow of a block-wise motion compensated prediction of at least part of the image, the block-wise motion compensated prediction being bidirectionally predicted; wherein said bidirectional optical flow comprises bit depth shifting of unequal weights used in said bidirectional prediction of said at least part of said image; and The at least part of the image is encoded or decoded using the sample-by-sample motion refinement prediction.

2. The method according to claim 1, wherein the bidirectional optical flow comprises: The unequal weights used in the bi-directional prediction are quantized.

3. The method according to claim 1, wherein the bidirectional optical flow comprises: Modified weights are derived from the unequal weights used in the bi-directional prediction to apply to the block-wise motion compensated prediction of the portion of the picture.

4. The method according to any one of claims 1-3, wherein the bidirectional prediction using unequal weights is one of generalized bidirectional prediction, bidirectional prediction with decoding unit level weights, bidirectional prediction with image level weights, and bidirectional prediction with image distance-based weights.

5. An apparatus comprising a memory and one or more processors, wherein the one or more processors are configured to: performing sample-by-sample motion refinement using bidirectional optical flow of block-wise motion compensated prediction of at least part of an image, the bidirectional optical flow comprising bit-depth shifting of unequal weights used in bidirectional prediction of the block-wise motion compensated prediction; and The at least part of the image is encoded or decoded using the sample-by-sample motion refinement prediction.

6. The apparatus according to claim 5, wherein the bidirectional optical flow comprises: The unequally weighted quantization used in the bi-directional prediction.

7. The apparatus according to claim 5, wherein the bidirectional optical flow comprises: Modified weights are derived from the unequal weights used in bi-directional prediction to apply to the block-wise motion compensated prediction of the portion of the picture.

8. An apparatus according to any one of claims 5 to 7, wherein the bidirectional prediction using unequal weights is one of generalized bidirectional prediction, bidirectional prediction with decoding unit level weights, bidirectional prediction with image level weights, and bidirectional prediction with image distance-based weights.

9. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors: performing sample-by-sample motion refinement using bidirectional optical flow of block-wise motion compensated prediction of at least part of an image, the bidirectional optical flow comprising bit-depth shifting of unequal weights used in bidirectional prediction of the block-wise motion compensated prediction; and The at least part of the image is encoded or decoded using the sample-by-sample motion refinement prediction.

10. The non-transitory computer-readable medium of claim 9, wherein the bidirectional optical flow comprises: The unequally weighted quantization used in the bi-directional prediction.

11. The non-transitory computer-readable medium of claim 9, wherein the bidirectional optical flow comprises: Modified weights are derived from the unequal weights used in bi-directional prediction to apply to the block-wise motion compensated prediction of the portion of the picture.

12. A non-temporary computer-readable medium according to any one of claims 9 to 11, wherein the bidirectional prediction using unequal weights is one of generalized bidirectional prediction, bidirectional prediction with decoding unit level weights, bidirectional prediction with picture level weights, and bidirectional prediction with picture distance-based weights.

13. A method comprising: performing sample-by-sample motion refinement using bidirectional optical flow of a block-wise motion compensated prediction of at least part of the image, the block-wise motion compensated prediction being bidirectionally predicted; wherein the bidirectional optical flow includes quantization of unequal weights used in the bidirectional prediction of the block-by-block motion compensated prediction; and The at least part of the image is encoded or decoded using the sample-by-sample motion refinement prediction.

14. The method of claim 13, wherein the bidirectional prediction using unequal weights is one of generalized bidirectional prediction, bidirectional prediction with coding unit level weights, bidirectional prediction with picture level weights, and bidirectional prediction with picture distance-based weights.

15. The method of claim 13, wherein the bidirectional optical flow comprises: Modified weights are derived from the unequal weights used in the bi-directional prediction to apply to the block-wise motion compensated prediction of the portion of the picture.

16. An apparatus comprising a memory and one or more processors, wherein the one or more processors are configured to: performing sample-by-sample motion refinement using bidirectional optical flow of a block-wise motion compensated prediction of at least part of the image, the block-wise motion compensated prediction being bidirectionally predicted; in, The bidirectional optical flow includes quantization of unequal weights used in the bidirectional prediction of the block-wise motion compensated prediction; and The at least part of the image is encoded or decoded using the sample-by-sample motion refinement prediction.

17. The device of claim 16, wherein the bidirectional prediction using unequal weights is one of generalized bidirectional prediction, bidirectional prediction with coding unit level weights, bidirectional prediction with picture level weights, and bidirectional prediction with picture distance-based weights.

18. The apparatus of claim 16, wherein the bidirectional optical flow comprises: Modified weights are derived from the unequal weights used in the bi-directional prediction to apply to the block-wise motion compensated prediction of the portion of the picture.

19. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors: performing sample-by-sample motion refinement using bidirectional optical flow of a block-wise motion compensated prediction of at least part of the image, the block-wise motion compensated prediction being bidirectionally predicted; in, The bidirectional optical flow includes quantization of unequal weights used in the bidirectional prediction of the block-wise motion compensated prediction; and The at least part of the image is encoded or decoded using the sample-by-sample motion refinement prediction.

20. The non-transitory computer-readable medium of claim 19, wherein the bidirectional prediction using unequal weights is one of generalized bidirectional prediction, bidirectional prediction with coding unit level weights, bidirectional prediction with picture level weights, and bidirectional prediction with picture distance-based weights.

21. The non-transitory computer-readable medium of claim 19, wherein the bidirectional optical flow comprises: Modified weights are derived from the unequal weights used in the bi-directional prediction to apply to the block-wise motion compensated prediction of the portion of the picture.

22. A method comprising: performing sample-by-sample motion refinement using bidirectional optical flow of a block-wise motion compensated prediction of at least part of the image, the block-wise motion compensated prediction being bidirectionally predicted; wherein the bidirectional optical flow comprises deriving modified weights from the unequal weights used in the bidirectional prediction to apply to the block-wise motion compensated prediction of the portion of the image; and The at least part of the image is encoded or decoded using the sample-by-sample motion refinement prediction.

23. The method of claim 22, wherein the bidirectional prediction using unequal weights is one of generalized bidirectional prediction, bidirectional prediction with coding unit level weights, bidirectional prediction with picture level weights, and bidirectional prediction with picture distance-based weights.

24. An apparatus comprising a memory and one or more processors, wherein the one or more processors are configured to: performing sample-by-sample motion refinement using bidirectional optical flow of a block-wise motion compensated prediction of at least a portion of an image, the bidirectional prediction comprising deriving modified weights to apply to the block-wise motion compensated prediction of the portion of the image from unequal weights used in bidirectional prediction of the block-wise motion compensated prediction; and The at least part of the image is encoded or decoded using the sample-by-sample motion refinement prediction.

25. The device of claim 24, wherein the bidirectional prediction using unequal weights is one of generalized bidirectional prediction, bidirectional prediction with coding unit level weights, bidirectional prediction with picture level weights, and bidirectional prediction with picture distance-based weights.

26. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors: performing sample-by-sample motion refinement using bidirectional optical flow of a block-wise motion compensated prediction of at least a portion of an image, the bidirectional prediction comprising deriving modified weights to apply to the block-wise motion compensated prediction of the portion of the image from unequal weights used in bidirectional prediction of the block-wise motion compensated prediction; and The at least part of the image is encoded or decoded using the sample-by-sample motion refinement prediction.

27. The non-transitory computer-readable medium of claim 26, wherein the bidirectional prediction using unequal weights is one of generalized bidirectional prediction, bidirectional prediction with coding unit level weights, bidirectional prediction with picture level weights, and bidirectional prediction with picture distance-based weights.

Images