Sample value generation

Abstract

There is provided a method for use in generating sample values of a block or a sub-block in a current picture. The method comprises (i) obtaining (s602) motion vector (MV) information indicating a first set of one or more MVs associated with the block or the sub- block, (ii) obtaining a first set of one or more MV adjustment values for adjusting one or more MVs, (iii) generating a second set of one or more MVs based on the first set of one or more MVs and the first set of one or more MV adjustment values. The method further comprises obtaining a second set of one or more MV adjustment values for adjusting one or more MVs, and generating a third set of one or more MVs based on the second set of one or more MVs and the second set of one or more MV adjustment values.

Description

SAMPLE VALUE GENERATION TECHNICAL FIELD

[0001] This disclosure relates to methods and apparatus for sample value generation. BACKGROUND

[0002] VVC and ECM

[0003] Versatile Video Coding (VVC) is a block-based video codec standardized by the International Telecommunication Union Telecommunication Standardization Sector (ITU- T) and the Moving Picture Experts Group (MPEG). Enhanced Coding Model (ECM) is an exploratory codec which is currently under development. The aim of ECM is to demonstrate and try providing evidence of video coding capabilities beyond VVC. The current ECM version is ECM-10.0.

[0004] Video and Picture

[0005] A video (a.k.a., “video sequence”) comprises a series of pictures. In VVC, each picture is identified with a picture order count (POC) value. The POC value also represents the display order of the picture. A picture with a smaller POC value is displayed before another picture with a larger POC value.

[0006] Components

[0007] Each component can be described as a two-dimensional rectangular array of sample values. It is common that each picture consists of three components; one luma component Y where the sample values are luma values and two chroma components Cb and Cr, where the sample values are chroma values.

[0008] It is also common that the dimensions of the chroma components are smaller than the luma components by a factor of two in each dimension. For example, the size of the luma component of a high definition (HD) picture would be 1920x1080 and the chroma components would each have the dimension of 960x540. Components are sometimes referred to as color components.

[0009] Coding Units and Coding Block

[0010] A block is a two-dimensional (2D) array (e.g., matrix) of sample values (or “samples” for short). In video coding, each component of a picture is split into blocks and thecoded video bitstream consists of a series of coded blocks. It is common in video coding that pictures are split into units that cover a specific area of the picture.

[0011] Each unit consists of all blocks from all components that make up that specific area of the picture and each block belongs fully to one unit. The Coding Unit (CU) in VVC is an example of a unit. In VVC, the CUs may be split recursively to smaller CUs. The CU at the top level is referred to as the coding tree unit (CTU). A CU usually contains three coding blocks, i.e., one coding block for luma and two coding blocks for chroma. The size of the luma coding block is the same as the size of the CU. In VVC, the CUs can have size of 4x4 up to 128x128. In current ECM, the CUs can have size of 4x4 up to 256x256.

[0012] Parameter Sets, Slice Headers, and Picture Headers

[0013] VVC specifies three types of parameter sets: the picture parameter set (PPS), the sequence parameter set (SPS), and the video parameter set (VPS). The PPS contains data that is common for a whole picture, the SPS contains data that is common for a coded layer video sequence (CLVS), and the VPS contains data that is common for multiple CLVSs, e.g., data for multiple layers in the bitstream.

[0014] The concept of slices divides the picture into independently coded slices, where decoding of one slice in a picture is independent of other slices of the same picture. Each slice has a slice header comprising syntax elements. Decoded slice header values from these syntax elements are used when decoding the slice. In VVC, a coded picture contains a picture header. The picture header contains parameters that are common for all slices of the coded picture.

[0015] Intra Prediction

[0016] In intra prediction, also known as spatial prediction, a current block is predicted using previous decoded blocks within the same picture. The samples from the previously decoded blocks within the same picture are used to predict the samples inside the current block. A picture consisting of only intra-predicted blocks is referred to as an intra picture.

[0017] Inter Prediction

[0018] In inter prediction, also known as temporal prediction, blocks of the current picture are predicted using blocks from previously decoded pictures (these blocks are referred to as reference blocks). The samples from the reference blocks in the previously decoded pictures are used to predict the samples inside the current block. A picture that comprises one or more inter-predicted blocks is referred to as an inter picture. The previous decoded picturesused for inter prediction are referred to as reference pictures.

[0019] The location of a referenced block inside a reference picture is indicated using a vector (i.e., a set of values) (which is referred to as “motion vector (MV)”). Each MV consists of two values: an x value (a.k.a., x component) and y value (a.k.a., y component) which represent the displacements between the current block and the referenced block in x and y dimensions. The value of a component may have a resolution finer than an integer position. When that is the case, a filtering (typically interpolation) is done to calculate values used for prediction. FIG. 9 shows an example of a MV for the current block C. The example MV consist of the values 2,1.

[0020] An inter picture may use several reference pictures. The reference pictures are usually put into two reference picture lists, L0 and L1. The reference pictures that are output before the current picture are typically the first pictures in L0. The reference pictures that are output after the current picture are typically the first pictures in L1.

[0021] Inter predicted blocks can use one of two prediction types, uni- and bi-prediction. Uni-predicted block predicts from one reference picture, either using L0 or L1. Bi-prediction predicts from two reference pictures, one from L0 and the other from L1. FIG. 10 shows an example of the prediction types -- uni- and bi- inter prediction.

[0022] Picture Coding Type (Low Delay Picture and Non-Low Delay Picture)

[0023] A low delay picture is a picture that has all its reference pictures displayed before the picture. In other words, for a low delay picture, all its reference pictures have smaller POC values than the current POC. A non-low delay picture is a picture that has at least one of its reference pictures displayed after the picture. In other words, a non-low delay picture has at least one reference picture with a larger POC value than the current POC.

[0024] Fractional MVs, Interpolation Filter and MV Rounding

[0025] The value of the MV’s x or y component may correspond to a sample position which has finer granularity than an integer (sample) position. Those positions are also referred to as fractional (sample) positions.

[0026] In VVC and current ECM, the MV can be at 1 / 16 sample position. FIG. 11A depicts several fractional positions in the horizontal (x-) dimension. The solid-square blocks represent integer positions. The circles represent 1 / 16-position. For example, MV = (4, 10) means the x component is at 4 / 16 position, the y component is at 10 / 16 position.

[0027] In video coding, a MV rounding process is sometimes used to convert a MV at one position to another target position. One example of rounding is to round a fractional MV position to the nearest integer position.

[0028] When an MV is at a fractional position, filtering (typically interpolation) is done to calculate the sample values at those positions. In VVC, the length (number of filter taps) of the interpolation filter for luma component is 8, as shown in table 1 below. In ECM, the length of the interpolation filter for luma component has been increased to 12. TABLE 1 Fractional sample interpolation filter coefficients position p fL[p][0] fL[p][1] fL[p][2] fL[p][3] fL[p][4] fL[p][5] fL[p][6] fL[p][7] 1 0 1 −3 63 4 −2 1 0 2 −1 2 −5 62 8 −3 1 0 3 −1 3 −8 60 13 −4 1 0 4 −1 4 −10 58 17 −5 1 0 5 −1 4 −11 52 26 −8 3 −1 6 −1 3 −9 47 31 −10 4 −1 7 −1 4 −11 45 34 −10 4 −1 8 −1 4 −11 40 40 −11 4 −1 9 −1 4 −10 34 45 −11 4 −1 10 −1 4 −10 31 47 −9 3 −1 11 −1 3 −8 26 52 −11 4 −1 12 0 1 −5 17 58 −10 4 −1 13 0 1 −4 13 60 −8 3 −1 14 0 1 −3 8 62 −5 2 −1 15 0 1 −2 4 63 −3 1 0

[0029] Residual, Transform, and Quantization

[0030] The difference between samples of a source block (contains original samples) and samples of the prediction block, also called residual block, is then typically compressed by a spatial transform to remove further redundancy. The transform coefficients are then quantized by a quantization parameter (QP) to control the fidelity of the residual block and thus also the bitrate required to compress the block. A coded block flag (CBF) is used to indicate if there are any non-zero quantized transform coefficients. All coding parameters are then entropy coded at the encoder and decoded at the decoder. If the coded block flag is one, a reconstructed block can then be derived by inverse quantization and inverse transformation of the quantized transform coefficients and then add that to the prediction block. If the coded block flag is zero, the reconstructed block is identical to the prediction block.

[0031] Inter Prediction Information / Motion Information

[0032] For an inter block inside an inter picture in VVC, its inter prediction information consists of the following three elements:

[0033] (1) a reference picture list flag (RefPicListFlag) which signals which reference picture list is used for the block (when the value of the flag is equal to 0, it means only L0 is used for predicting the current block, when the value of the flag is equal to 1, it means only L1 is used for predicting the current block, and when the value of the flag is equal to 2, it means both L0 and L1 are used for predicting the current block);

[0034] (2) a reference picture index (RefPicIdx) per reference picture list used (the index signals which reference picture inside the reference list to be used for predicting the current block), and

[0035] (3) a motion vector (MV) per reference picture used, which signals the position inside the reference picture that is used for predicting the current block.

[0036] The inter prediction information is also referred to as motion information. The decoder stores the motion information for each inter block. In other words, an inter block maintains its own motion information.

[0037] Encoder Decision and Rate Distortion (RD) Cost

[0038] In practice, for an encoder to decide the best prediction mode for a current block, the encoder would evaluate all the possible prediction modes for the current block and select the prediction mode that yields the smallest Rate-Distortion (RD) cost.

[0039] The RD cost is calculated as ^^ ^ ^^ ∗ ^^. D (Distortion) measures the difference between the reconstructed block and the corresponding source block. One commonly used metric for calculating D is the sum of squared error ^^ ^^ ^^ ൌ ∑௫,௬^ ^^^^ ^^, ^^^ െ ^^^^ ^^, ^^^^ଶ, where the ^^^and ^^^are the sample values in the two blocks A and B respectively. R (rate) is usually an estimation of the bits to be spent on encoding the mode, and ^^ is a trade-off parameter between R and D.

[0040] Motion Information Signaling

[0041] VVC and ECM includes several methods for implicit signaling of motion information for each block, including the merge method and the subblock merge method. A common motivation behind the implicit methods is to inherit or reuse motion information fromneighboring coded blocks. This often works in practice due to spatial correlation of close-by blocks, i.e., the fact that nearby blocks often behave similarly.

[0042] Motion Information Storage

[0043] The motion information is stored using a smallest block grid in the picture. Currently, in VVC and ECM, the smallest block size is 4x4. The smallest block is also referred to as a subblock. FIG. 11B shows an example of an 8x8 block which contains four 4x4 sub- blocks. Each sub-block stores a copy of the 8x8 block’s motion information (Mi).

[0044] Merge (Block Merge) Method and Merge Mode

[0045] The merge method derives a set of motion information from previously decoded blocks and use the derived motion information for generating the samples of the entire block. The merge method is sometimes referred to as the block merge method.

[0046] The merge method first generates a list of motion information candidates. The list is also referred to as the merge list. The candidates are derived from previously coded blocks. These previously coded blocks can be spatially adjacent neighboring blocks or temporal collocated blocks relative to the current block. FIG. 12 shows possible spatial neighboring blocks for fetching neighboring motion information: left (L), top (T), top-right (TR), left-bottom (LB) and top-left (TL).

[0047] The merge list construction process usually checks the previously coded blocks in a predefined order, for example: T, L, TR, LB, then TL. For each previously coded block being checked, if this previously coded block is inter coded and its motion information has no duplicates in the list, then the motion information of this previously coded block is added to the merge list.

[0048] After the merge list is generated, one of the candidates inside the list is used to derive the motion information of the current block. The candidate selection process is done on the encoder side. An encoder would select a best candidate from the list and encode an index (merge_index) in the bitstream to signal to a decoder. The decoder receives the index, it follows the same merge list derivation process as the encoder, and uses the index to retrieve the correct candidate. The blocks that use the block merge method are sometimes referred to as blocks in merge mode.

[0049] In the current ECM, non-adjacent spatial blocks are also considered as sources of motion information during the merge list construction. FIG.13 shows some examples (markedwith NA1, NA2 and NA3) of those non-adjacent spatial blocks.

[0050] Subblock Merge Method

[0051] VVC and ECM also include the subblock merge method. It splits a current block into a number of subblocks and allows each subblock to have its own motion information. FIGS. 12A and 12B show an example of a current block and its subblocks. Each subblock maintains its own motion information. It should be noted that the subblocks are all rectangular.

[0052] Merge With Motion Vector Difference (MMVD)

[0053] In VVC and ECM, MMVD is a tool which adds possibilities of explicitly signalling an MVD (motion vector difference) to the inherited motion vector (of the inherited motion information) for blocks coded using the block merge method.

[0054] The motivation of MMVD is to further adjust the inherited motion vector to better cater for the current inter block, since, in some cases, the motion vector used for a previously decoded block may not be well-suitable for the content within the current block. It should be noted that MMVD only modifies the motion vector. Other parameters in the motion information -- i.e., the inherited reference picture list flag and inherited reference picture index -- are kept unmodified.

[0055] MVD is signalled as a pair of distance and direction. In VVC, there are eight candidate distances (¼-sample, ½-sample, 1-sample, 2-sample, 4-sample, 8-sample, 16- sample, 32-sample), and four candidate directions (four horizontal / vertical directions). In ECM, the number of candidate MVDs are increased.

[0056] Overlapped Block Motion Compensation (OBMC)

[0057] In ECM, a tool called OBMC is introduced to improve prediction for samples that are close to a block boundary or a subblock boundary. For an inter coded block, its prediction samples are first generated using the block’s current motion information. Then, the OBMC tool further modifies the prediction samples that are adjacent to the block boundary or the subblock boundary with the help of neighboring motion information. OBMC operates at subblock basis within a current inter coded block as illustrated in FIGS. 14A and 14B. The size of the subblock that OBMC operates on is 4x4 in the current ECM.

[0058] For a subblock which is adjacent to a block boundary or a subblock boundary, if its corresponding adjacent neighboring subblock (the subblock on the other side of the boundary) is available and contain a piece of motion information that is not identical to thecurrent motion information, the OBMC tool is then activated to further modify the prediction samples of the subblock with the help of this neighboring motion information. FIG.15A shows examples of the block boundary and the subblock boundary, and FIGS. 15B-15C depict a simple example of such modification process.

[0059] In FIG.15B, the current subblock C has its motion information denoted a MV(C) and its left neighbor subblock is available. The left neighbor subblock has its motion information denoted as MV(Left). The prediction samples of the current subblock are further modified by blending in new prediction samples which are generated using the MV(Left). In other words, OBMC involves a blending process to take a weighted average of prediction samples generated using the current motion information (including the MV(C)) and prediction samples generated using the neighboring motion information (including the MV(Left)), as shown in FIG.15C.

[0060] FIG. 16A shows a more detailed example when OBMC is operating on a left subblock boundary of a subblock. The circles in the figure represent prediction samples. The prediction block A represents prediction samples that are generated using the current MV. The prediction block B represents prediction samples that are generated using the neighboring (in this case, the left) MV. OBMC then blends the prediction block B with the prediction block A to get a blended prediction block C for the current subblock. In other words, OBMC uses samples from the prediction block B to adjust the samples of the prediction block A. During blending, each column or row (that is adjacent to and also parallel to the subblock boundary) of prediction samples is assigned with a corresponding weight.

[0061] In the current ECM, when applying OBMC on a subblock boundary, the number of columns or rows (that are parallel to the boundary) that gets OBMC adjusted are fixed to 3. In other words, 3 columns or rows of prediction samples that are generated using the neighboring motion information are associated with non-zero weights during OBMC blending. FIG. 16B shows an example of OBMC operating on a subblock that is adjacent to an above subblock boundary. SUMMARY

[0062] Certain challenges presently exist. For example, sometimes the prediction accuracies of the MMVD and the OBMC are not good. In other words, values of samples (a.k.a., “sample values”) of a current picture, which are predicted (i.e., generated) based on sample values of reference picture(s) using the MMVD or the OBMC may not be close enough to original sample values (i.e., sample values of the current picture before encoding)of the current picture. Such inaccurate predictions of the original sample values may result in a poor reconstruction of the original picture. Therefore, there is a need to improve the prediction accuracies of the MMVD and the OBMC.

[0063] Accordingly, in one aspect of some embodiments of this disclosure, there is provided a method for use in generating sample values of a block or a sub-block in a current picture. The method comprises obtaining motion vector, MV, information indicating a first set of one or more MVs associated with the block or the sub-block; obtaining a first set of one or more MV adjustment values for adjusting one or more MVs; generating a second set of one or more MVs based on the first set of one or more MVs and the first set of one or more MV adjustment values; obtaining a second set of one or more MV adjustment values for adjusting one or more MVs; and generating a third set of one or more MVs based on the second set of one or more MVs and the second set of one or more MV adjustment values. The sample values of the block or the sub-block in the current picture are generated based on the third set of one or more MVs.

[0064] In another aspect, there is provided a method for generating sample values of a first block or a first sub-block in a current picture. The method comprises obtaining first motion vector, MV, information indicating a first MV associated with the first block or the first sub- block; based on the first MV, generating first prediction sample values of the first block or the first sub-block; obtaining second MV information indicating a second MV associated with a second block or a second sub-block in the current picture, wherein the second block is adjacent to the first block or the second sub-block is adjacent to the first sub-block; based on the second MV, generating second prediction sample values of the second block or the second sub-block; determining one or more regions of the first block based on one or more of the first MV, the second MV, or a boundary between the first block and the second block or between the first sub-block and the second sub-block; and generating sample values in said one or more regions (e.g., rows or columns) of the first block or the first sub-block based on some of the first prediction sample values and some of the second prediction sample values.

[0065] In a different aspect, there is provided a computer program comprising instructions which when executed by processing circuitry cause the processing circuitry to perform the method of any one of the above embodiments.

[0066] In a different aspect, there is provided a carrier containing the computer program of the above embodiment, wherein the carrier is one of an electronic signal, an optical signal,a radio signal, and a computer readable storage medium.

[0067] In a different aspect, there is provided an apparatus for use in generating sample values of a block or a sub-block in a current picture. The apparatus is configured to obtain motion vector, MV, information indicating a first set of one or more MVs associated with the block or the sub-block; obtain a first set of one or more MV adjustment values for adjusting one or more MVs; generate a second set of one or more MVs based on the first set of one or more MVs and the first set of one or more MV adjustment values; obtain a second set of one or more MV adjustment values for adjusting one or more MVs; and generate a third set of one or more MVs based on the second set of one or more MVs and the second set of one or more MV adjustment values. The sample values of the block or the sub-block in the current picture are generated based on the third set of one or more MVs.

[0068] In a different aspect, there is provided an apparatus for generating sample values of a first block or a first sub-block in a current picture, the apparatus being configured to obtain first motion vector, MV, information indicating a first MV associated with the first block or the first sub-block; based on the first MV, generate first prediction sample values of the first block or the first sub-block; obtain second MV information indicating a second MV associated with a second block or a second sub-block in the current picture, wherein the second block is adjacent to the first block or the second sub-block is adjacent to the first sub-block; based on the second MV, generate second prediction sample values of the second block or the second sub-block; determine one or more regions of the first block based on one or more of the first MV, the second MV, or a boundary between the first block and the second block or between the first sub-block and the second sub-block; and generate sample values in said one or more regions (e.g., rows or columns) of the first block or the first sub-block based on some of the first prediction sample values and some of the second prediction sample values.

[0069] In a different aspect, there is provided an apparatus comprising a memory; and processing circuitry coupled to the memory, wherein the apparatus is configured to perform the method of any one of the above embodiments.

[0070] Embodiments of this disclosure improve the prediction accuracies of the MMVD and the OBMC, thereby improving the quality of a reconstructed picture (i.e., a decoded picture). BRIEF DESCRIPTION OF THE DRAWINGS

[0071] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0072] FIG.1A shows a system according to some embodiments.

[0073] FIG.1B shows an example of the system according to some embodiments.

[0074] FIG.1C shows an example of the system according to some embodiments.

[0075] FIG.2 shows a process according to some embodiments.

[0076] FIG.3A shows a process of MMVD.

[0077] FIGS.3B-3E show a combined process of MMVD and BDOF.

[0078] FIG.4 shows a process according to some embodiments.

[0079] FIG.5A illustrates an exemplary way of applying OBMC.

[0080] FIG.5B illustrates an exemplary way of applying OBMC.

[0081] FIG.5C shows how sample values are generated using motion vectors.

[0082] FIG.5D shows how sample values are generated using motion vectors.

[0083] FIG.6 shows a process according to some embodiments.

[0084] FIG.7 shows a process according to some embodiments.

[0085] FIG.8 shows an apparatus according to some embodiments.

[0086] FIG.9 shows an example of an MV.

[0087] FIG.10 shows an example of prediction types.

[0088] FIG. 11A shows exemplary fractional positions of an MV in a horizontal (x-) dimension.

[0089] FIG. 11B shows an example of motion information storage in subblocks for an 8x8 block.

[0090] FIG.12 shows examples of spatial neighboring blocks.

[0091] FIG.13 shows some examples of adjacent spatial blocks and non-adjacent spatial blocks for fetching motion information.

[0092] FIGS.14A and 14B show an example of a current block and its subblocks.

[0093] FIG.15A shows a block boundary and subblock boundaries.

[0094] FIGS.15B and 15C show a simple example of OBMC process.

[0095] FIG. 16A shows an example of a blending process over a vertical subblock boundary in OBMC.

[0096] FIG. 16B shows an example of a blending process over an above subblock boundary in OBMC. DETAILED DESCRIPTION

[0097] FIG. 1A shows a system 100 according to some embodiments. The system 100 comprises a first entity 102, a second entity 104, and a network 110. The first entity 102 is configured to transmit to the second entity 104 a video stream (a.k.a., “a video bitstream,” “a bitstream,” “an encoded video, ” or “an encoded video bitstream”) 106.

[0098] The first entity 102 may be any computing device (e.g., a network node such as a server) capable of transmitting the encoded video 106 to the second entity 104 via the network 110. In some embodiments, the first entity 102 may include the encoder 112 for encoding a video, thereby generating the encoded video 106. However, in other embodiments, the encoder 112 may be provided outside the first entity 102, and the encoder 112 may be configured to encode a video, and transmit the encoded video 106 to the first entity 102. In a summary, in some embodiments, the first entity 102 may be configured to perform a video encoding, while, in other embodiments, another entity performs the video encoding.

[0099] The second entity 104 may be any computing device (e.g., a network node, a user equipment such as a mobile phone, a laptop, a tablet, a vehicle, etc.) capable of receiving the encoded video 106 and decoding the encoded video using a decoder 114. Each of the first entity 102 and the second entity 104 may be a single physical entity or a combination of multiple physical entities. The multiple physical entities may be located at the same location or may be distributed in a cloud.

[0100] In one example, as shown in FIG. 1B, the first entity 102 is a video streaming server 132 and the second entity 104 is a video streaming client -- e.g., a user equipment (UE) 134. The UE 134 may be any of a desktop, a laptop, a tablet, a mobile phone, or any other computing device. The video streaming server 132 may be configured to encode an original video and transmit an encoded video 136 (e.g., video streaming) to the UE 134. Upon receiving the encoded video 136, the UE 134 may decode the received encoded video 136, thereby reconstructing a video and displaying the reconstructed video for the video streaming.

[0101] In another example, as shown in FIG.1C, the first entity 102 and the second entity104 are first and second UEs 152 and 154. The first UE 152 may be an offeror of a video conferencing session or a caller of a video chat, and the second UE 154 may be an answerer of the video conference session or the answerer of the video chat. In the example shown in FIG. 1C, the first UE 152 may be configured to capture a video of the offeror or the caller, encode the captured video, and transmit an encoded video 156 for a video conference or a video chat to the second UE 154. Upon receiving the encoded video 156, the second UE 154 may decode the received encoded video 156, thereby reconstructing a video and displaying the reconstructed video for the video conference or the video chat.

[0102] In encoding a video and decoding the encoded video, the MMVD and / or the OBMC may be used. But, as mentioned above, the prediction accuracies of the MMVD and the OBMC may not be good. More specifically, let’s assume that the encoder 112 encodes an original video containing an original picture using the MMVD (or the OBMC). Due to the video compression performed during the encoding, this encoded video does not include the original sample values of the picture.

[0103] Thus, once the decoder 114 receives the encoded video, the decoder 114 tries to predict the original sample values of the picture (i.e., the current picture) using sample values of reference picture(s) (e.g., FIG. 8, reference pictures L0 and L1). In other words, the decoder 114 generates predicted sample values of the current picture based on the sample values of the reference picture(s). But sometimes the differences between the original sample values and the predicted sample values may be too large. In such scenario, the original picture may not be reconstructed accurately at the decoder 114. Such inaccurate reconstruction of the original picture may lead to, for instance, a poor video streaming / chat quality in the examples shown in FIGS. 1B and 1C. Therefore, there is a need to improve the prediction accuracies (i.e., the accuracies of predicting the original sample values of the current picture) of the MMVD and the OBMC.

[0104] Accordingly, in some embodiments of this disclosure, picture encoding and decoding are performed using one of the MMVD and the OBMC in combination with a bi- direction optical flow (BDOF) or decoder-side motion refinement (DMVR). The BDOF and the DMVR are explained below.

[0105] BDOF

[0106] The BDOF is a tool included in VVC and the current ECM. It can be used to refine prediction samples that are generated from a current Bi-MV of a current block. The BDOF relies on optical flow estimation to derive a pair of refinement parameter (Vx, Vy) asfollows.

[0107] First, horizontal and vertical gradients --డூ^ೖ^^ೖ^డ௫^ ^^, ^^^ andడூడ௬^ ^^, ^^^, ^^ ൌ 0,1 -- of the two prediction blocks are computed by between twoneighboring samples. The two prediction blocks are generated using the MV0 and MV1 of the bi-MV respectively. ^^ ^^^^^^ ^ ^ ^^^ ^^^ ^^, ^^^ ൌ ൫^ ^^ ^ ^ ^^ ^ 1, ^^^ ≫ ^^ℎ ^^ ^^ ^^1^ െ ^ ^^ ^ ^ ^^ െ 1, ^^^ ≫ ^^ℎ ^^ ^^ ^^1^൯wherein list ^^, ^^ ൌ0,1. The ^^ℎ ^^ ^^ ^^1 is a constant, and it is calculated based on the luma bit depth, bitDepth. Forexample, ^^ℎ ^^ ^^ ^^1 ൌ max^6, ^^ ^^ ^^ ^^ ^^ ^^ ^^ℎ െ 6^.

[0108] Then, the auto- and cross-correlation of the gradients -- ^^^, ^^ଶ, ^^ଷ, ^^ହand ^^^-- are calculated as follows: ^^^ ൌ ^ ^^ ^^ ^^^ ^^௫^^^, ^^^^where^^^ ^^^ ^^^^^ ^ ^ ^^^^^^^^^ ^^^ ^^^^^^^ ≫ ^^^where Ω is a 6×6of ^^^and ^^^are set to bemin^1, ^^ ^^ ^^ ^^ ^^ ^^ ^^ℎ െ 11^ and min^4, ^^ ^^ ^^ ^^ ^^ ^^ ^^ℎ െ 8^, respectively.

[0109] The refinement parameter ൫ ^^௫, ^^௬൯ is then derived using the cross- and auto- correlation terms using the following: ^^௫ൌ ^^^^ 0? ^^ ^^ ^^ ^^3 ^െ ^^ℎ^ᇱூை, ^^ℎ^ᇱூை,െ൫^ ^^ଷ∙ 2^್ି^ೌି^^ ≫ ⌊logଶ^^^⌋൯^ : 0prediction samples as follows: ^^ ^^ ^^ ^^^^ைி^ ^^, ^^^ ൌ ൫ ^^^^^^ ^^, ^^^ ^ ^^^^^^ ^^, ^^^ ^ ^^^ ^^, ^^^ ^ ^^^^^^^௧൯ ≫ ^^ℎ ^^ ^^ ^^refinement and the gradients: ^^ ^^ ^^^ ^^^ ^^^ ^^^ ^^^^ ^ ^^^^^^ ^^ ^^^^^^^ ^ ^^^ ^^^^^^ ^^ ^^^^^^^

[0112] DMVR is a tool included in VVC and the current ECM to refine motion vectors for a Bi-MV. DMVR operates on a certain grid, usually having the size of 16x16. As compared to the BDOF which relies on optical flow estimation, DMVR relies on bilateral matching of two reference blocks to refine the Bi-MV. The DMVR searches within a window around the Bi-MV to find whether there exists another Bi-MV (Bi-MV’) that gives a better match between the L0 reference block and the L1 reference block. If so, the Bi-MV’ is used instead of the Bi-MV for generating the prediction samples of the current block.

[0113] Referring back to FIG. 1A, once the decoder 114 receives the video bitstream 106, the decoder 114 may decode the video bitstream 106. More specifically, the video bitstream 106 may include a plurality of pictures each having a plurality of blocks, and the decoder 114 may be configured to decode each block (i.e., generating sample values of each block) in each picture of the video bitstream. In this disclosure, the block and the picture thatare currently subject to the decoding may be referred to as a current block and a current picture.

[0114] FIG.2 shows a process 200 for decoding a current block or a subblock within a current picture which is included in a coded video bitstream (e.g., the video bitstream 106). Since the encoding process is similar to the decoding process, the process 200 may also be used for encoding a video, thereby generating the coded video bitstream For simple explanation purpose, the embodiments below are explained using a decoding process. Also, even though the embodiments below are explained using a block, the principle of the embodiments below is equally applicable to a subblock. The process 200 may be performed by the decoder 114.

[0115] As explained above, decoding a current block means generating sample values of the current block. In generating the sample values, the decoder 114 does not receive the original sample values of the current block. Rather the decoder 114 predicts the original sample values of the current block using sample values in block(s) within reference picture(s). Thus, in this disclosure, the expressions -- decoding a block, predicting a block, predicting sample values of a block, and generating sample values of a block -- are used interchangeably.

[0116] The process 200 may begin with step s202. The step s202 comprises determining whether the current block is an inter coded block (i.e., whether sample values of the current block are to be derived from sample values of reference block(s) in reference picture(s)). Examples of the reference picture(s) are shown in FIG. 9 and 10.

[0117] In FIG. 10, in case the current block is a uni-inter predicted block, then the sample values of the current block may be determined based on the sample values of a reference block included in the reference picture 0. On the other hand, in case the current block is a bi-inter predicted block, the sample values of the current block may be determined based on the sample values of reference blocks included in the reference pictures 0 and 1.

[0118] In one example, the decoder 114 may check whether the current block is an inter coded block by checking whether the video bitstream includes motion information (MI) associated with the current block. Such information (whether a piece of motion information is associated with the current block) can also be indicated by a syntax element (which indicates which mode the current block is coded with) in the video bitstream. Note that, as explained above, MI associated with a block is the information related to an inter-prediction of the block. In response to determining that the current block is an inter coded block, theprocess 200 may proceed to step s204. Otherwise, the process 200 may proceed to performing a non-inter mode such as intra prediction mode, intra block copy mode, etc.

[0119] The step s204 comprises checking whether a motion vector difference (MVD) associated with the current block is present in the coded video bitstream. In some embodiments, the MVD may be signalled to the decoder 114 using the MMVD described above.

[0120] As explained above, the MVD may comprise a distance value and a direction value. Note that in case the current block is a bi-inter predicted block (i.e., the block of which sample values are to be generated based on two MVs -- MV1 and MV2), there may be two MVDs. Each of the first MVD (“MVD1”) and the second MVD (“MVD2”) may comprise a pair of a distance value and a direction value for each of the MV1 and the MV2. Note that since the MVD is used for adjusting MV(s), in this disclosure, the MVD is also referred to as MV adjustment value.

[0121] In response to determining that the MVD is present in the coded video bitstream, the process 200 may proceed to step s206. Otherwise, the process 200 may proceed to, for example, the step s210.

[0122] The step s206 comprises generating updated MVs based on (i) the MVDs and (ii) the MVs included in motion information (MI) associated with the current block.

[0123] As explained above, the video bitstream 106 that the decoder 114 receives may include MI associated with the current block. The MI associated with the current block may include a list of reference pictures, one or more indexes indicating certain one or more reference pictures in the list, and MV(s) associated with the certain one or more reference pictures.

[0124] In one example, in case the current block is a bi-inter coded block, the MI of the current block may include (i) two index values (I1 and I2) indicating two reference pictures (L1 and L2) and (ii) two MVs (MV1 and MV2) associated with the two reference pictures. In another example, in case the current block is a uni-inter coded block, the MI of the current block may include (i) a single index value indicating (i) one reference picture (L3) and (ii) one MV (MV3) associated with said one reference picture. For simple explanation purpose, the embodiments below are explained assuming that current block is a bi-inter coded block. But the principle of the embodiments is equally applicable to a uni-inter coded block.

[0125] The step 206 comprises updating the MV1 and the MV2 based on the MVD1 and the MVD2, thereby generating the updated MV1 (MV1’) and the updated MV2 (MV2’).There are different ways of generating the MV1’ and the MV2’ based on the MV1, the MV2, the MVD1, and the MVD2. In one example, each of the MV1’ and the MV2’ may be generated based on a sum of the MV1 and the MVD1 and a sum of the MV2 and the MVD2 respectively (e.g., MV1’ = MV1 + MVD1 and MV2’ = MV2 + MVD2).

[0126] FIGS. 3A-3C illustrate the concept of the MVs and the MVDs. As explained above and shown in FIG. 3A, sample values of the current block 302 in the current picture 352 can be obtained based on sample values of reference blocks 304 and 306 in reference pictures 354 and 356. Each MV allows the decoder 114 to obtain the location of each reference block.

[0127] FIG. 3B shows the reference picture 354. In the reference picture 354, there is a block 302’ at a location corresponding to the location of the current block 302 in the current picture 352. An MV 324 indicates a location of the reference block 304 with respect to the block 302’. In case there is no MVD, the sample values of the current block are generated based on the sample values of the reference block 304. But, in case there is an MVD 334, an updated MV’ 344 is generated based on the MV 324 and the MVD 334. The MV’ 344 indicates a location of a reference block 314 with respect to the block 302’.

[0128] FIG.3C shows the reference picture 356. In the reference picture 356, there is a block 302’’ at a location corresponding to the location of the current block 302 in the current picture 352. An MV 326 indicates a location of the reference block 306 with respect to the block 302’’. In case there is no MVD, the sample values of the current block are generated based on the sample values of the reference block 306. But, in case there is an MVD 336, an updated MV’ 346 is generated based on the MV 326 and the MVD 336. The MV’ 346 indicates a location of a reference block 316 with respect to the block 302’.

[0129] Referring back to FIG. 2, after generating the updated MVs, the process 200 may proceed to step s208. The step s208 comprises determining first prediction blocks each containing on a set of prediction samples based on the updated MVs. More specifically, in the example provided in FIGS. 3D and 3E, the sample values of the current block are generated based on the sample values (i.e., values of refBlk) of the reference blocks 314 and 316 of which the locations are determined by the updated MVs 344 and 346. Here, the samples in the reference blocks 314 and 316 correspond to the prediction samples, and the reference blocks 314 and 316 correspond to the first prediction blocks.

[0130] After determining the first prediction block, the process 200 may proceed to step s210. The step s210 comprises determining motion vector adjustment values (MvOffset)based on the samples values of the current block generated in the step s208 (i.e., the predicted sample values generated in the step s208). The MvOffsets may be generated using any one of the BDOF and the DMVR described above.

[0131] In case the BDOF is used for generating the MvOffsets, in the example shown in FIGS. 3A-3C, ^^^ଷ^ସ^^ ^^, ^^^ is the sample value at coordinate ^ ^^, ^^^ of the prediction block 314 and , ^^^ଷ^^^^ ^^, ^^^ is the sample value at coordinate ^ ^^, ^^^ of the prediction block 316. Also, in the BDOF, ൫ ^^௫, ^^௬൯ corresponds to each MvOffset associated with each of the prediction blocks 314 In the example shown in FIGS.3A-3C, horizontal and vertical gradients^యభర^^యభర^ ^యభల^ ^యభల^(డூడ௫^ ^^, ^^^,డூడ௬^ ^^, ^^^,డூడ௫^ ^^, ^^^,డூడ௫^ ^^, ^^^) of the two prediction blocks 314 and 316 each of the two prediction blocks 314 and316 (i.e., MvOffset 364 and MvOffset 366 shown in FIGS. 3D and 3E) may be determined based on the computed horizontal and vertical gradients. Note that since the MvOffsets 364 and 366 are used for further adjusting the MVs (e.g., the MVs 344 and 346), in this disclosure, an MvOffset is also referred to as an MV adjustment value.

[0132] After determining the MvOffsets, the process 200 may proceed to step s212. The step s212 comprises deriving adjusted motion vectors based on the MvOffsets determined in the step s210. More specifically, in the example shown in FIGS. 3A-3E, the MvOffsets 364 and 366 may be used to further modify the MVs 344 and 346, thereby generating the modified MVs 384 and 386. The adjusted motion vectors obtained in the step s212 may be assigned and stored for at least one subblock of the current block.

[0133] After deriving the adjusted motion vectors in the step s212, the process 200 may proceed to step s214. The step s214 comprises determining second prediction blocks based on the adjusted motion vectors and determining final prediction samples of the current block based on the second prediction blocks. More specifically, in the example provided in FIGS. 3D-3E, the sample values of the current block may be generated based on the sample values (i.e., values of refBlk) of the reference blocks 374 and 376 of which the locations are determined by the updated MVs 384 and 386. Here, the samples in the reference blocks 374 and 376 correspond to the prediction samples, and the reference blocks 374 and 376 correspond to the second prediction blocks.

[0134] In some embodiments, the steps s210 through s214 may be performed only when one or more of the following three conditions is satisfied.

[0135] The first condition is related to the size of the current block. More specifically,the first condition may indicate that the size of the current block is greater than a first size threshold value. In one example, the steps s210 through s214 are performed when W*H is greater than or equal to 64, where W is the width of the current block and H is the height of the current block.

[0136] The second and third conditions are related to the MVDs checked in the step s204.

[0137] The second condition may indicate that a magnitude of each or any of the MVDs is greater than a first magnitude threshold value (e.g., 4 in 1 / 16 sample precision).

[0138] The third condition may indicate that the precision of the MVD is greater than a first precision threshold value (e.g., a half-pel precision threshold).

[0139] As discussed above, the principle of the above embodiments is applicable not only to a block but also to a subblock. More specifically, for example, in case the current block is split into a number of subblocks having the size MxN, one or more of the steps s204- s214 may be performed for each MxN subblock.

[0140] Whether to perform one or more of the steps s204-s214 (e.g., all of the steps s204-s214) on a subblock basis can be determined based on whether one or more of the above three conditions is satisfied. For example, in some embodiments, if any one of the above three conditions is satisfied, the steps s204-s214 may be performed on a subblock basis (i.e., determining MVD(s) associated with a current subblock, generating updated MV(s) based on the MVD(s) …).

[0141] Examples of the conditions are that 1) the size of current block is greater than a second size threshold, e.g.,1024, 2) the magnitude of any of the MVDs is greater than a second magnitude threshold, e.g., 32 in 1 / 16 sample precision, 3) the precision of any of the MVDs is greater than a second precision threshold, e.g., a full-pel precision threshold. Here, the second size threshold may be greater than the first size threshold, the second magnitude threshold may greater than the first magnitude threshold, and the second precision threshold may be greater than the first precision threshold.

[0142] In a summary, the above embodiments related to the process 200 may be summarized as follows.

[0143] For blocks that are coded with the MMVD, the BDOF may be used to refine MVs associated with the blocks and prediction samples of the blocks. More specifically, first, a current block may be split into a group of subblocks. For each subblock, the BDOF may be applied to a get an MV adjustment for the subblock. Then the MV adjustment value isadded on top of the explicitly signaled MVD of the MMVD to get an adjusted MV for the subblock. The adjusted MV is not only used for deriving the final prediction block of the current block but is also stored within the current block such that it can be used for prediction of next blocks. Here, the motivation is that even though the MVD is explicitly signaled from the encoder 112 to the decoder 114, the MVDs that can be signaled are within a limited set (which gives limited adjustment possibilities) which may still not be well suited for the current block. Using the BDOF allows an MV adjustment at a finer granularity / precision which may result in better final MVs.

[0144] FIG. 4 shows a process 400 for decoding a current block or a sub-block within a current picture included in a coded video bitstream (e.g., the video bitstream 106), according to some embodiments. For simple explanation purpose, the process 400 is explained below with respect to a subblock. However, the principle of the process 400 is equally applicable to a block. Also, since the encoding process is similar to the decoding process, the process 400 may also be used for encoding a video, thereby generating the coded video bitstream.

[0145] The process 400 may be performed by the decoder 114 and may further improve the process of the OBMC. The process 400 is explained below using FIG. 5A. The process 400 begins with step s402.

[0146] The step s402 comprises determining a neighboring subblock 504 (shown in FIG. 5A) adjacent to a boundary 506 of a current subblock 502. After determining the neighboring subblock 504, the process 400 may proceed to step s404.

[0147] The step s404 comprises determining MI of the neighboring subblock 504 (MI_neighbor). As explained above, the MI_neighbor may include a list of reference pictures, one or more indices indicating one or more reference pictures included in the list, and MVs for the neighboring subblock 504 (MV_neighbor). After determining the MI_neighbor, the process 400 may proceed to step s406.

[0148] The step s406 comprises generating sample values with respect to the sample positions of the current block using the MI_neighbor associated with the neighboring subblock 504 (hereinafter, “the sample values associated with MI_neighbor”).

[0149] Conventionally, as shown in FIG. 5C, in case the neighboring subblock 554 is a uni-inter coded subblock, the MI_neighbor includes a single MV – i.e., MV_neighbor 538 and identifies a single reference picture 572. As illustrated in FIG. 5C, an area 532 within the reference picture 572 is determined by positions of samples included in the neighboringsubblock 554 and the MV_neighbor 538, and the sample values corresponding to the area 532 in the reference picture 572 are used as the sample values of the neighboring subblock 554. Similarly, in case the current subblock 560 is a uni-inter coded subblock, the MI of the current subblock 560 (MI_current) may include a single MV 536 (MV_current) and indicates a single reference picture 574 (which may be same as or different from the reference picture 572). As illustrated in FIG. 5C, an area 534 within the reference picture 574 is determined by positions of samples included in the current subblock 560 and the MV_current 536, and the sample values corresponding to the area 534 in the reference picture 574 are used as the sample values of the current subblock 560.

[0150] In the step s406, an area 576 (shown in FIG. 5D) within the reference picture 572 is determined by sample positions of the current subblock 560 and the MV_neighbor 538. The sample values corresponding to the area 576 in the reference picture 572 are the sample values generated in the step s406

[0151] The step s408 comprises determining, from among values of samples that are located adjacent to and parallel to the boundary 506, which sample values of the current subblock 502 to modify using the sample values associated with MI_neighbor.

[0152] For example, if N indicates a number of columns of samples in the current subblock 502, of which values are to be modified based on the sample values associated with MI_neighbor, the step s408 may set the value of N. In some embodiments, the value of N may be set based on one or more of a characteristic of the boundary 506, a characteristic of the MI_neighbor, and / or a characteristic of the MI of the current subblock 502 (MI_current). Note that, in FIG.5A, N indicates a number of columns of samples because the neighboring subblock 503 is located on the right side of the current subblock 502. But, in case, the neighboring subblock is located above or below the current subblock, N may indicate a number of rows of samples.

[0153] In some embodiments, two candidate values may be provided for N, and one of the two candidate values may be selected as the value of N based on whether one or more conditions related to the characteristic of the boundary 506, the characteristic of the MI_neighbor, and / or the characteristic of the MI_current is satisfied.

[0154] In case one or more conditions related to the characteristic of the boundary 506, the characteristic of the MI_neighbor, and / or the characteristic of the MI_current is satisfied, N may be set to 4. In this case, the values of samples 514 (having the selected number of columns, i.e., 4) in the current subblock 502 may be modified based on the values of samples510 in the neighboring subblock 504. On the other hand, in case one or more conditions related to the characteristic of the boundary 506, the characteristic of the MI_neighbor, and / or the characteristic of the MI_current is not satisfied, N may be set to 3. In this case, the values of samples 512 (having the selected number of columns, i.e., 3) in the current subblock 502 may be modified based on the values of samples 508 in the neighboring subblock 504.

[0155] In one example, the condition is that the boundary 506 is determined to be aligned with a grid of DMVR. If the condition is satisfied, N may be set to 4. Otherwise, N may be set to 3.

[0156] In a different example, the condition may indicate that a motion vector difference -- the difference between an MV of the current subblock 502 (i.e., an MV included in the MI_current) and an MV of the neighboring subblock 504 (i.e., an MV included in the MI_neighbor) is greater than a certain value. If such condition is satisfied, then N may be set to 4. Otherwise, N may be set to 3.

[0157] In case each of the current subblock 502 and the neighboring subblock 504 is a bi-inter coded subblock, then there are two MVs provided for each block. In such case, the condition may indicate that a combination of a first difference between one of the MVs of the current subblock and one of the MVs of the neighboring subblock and a second difference between another of the MVs of the current subblock and another one of the MVs of the neighboring subblock is greater than a certain value. Alternatively, the condition may indicate that each or any of the first and second differences is greater than a certain value.

[0158] In the above embodiments, one of the two candidate values is selected as the value of N based on whether one or more of the above conditions is satisfied. But, in some embodiments, the value of N may be set differently. For example, in case the boundary 506 is determined to be aligned with the grid of the DMVR, the value of N (i.e., the number of columns or rows of samples of which values are to be modified based on the sample values associated with MI_neighbor (i.e., sample values that are generated with respect to the sample positions of the current block using the MI_neighbor associated with the neighboring subblock)) may be set to be same as the number of columns or rows of samples of which values are to be modified when the boundary 506 is determined to be aligned with a block boundary (e.g., a CU boundary). Furthermore, when the boundary 506 is determined to be aligned with the grid of DMVR, the weights associated with the columns or rows for the modification may be set to be same as the weights used in cases when the boundary 506 is aligned with a block boundary (e.g., a CU boundary).

[0159] As explained above, when the OBMC is applied, some sample values of the current subblock are generated based on some the sample values associated with MI_neighbor (i.e., sample values that are generated with respect to the sample positions of the current block using the MI_neighbor associated with the neighboring subblock) . For example, as shown in FIG. 5A, the values of the samples 512 of the current subblock 502 may be generated based on the values of the samples 512 generated based on the MI_current and the values of the samples 508 generated based on the MI_neighbor with respect to the sample positions of the current block. More specifically, the value of each of the samples 512 may be determined based on W_current * the sample value generated using the MI_current + W_neighbor * the sample value generated using the MI_neighbor.

[0160] In some embodiments, these weight values (W_current and W_neighbor) may be set such that the same weight is given to the sample values generated according to the MI_current as the sum of weights given to the sample values generated according to MIs of all surrounding neighbor subblocks. More specifically, in FIG.5B, the current subblock 560 is surrounded by a left subblock 554, a top subblock 552, a right subblock 556, and a bottom subblock 558.

[0161] In this example, let’s assume that the value of the middle sample among the three samples at a sample region 588 is generated based on W1_current * V1_sample_current + W_top_neighbor * V1_sample_top_neighbor. Here, W1_current is the weight value assigned to sample values associated with the sample region 588 of the current subblock 560, and V1_sample_current is the sample value which (1) is associated with the middle position in the sample region 588 and (2) is generated based on the MV(s) of the current subblock 560 and the reference picture(s) indicated by the MI of the current subblock 560. W_top_neighbor is the weight value assigned to sample values associated with a sample region 590 (which is at the same location as the location of the sample region 588 within in a picture) of the top subblock 552 and V1_sample_top_neighbor is the sample value which (1) is associated with the middle position in the sample region 588 and (2) is generated based on the MV(s) of the top subblock 552 and the reference picture(s) indicated by the MI of the top subblock 552.

[0162] Let’s further assume that the value of the middle sample among the three samples at a sample region 586 is generated based on W2_current * V2_sample_current + W_left_neighbor * V1_sample_left_neighbor. Here, W2_current is the weight value assigned to sample values associated with the sample region 586 of the current subblock 560,and V2_sample_current is the sample value which (1) is associated with the middle position in the sample region 586 and (2) is generated based on the MV(s) of the current subblock 560 and the reference picture(s) indicated by the MI of the current subblock 560. W_left_neighbor is the weight value assigned to sample values associated with a sample region 592 (which is at the same location as the location of the sample region 586 within in a picture) of the left subblock 554 and V1_sample_left_neighbor is the sample value which (1) is associated with the middle position in the sample region 586 and (2) is generated based on the MV(s) of the left subblock 554 and the reference picture(s) indicated by the MI of the left subblock 554.

[0163] Let’s further assume that the value of the middle sample among the three samples at a sample region 584 is generated based on W3_current * V3_sample_current + W_left_neighbor * V1_sample_left_neighbor. Here, W3_current is the weight value assigned to sample values associated with the sample region 584 of the current subblock 560, and V2_sample_current is the sample value which (1) is associated with the middle position in the sample region 584 and (2) is generated based on the MV(s) of the current subblock 560 and the reference picture(s) indicated by the MI of the current subblock 560. W_bottom_neighbor is the weight value assigned to sample values associated with a sample region 594 (which is at the same location as the location of the sample region 584 within in a picture) of the bottom subblock 558 and V1_sample_bottom_neighbor is the sample value which (1) is associated with the middle position in the sample region 582 and (2) is generated based on the MV(s) of the bottom subblock 558 and the reference picture(s) indicated by the MI of the bottom subblock 558.

[0164] Let’s further assume that the value of the middle sample among the three samples at a sample region 582 is generated based on W4_current * V4_sample_current + W_right_neighbor * V1_sample_right_neighbor. Here, W4_current is the weight value assigned to sample values associated with the sample region 582 of the current subblock 560, and V4_sample_current is the sample value which (1) is associated with the middle position in the sample region 582 and (2) is generated based on the MV(s) of the current subblock 560 and the reference picture(s) indicated by the MI of the current subblock 560. W_right_neighbor is the weight value assigned to sample values associated with a sample region 596 (which is at the same location as the location of the sample region 582 within in a picture) of the right subblock 556 and V1_sample_right_neighbor is the sample value which (1) is associated with the middle position in the sample region 582 and (2) is generated basedon the MV(s) of the right subblock 556 and the reference picture(s) indicated by the MI of the right subblock 556.

[0165] In the above example, according to some embodiments, the weights for the neighboring subblocks may be set such that W_top_neighbor + W_left_neighbor + W_bottom_neighbor + W_right_neighbor is equal to 0.5 (meaning that the sum of weights given to the neighboring subblocks surrounding the current subblock is same as the weight given to the current subblock).

[0166] In one example, when it is determined that N = 4, the weights associated with the columns or the rows are set to be {32 / 128, 20 / 128, 12 / 128, 4 / 128}, i.e., {8 / 32, 5 / 32, 3 / 32, 1 / 32}. Note that a higher weight is given to the column or the row that is closer to the boundary. In another example, the weights associated with the column or the rows are set to be {28 / 128, 20 / 128, 12 / 128, 4 / 128}, i.e., {7 / 32, 5 / 32, 3 / 32, 1 / 32}.

[0167] As explained above with respect to the step s406, the sample values associated with MI_neighbor may be generated using the MV(s) included in the MI_neighbor. However, in some embodiments, the sample values associated with MI_neighbor may further be adjusted using the BDOF. The rationale here is as follows: The MI_neighbor is used for coding of the neighboring block or subblock and it is tweaked to reference an area which better suits the content or position of the neighboring block or subblock. When directly using the MI_neighbor to generate sample values with respect to the current sample position, the referenced area may not fit well for the current block and thus may introduce some loss of prediction accuracy. Accordingly, further refinement of the sample values may be needed, and the BDOF provides this additional refinement.

[0168] Note that even though the BDOF is applied to generate sample values of a subblock or a block in the process 200 and the process 400, the way the BDOF is applied in the process 400 may be different from the way the BDOF is applied in the process 200. More specifically, in the process 200, the MvOffset(s) determined in the step s210 are used to derive adjusted MV(s) in the step s212, and the adjusted MV(s) are used to generate sample values of a block or a subblock in the step s214.

[0169] But, in the process 400, the MvOffset(s) (൫^^௫, ^^௬൯) are not used to derive adjusted MV(s). Rather the MvOffset(s) are directly the sample values( ^^^ ^^, ^^^) using the MVs of the neighboring subblock 504.

[0170] Also, in those embodiments where the BDOF is used for generating the sample values associated with MI_neighbor, the sample values associated with MI_neighbor may begenerated using the BDOF only when one or more conditions is satisfied. The conditions may be related to the characteristic of the boundary 506, the characteristic of the MI_neighbor, and / or the characteristic of the MI_current.

[0171] In one example, the condition is that the boundary 506 is determined to be aligned with a grid of DMVR. If the condition is satisfied, the BDOF is used to generate the sample values associated with MI_neighbor. Otherwise, the sample values associated with MI_neighbor may be generated without using the BDOF, as explained with respect to the step s406.

[0172] In a different example, the condition indicates that a motion vector difference - - the difference between a MV of the current subblock 502 (i.e., a MV included in the MI_current) and a MV of the neighboring subblock 504 (i.e., a MV included in the MI_neighbor) is greater than a certain value. If such condition is satisfied, the BDOF is used to generate the sample values associated with MI_neighbor. Otherwise, the step s406 is used to generate the sample values associated with MI_neighbor.

[0173] In case each of the current subblock 502 and the neighboring subblock 502 is a bi-inter coded subblock, then there are two MVs provided for each subblock. In such case, the condition may indicate that a combination of a difference between one of the MVs of the current subblock and one of the MVs of the neighboring subblock and a difference between another of the MVs of the current subblock and another one of the MVs of the neighboring subblock is greater than a certain value. Alternatively, the condition may indicate that each or any of a difference between one of the MVs of the current subblock and one of the MVs of the neighboring subblock and a difference between another of the MVs of the current subblock and another one of the MVs of the neighboring subblock is greater than a certain value.

[0174] The embodiments related to the process 400 shown in FIG. 4 may be summarized as follows. First, in some of these embodiments, the BDOF may be used to further refine prediction samples obtained using neighboring MI in the OBMC. Second, in some of these embodiments, an adaptive OBMC blending scheme is adapted.

[0175] For example, for a subblock that is adjacent to a subblock boundary, if the boundary is aligned with DMVR operating grid, the blending area (the area that the current prediction samples are adjusted using the prediction samples coming out from using the neighboring motion vector) may be increased to be 4 columns or rows. On the other hand, for subblock boundaries that are not aligned with the DMVR operating grid, the blendingarea is kept at 3 columns or rows. Here, the motivation is that it is found that DMVR boundaries are more like CU boundaries given that the DMVR search may results in relatively large MV differences between two adjacent subblocks. In such cases, using a larger blending area may help smooth out the boundary and improve the prediction result.

[0176] In the process 400 shown in FIG. 4, the step s408 is performed after the step s406 is performed. But, in some embodiments, the step s408 may be performed before the step s406 is performed. In such embodiments, first, it is determined which sample values of the current subblock 502 are to be modified based on sample values associated with the MI_neighbor.

[0177] More specifically, in one example, two candidate values are provided for N, and one of the two candidate values is selected as the value of N based on whether one or more conditions related to the characteristic of the boundary 506, the characteristic of the MI_neighbor, and / or the characteristic of the MI_current is satisfied. Here, N indicates a number of columns or rows of the current subblock 502’s samples of which values are to be modified.

[0178] After selecting N, in the step s406, sample values are generated using the MI_neighbor associated with the neighboring subblock. Note that in case the step s406 is performed before the step s408 is performed (as shown in FIG. 4), in step s406, all sample values corresponding to all sample points of the current subblock 502 are generated using the MI_neighbor. On the contrary, in case the step s406 is performed after the step s408 is performed, only the sample values corresponding to the number of sample points that are determined to be modified in the step s408 are generated.

[0179] For instance, in FIG.5B, in case the step s406 is performed before the step s408 is performed, sample values associated with the nine sample points included in the subblock 560 are generated using the MI_neighbor associated with the subblock 554. On the contrary, in case the step s406 is performed after the step s408 is performed and in case it is determined in the step s408 that only the sample values of the portion 586 are to be modified, then, in the step s406, only the sample values corresponding to the portion 592 are generated using the MI_neighbor.

[0180] The embodiments explained above may be summarized as follows. Embodiment 1 (element 1)For decoding a current block within a current picture inside a coded video bitstream, the decoder may implement the proposed method with at least one of the following steps: 1. Determine whether the current block is an inter coded block. 2. In responsive to the determination that the current block is an inter block, determine whether a motion vector difference value which associates with the current block is present in the coded video bitstream and determine the motion vector difference value MVD0 if it is present. 3. Determine an initial motion vector MV0 based on the motion vector difference value MVD0. 4. Determine a first prediction block PB0 containing a set of prediction samples based on the motion vector MV0, where the determination comprises determining a set refBlk of reference samples from one for more reference picture. 5. Determine a motion vector adjustment value MvOffset based on the reference samples within the reference samples set refBlk. 6. Derive an adjusted motion vector (denoted as MV0_new) based on the adjustment value MvOffset. 7. Assign and store the adjusted motion vector MV0_new for at least one subblock of current block. 8. Determine a second prediction block PB1 based on the adjusted motion vector MV0_new. 9. Determine the final prediction samples of the current block based on PB1. Embodiment 2 As in Embodiment 1, where the steps may comprise: 1. Split the current block into a number of subblocks with size MxN. 2. For each MxN subblock: A. Determine a set refBlkSub containing a set of reference samples from a reference picture based on the initial motion vector MV0. B. Determine a motion vector refinement value MvOffsetSub based on the reference samples within the reference set refBlkSub. C. Derive an adjusted motion vector (denoted as MV0Sub_new) for the current subblock based on MvOffsetSub. D. Assign and store the adjusted motion vector (denoted as MV0Sub_new) for the current subblock. E. Determine a second prediction block PbSub for the subblock based on the motion vector adjustment value MvOffsetSub. F. Determine the final prediction samples of the current subblock based on PbSub. In one example, M = N = 8. The difference between Embodiment 1 and Embodiment 2 is that in Embodiment 1, the derivation of motion vector adjustment value is done on the block level, while in Embodiment 2, the derivation of motion vector adjustment value is done on the subblock level. Embodiment 3As in Embodiment 1 – 2, where the method is applied when the current block is coded in MMVD mode. Embodiment 4 As in Embodiment 1-3, where the method is applied when the size of the current block that is greater than a certain threshold. In one example, the method is applied when the W*H >= 64, where W is the current block width and H is the current block height. Embodiment 5 As in Embodiment 1-4, where the method is applied when the motion vector difference value MVD0 fulfils a criterion. In one alternative, the method is applied when the magnitude of the MVD0 is greater than a threshold, for example, 4 in 1 / 16 sample precision. Embodiment 6 As in Embodiment 1-5, where the method is applied when the precision of motion vector difference MVD0 is greater than a certain threshold. For example, a half-pel precision threshold. Embodiment 7a As in Embodiment 1 – 6, where the determination of the motion vector adjustment value MvOffset is done using BDOF scheme. In a more general example, the determination is based on the reference sample values as well as the gradience of the reference sample values. Embodiment 7b As in Embodiment 1 – 6, where the determination of the motion vector adjustment value MvOffset is done using DMVR scheme. Embodiment 8 (OBMC modifications, elements 2 and 3) The method described in this embodiment modifies prediction samples that are within a subblock sbA and adjacent to a subblock boundary BoundT of sbA using OBMC, the decoder may implement the proposed method by the following steps: For a subblock sbA that is within the current block and is adjacent to a boundary BoundT, denote the motion information associated with the subblock sbA as MiC, the decoder may implement the proposed method by the following steps: 1) Determine a neighboring subblock or block Nb that is also adjacent to the boundary BoundT. 2) Determine a set of motion information MiNb from the neighboring subblock or block. 3) Determine a set of prediction samples PbNb using the motion information MiNb. 4) Determine whether to use the prediction samples PbNb to modify the at least N column or row of samples that are adjacent and parallel to the boundary T of the current subblock sbA based on the characteristics of BoundT or the characteristics of the motion information MiNb and MiC. In one alternative, when the characteristics of BoundT or the characteristics of the motion information satisfy a criterion, it is determined to modify 4 columns or rowsof samples that are parallel to the boundary BoundT. When the characteristics of BoundT or the characteristics of the motion information does not satisfy a criterion, it is determined to modify 3 columns or rows of samples that are parallel to the boundary BoundT. Embodiment 9 As in Embodiment 8, when the BoundT is determined to be aligned with the grid of DMVR, then 4 columns or rows of samples are determined to be modified by using the prediction samples of using the neighboring motion information. When the BoundT is determined to not be aligned with the grid of DMVR, then 3 columns or rows of samples are determined to be modified by using the prediction samples of using the neighboring motion information. Embodiment 10 As in Embodiment 8-9, when the motion vector difference (the difference between the motion vector of the current motion information MiC and the motion vector of the neighboring motion information MiNb) is exceeding a certain threshold, then 4 columns or rows of samples are determined to be modified. Otherwise, 3 columns or rows of samples are determined to be modified. Embodiment 11 As in Embodiment 8-10, when the BoundT is aligned with the grid of DMVR, the number of columns or rows of samples that are determined to be modified are set to be the same as the number of modified columns or rows of samples when the BoundT is aligned with a block (i.e., CU) boundary. Furthermore, when the BoundT is aligned with the grid of DMVR, the weights that associates with the columns or rows (that are needed to be modified) are set to be the same as the weights used in cases when the BoundT is aligned with a block (i.e., CU) boundary. Embodiment 12 As in Embodiment 8 – 11, assuming all boundaries of the current subblock are with OBMC applied, for each sample position inside the current subblock, denoting the weightA as the weight (associated with the prediction samples generated using the above neighboring motion information) for above boundary, weightL as the weight (associated with the prediction samples generated using the left neighboring motion information) for left boundary, weightB (associated with the prediction samples generated using the bottom neighboring motion information) for the bottom boundary and weightR (associated with the prediction samples generated using the right neighboring motion information) for the right boundary. It is preferred that the sum weightA +weightL+wegihtB+weightR is 1 / 2. In other words, same amount of weight (importance) is given to a sample predicted according to the motion of the current subblock as the sum of weights for the same sample predicted according to the neighboring subblock motions. For example, using 8 at most as the weight for the closest column or row to a boundary when the weight denominator is 32 (2^5) and DMVR operating on a grid of 8x8 or 16x16 or above. In one example, when it is determined that 4 columns or rows of samples that are adjacent and parallel to BoundT are to be modified. The weights associated witheach column or rows are {32 / 128, 20 / 128, 12 / 128, 4 / 128}, i.e., {8 / 32, 5 / 32, 3 / 32, 1 / 32}. Higher weights are associated with columns or rows (of the prediction samples generated using the neighboring motion information) that are closer to the boundary. In another example, the weights associated with each column or rows are {28 / 128, 20 / 128, 12 / 128, 4 / 128}, i.e., {7 / 32, 5 / 32, 3 / 32, 1 / 32}. Embodiment 13 As in Embodiment 8 – 12, wherein the process of determining a set PbNb of prediction samples using the neighboring motion information MiNb may further consists of following steps: 1. Determine a prediction block PB2 containing a set of prediction samples based on the motion information MiNb, where the determination comprises determining a set RefBLK of reference samples from a reference picture. 2. Determine a motion vector adjustment value MvOffset based on the reference set RefBLK. 3. Determine an adjusted prediction block PB2_new based on the motion vector adjustment value MvOffset and determine the set PbNb to be PB2_new. Embodiment 14 In one alternative, the method described in Embodiment 13 can be implemented without the step 4 of Embodiment 8. The decoder may implement the proposed method by the following steps: For a subblock sbA that is within the current block and is adjacent to a boundary BoundT, denote the motion information associated with the subblock sbA as MiC, the decoder may implement the proposed method by the following steps: 1) Determine a neighboring subblock or block Nb that is also adjacent to the boundary BoundT. 2) Determine a set of motion information MiNb from the neighboring subblock or block. 3) Determine a set of prediction samples PbNb using the motion information MiNb, where the determination consists of: a. Determine a prediction block PB2 containing a set of prediction samples based on the motion information MiNb, where the determination comprises determining a set RefBLK of reference samples from a reference picture. b. Determine a motion vector adjustment value MvOffset based on the reference set RefBLK. c. Determine an adjusted prediction block PB2_new based on the motion vector adjustment value MvOffset and determine the set PbNb to be PB2_new. Embodiment 15 As in Embodiment 13 - 14, the method is invoked based on the characteristics of BoundT or the characteristics of the motion information MiNb or MiC. In one example, the method is applied when the BoundT is determined to be aligned with the grid of DMVR.In another example, the method is applied when the motion vector difference (the difference between the motion vector of the current motion information MiC and the motion vector of the neighboring motion information MiNb) is exceeding a certain threshold. Embodiment 16 As in Embodiment 13 – 15, where the determination of the motion vector adjustment value MvOffset is done using BDOF scheme. In a more general example, the determination is based on the reference sample values as well as the gradience of the reference sample values.

[0181] FIG. 6 shows a process 600 for use in generating sample values of a block or a sub-block in a current picture, according to some embodiments. The process 600 may begin with step s602. The step s602 comprises (i) obtaining motion vector, MV, information indicating a first set of one or more MVs associated with the block or the sub-block. Step s604 comprises (ii) obtaining a first set of one or more MV adjustment values for adjusting one or more MVs. Step s606 comprises (iii) generating a second set of one or more MVs based on the first set of one or more MVs and the first set of one or more MV adjustment values. Step s608 comprises (iv) obtaining a second set of one or more MV adjustment values for adjusting one or more MVs. Step s610 comprises (v) generating a third set of one or more MVs based on the second set of one or more MVs and the second set of one or more MV adjustment values. The sample values of the block or the sub-block in the current picture are generated based on the third set of one or more MVs.

[0182] In some embodiments, each of the first, second, and third sets of MVs includes an MV associated with a first reference picture and an MV associated with a second reference picture, and the sample values of the block or the sub-block in the current picture are generated based on performing a bi-inter prediction using the first reference picture and the second reference picture.

[0183] In some embodiments, the process 600 comprises obtaining reference sample values in one or more reference pictures; and generating first prediction sample values of the block or the sub-block based on at least some of the reference sample values, and the second set of one or more MVs, wherein obtaining the second set of one or more MV adjustment values comprises calculating the second set of one or more MV adjustment values based on the first prediction sample values of the block or the sub-block.

[0184] In some embodiments, the second set of MV adjustment values is calculated based on horizontal and vertical gradients of the first prediction sample values of the block or the sub-block.

[0185] In some embodiments, the second set of MV adjustment values is obtained using a bi-directional optical flow, BDOF, scheme.

[0186] In some embodiments, the first prediction sample values of the block or the sub- block are generated using merge with motion vector difference, MMVD, scheme.

[0187] In some embodiments, the process 600 comprises determining the block in the current picture is an inter coded block, wherein the steps (iv)-(v) are performed based on determining that the block in the current picture is an inter coded block.

[0188] In some embodiments, the process 600 comprises determining whether the size of the block or the size of the sub-block in the current picture satisfies a condition, wherein the steps (iv)-(v) are performed based on determining that the size of the block or the size of the sub-block in the current picture satisfies the condition.

[0189] In some embodiments, the condition indicates that the size of the block or the size of the sub-block in the current picture is greater than a threshold size.

[0190] In some embodiments, the process 600 comprises determining whether one or more of the first set of one or more MV adjustment values satisfy one or more conditions, wherein the steps (iv)-(v) are performed based on determining that said one or more of the first set of one or more MV adjustment values satisfy said one or more conditions.

[0191] In some embodiments, the first set of one or more MV adjustment values comprises a first MV adjustment value, and said one or more conditions indicates that a magnitude of the first MV adjustment value is greater than a threshold value and / or a precision of the first MV adjustment value is greater than a threshold value.

[0192] FIG.7 shows a process 700 for generating sample values of a first block or a first sub-block in a current picture, according to some embodiments. The process 700 may begin with step s702. The step s702 comprises obtaining first motion vector, MV, information indicating a first MV associated with the first block or the first sub-block. Step s704 comprises, based on the first MV, generating first prediction sample values of the first block or the first sub-block. Step s706 comprises obtaining second MV information indicating a second MV associated with a second block or a second sub-block in the current picture, wherein the second block is adjacent to the first block or the second sub-block is adjacent to the first sub-block. Step s708 comprises, based on the second MV, generating second prediction sample values of the second block or the second sub-block. Step s710 comprises determining one or more regions of the first block based on one or more of the first MV, the second MV, or a boundary between the first block and the second block or between the first sub-block and the second sub-block. Step s712 comprises generating sample values in said one or more regions (e.g., rows or columns) of the first block or the first sub-block based on some of the first prediction sample values and some of the second prediction sample values.

[0193] In some embodiments, determining said one or more regions of the first block or the first sub-block comprises determining a number of columns of samples or a number of rows of samples included in said one or more regions, the columns of samples or the rows of samples are parallel to the boundary between the first block and the second block or between the first sub-block and the second sub-block.

[0194] In some embodiments, the process 700 comprises determining any one or a combination of (i) if the boundary between the first block and the second block or between the first sub-block and the second sub-block satisfies a first condition, (ii) if the first MV satisfies a second condition, (iii) if the second MV satisfies a third condition, or (iv) whether a difference between the first MV and the second MV satisfies a fourth condition, wherein the number of columns of samples and / or the number of rows of samples included in said one or more regions are determined based on the above determination.

[0195] In some embodiments, the process 700 comprises determining whether the boundary between the first block and the second block or between the first sub-block and the second sub-block satisfies the first condition, wherein the first condition is that the boundary between the first block and the second block or between the first sub-block and the second sub- block aligns with a boundary of a grid of decoder-side motion refinement, DMVR, scheme, or aligns with a boundary of a block, and the number of columns of samples and / or the number of rows of samples included in said one or more regions are set to be a first value in case the first condition is satisfied, the number of columns of samples and / or the number of rows of samples included in said one or more regions are set to be a second value in case the first condition is not satisfied, and the first value is greater than the second value.

[0196] In some embodiments, the process 700 comprises determining whether the difference between the first MV and the second MV satisfies the fourth condition, wherein the fourth condition indicates that the difference between the first MV and the second MV is greater than a certain value, and the number of columns of samples and / or the number of rows of samples included in said one or more regions are set to be a first value in case the fourth condition is satisfied, the number of columns of samples and / or the number of rows of samples included in said one or more regions are set to be a second value in case the fourth condition is not satisfied, and the first value is greater than the second value.

[0197] In some embodiments, each of said one or more regions of the first block or the first sub-block is associated with a first weight value, the process 700 comprises applying the first weight values to said at least some of the first prediction sample values, thereby obtaining weighted first prediction sample values, and the sample values in said one or more regions of the first block or the first sub-block are generated based on said at least some of the second prediction sample values and the weighted first prediction sample values.

[0198] In some embodiments, the first weight values applied to said at least some of the first prediction sample values are different from each other.

[0199] In some embodiments, each of said one or more regions of the first block or the first sub-block is associated with a second weight value, the process 700 comprises applying the second weight values to said at least some of the second prediction sample values, thereby obtaining weighted second prediction sample values, and the sample values in said one or more regions of the first block or the first sub-block are modified based on the weighted first prediction sample values and the weighted second prediction sample values.

[0200] In some embodiments, the second weight values applied to said at least some of the second prediction sample values are different from each other.

[0201] In some embodiments, generating the second prediction sample values of the second block or the second sub-block comprises: obtaining reference sample values in a reference picture; generating third prediction sample values of the second block or the second sub-block based on at least some of the reference sample values and the second MV, and obtaining a set of MV adjustment values based on the third prediction sample values of the second block or the second sub-block, and determining the second prediction sample values based on the set of MV adjustment values and the third prediction sample values.

[0202] In some embodiments, the set of MV adjustment values is calculated based on horizontal and vertical gradients of the third prediction sample values of the second block or the second sub-block.

[0203] In some embodiments, the process 700 comprises determining any one or a combination of (i) whether the boundary of the first block or the first sub-block satisfies a first condition; (ii) whether the first MV satisfies a second condition; or (iii) whether the second MV satisfies a third condition or (iv) whether a difference between the first MV and the second MV satisfies a fourth condition, wherein the set of MV adjustment values is obtained based on the above determination.

[0204] In some embodiments, the first condition indicates that the boundary between thefirst block and the second block or between the first sub-block and the second sub-block aligns with a boundary of a grid of DMVR scheme.

[0205] In some embodiments, the fourth condition indicates that the difference between the first MV and the second MV is greater than a certain value.

[0206] In some embodiments, the set of MV adjustment values is obtained using bi- directional optical flow, BDOF, scheme.

[0207] FIG.8 is a block diagram of the encoder 112, the decoder 114, or any other entity capable of encoding and / or decoding a video using the process 200 shown in FIG, according to some embodiments. As shown in FIG. 8, network node 800 may comprise: processing circuitry (PC) 802, which may include one or more processors (P) 855 (e.g., one or more general purpose microprocessors and / or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (e.g., network node 800 may be a distributed computing apparatus comprising two or more computers or a monolithic computing apparatus consisting of a single computer); at least one network interface 848 (e.g., a physical interface or air interface) comprising a transmitter (Tx) 845 and a receiver (Rx) 847 for enabling network node 800 to transmit data to and receive data from other nodes connected to network 110 (e.g., an Internet Protocol (IP) network) to which network interface 848 is connected (physically or wirelessly) (e.g., network interface 848 may be coupled to an antenna arrangement comprising one or more antennas for enabling network node 800 to wirelessly transmit / receive data); and a storage unit (a.k.a., “data storage system”) 808, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where PC 802 includes a programmable processor, a computer readable storage medium (CRSM) 842 may be provided. CRSM 842 may store a computer program (CP) 843 comprising computer readable instructions (CRI) 844. CRSM 842 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 844 of computer program 843 is configured such that when executed by PC 802, the CRI causes network node 800 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, network node 800 may be configured to perform steps described herein without the need for code. That is, for example, PC 802 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein maybe implemented in hardware and / or software.

[0208] Summary of Embodiments A1. A method (600) for use in generating sample values of a block or a sub-block in a current picture, the method comprising: (i) obtaining (s602) motion vector, MV, information indicating a first set of one or more MVs associated with the block or the sub-block; (ii) obtaining (s604) a first set of one or more MV adjustment values for adjusting one or more MVs; (iii) generating (s606) a second set of one or more MVs based on the first set of one or more MVs and the first set of one or more MV adjustment values; (iv) obtaining (s608) a second set of one or more MV adjustment values for adjusting one or more MVs; and (v) generating (s610) a third set of one or more MVs based on the second set of one or more MVs and the second set of one or more MV adjustment values, wherein the sample values of the block or the sub-block in the current picture are generated based on the third set of one or more MVs. A2. The method of embodiment A1, wherein each of the first, second, and third sets of MVs includes an MV associated with a first reference picture and an MV associated with a second reference picture, and the sample values of the block or the sub-block in the current picture are generated based on performing a bi-inter prediction using the first reference picture and the second reference picture. A3. The method of embodiment A1 or A2, the method further comprising: obtaining reference sample values in one or more reference pictures; and generating first prediction sample values of the block or the sub-block based on at least some of the reference sample values, and the second set of one or more MVs, wherein obtaining the second set of one or more MV adjustment values comprises calculating the second set of one or more MV adjustment values based on the first prediction sample values of the block or the sub-block.A4. The method of embodiment A3, wherein the second set of MV adjustment values is calculated based on horizontal and vertical gradients of the first prediction sample values of the block or the sub-block. A5. The method of any one of embodiments A1-A4, wherein the second set of MV adjustment values is obtained using a bi-directional optical flow, BDOF, scheme. A6. The method of any one of embodiments A1-A5, wherein the first prediction sample values of the block or the sub-block are generated using merge with motion vector difference, MMVD, scheme. A7. The method of any one of embodiments A1-A6, the method comprising: determining the block in the current picture is an inter coded block, wherein the steps (iv)-(v) are performed based on determining that the block in the current picture is an inter coded block. A8. The method of any one of embodiments A1-A7, the method comprising: determining whether the size of the block or the size of the sub-block in the current picture satisfies a condition, wherein the steps (iv)-(v) are performed based on determining that the size of the block or the size of the sub-block in the current picture satisfies the condition. A9. The method of embodiment A8, wherein the condition indicates that the size of the block or the size of the sub-block in the current picture is greater than a threshold size. A10. The method of any one of embodiments A1-A7, the method comprising: determining whether one or more of the first set of one or more MV adjustment values satisfy one or more conditions, wherein the steps (iv)-(v) are performed based on determining that said one or more of the first set of one or more MV adjustment values satisfy said one or more conditions. A11. The method of embodiment A10, whereinthe first set of one or more MV adjustment values comprises a first MV adjustment value, and said one or more conditions indicates that a magnitude of the first MV adjustment value is greater than a threshold value and / or a precision of the first MV adjustment value is greater than a threshold value. B1. A method (700) for generating sample values of a first block or a first sub-block in a current picture, the method comprising: obtaining (s702) first motion vector, MV, information indicating a first MV associated with the first block or the first sub-block; based on the first MV, generating (s704) first prediction sample values of the first block or the first sub-block; obtaining (s706) second MV information indicating a second MV associated with a second block or a second sub-block in the current picture, wherein the second block is adjacent to the first block or the second sub-block is adjacent to the first sub-block; based on the second MV, generating (s708) second prediction sample values of the second block or the second sub-block; determining (s710) one or more regions of the first block based on one or more of the first MV, the second MV, or a boundary between the first block and the second block or between the first sub-block and the second sub-block; and generating (s712) sample values in said one or more regions (e.g., rows or columns) of the first block or the first sub-block based on some of the first prediction sample values and some of the second prediction sample values. B2. The method of embodiment B1, wherein determining said one or more regions of the first block or the first sub-block comprises determining a number of columns of samples or a number of rows of samples included in said one or more regions, and the columns of samples or the rows of samples are parallel to the boundary between the first block and the second block or between the first sub-block and the second sub-block. B3. The method of embodiment B2, the method comprising:determining any one or a combination of (i) if the boundary between the first block and the second block or between the first sub-block and the second sub-block satisfies a first condition, (ii) if the first MV satisfies a second condition, (iii) if the second MV satisfies a third condition, or (iv) whether a difference between the first MV and the second MV satisfies a fourth condition, wherein the number of columns of samples and / or the number of rows of samples included in said one or more regions are determined based on the above determination. B4. The method of embodiment B3, the method comprising: determining whether the boundary between the first block and the second block or between the first sub-block and the second sub-block satisfies the first condition, wherein the first condition is that the boundary between the first block and the second block or between the first sub-block and the second sub-block aligns with a boundary of a grid of decoder-side motion refinement, DMVR, scheme, or aligns with a boundary of a block, the number of columns of samples and / or the number of rows of samples included in said one or more regions are set to be a first value in case the first condition is satisfied, the number of columns of samples and / or the number of rows of samples included in said one or more regions are set to be a second value in case the first condition is not satisfied, and the first value is greater than the second value. B5. The method of embodiment B3, the method comprising: determining whether the difference between the first MV and the second MV satisfies the fourth condition, wherein the fourth condition indicates that the difference between the first MV and the second MV is greater than a certain value, the number of columns of samples and / or the number of rows of samples included in said one or more regions are set to be a first value in case the fourth condition is satisfied, the number of columns of samples and / or the number of rows of samples included in said one or more regions are set to be a second value in case the fourth condition is not satisfied, and the first value is greater than the second value.B6. The method of any one of embodiments B1-B5, wherein each of said one or more regions of the first block or the first sub-block is associated with a first weight value, the method comprises applying the first weight values to said at least some of the first prediction sample values, thereby obtaining weighted first prediction sample values, and the sample values in said one or more regions of the first block or the first sub-block are generated based on said at least some of the second prediction sample values and the weighted first prediction sample values. B7. The method of embodiment B6, wherein the first weight values applied to said at least some of the first prediction sample values are different from each other. B8. The method of embodiment B6 or B7, wherein each of said one or more regions of the first block or the first sub-block is associated with a second weight value, the method comprises applying the second weight values to said at least some of the second prediction sample values, thereby obtaining weighted second prediction sample values, and the sample values in said one or more regions of the first block or the first sub-block are modified based on the weighted first prediction sample values and the weighted second prediction sample values. B9. The method of embodiment B8, wherein the second weight values applied to said at least some of the second prediction sample values are different from each other. B10. The method of any one of embodiments B1-B9, wherein generating the second prediction sample values of the second block or the second sub-block comprises: obtaining reference sample values in a reference picture; generating third prediction sample values of the second block or the second sub-block based on at least some of the reference sample values and the second MV; obtaining a set of MV adjustment values based on the third prediction sample values of the second block or the second sub-block; anddetermining the second prediction sample values based on the set of MV adjustment values and the third prediction sample values. B11. The method of embodiment B10, wherein the set of MV adjustment values is calculated based on horizontal and vertical gradients of the third prediction sample values of the second block or the second sub-block. B12. The method of any one of embodiments B10 and B11, the method comprising: determining any one or a combination of (i) whether the boundary of the first block or the first sub-block satisfies a first condition; (ii) whether the first MV satisfies a second condition; or (iii) whether the second MV satisfies a third condition or (iv) whether a difference between the first MV and the second MV satisfies a fourth condition, wherein the set of MV adjustment values is obtained based on the above determination. B13 The method of embodiment B12 wherein the first condition indicates that the boundary between the first block and the second block or between the first sub-block and the second sub-block aligns with a boundary of a grid of DMVR scheme. B14. The method of embodiment B12 or B13, wherein he fourth condition indicates that the difference between the first MV and the second MV is greater than a certain value. B15. The method of any one of embodiments B10-B14, wherein the set of MV adjustment values is obtained using bi-directional optical flow, BDOF, scheme. C1. A computer program (643) comprising instructions (644) which when executed by processing circuitry (602) cause the processing circuitry (602) to perform the method of any one of embodiments A1-B15. C2. A carrier containing the computer program of embodiment C1, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (642).D1. An apparatus for use in generating sample values of a block or a sub-block in a current picture, the apparatus being configured to: (i) obtain motion vector, MV, information indicating a first set of one or more MVs associated with the block or the sub-block; (ii) obtain a first set of one or more MV adjustment values for adjusting one or more MVs; (iii) generate a second set of one or more MVs based on the first set of one or more MVs and the first set of one or more MV adjustment values; (iv) obtain a second set of one or more MV adjustment values for adjusting one or more MVs; and (v) generate a third set of one or more MVs based on the second set of one or more MVs and the second set of one or more MV adjustment values, wherein the sample values of the block or the sub-block in the current picture are generated based on the third set of one or more MVs. D2. The apparatus of embodiment D1, wherein the apparatus is configured to perform the method of any one of embodiments A2-A11. E1. An apparatus for generating sample values of a first block or a first sub-block in a current picture, the apparatus being configured to: obtain first motion vector, MV, information indicating a first MV associated with the first block or the first sub-block; based on the first MV, generate first prediction sample values of the first block or the first sub-block; obtain second MV information indicating a second MV associated with a second block or a second sub-block in the current picture, wherein the second block is adjacent to the first block or the second sub-block is adjacent to the first sub-block; based on the second MV, generate second prediction sample values of the second block or the second sub-block; determine one or more regions of the first block based on one or more of the first MV, the second MV, or a boundary between the first block and the second block or between the first sub-block and the second sub-block; andgenerate sample values in said one or more regions (e.g., rows or columns) of the first block or the first sub-block based on some of the first prediction sample values and some of the second prediction sample values. E2. The apparatus of embodiment E1, wherein the apparatus is configured to perform the method of any one of embodiments B2-B15. F1. An apparatus (600) comprising: a memory (642); and processing circuitry (602) coupled to the memory (642), wherein the apparatus (600) is configured to perform the method of any one of embodiments A1-B15.

[0209] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0210] As used herein transmitting a message “to” or “toward” an intended recipient encompasses transmitting the message directly to the intended recipient or transmitting the message indirectly to the intended recipient (i.e., one or more other nodes are used to relay the message from the source node to the intended recipient). Likewise, as used herein receiving a message “from” a sender encompasses receiving the message directly from the sender or indirectly from the sender (i.e., one or more nodes are used to relay the message from the sender to the receiving node). Further, as used herein “a” means “at least one” or “one or more.” Also, as used here, “based on” does not necessarily mean “based solely on.” It may also mean based at least on.”

[0211] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

Claims

CLAIMS 1. A method (600) for use in generating sample values of a block or a sub-block in a current picture, the method comprising: (i) obtaining (s602) motion vector, MV, information indicating a first set of one or more MVs associated with the block or the sub-block; (ii) obtaining (s604) a first set of one or more MV adjustment values for adjusting one or more MVs; (iii) generating (s606) a second set of one or more MVs based on the first set of one or more MVs and the first set of one or more MV adjustment values; (iv) obtaining (s608) a second set of one or more MV adjustment values for adjusting one or more MVs; and (v) generating (s610) a third set of one or more MVs based on the second set of one or more MVs and the second set of one or more MV adjustment values, wherein the sample values of the block or the sub-block in the current picture are generated based on the third set of one or more MVs.

2. The method of claim 1, wherein each of the first, second, and third sets of MVs includes an MV associated with a first reference picture and an MV associated with a second reference picture, and the sample values of the block or the sub-block in the current picture are generated based on performing a bi-inter prediction using the first reference picture and the second reference picture.

3. The method of claim 1 or 2, the method further comprising: obtaining reference sample values in one or more reference pictures; and generating first prediction sample values of the block or the sub-block based on at least some of the reference sample values, and the second set of one or more MVs, wherein obtaining the second set of one or more MV adjustment values comprises calculating the second set of one or more MV adjustment values based on the first prediction sample values of the block or the sub-block.

4. The method of claim 3, wherein the second set of MV adjustment values is calculated based on horizontal and vertical gradients of the first prediction sample values of the block or the sub-block.

5. The method of any one of claims 1-4, wherein the second set of MV adjustment values is obtained using a bi-directional optical flow, BDOF, scheme.

6. The method of any one of claims 1-5, wherein the first prediction sample values of the block or the sub-block are generated using merge with motion vector difference, MMVD, scheme.

7. The method of any one of claims 1-6, the method comprising: determining the block in the current picture is an inter coded block, wherein the steps (iv)-(v) are performed based on determining that the block in the current picture is an inter coded block.

8. The method of any one of claims 1-7, the method comprising: determining whether the size of the block or the size of the sub-block in the current picture satisfies a condition, wherein the steps (iv)-(v) are performed based on determining that the size of the block or the size of the sub-block in the current picture satisfies the condition.

9. The method of claim 8, wherein the condition indicates that the size of the block or the size of the sub-block in the current picture is greater than a threshold size.

10. The method of any one of claims 1-7, the method comprising: determining whether one or more of the first set of one or more MV adjustment values satisfy one or more conditions, wherein the steps (iv)-(v) are performed based on determining that said one or more of the first set of one or more MV adjustment values satisfy said one or more conditions.

11. The method of claim 10, whereinthe first set of one or more MV adjustment values comprises a first MV adjustment value, and said one or more conditions indicates that a magnitude of the first MV adjustment value is greater than a threshold value and / or a precision of the first MV adjustment value is greater than a threshold value.

12. A method (700) for generating sample values of a first block or a first sub-block in a current picture, the method comprising: obtaining (s702) first motion vector, MV, information indicating a first MV associated with the first block or the first sub-block; based on the first MV, generating (s704) first prediction sample values of the first block or the first sub-block; obtaining (s706) second MV information indicating a second MV associated with a second block or a second sub-block in the current picture, wherein the second block is adjacent to the first block or the second sub-block is adjacent to the first sub-block; based on the second MV, generating (s708) second prediction sample values of the second block or the second sub-block; determining (s710) one or more regions of the first block based on one or more of the first MV, the second MV, or a boundary between the first block and the second block or between the first sub-block and the second sub-block; and generating (s712) sample values in said one or more regions (e.g., rows or columns) of the first block or the first sub-block based on some of the first prediction sample values and some of the second prediction sample values.

13. The method of claim 12, wherein determining said one or more regions of the first block or the first sub-block comprises determining a number of columns of samples or a number of rows of samples included in said one or more regions, and the columns of samples or the rows of samples are parallel to the boundary between the first block and the second block or between the first sub-block and the second sub-block.

14. The method of claim 13, the method comprising:determining any one or a combination of (i) if the boundary between the first block and the second block or between the first sub-block and the second sub-block satisfies a first condition, (ii) if the first MV satisfies a second condition, (iii) if the second MV satisfies a third condition, or (iv) whether a difference between the first MV and the second MV satisfies a fourth condition, wherein the number of columns of samples and / or the number of rows of samples included in said one or more regions are determined based on the above determination.

15. The method of claim 14, the method comprising: determining whether the boundary between the first block and the second block or between the first sub-block and the second sub-block satisfies the first condition, wherein the first condition is that the boundary between the first block and the second block or between the first sub-block and the second sub-block aligns with a boundary of a grid of decoder-side motion refinement, DMVR, scheme, or aligns with a boundary of a block, the number of columns of samples and / or the number of rows of samples included in said one or more regions are set to be a first value in case the first condition is satisfied, the number of columns of samples and / or the number of rows of samples included in said one or more regions are set to be a second value in case the first condition is not satisfied, and the first value is greater than the second value.

16. The method of claim 14, the method comprising: determining whether the difference between the first MV and the second MV satisfies the fourth condition, wherein the fourth condition indicates that the difference between the first MV and the second MV is greater than a certain value, the number of columns of samples and / or the number of rows of samples included in said one or more regions are set to be a first value in case the fourth condition is satisfied, the number of columns of samples and / or the number of rows of samples included in said one or more regions are set to be a second value in case the fourth condition is not satisfied, and the first value is greater than the second value.

17. The method of any one of claims 12-16, wherein each of said one or more regions of the first block or the first sub-block is associated with a first weight value, the method comprises applying the first weight values to said at least some of the first prediction sample values, thereby obtaining weighted first prediction sample values, and the sample values in said one or more regions of the first block or the first sub-block are generated based on said at least some of the second prediction sample values and the weighted first prediction sample values.

18. The method of claim 17, wherein the first weight values applied to said at least some of the first prediction sample values are different from each other.

19. The method of claim 17 or 18, wherein each of said one or more regions of the first block or the first sub-block is associated with a second weight value, the method comprises applying the second weight values to said at least some of the second prediction sample values, thereby obtaining weighted second prediction sample values, and the sample values in said one or more regions of the first block or the first sub-block are modified based on the weighted first prediction sample values and the weighted second prediction sample values.

20. The method of claim 19, wherein the second weight values applied to said at least some of the second prediction sample values are different from each other.

21. The method of any one of claims 12-20, wherein generating the second prediction sample values of the second block or the second sub-block comprises: obtaining reference sample values in a reference picture; generating third prediction sample values of the second block or the second sub-block based on at least some of the reference sample values and the second MV; obtaining a set of MV adjustment values based on the third prediction sample values of the second block or the second sub-block; anddetermining the second prediction sample values based on the set of MV adjustment values and the third prediction sample values.

22. The method of claim 21, wherein the set of MV adjustment values is calculated based on horizontal and vertical gradients of the third prediction sample values of the second block or the second sub-block.

23. The method of any one of claims 21 and 22, the method comprising: determining any one or a combination of (i) whether the boundary of the first block or the first sub-block satisfies a first condition; (ii) whether the first MV satisfies a second condition; or (iii) whether the second MV satisfies a third condition or (iv) whether a difference between the first MV and the second MV satisfies a fourth condition, wherein the set of MV adjustment values is obtained based on the above determination.

24. The method of claim 23 wherein the first condition indicates that the boundary between the first block and the second block or between the first sub-block and the second sub-block aligns with a boundary of a grid of DMVR scheme.

25. The method of claim 23 or 24, wherein he fourth condition indicates that the difference between the first MV and the second MV is greater than a certain value.

26. The method of any one of claims 21-25, wherein the set of MV adjustment values is obtained using bi-directional optical flow, BDOF, scheme.

27. A computer program (643) comprising instructions (644) which when executed by processing circuitry (602) cause the processing circuitry (602) to perform the method of any one of claims 1-26.

28. A carrier containing the computer program of claim 27, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (642).

29. An apparatus for use in generating sample values of a block or a sub-block in a current picture, the apparatus being configured to: (i) obtain motion vector, MV, information indicating a first set of one or more MVs associated with the block or the sub-block; (ii) obtain a first set of one or more MV adjustment values for adjusting one or more MVs; (iii) generate a second set of one or more MVs based on the first set of one or more MVs and the first set of one or more MV adjustment values; (iv) obtain a second set of one or more MV adjustment values for adjusting one or more MVs; and (v) generate a third set of one or more MVs based on the second set of one or more MVs and the second set of one or more MV adjustment values, wherein the sample values of the block or the sub-block in the current picture are generated based on the third set of one or more MVs.

30. The apparatus of claim 29, wherein the apparatus is configured to perform the method of any one of claims 2-11.

31. An apparatus for generating sample values of a first block or a first sub-block in a current picture, the apparatus being configured to: obtain first motion vector, MV, information indicating a first MV associated with the first block or the first sub-block; based on the first MV, generate first prediction sample values of the first block or the first sub-block; obtain second MV information indicating a second MV associated with a second block or a second sub-block in the current picture, wherein the second block is adjacent to the first block or the second sub-block is adjacent to the first sub-block; based on the second MV, generate second prediction sample values of the second block or the second sub-block; determine one or more regions of the first block based on one or more of the first MV, the second MV, or a boundary between the first block and the second block or between the first sub-block and the second sub-block; andgenerate sample values in said one or more regions (e.g., rows or columns) of the first block or the first sub-block based on some of the first prediction sample values and some of the second prediction sample values.

32. The apparatus of claim 31, wherein the apparatus is configured to perform the method of any one of claims 13-26.

33. An apparatus (600) comprising: a memory (642); and processing circuitry (602) coupled to the memory (642), wherein the apparatus (600) is configured to perform the method of any one of claims 1-26.

Images