Motion vector derivation

Abstract

There is provided a method for obtaining, for a current subblock of samples, a subblock of motion compensated predicted samples. The method comprises, based on a boundary distortion criterion, selecting a motion vector, MV, pair for the current subblock of samples from a set of candidate MV pairs. The selecting comprises, for each candidate MV pair, determining a distortion value for the candidate MV pair, wherein determining a distortion value for each candidate MV pair comprises: determining a first distortion value, DV1, for a first candidate MV pair and determining a second distortion value for a second candidate MV pair, wherein DV1 is equal to a sum of a current subblock distortion value, CurrentDistortion, and a subblock boundary distortion value, BoundaryDistortion. The method comprises determining the candidate MV pair having the lowest distortion value, wherein the determined candidate MV pair is the MV pair selected for the current subblock of samples. The method comprises using the selected MV pair to produce the subblock of motion compensated prediction samples.

Description

MOTION VECTOR DERIVATION TECHNICAL FIELD

[0001] This disclosure relates to methods and apparatus for motion vector derivation. BACKGROUND

[0002] VVC and ECM

[0003] Versatile Video Coding (VVC) is a block-based video codec standardized by ITU-T and MPEG. Enhanced Coding Model (ECM) is an exploratory codec which is currently under development. The aim of ECM is to demonstrate and try providing evidence of video coding capabilities beyond VVC. The current ECM version is ECM-9.0.

[0004] Video and Picture

[0005] A video (a.k.a., “video sequence”) comprises of a series of pictures. In VVC, each picture is identified with a picture order count (POC) value. The POC value also represents the display order of the picture. A picture with a smaller POC value is displayed before another picture with a larger POC value.

[0006] Components

[0007] It is common that each picture consists of three components; one luma component Y where the sample values are luma values and two chroma components Cb and Cr, where the sample values are chroma values. Each component can be described as a two-dimensional rectangular array of sample values. It is also common that the dimensions of the chroma components are smaller than the luma components by a factor of two in each dimension. For example, the size of the luma component of an HD picture would be 1920x1080 and the chroma components would each have the dimension of 960x540. Components are sometimes referred to as color components.

[0008] Blocks, Subblocks, and Units

[0009] A block is a two-dimensional (2D) matrix of sample values (or “samples” forshort). A block may be divided into two or more subblocks, where each subblock is a matrix of samples. In video coding, each component of a picture is split into blocks and the coded video bitstream consists of a series of coded blocks. It is common in video coding that pictures are split into units that cover a specific area of the picture.

[0010] Each unit consists of all blocks from all components that make up that specific area of the picture and each block belongs fully to one unit. The Coding Unit (CU) in VVC is an example of a unit. In VVC, the CUs may be split recursively to smaller CUs. The CU at the top level is referred to as the coding tree unit (CTU). A CU usually contains three coding blocks, i.e., one coding block for luma and two coding blocks for chroma. In VVC, the CUs can have size of 4x4 up to 128x128. In current ECM, the CUs can have size of 4x4 up to 256x256.

[0011] Parameter sets, slice headers, and picture headers

[0012] VVC specifies three types of parameter sets: the picture parameter set (PPS), the sequence parameter set (SPS), and the video parameter set (VPS). The PPS contains data that is common for all units of a picture, the SPS contains data that is common for a coded layer video sequence (CLVS), and the VPS contains data that is common for multiple CLVSs, e.g., data for multiple layers in the bitstream.

[0013] The concept of slices divides the picture into independently coded slices, where decoding of one slice in a picture is independent of other slices of the same picture. Each slice has a slice header comprising syntax elements. Decoded slice header values from these syntax elements are used when decoding the slice. In VVC, a coded picture contains a picture header. The picture header contains parameters that are common for all slices of the coded picture.

[0014] Intra prediction

[0015] In intra prediction, also known as spatial prediction, a current block is predicted using previous decoded blocks within the same picture. The samples from the previously decoded blocks within the same picture are used to predict the samples inside the current block. A picture consisting of only intra-predicted blocks is referred to as an intra picture.

[0016] Inter prediction

[0017] In inter prediction, also known as temporal prediction, a current block of thecurrent picture is predicted using blocks from previously decoded pictures (these blocks are referred to as reference blocks). The samples from the reference blocks in the previously decoded pictures are used to predict the samples inside the current block. A picture that comprises one or more inter-predicted blocks is referred to as an inter picture. The previous decoded pictures used for inter prediction are referred to as reference pictures.

[0018] The location of a referenced block inside a reference picture is indicated using a vector (i.e., a set of values) (which is referred to as “motion vector (MV)”). Each MV consists of two values: an x value (a.k.a., x component) and y value (a.k.a., y component) which represents the displacements between current block and the referenced block in x or y dimension. The value of a component may have a resolution finer than an integer position. When that is the case, a filtering (typically interpolation) is done to calculate values used for prediction. FIG.1 shows an example of a MV for the current block C. The example MV consist of the values 2,1.

[0019] An inter picture may use several reference pictures. The reference pictures are usually put into two reference picture lists, L0 and L1. The reference pictures that are output before the current picture are typically the first pictures in L0. The reference pictures that are output after the current picture are typically the first pictures in L1.

[0020] Inter predicted blocks can use one of two prediction types, uni- and bi-prediction. Uni-predicted block predicts from one reference picture, either using L0 or L1. Bi-prediction predicts from two reference pictures, one from L0 and the other from L1. FIG.2 shows an example of the prediction types.

[0021] Picture coding type (Low delay picture and non-low delay picture)

[0022] A low delay picture is a picture that has all its reference pictures displayed before the picture. In other words, for a low delay picture, all its reference pictures have smaller POC values than the current POC. A non-low delay picture is a picture that has at least one of its reference pictures displayed after the picture. In other words, a non-low delay picture has at least one reference picture with a larger POC value than the current POC.

[0023] Fractional MVs, Interpolation filter and MV rounding

[0024] The value of the MV’s x or y component may corresponds to a sample positionwhich has finer granularity than integer (sample) position. Those positions are also referred to as fractional (sample) positions.

[0025] In VVC and current ECM, the MV can be at 1 / 16 sample position. FIG.3 depicts several fractional positions in the horizontal (x-) dimension. The solid-square blocks represent integer positions. The circles represent 1 / 16-position. For example, MV = (4, 10) means the x component is at 4 / 16 position, the y component is at 10 / 16 position.

[0026] In video coding, a MV rounding process is sometimes used to convert a MV at one position to another target position. One example of rounding is to round a fractional MV position to the nearest integer position.

[0027] When an MV is at a fractional position, filtering (typically interpolation) is done to calculate the sample values at those positions. In VVC, the length (number of filter taps) of the interpolation filter for luma component is 8, as shown in table 1 below. In ECM, the length of the interpolation filter for luma component has been increased to 12. TABLE 1 Fractional interpolation filter coefficients sample position p fL[p][0] fL[p][1] fL[p][2] fL[p][3]fL[p][4] fL[p][5] fL[p][6] fL[p][7] 1 0 1 −3 63 4 −2 1 0 2 −1 2 −5 62 8 −3 1 0 3 −1 3 −8 60 13 −4 1 0 4 −1 4 −10 58 17 −5 1 0 5 −1 4 −11 52 26 −8 3 −1 6 −1 3 −9 47 31 −10 4 −17 −1 4 −11 45 34 −10 4 −1 8 −1 4 −11 40 40 −11 4 −1 9 −1 4 −10 34 45 −11 4 −1 10 −1 4 −10 31 47 −9 3 −1 11 −1 3 −8 26 52 −11 4 −1 12 0 1 −5 17 58 −10 4 −1 13 0 1 −4 13 60 −8 3 −1 14 0 1 −3 8 62 −5 2 −1 15 0 1 −2 4 63 −3 1 0

[0028] Residual, transform and quantization

[0029] The difference between samples of a source block (contains original samples) and samples of the prediction block, also called residual block, is then typically compressed by a spatial transform to remove further redundancy. The transform coefficients are then quantized by a quantization parameter (QP) to control the fidelity of the residual block and thus also the bitrate required to compress the block. A coded block flag (CBF) is used to indicate if there are any non-zero quantized transform coefficients. All coding parameters are then entropy coded at the encoder and decoded at the decoder. If the coded block flag is one, a reconstructed block can then be derived by inverse quantization and inverse transformation of the quantized transform coefficients and then add that to the prediction block. If the coded block flag is zero, the reconstructed block is identical to the prediction block.

[0030] Inter prediction information / Motion information

[0031] For an inter block inside an inter picture in VVC, its inter prediction information consists of the following three elements:

[0032] (1) a reference picture list flag (RefPicListFlag) which signals which reference picture list is used for the block (when the value of the flag is equal to 0, it means only L0 is used for predicting the current block, when the value of the flag is equal to 1, it means only L1 is used for predicting the current block, and when the value of the flag is equal to 2, it means both L0 and L1 are used for predicting the current block);

[0033] (2) a reference picture index (RefPicIdx) per reference picture list used (the index signals which reference picture inside the reference list to be used for predicting the current block), and

[0034] (3) a motion vector (MV) per reference picture used, which signals the position inside the reference picture that is used for predicting the current block.

[0035] The inter prediction information is also referred to as motion information. The decoder stores the motion information for each inter block. In other words, an inter block maintains its own motion information.

[0036] Encoder decision and Rate Distortion (RD) cost

[0037] In practice, for an encoder to decide the best prediction mode for a current block, the encoder would evaluate all the possible prediction modes for the current block and select the prediction mode that yields the smallest Rate-Distortion (RD) cost.

[0038] The RD cost is calculated as ^^ + ^^ ∗ ^^. D (Distortion) measures the difference between the reconstructed block and the corresponding source block. One commonly used metric for calculating D is the sum of squared error ^^ ^^ ^^ = ∑௫,௬( ^^^( ^^, ^^) − ^^^( ^^, ^^))ଶ, where theP AandP Bare the sample values in the two blocks Ais usually of the bits to be spent on encoding the mode, and ^^ is a trade-off parameter between R and D.

[0039] Motion information signaling

[0040] VVC and ECM includes several methods for implicit signaling of motion information for each block, including the merge method and the subblock merge method. A common motivation behind the implicit methods is to inherit or reuse motion information fromneighboring coded blocks. This often works in practice due to spatial correlation of close-by blocks, i.e., the fact that nearby blocks often behave similarly.

[0041] Merge (block merge) method and merge mode

[0042] The merge method derives a set of motion information from previously decoded blocks and use the derived motion information for generating the samples of the entire block. The merge method is sometimes referred to as the block merge method.

[0043] The merge method first generates a list of motion information candidates. The list is also referred to as the merge list. The candidates are derived from previously coded blocks. These previously coded blocks can be spatially adjacent neighboring blocks or temporal collocated blocks relative to the current block. FIG.4 shows the spatial neighboring blocks: left (L), top(T), top-right(TR), left-bottom (LB) and top-left (TL).

[0044] The merge list construction process usually checks the previously coded blocks in a predefined order, for example: T, L, TR, LB, then TL. For each previously coded block being checked, if this previously coded block is inter coded and its motion information has no duplicates in the list, then the motion information of this previously coded block is added to the merge list.

[0045] After the merge list is generated, one of the candidates inside the list is used to derive the motion information of the current block. The candidate selection process is done on the encoder side. An encoder would select a best candidate from the list and encode an index (merge_index) in the bitstream to signal to a decoder. The decoder receives the index, it follows the same merge list derivation process as the encoder and uses the index to retrieve the correct candidate. The blocks that use the block merge method are sometimes referred to as blocks in merge mode.

[0046] In the current ECM, non-adjacent spatial blocks are also considered as sources of motion information during the merge list construction. FIG.5 shows some examples (marked with NA1, NA2 and NA3) of those non-adjacent spatial blocks.

[0047] Subblock merge method

[0048] VVC and ECM also include the subblock merge method. It splits a current blockinto a number of subblocks and allows each subblock to have its own motion information. FIG.6 shows an example of a current block and its subblocks. Each subblock maintains its own motion information. It should be noted that the subblocks are all rectangular.

[0049] Overlapped Block Motion Compensation (OBMC)

[0050] OBMC is a tool included in ECM which operates at the block boundaries or subblock boundaries of a current inter block. OBMC blends the current block or subblock’s prediction samples, P_CUR, (generated using the current associated motion information) with another set of prediction samples, P_NB, which are generated using motion information from a neighboring block or a neighboring subblock at the current block or subblock boundaries. The OBMC blending process takes weighted average of P_CUR and P_NB to produce a set of OBMC modified prediction samples, P_OBMC. In other words, the set P_CUR or P_NB each associate with a respective weighting factor, and the set of OBMC modified prediction samples P_OBMC is derived as w1*P_CUR + w2*P_NB (where w1 + w2 = 1). The OBMC may give better prediction for samples that are close to the block or subblock boundary.

[0051] Bi-Directional Optical Flow (BDOF)

[0052] BDOF is a tool included in VVC and the current ECM that can be used to refine prediction samples that are generated from a Bi-MV. BDOF relies on optical flow estimation to derive a pair of refinement parameter (Vx, Vy) which can be further used to refine the prediction samples.

[0053] Decoder-side Motion Vector Refinement (DMVR)

[0054] DMVR is a tool included in VVC and the current ECM to refine motion vectors for a Bi-MV. DMVR operates on subblock level, usually 16x16. Different from BDOF, which relies on optical flow estimation, DMVR relies on bilateral matching of two reference blocks to refine the Bi-MV. The matching is based on SAD (sum of absolute differences). The DMVR searches within a window around the Bi-MV to find whether there exists another Bi-MV (Bi- MV’) that gives a better match between the L0 reference block and the L1 reference block. If so, the Bi-MV’ is further used instead for generating the prediction samples of the current block. After that a subpel adjustment is made based on the SAD costs around the motion with leastSAD to determine subblock motion with sub pixel accuracy.

[0055] Multi-pass DMVR

[0056] In ECM, DMVR has been further evolved to use multi-pass optimization, first bilateral block matching in a search area to find the refinement of the merge motion that gives best match, then bilateral 16x16 subblock matching, then bi-directional optical flow on 8x8 subblocks to refine motion further. The multi-pass DMVR also allows for modification of only one of the bi-predictive motions. It is also allowed to use DMVR when BCW (other weightings than just average) is used.

[0057] Template matching (TM)

[0058] In ECM block or subblock motion can also be refined by matching a template outside the current block with a corresponding template on the reference picture. The search is limited to a small range to find a better motion without signaling additional motion information. This can also be used cascaded with the bilateral matching in multi-pass DMVR. SUMMARY

[0059] Certain challenges presently exist. For example, a problem with DMVR and methods that determine motion based on distortion only inside the current block (bilateral block matching) or only outside the current block (template matching) risk obtaining an inaccurate motion that results in a current block that does not fit well with the surrounding blocks. Visual artifacts can occur at block (e.g., subblock) boundaries and can also result in that some subblocks completely deviate in appearance from the surrounding subblocks due to completely wrong subblock motion in cases where the true motion is too large or cases where one of the references blocks is too smooth to find a good match.

[0060] According to a first aspect of the present disclosure, there is provided a method for obtaining, for a current subblock of samples, a subblock of motion compensated predicted samples. The method comprises, based on a boundary distortion criterion, selecting a motion vector, MV, pair for the current subblock of samples from a set of candidate MV pairs. The selecting comprises: for each candidate MV pair, determining a distortion value for the candidate MV pair, wherein determining a distortion value for each candidate MV pair comprises:determining a first distortion value, DV1, for a first candidate MV pair and determining a second distortion value for a second candidate MV pair, wherein DV1 is equal to a sum of a current subblock distortion value, CurrentDistortion, and a subblock boundary distortion value, BoundaryDistortion. The method further comprises determining the candidate MV pair having the lowest distortion value, wherein the determined candidate MV pair is the MV pair selected for the current subblock of samples. The method comprises using the selected MV pair to produce the subblock of motion compensated prediction samples.

[0061] According to a second aspect of the present disclosure, there is provided a computer program comprising instructions which when executed by processing circuitry of an apparatus causes the apparatus to perform the method according to the first aspect.

[0062] According to a third aspect of the present disclosure, there is provided a carrier containing the computer program wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

[0063] According to a fourth aspect of the present disclosure, there is provided an apparatus that is configured to perform the method according to the first aspect. The apparatus may include memory and processing circuitry coupled to the memory.

[0064] An advantage of the embodiments disclosed herein is that they provide better subjective quality in form of more coherent subblocks and avoiding dislocated subblocks as in approaches where motion is derived on decoder side, such as in DMVR. In some cases, it can also provide better objective performance in terms of BDR (Bjontegaard delta rate). BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0066] FIG.1 shows an example of a motion vector (MV).

[0067] FIG.2 illustrates uni-inter prediction and bi-inter prediction.

[0068] FIG.3 depicts several fractional positions in the horizontal (x-) dimension.

[0069] FIG.4 shows spatial neighboring blocks.

[0070] FIG.5 shows some examples of non-adjacent spatial blocks.

[0071] FIG.6 shows an example of a current block and its subblocks.

[0072] FIG.7 illustrates a system according to some embodiments.

[0073] FIG.8 illustrates an example current subblock and its neighboring subblocks.

[0074] FIG.9 is a flowchart illustrating a process according to some embodiments.

[0075] FIG.10 is a flowchart illustrating a process according to some embodiments.

[0076] FIG.11 is a block diagram of an apparatus according to some embodiments. DETAILED DESCRIPTION

[0077] FIG.7 illustrates a system 700 according to an embodiment. System 700 includes an encoder 702 and a decoder 704, wherein encoder 702 is in communication with decoder 704 via a network 110, such as the Internet or other network. Encoder 702 encodes a source video sequence 701 (e.g., encodes blocks of units of pictures of video sequence 701) into a bitstream comprising an encoded video sequence and transmits the bitstream to decoder 704 via network 708. In some embodiments, encoder 702 is not in communication with decoder 704, and, in such an embodiment, rather than transmitting bitstream to decoder 704, the bitstream is stored in a data storage unit for later use. Decoder 704 decodes the pictures included in the encoded video sequence to produce video data for display and / or for further image processing (e.g. a machine vision task). Accordingly, decoder 704 may be part of a device 703 having an image processor 705 and / or a display 706. The image processor 705 may perform machine vision tasks on the decoded pictures. The device 703 may be a mobile device, a set-top device, a head-mounted display, or any other device.

[0078] As noted above, the existing DMVR scheme can lead to obtaining an inaccurate motion that results in a current block that does not fit well with the surrounding blocks.

[0079] Accordingly, this disclosure provides a method that uses at least one block boundary distortion criteria when selecting a motion vector for producing motion compensated prediction of a block or subblock of samples. This can be used both as an encoder only solution for finding better motion vectors but also as a both encoder and decoder method for codecs thatdetermine motion vectors for motion compensated prediction both on encoder and decoder side to reduce transmission of motion vector information.

[0080] That is, certain embodiments of this disclosure propose adding one or more criterions that penalize misalignment of at least one boundary of the current predicted subblock compared to adjacent samples of a neighbouring subblock. The penalization can be performed by adding a misalignment distortion to the block matching distortion for a given motion of the subblock. For a given range of motion, the subblock motion vector that minimizes the total distortion is then selected.

[0081] One example criterion to estimate the misalignment distortion is pixel wise difference calculation for at least one sample at the block boundary but inside the current subblock and an adjacent sample in a neighbouring subblock. Any differences indicates that there is some misalignment between current subblock and neighbouring subblock. Another example criterion is the difference in characteristics (texture) between the current subblock boundary and an adjacent neighbouring subblock boundary. One example characteristics is slope and another is 2:nd order variation (hill or valley).

[0082] As noted above, the embodiments can be used solely at the encoder or the embodiments may be used at both the encoder and the decoder. The embodiments can be used for both blocks and subblocks. The embodiments described below are mainly described for subblocks but similar approach can be used for blocks in general. For samples, the embodiments mainly refer to predicted samples from previously coded pictures that have been motion compensated. The predicted samples used in the embodiments could either be after bi-prediction, or separately on the individual uni-directional predictions before bi-prediction. Neighbouring samples could also be reconstructed samples.

[0083] Embodiment 1

[0084] For a current subblock, the determination of its motion compensated prediction samples may involve the following three steps:

[0085] Step 1: Determine whether the current subblock has a neighboring subblock that is adjacent to one of the current subblock’s boundaries.

[0086] Step 2: As a result of determining that a neighboring subblock is adjacent to one of the current subblock’s boundaries, select at least one sample (Snb) from the neighboring subblock (the sample can either be a prediction sample or a reconstructed sample of the neighboring subblock).

[0087] Step 3: derive a refined pair of motion vectors based on a subblock boundary distortion criteria. For example, in one embodiment, deriving the refined MV pairs comprises: i) for each MV pair in a set of candidate MV pairs, determine a distortion value (a.k.a., total distortion value), TotalDistortion, for the MV pair, where the total distortion value is a function of a current block distortion value, CurrentDistortion, (a.k.a., the block matching distortion) and, potentially, a subblock boundary distortion value, BoundaryDistortion, and ii) select from the set of candidate MV pairs, the MV pair having the lowest TotalDistortion.

[0088] Step 4: producing a motion compensated prediction sample using the selected MV pair.

[0089] In one embodiment, the CurrentDistortion for an MV pair is a difference (e.g., absolute difference) between a first candidate prediction block, CPB1, and a second candidate prediction block, CPB2, where CPB1 is derived using the first MV of the MV pair and CPB2 is derived using the second MV of the MV pair.

[0090] In one embodiment, subblock boundary distortion refers to boundary differences based on samples on both sides of a subblock boundary. The differences can be in terms of a sum of absolute sample difference across the subblock boundary and / or a sum of absolute differences in characteristics along the boundary on both sides of the boundary, such as, for example, first order (slope) or second order (quadratic function) or higher order differences.

[0091] One example of sample difference for a horizontal subblock boundary (e.g., top or bottom subblock boundary) is a sum of absolute difference across the horizontal boundary. The sample differences can include all boundary samples or a subset of horizontal boundary samples. One example of sample difference for a vertical subblock boundary (e.g., left or right subblock boundary) is a sum of absolute difference across the vertical subblock boundary. The sample differences can include all boundary samples or a subset of vertical boundary samples.

[0092] One example of characteristics differences is to compute the absolute difference between i) the difference between the top-right and the bottom right sample of the left neighbouring subblock and ii) the difference between the top-left and the bottom-left sample of the current subblock. This can indicate difference in slope or gradient of the current and neighbouring subblock boundary samples.

[0093] Another example of characteristic differences is to compute the absolute difference between i) the Laplacian (a -2*b +c), where a, b and c are three samples, based on the top-right sample, a middle sample and the bottom-right sample of the left neighbouring subblock and ii) the Laplacian based on the top-left sample, a middle sample and the bottom-left sample of the current subblock. This can indicate difference in second order characteristics such as a hill or a valley.

[0094] More generally one can compute the response of a set of base functions along respective boundary and then take the sum of absolute differences between the responses as the difference in boundary characteristics.

[0095] Embodiment 2

[0096] In one embodiment, the selection of subblock motion vectors is made by minimization of a subblock boundary distortion, distortion from template-based motion derivation, and / or block matching based motion derivation. The relative contribution of the subblock boundary distortion can be scaled by a scaling factor, SF.

[0097] That is, in one embodiment, for each candidate MV pair, the TotalDistortion for the MV pair is calculated as: TotalDistortion = CurrentDistortion + BoundaryDistortion, where BoundaryDistortion is a function of SF, vBoundaryDistortion, and hBoundaryDistortion.

[0098] In one example: BoundaryDistortion = 16*((vBoundaryDistortion + hBoundaryDistortion + 2)>>2).

[0099] In one embodiment: vBoundaryDistortion = (sumVertBoundarySampleDiff)>>Rv+4*(vertBoundaryGradDiff+vertBoundaryLaplacianDiff), and HBoundaryDistortion = (sumHorizBoundarySampleDiff)>> Rh+4*(horizBoundaryGradDiff+horizBoundaryLaplacianDiff). In one embodiment, Rv = log2(N)-K+1 and Rh = log2(M)-K+1, where N is the length of the vertical subblock boundary, M is the length of the horizontal subblock boundary, and K is a selected value (e.g., K=1, 2, 4, 8, or 16). Hence, when K=1 and N=M=16, then Rv=Rh=4.

[0100] Using FIG.8 as an example, in one embodiment: ^^ sumHorizBoundarySampleDiff =∑ெ^ୀି^^^^ ^^ ^^ (Ai,0 - Bi,N-1) (or more generically = ^^−1 ^^=0^^ ^^ ^^ (Ai*k,0 - Bi*k,N-1), where K > 0 and K ≤ M (e.g., K=1, 2, 4, 8, or 16));^^sumVertBoundarySampleDiff =∑ே^ୀି^^^^ ^^ ^^ (A0,j- DM-1,j) (or more generically = ∑ ^^−1 ^^=0^^ ^^ ^^ (A0, j*k– DM-1, j*k), where K > 0 and K ≤ N2, 4, 8, or 16));vertBoundaryGradDiff = abs(DM-1,0 – DM-1,N-1 – (A0,0 – A0,N-1); vertBoundaryLaplacianDiff = abs(DM-1,0–2* DM-1,N / 2+ DM-1,N-1– (A0,0– 2*A0,N / 2+ A0,N-1)); horizBoundaryGradDiff = abs(B0,N-1 – BM-1,N-1 – (A0,0 – AM-1,0); and horizBoundaryLaplacianDiff = abs(B0,N-1 –2* BM / 2,N-1 + BM-1,N-1 – (A0,0 – 2*AM / 2,0 + AM-1,0)).

[0101] In the embodiment the matrix of the A values (i.e., Ai,j for i=0 to N-1 and for j=0 to M-1 is equal to the average of CPB1 and CPB2 (i.e., the first candidate prediction block derived from the first MV of the MV pair and the second candidate prediction block derived from the second MV of the MV pair). In some embodiments A is equal to CPB1 or CPB2. In some embodiments, A is equal to a weighted average of CPB1 and CPB2 (i.e., CPB1 and CPB2 each associate with a weighting factor).

[0102] The weighting of the boundary distortion (e.g. SF) can also be QP dependent so that a relative lower weight is used for higher QP when distortion of the current subblock may be smaller due to less details in the prediction.

[0103] Embodiment 3

[0104] In some embodiments, the above-described embodiments are used when a subblock boundary criterion based on neighboring samples is fulfilled.

[0105] The use of subblock boundary distortion of a horizontal subblock boundary at the top of the current subblock can be determined by the magnitude of the difference (i.e., the absolute difference) between i) the top-right sample of the neighbouring subblock to the left of current subblock (e.g., DM-1,0 shown in FIG.8) and ii) the bottom-right sample of the top-left neighboring subblock of the current subblock (e.g., CM-1,N-1shown in FIG.8). When the absolute difference is greater than a threshold, then the horizontal boundary is determined to be a true edge and thus no horizontal subblock boundary distortion is calculated (e.g., hBoundaryDistortion is set to zero).

[0106] The use of subblock boundary distortion of a vertical subblock boundary at the left of the current subblock can be determined by the magnitude of the difference between i) the bottom-left sample of the neighbouring subblock above current subblock (e.g., B0,N-1 shown in FIG.8) and ii) the bottom-right sample of the top-left neighboring subblock of the current subblock (e.g., CM-1,N-1 shown in FIG.8). When the absolute difference is greater than a threshold, then the vertical subblock boundary is determined to be a true edge and thus no vertical subblock boundary distortion is calculated (e.g., vBoundaryDistortion is set to zero).

[0107] One example threshold is 15 for 10-bit video.

[0108] Embodiment 4

[0109] In some embodiments, the above-described embodiments are used for a horizontal subblock boundary when a subblock boundary criterion (a.k.a., “motion difference” criterion) based on an absolute difference between a component the motion vector of the current subblock and a corresponding component of the motion vector of the above neighbouring subblock, is greater than or equal to a threshold. That is, for example, if the absolute difference between the motion vector of the current subblock and the motion of the above neighbouring subblock is not greater than or equal to a threshold, then hBoundaryDistortion is set to zero.

[0110] In some embodiments, the above-described embodiments are used for a vertical subblock boundary when a subblock boundary criterion based on absolute difference between themotion of current subblock and the motion of the left neighbouring subblock is greater or equal to a threshold.

[0111] One example threshold is 16 when motion is given in 16-pel accuracy, e.g. corresponding to a motion of 1 sample.

[0112] In one specific example, let mvCand denote a first MV of a candidate MV pair of the current subblock, and let mvNb denote a first MV of a MV pair of the neighbouring subblock (left or above neighbouring subblock). The motion difference can be calculated as taking the absolute difference between the corresponding components of mvCand and mvNb, i.e., |mvCand.x – mvNb.x| and |mvCand.y – mvNb.y|. When at least one of the absolute differences is greater or equal to the thrheshold, i.e., |mvCand.x – mvNb.x| >= threshold or |mvCand.y – mvNb.y| >= threshold, it is then determined that there exists sufficient motion difference.

[0113] Embodiment 5

[0114] Here an example order of steps to take for using subblock boundary distortion in subblock bilateral matching to make sure that selected subblock motion vectors always produce a subblock aligned with neighboring subblocks is shown. That is, below are steps for how boundary distortion is used to select a motion vector candidate.

[0115] Firstly, spatial activity for the reference subblocks centered at the block motion vector is determined. In one embodiment, the reference subblocks can be identified by an initial MV pair of the current block. In another embodiment the reference subblocks can be identify a candidate MV pair. If the spatial activity is lower than a threshold (e.g., there is a possibility of having an unreliable), then implement one of the above embodiments, otherwise do not. In such cases the subblock distortion can be added when some other criterions (e.g., as described in embodiments 3 and 4) are fulfilled as part of the search of best candidate MVs.

[0116] When the subblocks risk to get unreliable motion vectors, each candidate motion vector for the subblock is also compared regarding the absolute difference between the current subblock’s motion vector components with corresponding motion vector components of the above and the left subblocks motion vectors.

[0117] If the absolute difference for one component is greater than a first threshold (e.g,T1=16) across a subblock boundary, then boundary checks are made for that subblock boundary.

[0118] If boundary checks on neighbouring samples indicates that there not is a true edge based on a second threshold comparison (e.g., T2=15) along the subblock boundary, then subblock boundary distortion is determined across that subblock boundary.

[0119] If the subblock boundary distortion is greater than a third threshold (e.g., T3=15), then the subblock boundary distortion is added to the block matching distortion and the motion that minimizes the total distortion is selected.

[0120] Embodiment 6

[0121] In some embodiments, when the boundary distortion on the horizontal subblock boundary is greater than a threshold, then the subblock is divided into two horizontal subblocks. One example threshold is 15 (per sample). In some embodiments the threshold is dependent on QP such that it is greater for lower QP than higher QPs.

[0122] Similarly, when the boundary distortion on the vertical subblock boundary is greater than a threshold, then the subblock is divided into two vertical smaller blocks.

[0123] If both horizontal and vertical subblock boundary have sufficient boundary distortion, then the subblock is divided into 4 smaller blocks.

[0124] One example is to reuse one of the subblock motions as the motion for respective smaller block that gives least distortion.

[0125] Another example is to add additional motion search for the smaller parts. That is, because, the subblock is divided into smaller blocks, split into two smaller blocks, half vertically or horizontally or into 4 smaller blocks, quarter blocks, one can search for motion vector among candidate motion vector for each smaller block that minimizes distortion for respective smaller block.

[0126] Embodiment 7

[0127] In some embodiments, the selection of subblock motion vectors may comprise the following steps.1) Select a first candidate refined MV pair, candRefinedMV, by minimizing the current distortion (distortion from template-based motion derivation or block matching motionderivation) within a first set of MV candidates.2) Select a final refined MV pair, refinedMV, by minimizing the subblock boundary distortion within a second set of MV candidates. The second set of MV candidates may include MV candidates that are within a window around the first candidate refined MV pair candRefinedMV. The final refined MV pair, refinedMV, is then further used to produce motion compensated prediction samples of the current block.

[0128] In some embodiments, the selection of subblock motion vectors may comprise the following steps.1) Select a number (N) of candidate refined MV pairs by minimizing the current distortion (distortion from template-based motion derivation or block matching motion derivation) within a first set of MV candidates. N is preferred to be greater than 0 but smaller than the total number of MV candidates in the first set.2) Select a final refined MV pair, refinedMV, from the N refined MV pairs by minimizing a subblock boundary distortion. The final refined MV pair, refinedMV, is then further used to produce motion compensated prediction samples of the current block.

[0129] Embodiment 8

[0130] In some embodiments, when neighboring samples indicate that the motion boundary may be aligned with a true edge (see, e.g., embodiment 3), then use of OBMC across a motion boundary is avoided or used carefully (i.e., having a relatively small weight than usual on the neighbouring motion compared to the current subblock motion or completely avoid using OBMC in this case).

[0131] In a possible alternative of the OBMC control, when i) the neighboring samples indicates that the boundary of the current block is likely to correspond to a true edge and ii) blending the current prediction samples with the prediction samples generated using the neighboring motion, the weighting factor that associates with the prediction samples that are generated using the neighboring motion is reduced (e.g., the weighting factor “w2” described above). In other words, OBMC tries to reduce boundary differences by using a weighted average of motion compensated prediction of boundary samples of the current block by current blocks motion vector and neighboring block motion vector. Accordingly, if, for example embodiment 3 indicates a true edge, then the weight on the motion compensated prediction from the neighboring motion vector is reduced.

[0132] Embodiment 9

[0133] The above-described embodiments can be used to control the usage of template matching for blocks or subblocks.

[0134] For example, when the neighboring samples indicates the boundary may be aligned with a true edge, the template matching process is not invoked for the current block. As the template matching uses the sample information from the neighboring block to refine the motion for the current block, it is based on the assumption that the current block and the neighboring block share similar characteristics. And for the case when there is a true edge between the current block and the neighboring block, it is more likely that the characteristics of the current block will be different from the neighboring block and use the template matching process may result in unsuitable motion.

[0135] In some embodiments, the selection of template area is based on neighbouring samples. If the neighbouring samples indicate that the border between the above block and the current block is a true edge, then the template area above of the current block is not used. If the neighbouring samples indicate that the border between the left block and the current block may be aligned with a true edge, then the template area left of the current block is not used. In case at least one template area can be used the template matching is performed using the allowed template areas to find the best motion for the current subblock.

[0136] FIG.9 is a flowchart illustrating process 900, according to some embodiments, for obtaining, for a current subblock, a subblock of motion compensated predicted samples. The process 900 may begin with step s902.

[0137] Step s902 comprises selecting a motion vector for the current subblock from a set of candidate motion vectors based on a boundary distortion criterion. The selecting comprises: for each candidate MV pair, determining a distortion value for the candidate MV pair, wherein determining a distortion value for each candidate MV pair comprises: determining a first distortion value, DV1, for a first candidate MV pair and determining a second distortion value for a second candidate MV pair, wherein DV1 is equal to a sum of a current subblock distortion value, CurrentDistortion, and a subblock boundary distortion value, BoundaryDistortion. Theselecting comprises determining the candidate MV pair having the lowest distortion value, wherein the determined candidate MV pair is the MV pair selected for the current subblock of samples.

[0138] Step s904 comprises using the selected MV pair to produce the subblock of motion compensated prediction samples. For example, this step may use the well-known method of producing the motion compensated prediction samples.

[0139] FIG.10 is a flowchart illustrating process 1000, according to some embodiments, for obtaining, for a current subblock, a subblock of motion compensated predicted samples. The process 1000 may begin with step s1002.

[0140] Step s1002 comprises determining if there is a risk that the current subblock will have an unreliable motion vector (e.g., determining spatial activity for the reference subblocks centered at the block motion vector is determined, and determining whether the spatial activity is lower than a threshold). If there is such a risk, the process proceeds to step s1004.

[0141] Step s1004 comprises determining if the absolute difference between a motion vector component of the current subblock and a corresponding motion vector component of a neighboring subblock differs more than a second threshold (e.g., 16). If so, the process proceeds to step s1006.

[0142] Step s1006 comprises determining whether there is a true edge along a subblock boundary (e.g., boundary between the current subblock and an above or left subblock). If there is a true edge along the subblock boundary, then the process proceeds to step s1008.

[0143] Step s1008 comprises determining subblock boundary distortion across that subblock boundary.

[0144] Step s1010 comprises determining if the subblock boundary distortion is greater than a third threshold. If the subblock boundary distortion is greater than a third threshold, then the process proceeds to step s1012.

[0145] Step s1012 comprises adding the subblock boundary distortion to the block matching distortion.

[0146] FIG.11 is a block diagram of an apparatus 1100 for implementing encoder 702 and / or decoder 704, according to some embodiments. When apparatus 1100 implementsencoder 702, apparatus 1100 may be referred to as an encoder apparatus, and when apparatus 1100 implements decoder 704, apparatus 1100 may be referred to as a decoder apparatus. As shown in FIG.11, apparatus 1100 may comprise: processing circuitry (PC) 1102, which may include one or more processors (P) 1155 (e.g., one or more general purpose microprocessors and / or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (i.e., encoder apparatus 1100 may be a distributed computing apparatus); at least one network interface 1148 (e.g., a physical interface or air interface) comprising a transmitter (Tx) 1145 and a receiver (Rx) 1147 for enabling apparatus 1100 to transmit data to and receive data from other nodes connected to a network 110 (e.g., an Internet Protocol (IP) network) to which network interface 1148 is connected (physically or wirelessly) (e.g., network interface 1148 may be coupled to an antenna arrangement comprising one or more antennas for enabling encoder apparatus 1100 to wirelessly transmit / receive data); and a storage unit (a.k.a., “data storage system”) 1108, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where PC 1102 includes a programmable processor, a computer readable storage medium (CRSM) 1142 may be provided. CRSM 1142 may store a computer program (CP) 1143 comprising computer readable instructions (CRI) 1144. CRSM 1142 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 1144 of computer program 1143 is configured such that when executed by PC 1102, the CRI causes encoder apparatus 1100 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, encoder apparatus 1100 may be configured to perform steps described herein without the need for code. That is, for example, PC 1102 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and / or software.

[0147] Summary of Various Embodiments A1 . A method for obtaining, for a current matrix (e.g., block or subblock), a matrix of motion compensated predicted samples, the method comprising:based on a boundary distortion criterion, selecting a motion vector for the current matrix from a set of candidate motion vectors; and using the motion vector selected for the current matrix to produce the matrix of motion compensated prediction samples (e.g., this step uses the well known method of producing the motion compensated prediction samples). A2. The method of embodiment A1, wherein the set of candidate motion vectors, MVs, comprises a set of candidate MV pairs, selecting a motion vector for the current matrix comprises selecting, from the set of candidate MV pairs, an MV pair for the current matrix. A3. The method of embodiment A2, wherein selecting an MV pair for the current matrix comprises for each candidate MV pair included in the set of candidate MV pairs, determining a distortion value (e.g., the TotalDistortion value) for the candidate MV pair; and determining the candidate MV pair having the lowest distortion value, wherein the determined candidate MV pair having the lowest distortion value is the MV pair selected for the current matrix. A4. The method of embodiment A3, wherein determining a distortion value for each candidate MV pair comprises: determining a first distortion value for a first candidate MV pair, and determining a second distortion value for a second candidate MV pair. A5. The method of embodiment A4, wherein the first distortion value, DV1, for the first candidate MV pair is a function of i) a current block distortion value (denoted “CurrentDistortion”) and ii) a subblock boundary distortion value (denoted “BoundaryDistortion”). A6. The method of embodiment A5, wherein DV1 = CurrentDistortion + BoundaryDistortion.A7. The method of embodiment A6, wherein BoundaryDistortion is a function of: a scaling factor, SF, a vertical boundary distortion value, vBD, and a horizontal boundary distortion value, hBD. A8. The method of embodiment A7, wherein BoundaryDistortion = SF × ((vBD + hBD + 2)>>2). A9. The method of embodiment A7 or A8, wherein vBD = (sumVBSDiff)>>4+4*(vBGDiff+vBLDiff), hBD = (sumHBSDiff)>>4+4*( hBGDiff+hBLDiff), sumHBSDiff = ∑ெ^ୀି^^^^ ^^ ^^ (Ai,0- Bi,N-1),, vBGDiff =– A0,N-1), vBLDiff = abs(DM-1,0 –2* DM-1,N / 2 + DM-1,N-1 – (A0,0 – 2*A0,N / 2 + A0,N-1)), hBGDiff = abs(B0,N-1– BM-1,N-1– (A0,0– AM-1,0), and hBLDiff = abs(B0,N-1 –2* BM / 2,0 + BM-1,0 – (A0,0 – 2*AM / 2,0 + AM-1,0)), wherein A is the current matrix, A is an MxN matrix of values Ai,j for i=0 to M-1 and j=0 to N-1, B is a neighbouring matrix above A, B is an MxN matrix of values Bi,j for i=0 to M-1 and j=0 to N-1, D is a neighbouring matrix left of A, and D is an MxN matrix of values Bi,j for i=0 to M-1 and j=0 to N-1. A10. The method of embodiment A9, wherein A is derived based on CPB1 and / or CPB2, where CPB1 is a first candidate prediction block derived from the first MV of the first candidate MV pair and CPB2 is a second candidate prediction block derived from the second MV of the first candidate MV pair.A11. The method of embodiment A10, wherein A is equal to the average of CPB1 and CPB2, A is equal to a weighted average of CPB1 and CPB2, A is equal to CPB1, or A is equal to CPB2. A12. The method of any one of embodiment A4-A11, wherein determining the second distortion value for the second candidate MV pair comprises determining whether the second distortion value should be determined using a second subblock boundary distortion value. A13. The method of embodiment A12, wherein determining whether the second distortion value should be determined using a second subblock boundary distortion value comprises: determining whether a spatial activity for a first reference block identified by a first MV of the second candidate MV pair satisfies a first criterion (e.g., determine whether the spatial activity is less than a first threshold); determining whether an absolute difference between i) a first component of the first MV of the second candidate MV pair and ii) a corresponding component of a MV of a neighboring subblock satisfies a second criterion (e.g., determine whether the absolute difference is greater than a second threshold); determining whether a boundary between the current subblock and a neighboring subblock is a true edge; and / or determining whether a subblock boundary distortion value for the second candidate MV pair satisfies a third criterion. A14. The method of claim A13, further comprising: adding the subblock boundary distortion value to a current block distortion value to produce a total distortion value as a result of: determining that the spatial activity for the first reference block satisfies the first criterion;determining that the absolute difference between i) the first component of the first MV of the second candidate MV pair and ii) the corresponding component satisfies the second criterion; determining that the boundary between the current subblock and a neighboring subblock is not a true edge; and / or determining that the subblock boundary distortion value satisfies the third criterion. B1. A computer program (1143) comprising instructions (1144) which when executed by processing circuitry (1102) cause the processing circuitry (1102) to perform the method of any one of the above embodiments. B2. A carrier containing the computer program of embodiment B1, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (1142). C1. An apparatus (1100) for encoding a picture, the apparatus configured to perform a method comprising: C2. The apparatus of embodiment C1, wherein the apparatus is further configured to perform the method of any one of embodiments A2-A14. D1. An apparatus (1100), the apparatus comprising: a memory (1142); and processing circuitry (1102) coupled to the memory (1142), wherein the apparatus (1100) is configured to perform the method of any one of embodiments A1-A14.

[0148] Results

[0149] As noted above, the embodiments provide better subjective quality in form of more coherent subblocks and avoiding dislocated subblocks as in approaches where motion isderived on decoder side such as in DMVR. In some cases, it can also provide better objective performance in terms of BDR.

[0150] Below when applied on top of multi-pass DMVR in ECM-9.0 with objective performance in terms of BDR for a higher QP range (QP 40, 43, 47 and 50) compared to ECM- 9.0. A negative BDR means a reduction in bitrate at same PSNR. The method can reduce the bitrate by about 0.1% on higher QPs at same PSNR luma PSNR (BDR Y).

[0151] RaceHorses_416x240_30: BDR Y -0.165898 U 1.826454 V 1.459041

[0152] BasketballPass_416x240_50: BDR Y -0.249260 U -0.748526 V 0.033722

[0153] BQSquare_416x240_60: BDR Y -0.135372 U -0.120413 V -0.158208

[0154] BlowingBubbles_416x240_50: BDR Y 0.002110 U -0.470081 V -0.197244

[0155] BasketballDrill_832x480_50: BDR Y -0.183240 U 0.028203 V -0.019215

[0156] BQMall_832x480_60: BDR Y -0.077460 U -0.326442 V -0.582697

[0157] PartyScene_832x480_50: BDR Y -0.052509 U -0.064285 V -0.066123

[0158] RaceHorses_832x480_30: BDR Y -0.121897 U 0.429449 V 0.551223

[0159] MarketPlace_1920x1080_60fps_10bit_420: BDR Y -0.042158 U 0.160112 V - 0.152783

[0160] RitualDance_1920x1080_60fps_10bit_420: BDR Y -0.115280 U 0.126181 V 0.129702

[0161] Cactus_1920x1080_50: BDR Y 0.023667 U 0.176786 V 0.084542

[0162] BasketballDrive_1920x1080_50: BDR Y -0.060464 U -0.425289 V 0.094443

[0163] BQTerrace_1920x1080_60: BDR Y -0.099447 U -0.093390 V -0.076694

[0164] AVG: BDR Y -0.098247 U 0.038366 V 0.084593

[0165] For QP 22, 27, 32 and 37 the method provides similar performance compared to ECM-9.0:

[0166] BasketballDrill_832x480_50: BDR Y 0.039411 U -0.030885 V -0.028959

[0167] BQMall_832x480_60: BDR Y 0.007278 U -0.242964 V -0.219833

[0168] PartyScene_832x480_50: BDR Y 0.041937 U 0.007310 V -0.022749

[0169] RaceHorses_832x480_30: BDR Y 0.053528 U 0.054025 V 0.041668

[0170] BasketballPass_416x240_50: BDR Y -0.000496 U -0.094225 V -0.016754

[0171] BQSquare_416x240_60: BDR Y -0.006448 U -0.064699 V 0.077713

[0172] BlowingBubbles_416x240_50: BDR Y -0.042204 U -0.101248 V 0.011295

[0173] RaceHorses_416x240_30: BDR Y -0.068343 U -0.057497 V -0.112463

[0174] AVG: BDR Y 0.003083 U -0.066273 V -0.033760

[0175] Conclusion

[0176] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0177] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

[0178] Additional Disclosure:

[0179] Multi-pass DMVR in ECM can, similarly to DMVR in VVC, sometimes experience a dislocated subblock due to an unreliable motion vector. It is therefore proposed to include additional criterions for the bilateral block matching in cases where there is a risk of having an unreliable motion vector to make sure that selected motion vectors also results in subblocks that are aligned with neighbouring subblocks. To that end we inherit low activity check, subblock motion vector difference check, and boundary difference checks from JVET-AD0045. When there is a risk for unreliable motion vectors, a boundary distortion metric is added to the block matching SAD to bias the selection of motion vectors such that they produce subblocks that are aligned with above and left neighbouring subblocks.

[0180] Objective performance (BDR) versus ECM-9.0: RA: Class C and D: BDR Y 0.003083 U -0.066273 V -0.033760 For higher QP range QP= 40, 43, 47, 50 RA: Class B, C and D: Y -0.098247 % U 0.038366% V 0.084593%

[0181] Introduction

[0182] In JVET-AD0045 (reference [1]) an encoder only solution was suggested to fix visual artifacts from DMVR. This was then added to the VVC reference software. In ECM [2], DMVR has been further developed to multi-pass DMVR which combine DMVR refinements on both block and subblock basis, and also use a larger search range than for VVC followed with refinements by BDOF in 8x8 subblocks.

[0183] Still DMVR multi-pass in ECM can produce visual artifacts similar to those for VVC for larger block sizes such as 32x32 or greater. Since ECM is work in progress and not a standard it is better to adress the issue normativly rather than trying to avoid use of DMVR when there is a risk to get unreliable motion vectors. This can then potentially also rather improve coding efficiency rather than reduce coding efficiency especially at higher QPs.

[0184] Proposal

[0185] A proposal is to add subblock boundary distortion to the subblock bilateral matching distortion to make sure that selected subblock motion vectors always produce a subblock aligned with neighboring subblocks.

[0186] Firstly, spatial activity for the reference subblocks centered at the block motion vector is determined, and, if the spatial activity is lower than a threshold, then the subblock risks to have an unreliable motion vector. When the subblocks risk getting unreliable motion vectors, each candidate motion vector for the subblock is also compared regarding the absolute differencebetween the current subblock’s motion vector components with corresponding motion vector components of the above and the left subblocks motion vectors. If the absolute difference for one component is greater than 16 across a subblock boundary then also boundary checks are made for that subblock boundary. If boundary checks on neighbouring samples indicates that there not is a true edge along the subblock boundary, then subblock boundary distortion is determined across that subblock boundary. If the subblock boundary distortion is greater than a threshold the subblock boundary distortion is added to the block matching distortion and the motion that minimizes the total distortion is selected.

[0187] Results

[0188] Subjective impact: the proposal can improve subjective quality by a robustification of the motion vector derivation for multi-pass DMVR.

[0189] Objective impact: The performance has been compared both for CTC at normal QP range and CTC at a higher QP range (40, 43, 47 and 50). For CTC at normal QP range there is very similar objective performance but for the higher QP range some objective benefit can be shown.

[0190] It has been shown that robustification of the derivation of motion vectors in multi- pass DMVR in cases where it is a risk to get unreliable motion vectors can maintain objective performance of ECM for CTC QP range and can achieve objective improvements for a higher QP range where DMVR also is used more often.

[0191] References [1] K. Andersson, R, Yu, “AHG10: Encoder MV selections and DMVR revisited”, JVET- AD0045, Antalya Turkey, April 2023. [2] ECM-9.0, available at https: / / vcgit.hhi.fraunhofer.de / ecm / ECM / - / tags / ECM-9.0. [3] M. Karczewicz, Y. Yan, “Common Test Conditions and evaluation procedures for enhanced compression tool testing”, JVET-Y2017, Jan.2022.

Claims

CLAIMS 1. A method for obtaining, for a current subblock of samples, a subblock of motion compensated predicted samples, the method comprising: based on a boundary distortion criterion, selecting a motion vector, MV, pair for the current subblock of samples from a set of candidate MV pairs, wherein the selecting comprises: for each candidate MV pair, determining a distortion value for the candidate MV pair, wherein determining a distortion value for each candidate MV pair comprises: determining a first distortion value, DV1, for a first candidate MV pair and determining a second distortion value for a second candidate MV pair, wherein DV1 is equal to a sum of a current subblock distortion value, CurrentDistortion, and a subblock boundary distortion value, BoundaryDistortion; determining the candidate MV pair having the lowest distortion value, wherein the determined candidate MV pair is the MV pair selected for the current subblock of samples; and using the selected MV pair to produce the subblock of motion compensated prediction samples.

2. The method of claim 1, wherein BoundaryDistortion is a function of: a scaling factor, SF, a vertical boundary distortion value, vBD, and a horizontal boundary distortion value, hBD.

3. The method of claim 2, wherein BoundaryDistortion = SF × ((vBD + hBD + 2)>>2).

4. The method of any of claims 2-3, wherein vBD = (sumVBSDiff)>>4+4*(vBGDiff+vBLDiff), hBD = (sumHBSDiff)>>4+4*( hBGDiff+hBLDiff), sumHBSDiff = ∑ெ^ୀି^^^^ ^^ ^^ (Ai,0- Bi,N-1),sumVBSDiff =∑ே^ୀି^^^^ ^^ ^^ (A0,j - DM-1,j), vBGDiff =– A0,N-1), vBLDiff = abs(DM-1,0–2* DM-1,N / 2+ DM-1,N-1– (A0,0– 2*A0,N / 2+ A0,N-1)), hBGDiff = abs(B0,N-1 – BM-1,N-1 – (A0,0 – AM-1,0), and hBLDiff = abs(B0,N-1 –2* BM / 2,0 + BM-1,0 – (A0,0 – 2*AM / 2,0 + AM-1,0)), wherein A is the current subblock, A is an MxN subblock of values Ai,j for i=0 to M-1 and j=0 to N-1, B is a neighbouring subblock above A, B is an MxN subblock of values Bi,j for i=0 to M-1 and j=0 to N-1, D is a neighbouring subblock left of A, and D is an MxN subblock of values Bi,j for i=0 to M-1 and j=0 to N-1.

5. The method of claim 4, wherein A is derived based on CPB1 and / or CPB2, where CPB1 is a first candidate prediction subblock derived from the first MV of the first candidate MV pair and CPB2 is a second candidate prediction subblock derived from the second MV of the first candidate MV pair.

6. The method of claim 5, wherein A is equal to the average of CPB1 and CPB2, A is equal to a weighted average of CPB1 and CPB2, A is equal to CPB1, or A is equal to CPB2.

7. The method of any one of claims 1-6, wherein determining the second distortion value for the second candidate MV pair comprises determining whether the second distortion value should be determined using a second subblock boundary distortion value.

8. The method of claim 7, wherein determining whether the second distortion value should be determined using a second subblock boundary distortion value comprises:determining whether a spatial activity for a first reference subblock identified by a first MV of the second candidate MV pair satisfies a first criterion; determining whether an absolute difference between i) a first component of the first MV of the second candidate MV pair and ii) a corresponding component of a MV of a neighboring subblock satisfies a second criterion; determining whether a boundary between the current subblock and a neighboring subblock is a true edge; and / or determining whether a subblock boundary distortion value for the second candidate MV pair satisfies a third criterion.

9. The method of claim 8, further comprising: adding the subblock boundary distortion value to a current subblock distortion value to produce a total distortion value as a result of: determining that the spatial activity for the first reference subblock satisfies the first criterion; determining that the absolute difference between i) the first component of the first MV of the second candidate MV pair and ii) the corresponding component satisfies the second criterion; determining that the boundary between the current subblock and a neighboring subblock is not a true edge; and / or determining that the subblock boundary distortion value satisfies the third criterion.

10. A computer program (1143) comprising instructions (1144) which when executed by processing circuitry (1102) cause the processing circuitry (1102) to perform the method of any one of the above claims.

11. A carrier containing the computer program of claim 10, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (1142).

12. An apparatus (1100) for encoding a picture, the apparatus configured to perform the method of any one of claims 1-9.

13. An apparatus (1100), the apparatus comprising: a memory (1142); and processing circuitry (1102) coupled to the memory (1142), wherein the apparatus (1100) is configured to perform the method of any one of claims 1-9.

Images