Motion vector derivation based on block boundary distortion

Abstract

There is provided a method for obtaining a current block of motion compensated predicted samples. The method comprises selecting (s902) a motion vector from a set of candidate motion vectors, the set of candidate motion vectors comprising a first candidate motion vector and a second candidate motion vector. The selecting of a motion vector from the set of candidate motion vectors comprises calculating a first misalignment distortion value for the first candidate motion vector. The calculating of the first misalignment distortion value for the first candidate motion vector is based on differences of sample values on the boundary between the first neighboring block and the current block. The method comprises using (s904) the selected motion vector to produce the block of motion compensated prediction samples.

Description

MOTION VECTOR DERIVATION BASED ON BLOCK BOUNDARY DISTORTIONTECHNICAL FIELD

[0001] This disclosure relates to methods and apparatus for motion vector derivation.BACKGROUND

[0002] VVC and ECM

[0003] Versatile Video Coding (VVC) is a block-based video codec standardized by ITU-T and MPEG. Enhanced Coding Model (ECM) is an exploratory codec which is currently under development. The aim of ECM is to demonstrate and try providing evidence of video coding capabilities beyond VVC. The current ECM version is ECM- 10.0.

[0004] Video and Picture

[0005] A video (a.k.a., “video sequence”) comprises of a series of pictures. In VVC, each picture is identified with a picture order count (POC) value. The POC value also represents the display order of the picture. A picture with a smaller POC value is displayed before another picture with a larger POC value.

[0006] Components

[0007] It is common that each picture consists of three components; one luma component Y where the sample values are luma values and two chroma components Cb and Cr, where the sample values are chroma values. Each component can be described as a two-dimensional rectangular array of sample values. It is also common that the dimensions of the chroma components are smaller than the luma components by a factor of two in each dimension. For example, the size of the luma component of an HD picture would be 1920x1080 and the chroma components would each have the dimension of 960x540. Components are sometimes referred to as color components.

[0008] Blocks, Subblocks, and Units

[0009] A block is a two-dimensional (2D) matrix of sample values (or “samples” forshort). A block may be divided into two or more subblocks, where each subblock is a matrix of samples. In video coding, each component of a picture is split into blocks and the coded video bitstream consists of a series of coded blocks. It is common in video coding that pictures are split into units that cover a specific area of the picture.

[0010] Each unit consists of all blocks from all components that make up that specific area of the picture and each block belongs fully to one unit. The Coding Unit (CU) in VVC is an example of a unit. In VVC, the CUs may be split recursively to smaller CUs. The CU at the top level is referred to as the coding tree unit (CTU).

[0011] A CU usually contains three coding blocks, i.e., one coding block for luma and two coding blocks for chroma.

[0012] In VVC, the CUs can have size of 4x4 up to 128x128. In current ECM, the CUs can have size of 4x4 up to 256x256.

[0013] Parameter sets, slice headers, and picture headers

[0014] VVC specifies three types of parameter sets: the picture parameter set (PPS), the sequence parameter set (SPS), and the video parameter set (VPS). The PPS contains data that is common for all units of a picture, the SPS contains data that is common for a coded layer video sequence (CLVS), and the VPS contains data that is common for multiple CLVSs, e.g., data for multiple layers in the bitstream.

[0015] The concept of slices divides the picture into independently coded slices, where decoding of one slice in a picture is independent of other slices of the same picture. Each slice has a slice header comprising syntax elements. Decoded slice header values from these syntax elements are used when decoding the slice.

[0016] In VVC, a coded picture contains a picture header. The picture header contains parameters that are common for all slices of the coded picture.

[0017] Intra prediction

[0018] In intra prediction, also known as spatial prediction, a current block is predicted using previous decoded blocks within the same picture. The samples from the previouslydecoded blocks within the same picture are used to predict the samples inside the current block. A picture consisting of only intra-predicted blocks is referred to as an intra picture.

[0019] Inter prediction

[0020] In inter prediction, also known as temporal prediction, a current block of the current picture is predicted using blocks from previously decoded pictures (these blocks are referred to as reference blocks). The samples from the reference blocks in the previously decoded pictures are used to predict the samples inside the current block. A picture that comprises one or more inter-predicted blocks is referred to as an inter-picture. The previous decoded pictures used for inter prediction are referred to as reference pictures.

[0021] The location of a referenced block inside a reference picture is indicated using a vector (i.e., a set of values) (which is referred to as “motion vector (MV)”). Each MV consists of two values: an x value (a.k.a., x component) and y value (a.k.a., y component) which represents the displacements between current block and the referenced block in x or y dimension. The value of a component may have a resolution finer than an integer position. When that is the case, a filtering (typically interpolation) is done to calculate values used for prediction. FIG. 1 shows an example of a MV for the current block. The example MV consist of the values 2, 1.

[0022] An inter picture may use several reference pictures. The reference pictures are usually put into two reference picture lists, L0 and LI. The reference pictures that are output before the current picture are typically the first pictures in L0. The reference pictures that are output after the current picture are typically the first pictures in LI.

[0023] Inter predicted blocks can use one of two prediction types, uni- and bi-prediction. Uni -predicted block predicts from one reference picture, either using L0 or LI. Bi-prediction predicts from two reference pictures, one from L0 and the other from LI. FIG. 2 shows an example of the prediction types.

[0024] Picture coding type (Low delay picture and non-low delay picture)

[0025] A low delay picture is a picture that has all its reference pictures displayed before the picture. In other words, for a low delay picture, all its reference pictures have smaller POC values than the current POC.

[0026] A non-low delay picture is a picture that has at least one of its reference pictures displayed after the picture. In other words, a non-low delay picture has at least one reference picture with a larger POC value than the current POC.

[0027] Fractional MVs, Interpolation filter and MV rounding

[0028] The value of the MV’s x or y component may corresponds to a sample position which has finer granularity than integer (sample) position. Those positions are also referred to as fractional (sample) positions.

[0029] In VVC and current ECM, the MV can be at 1 / 16 sample position. FIG. 3 depicts several fractional positions in the horizontal (x-) dimension. The solid-square blocks represent integer positions. The circles represent 1 / 16-position. For example, MV = (4, 10) means the x component is at 4 / 16 position, the y component is at 10 / 16 position.

[0030] In video coding, a MV rounding process is sometimes used to convert a MV at one position to another target position. One example of rounding is to round a fractional MV position to the nearest integer position.

[0031] When an MV is at a fractional position, filtering (typically interpolation) is done to calculate the sample values at those positions. In VVC, the length (number of filter taps) of the interpolation filter for luma component is 8, as shown in table 1 below. In ECM, the length of the interpolation filter for luma component has been increased to 12.TABLE 1

[0032] Residual, transform and quantization

[0033] A residual block is generated by computing the difference between samples of a source block (a.k.a., “input block”), which contains original samples, and samples of a prediction block (e.g., an inter-prediction block or an intra-prediction block). To remove further redundancy, the residual block is then typically compressed by a spatial transform, thereby producing transform coefficients. The transform coefficients are then quantized based on a quantization parameter (QP) to control the fidelity of the residual block and thus also the bitrate required to compress the block. A coded block flag (CBF) is used to indicate if there are any non-zero quantized transform coefficients. All coding parameters are then entropy coded at the encoder and decoded at the decoder. If the coded block flag is one, a reconstructed block can then be derived by inverse quantization and inverse transformation of the quantized transform coefficients and then add that to the prediction block. If the coded block flag is zero, the reconstructed block is identical to the prediction block.

[0034] Inter prediction information / Motion information

[0035] For an inter block inside an inter picture in VVC, its inter prediction information consists of the following three elements:

[0036] (1) a reference picture list flag (RefPicListFlag) which signals which reference picture list is used for the block (when the value of the flag is equal to 0, it means only L0 is used for predicting the current block, when the value of the flag is equal to 1, it means only LI is used for predicting the current block, and when the value of the flag is equal to 2, it means both L0 and LI are used for predicting the current block);

[0037] (2) a reference picture index (RefPicIdx) per reference picture list used (the index signals which reference picture inside the reference list to be used for predicting the current block), and

[0038] (3) a motion vector (MV) per reference picture used, which signals the position inside the reference picture that is used for predicting the current block.

[0039] The inter prediction information is also referred to as motion information. The decoder stores the motion information for each inter block. In other words, an inter block maintains its own motion information.

[0040] Encoder decision and Rate Distortion (RD) cost

[0041] In practice, for an encoder to decide the best prediction mode for a current block, the encoder would evaluate all the possible prediction modes for the current block and select the prediction mode that yields the smallest Rate-Distortion RD) cost.

[0042] The RD cost is calculated as D + * R. D (Distortion) measures the difference between the reconstructed block and the corresponding source block. One commonly used metric for calculating D is the sum of squared error SSE =(v y))2, where the and are the sample values in the two blocks F and G respectively. R (rate) is usually an estimation of the bits to be spent on encoding the mode, and is a trade-off parameter between R and D.

[0043] Motion information signaling

[0044] VVC and ECM includes several methods for implicit signaling of motion information for each block, including the merge method and the subblock merge method. A common motivation behind the implicit methods is to inherit or reuse motion information fromneighboring coded blocks. This often works in practice due to spatial correlation of close-by blocks, i.e., the fact that nearby blocks often behave similarly.

[0045] Merge (block merge) method and merge mode

[0046] The merge method derives a set of motion information from previously decoded blocks and use the derived motion information for generating the samples of the entire block. The merge method is sometimes referred to as the block merge method.

[0047] The merge method first generates a list of motion information candidates. The list is also referred to as the merge list. The candidates are derived from previously coded blocks. These previously coded blocks can be spatially adjacent neighboring blocks or temporal collocated blocks relative to the current block. FIG. 4 shows the spatial neighboring blocks: left (L), top(T), top-right(TR), left-bottom (LB) and top-left (TL).

[0048] The merge list construction process usually checks the previously coded blocks in a predefined order, for example: T, L, TR, LB, then TL. For each previously coded block being checked, if this previously coded block is inter coded and its motion information has no duplicates in the list, then the motion information of this previously coded block is added to the merge list.

[0049] After the merge list is generated, one of the candidates inside the list is used to derive the motion information of the current block. The candidate selection process is done on the encoder side. An encoder would select a best candidate from the list and encode an index (merge index) in the bitstream to signal to a decoder. The decoder receives the index, it follows the same merge list derivation process as the encoder and uses the index to retrieve the correct candidate. The blocks that use the block merge method are sometimes referred to as blocks in merge mode.

[0050] In the current ECM, non-adjacent spatial blocks are also considered as sources of motion information during the merge list construction. FIG. 5 shows some examples (marked with NA1, NA2 and NA3) of those non-adjacent spatial blocks.

[0051] Subblock merge method

[0052] VVC and ECM also include the subblock merge method. It splits a current blockinto a number of subblocks and allows each subblock to have its own motion information. FIG. 6 shows an example of a current block and its subblocks. Each subblock maintains its own motion information. It should be noted that the subblocks are all rectangular.

[0053] Overlapped Block Motion Compensation (OBMC)

[0054] OBMC is a tool included in ECM which operates at the block boundaries or subblock boundaries of a current inter block. OBMC blends the current block or subblock’s prediction samples, P CUR, (generated using the current associated motion information) with another set of prediction samples, P_NB, which are generated using motion information from a neighboring block or a neighboring subblock at the current block or subblock boundaries. The OBMC blending process takes weighted average of P CUR and P_NB to produce a set of OBMC modified prediction samples, P OBMC. In other words, the set P CUR or P_NB each associate with a respective weighting factor, and the set of OBMC modified prediction samples P_OBMC is derived as wl*P_CUR + w2*P_NB (where wl + w2 = 1). The OBMC may give better prediction for samples that are close to the block or subblock boundary.

[0055] Bi-Directional Optical Flow (BDOF)

[0056] BDOF is a tool included in VVC and the current ECM that can be used to refine prediction samples that are generated from a Bi-MV. BDOF relies on optical flow estimation to derive a pair of refinement parameter (Vx, Vy) which can be further used to refine the prediction samples.

[0057] Decoder-side Motion Vector Refinement (DMVR)

[0058] DMVR is a tool included in VVC and the current ECM to refine motion vectors for a Bi-MV. DMVR operates on subblock level, usually 16x16. Different from BDOF, which relies on optical flow estimation, DMVR relies on bilateral matching of two reference blocks to refine the Bi-MV. The matching is based on SAD (sum of absolute differences). The DMVR searches within a window around the Bi-MV to find whether there exists another Bi-MV (Bi- MV’) that gives a better match between the L0 reference block and the LI reference block. If so, the Bi-MV’ is further used instead for generating the prediction samples of the current block. After that a subpel adjustment is made based on the SAD costs around the motion with leastSAD to determine subblock motion with sub pixel accuracy.

[0059] Multi-pass DMVR

[0060] In ECM, DMVR has been further evolved to use multi-pass optimization, first bilateral block matching in a search area to find the refinement of the merge motion that gives best match, then bilateral 16x16 subblock matching, then bi-directional optical flow on 8x8 subblocks to refine motion further. The multi-pass DMVR also allows for modification of only one of the bi-predictive motions. It is also allowed to use DMVR when BCW (other weightings than just average) is used.

[0061] Template matching (TM)

[0062] In ECM block or subblock motion can also be refined by matching a template outside the current block with a corresponding template on the reference picture. The search is limited to a small range to find a better motion without signaling additional motion information. This can also be used cascaded with the bilateral matching in multi-pass DMVR.SUMMARY

[0063] Certain challenges presently exist. For example, a problem with DMVR and methods that determine motion based on distortion only inside the current block (bilateral block matching) or only outside the current block (template matching) risk obtaining an inaccurate motion that results in a current block that does not fit well with the surrounding blocks. Visual artifacts can occur at block (e.g., subblock) boundaries and can also result in that some subblocks completely deviate in appearance from the surrounding subblocks due to completely wrong subblock motion in cases where the true motion is too large or cases where one of the references blocks is too smooth to find a good match.

[0064] Accordingly, according to a first aspect of the present disclosure, there is provided a method for obtaining a current block of motion compensated predicted samples. The method comprises selecting a motion vector from a set of candidate motion vectors, the set of candidate motion vectors comprising a first candidate motion vector and a second candidate motion vector. The method comprises using the selected motion vector to produce the block of motion compensated prediction samples, wherein the selecting of a motion vector from the set ofcandidate motion vectors comprises calculating a first misalignment distortion value, CM Distl, for the first candidate motion vector. The calculating of the first misalignment distortion value for the first candidate motion vector comprises calculating a boundary difference value using n and c and calculating the first misalignment distortion value using the boundary difference value, where n is a sample value that is from a first neighboring block neighboring the current block and that is on a boundary between the first neighboring block and the current block, and c is a first candidate motion compensated prediction sample value that was determined using the first candidate motion vector and that is located in the current block on the boundary between the first neighboring block and the current block; and / or calculating a first difference, dl, between nl and n2; calculating a second difference, d2, between cl and c2; calculating a third difference, d3, between dl and d2; and calculating the first misalignment distortion value using d3, where nl and n2 are sample values that are from the first neighboring block and that are on the boundary between the first neighboring block and the current block, cl is a first candidate motion compensated prediction sample value that was determined using the first candidate motion vector and that is located in the current block on the boundary between the first neighboring block and the current block, and c2 is a second candidate motion compensated prediction sample value that was determined using the first candidate motion vector and that is located in the current block on the boundary between the first neighboring block and the current block.

[0065] According to a second aspect of the present disclosure, there is provided a computer program comprising instructions which when executed by processing circuitry cause the processing circuitry to perform the method according to the first aspect.

[0066] According to a third aspect of the present disclosure, there is provided a carrier containing the computer program according to the second aspect, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

[0067] According to a fourth aspect of the present disclosure, there is provided an apparatus configured to perform the method according to the first aspect.

[0068] An advantage of the embodiments disclosed herein is that they provide more coherent subblocks and avoid dislocated subblocks as compared to approaches where motion is derived on decoder side such as in DMVR. In some cases, it can also provide better objective performance in terms of the bjontegaard delta rate (BDR) (bit rate reduction at equal quality) aswell as better subjective quality.BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0070] FIG. 1 shows an example of a motion vector (MV).

[0071] FIG. 2 illustrates uni-inter prediction and bi-inter prediction.

[0072] FIG. 3 depicts several fractional positions in the horizontal (x-) dimension.

[0073] FIG. 4 shows spatial neighboring blocks.

[0074] FIG. 5 shows some examples of non-adjacent spatial blocks.

[0075] FIG. 6 shows an example of a current block and its subblocks.

[0076] FIG. 7 illustrates a system according to some embodiments.

[0077] FIG. 8 illustrates an example current subblock of motion compensated values and its neighboring subblocks.

[0078] FIG. 9 is a flowchart illustrating a process according to some embodiments.

[0079] FIG. 10 is a schematic block diagram of an encoder according to an embodiment.

[0080] FIG. 11 is a schematic block diagram of a decoder according to an embodiment.

[0081] FIG. 12 is a block diagram of an apparatus according to some embodiments.DETAILED DESCRIPTION

[0082] FIG. 7 illustrates a system 700 according to an embodiment. System 700 includes an encoder 702 and a decoder 704, wherein encoder 702 is in communication with decoder 704 via a network 110 (e.g., the Internet or other network). Encoder 702 encodes a source video sequence 701 (e.g., encodes blocks of units of pictures of video sequence 701) into a bitstream comprising an encoded video sequence (e.g., encoded blocks) and transmits the bitstream to decoder 704 via network 708. In some embodiments, encoder 702 is not in communication with decoder 704, and, in such an embodiment, rather than transmittingbitstream to decoder 704, the bitstream is stored in a data storage unit for later use. Decoder 704 decodes the pictures included in the encoded video sequence to produce video data for display and / or for further image processing (e.g. a machine vision task). Accordingly, decoder 704 may be part of a device 703 having an image processor 705 and / or a display 706. The image processor 705 may perform machine vision tasks on the decoded pictures. The device 703 may be a mobile device, a set-top device, a head-mounted display, or any other device.

[0083] As noted above, the existing DMVR scheme can lead to obtaining an inaccurate motion that results in a current block that does not fit well with the surrounding blocks.

[0084] Accordingly, this disclosure provides a method that penalizes misalignment of at least one boundary of the current predicted subblock with respect to adjacent samples of a neighboring subblock. The penalization can be performed by considering a cumulative misalignment distortion (which is also sometimes referred to as a “subblock boundary distortion”) when determining the total distortion for each given candidate motion vector of the current subblock. The candidate motion vector, for a given range of motion, that minimizes the total distortion is then selected and used to produce a subblock of motion compensated prediction samples. Producing the subblock of motion compensated prediction samples can be performed using any well-known method.

[0085] In one embodiment, the cumulative misalignment distortion is based on a misalignment distortion that is based on a difference in characteristics (texture, sample activity variations) between the current subblock boundary and an adjacent neighboring subblock boundary. The difference in boundary characteristics is preferably phase sensitive (sensitive to movement) so that it’s not only required that the texture is same, but also aligned in phase. One example is pixel wise differences calculation on pairs of two samples at the block boundary both inside the current subblock and similarly on corresponding pairs of two samples in a neighboring subblock. For instance, the misalignment distortion may be based on a value equal to the absolute value of ((NO-N1)-(C1-C2)), where NO and N1 are samples along the block boundary in the bottom of above neighboring subblock and Cl and C2 are sample along the block boundary in the top of the current block. Any differences indicates that there is some misalignment of texture between current subblock and neighboring subblock. That is, to get a low distortion itsrequired to get same relation between sample values CO and Cl and between NO and Nl. If there is an increase in sample value between NO and Nl there should also be an increase in sample value between CO and Cl.

[0086] When the boundary samples are close to a constant value (low spatial activity) the difference in characteristics between the samples of the current subblock boundary and an samples of the adjacent subblock boundary cannot discriminate the correct phase / motion; in this situation there can still be a constant difference between the subblock boundaries which could cause a visual artifact, and, in this case, it can be favorable to include another misalignment distortion criteria that punishes any constant difference between the boundary samples. Accordingly, in another embodiment, the misalignment distortion is based on a pixel wise difference calculation for at least one sample at the block boundary but inside the current subblock and an adjacent sample in a neighboring subblock. Any differences indicates that there is some misalignment between current subblock and neighboring subblock.

[0087] The embodiments disclosed herein can be used solely at the encoder or the embodiments may be used at both the encoder and the decoder. The embodiments can be used for both blocks and subblocks. The embodiments described below are mainly described for subblocks but similar approach can be used for blocks in general. For samples, the embodiments mainly refer to predicted samples from previously coded pictures that have been motion compensated. The predicted samples used in the embodiments could either be after bi-prediction, or separately on the individual uni-directional predictions before bi-prediction. Neighboring samples could also be reconstructed samples.

[0088] Additional Details

[0089] For a current subblock (e.g., a current input subblock corresponding to a current region of a picture), the determination of its motion compensated prediction samples may involve the following three steps:

[0090] Step 1 : Determine whether the current subblock has a neighboring subblock that is adjacent to one of the current subblock’s boundaries.

[0091] Step 2: As a result of determining that a neighboring subblock is adjacent to oneof the current subblock’s boundaries, then, for each candidate motion vector (MV) pair (e.g., for each candidate MV offset that is used to derive an MV pair) for the current subblock: i) determine at least two samples from the neighboring subblock (the samples can either be a prediction sample or a reconstructed sample of the neighboring subblock), ii) determine at least two samples from the current subblock based on the candidate MV pair (e.g., based on a motion vector offset), iii) calculate a cumulative misalignment distortion for the candidate MV pair based on the samples.

[0092] Step 3: selecting a one of the candidate MV pairs based on a distortion criteria (e.g., select the candidate MV offset that has the smallest total distortion, wherein the total distortion is a function of the cumulative misalignment distortion).

[0093] For example, in one embodiment, selecting the candidate MV offset comprises: i) for each candidate MV offset in a set of candidate MV offset, determine a total distortion value (denoted “T Disf ’) for the MV offset, where the total distortion value is a function of a current block distortion (denoted “CB Dist”) (a.k.a., the block matching distortion) and the cumulative misalignment distortion (denoted “CM Disf ’) and ii) select from the set of candidates, the candidate having the lowest T Dist. An MV offset may be a vector (e.g., a vector of length 2 (a.k.a., 2-tuple)) or a scalar. For instance, in one embodiment, each candidate MV offset is 2- tuple comprising a first value (xOffset) specifying a horizontal offset (a.k.a., horizontal component) and second value (yOffset) specifying a vertical offset (a.k.a., vertical component).

[0094] Step 4: producing a motion compensated prediction sample using the selected candidate. For example, if the selected MV candidate is an MV offset, at least one MV may be derived from the selected MV offset and a “default” MV, and this at least one derived MV is used in a process for producing a motion compensated prediction sample. For instance, a pair of MVs may be derived from the selected MV offset and a default MV or MV Pair, and this derived pair of MVs is used to produce the motion compensated prediction sample.

[0095] Misalignment distortion indicates a difference in boundary characteristics (texture, sample activity variations) on both sides of a subblock boundary. The differences can be in terms of sum of absolute differences of boundary characteristics across the subblock boundary. The distortion metric is preferably sensitive to phase / movement in direction parallel tothe boundary but insensitive to variations orthogonal to the boundary. In that way the minimization of distortion will keep natural structure / details both parallel and across the subblock boundary.

[0096] One example of difference of boundary characteristics for a horizontal subblock boundary is the sum of absolute differences of sample differences across the horizontal boundary and for a vertical subblock boundary the sum of absolute differences of sample differences across the vertical subblock boundary. At least two samples at the boundary of the current subblock and two adjacent samples in the neighboring subblock is used for the sample difference calculation on respective side of the boundary.

[0097] In case the boundary samples are close to a constant value, e.g. has low spatial activity, there can still be a misalignment between the current and neighboring subblock which can be considered in the cumulative misalignment distortion by including a metric that measures the constant difference between boundary samples. One example metric is sample difference across the subblock boundary. At least one samples at the boundary of the current subblock and one adjacent sample in the neighboring subblock can be used for the sample difference calculation across the boundary.

[0098] In one embodiment, at least two different distances between samples are used to calculate the cumulative misalignment distortion, e.g. short distance between samples and another with longer distances between samples. For example, sample difference between adjacent samples and sample difference between samples not adjacent. Sample differences for samples closer to each other focus on high frequency variation and sample differences for samples with longer distances between samples focus on low frequency variation. Both can be useful for determining misalignment distortion.

[0099] In some embodiments a greater importance is given to low frequency variations, for example difference between samples with longer distance between samples. Because a misalignment of samples with low frequency can be more visible than a misalignment of samples with high frequency since human vision is more sensitive to artifacts with low frequency.

[0100] In some embodiments, the cumulative misalignment distortion is computed usingsample differences between samples with same distance between samples is non-overlapping. As an example of sample differences with the same distance between samples that are nonoverlapping and covers samples PO, Pl, P2, P3. One set is P0-P1 and another non-overlapping set with same sample distance is P2-P3. An example of sets with overlap covering same range would be to have P0-P1, P1-P2 and P2-P3. The case of no-overlap with same distance between samples can cover a larger range of samples with less number of sets than the case of overlap with same distance between samples.

[0101] Examples for Calculating the Cumulative Misalignment Distortion (CM Dist)

[0102] In one embodiment, CM Dist for a candidate MV is based on i) a misalignment distortion (MD) at a horizontal (top or bottom) boundary (hMD) between the current subblock and a top / bottom subblock (denoted “B”) and ii) a misalignment distortion (MD) at a vertical boundary (vMD) between the current subblock and a left / right subblock (denoted “D’). For example, in one embodiment:CM_Dist = SF*(hMD + vMD), where SF is > 0 (equation 1)

[0103] In another embodiment:CM Dist = SFl*hMD + SF2*vMD, where SF1 > 0 and SF2 > 0 (equation 2).

[0104] In one embodiment, vMD = vMDl+vMD2+vMD3+vMD4, and hMD = hMDl+hMD2+hMD3+hMD4.

[0105] Using FIG. 8 as an example, in one embodiment, hMDl, hMD2, hMD3, hMD4 are calculated as shown below:where, in this specific example, B is the subblock above the current subblock and the dimension of the subblocks are MxN, and A is a set of candidate motion compensated prediction sample values that were generated using the candidate motion vector (e.g., generated using one or both of the candidate motion vectors derived from candidate motion vector offset).

[0106] Again using the example sown in FIG. 8, in one embodiment, vMDl, vMD2, vMD3, vMD4 are calculated as shown below:where, in this specific example, D is the subblock to the left of the current subblock.

[0107] The metrics can also be useful for other block and subblock sizes where M and N are multiples of 8. For example, as 8xN, Mx8, 16xN, Mxl6, 32xN, Mx32, 64xN, Mx64, 128xN, Mxl28 where N and M is equal to 8, 16, 32, 64 or 128. For smaller blocks or subblocks such as 4x4 the first three metrics (vMDl, vMD2, vMD3 and hMDl, hMD2, hMD3) can be used.

[0108] In alternative embodiments the right and bottom block or subblock boundaries could also be considered instead of or in addition to the left and top boundaries.

[0109] With respect to the scaling factor (SF) in equation 1 above, in one embodiment SF=8 when subblock misalignment distortion is based in total for of 19*2=38 spatial differences corresponding to 38*8 which is close to the number of samples 256 used for block matching based motion derivation for a subblock size of 16x16. Meaning that the importance of the misalignment distortion is similar as the importance of block matching with 256 samples. More generally, SF=N*M / number of spatial differences be used. In one example, the number of spatial differences for the 4 sets are (M-2) / 2+(M-2) / 2-l+M / 4+M / 8+(M-2) / 2+(N-2) / 2-l+N / 4+N / 8. For a8x8 subblock SF would be 8x8 / 16=4, for a 4x4 subblock 4x4 / 4=4 and for a 32x32 subblock SF would be 32x32 / 82=12.

[0110] In some embodiments, SF is less if the current subblock only has one neighboring subblock left of the current subblock or only one neighboring subblock above the current subblock since then accurate misalignment distortion can only be determined vertically or horizontally and not both vertically and horizontally. One example SF in this case is SF=2, which then corresponds to 19*2. Meaning that the importance of the misalignment distortion is about 1 / 6 (19*2 / 256) of the importance of block matching with 256 samples. That would mean generally a SF of (N*M / 6) / number of spatial differences where an example of the number of spatial differences is given by vMDl, vMD2, vMD3 and vMD4 if the current subblock only has one neighboring subblock left of the current subblock and hMDl, hMD2, hMD3 and hMD4 for a single neighboring subblock above the current subblock as given above.[OHl] As shown in equation 2, the scaling factor can also be individual for respective misalignment distortion (left or above). So that left subblock misalignment distortion has a scaling factor of SFleft and the top subblock misalignment distortion has a scaling factor of SFtop. One example to have individual scaling factor for different boundaries, top and left subblock boundaries as examples, is to make it dependent on spatial activity of boundary samples such that a boundary with neighboring subblock boundary samples with spatial activity less than a threshold (th2) is given a higher scaling factor than a boundary with neighboring subblock boundary samples with spatial activity equal or higher than th2 since areas of low spatial activity is typically more visually sensitive for artifacts. Alternatively the scaling factor for the current block is increased if both top and left subblock boundaries have samples with low spatial activity.

[0112] In one embodiment, different importance is given to boundary characteristics determined by sample differences measured with different distances between samples.

[0113] Consider four sets of differences of samples activities, each set including sets with sample activities computed with same distance between samples. Then each set is given a local scaling factor (f) to control the importance of respective set. That is, for example,

[0114] hMD = fl*hMDl+f2*hMD2+f3*hMD3+f4*hMD4, and

[0115] vMD = fl*vMDl+f2*vMD2+f3*vMD3+f4*vMD4.

[0116] The scaling factor fi (i=l, 2, 3, 4) can be implemented either by multiplication or by right or left shift of the corresponding misalignment distortion. A multiplication by a scaling factor greater that 1 would increase the importance of that set and a scaling factor less than 1 would decrease the importance of that set. A left shift (e.g., hMDl « fl) would similarly increase the importance and a right shift (e.g., hMDl » fl) would decrease the importance.

[0117] For example, fl and f2 could be 1 which would mean that the scaling factor for hMDl, hMD2, vMDl, and vMD2 is not needed. f3 could be a multiplication of 2 or a left shift of 1, increasing importance of the corresponding hMD3 and vMD3 by a factor of 2. f4 could be a multiplication of 4 or a left shift of 2.

[0118] One reason for controlling the importance could be to equivalize the impact of different sets. If for example hMDl and hMD2 corresponds to 8 absolute differences of boundary characteristics and hMD3 corresponds to 4 absolute differences of boundary characteristics and hMD4 corresponds to two absolute differences of boundary characteristics a scaling factor of 2 for f3 and a scaling factor of 4 for f4 would make all sets to have same impact on the total boundary distortion hMDl+ hMD2+2*hMD3+4*hMD4 (the same is true for vMD). Another reason could be to prioritize differences in low-frequency boundary characteristics over differences in high-frequency boundary characteristics.

[0119] Low Spatial Activity Embodiment

[0120] When the spatial activity is low (or a proxy of spatial activity is low) the use of difference in texture (boundary characteristics) is less useful since it cannot detect a DC / constant difference between the neighboring and current subblocks boundary samples. Accordingly, in one embodiment, a difference between neighboring subblock boundary samples and current subblock samples is used as part of misalignment distortion if the spatial activity on boundary samples on at least neighboring subblock boundary samples is less than a threshold (th3).

[0121] One example is to only use the difference between boundary samples when the spatial activity in neighboring boundary samples is less than th3.

[0122] That is, for example, instead of hMD being equal to (hMDl+ hMD2+ hMD3+ hMD4) (or hMD a for short), when a spatial activity measure is less than th3, hMD is calculated as: hMD = fb * hMD b, were fb is scale factor and hMD b is a boundary difference value.

[0123] Otherwise when the spatial activity measure is not less than th3, hMD is equal to hMD_a (i.e., hMD = hMDl+ hMD2+ hMD3+ hMD4). That is, the misalignment distortion is based only on the differences in boundary characteristics.

[0124] Likewise, instead of vMD being equal to (vMDl+ vMD2+ vMD3+ vMD4) (or vMD a for short), when a spatial activity measure is less than th3, vMD is calculated as: vMD = fb*vMD_b, were fb is scale factor and vMD b is a boundary difference value.

[0125] Otherwise when the spatial activity measure is not less than th3, vMD is equal to vMD a (i.e., vMDl+ vMD2+ vMD3+ vMD4). That is, the misalignment distortion is based only on the differences in boundary characteristics.

[0126] In one embodiment, vMD_b = abs(S^)1(DM-i,i - Ao,0) andIt can be noted that the boundary differences are summed before taking the absolute value in this case. Since the focus here is on catching a constant difference between current and neighboring samples that is sufficient which works as long as other complementary differences in boundary characteristics are used since those will capture non-constant differences. The absolute value could also be taken in each individual case but that would increase complexity. When the misalignment distortion also is computed by differences in boundary characteristics it is usually sufficient to use very few boundary differences but at least 1.

[0127] An alternative way to calculate the boundary difference value is to use a Laplacian to be insensitive to a slope (linearly increasing or decreasing samples 1 2 3) across the subblock boundary and only give non-zero difference for a constant difference across the subblock boundary (1 1 2) as follows determined for a top boundary, between neighboringsubblock A and current subblock B as: hMD_b=abs(S" o1( Bi,N-2-2*Bi,N-i + Ai;0)) and for a left boundary between subblock D and B as: vMD_b=abs(S- “01( DM-2,i -2*DM-i,i - A0,i)).

[0128] It can be noted that the boundary differences are summed before taking the absolute value in this case. Since the focus here is on catching a constant difference between current and neighboring samples that is sufficient which works as long as other complementary differences in boundary characteristics are used since those will capture non-constant differences. The absolute value could also be taken in each individual case but that would increase complexity. When the boundary distortion also is computed by differences in boundary characteristics it is usually sufficient to use very few boundary differences but at least 1.

[0129] In another embodiment the difference between boundary samples is added to the misalignment distortion based on difference in boundary characteristics when a spatial activity value in neighboring boundary samples is less than th3, otherwise misalignment distortion is only based on differences in boundary characteristics. That is, for example:

[0130] Measures of Spatial Activity

[0131] As a proxy of spatial activity in neighboring samples, the difference in boundary characteristics for a boundary could be used. This can be handy if also differences in boundary characteristics anyway is computed. It can be noted that this do not measure spatial activity as such but if there is good alignment according to the boundary characteristics the difference can only be a constant value which cannot be detected by the boundary characteristics.

[0132] For a top boundary, between neighboring subblock A and current subblock B the misalignment distortion can be given by:

[0133] For a left boundary, between neighbouring subblock D and current subblock B the misalignment distortion can be given by:

[0134] A measure of spatial activity is to take only use the boundary characteristics based on the neighboring boundary samples. This is also relatively straight forward if also differences in boundary characteristics anyway is computed. The spatial activity for the neighboring samples of subblock B can be based on:saH = saTop = saTop l+saTop2+saTop3+saTop4 where B is the subblock above of the current subblock and the dimension of the subblocks are MxN.

[0135] The spatial activity for the neighboring samples of subblock D can be based on:saV = saLeft = saLeftl +saLeft2+saLeft3+saLeft4 where D is the subblock left of the current subblock.

[0136] Alternatively, the spatial activity for neighboring samples can be based on: saHsaV = saLeft

[0137] Alternatively average of spatial activities can also be taken on spatial activities above to be less noise sensitive. For example, for the case of M=N=16: saHsaV = saLeftabs(DM-i,i - DM-i,i-i)) » 4where more generally the right shift with 4 (similar to division by 16) can be given by log2(M) and log2(N).

[0138] Alternatively, an exact average can be computed, but that’s more complex. Since there are M-l respectively N-l spatial differences computed in the example which means that the exact average is given by division by M-l respectively N-l. saHsaV = saLeft = (S ? abs (DM-i,i - DM-i,i-i) ) / (N-l)

[0139] Alternatively, the spatial activity for neighboring samples can be based on: saH = saTop = abs(CM-i,N-i - DM-I,O) saV = saLeft = abs(CM-i,N-i - BO,N-I) where C M-I,N-I is the bottom right sample from the subblock C top left of the current block A. The reason for including it is to avoid aligning boundary samples of current subblock and neighboring subblock B in case there is a true edge between subblock C and D and similarly avoid aligning boundary samples of current subblock and neighboring subblock D in case there is a true edge between subblock C and B.

[0140] Alternatively, the spatial activity for neighboring samples can be based on:

[0141] Alternatively, the spatial activity for neighboring samples can be based on:+abs(CM-i,N-i - Bo, N-i), and saLeft = saTopwhere the spatial activity surrounding the current block both above and left and top left is considered.

[0142] The method as described in other embodiments can be determined to be used for a horizontal subblock boundary and / or a vertical subblock boundary when a criterion based on absolute difference between the motion of current subblock and the motion of the above neighboring subblock or left neighboring subblock is greater or equal to a threshold (thl). One example thl is 8 when motion is given in 16-pel accuracy, e.g. corresponding to a motion of 1 sample.

[0143] The method as described in other embodiments can be determined to be used for a horizontal subblock boundary and / or a vertical subblock boundary when a criterion based on detection of a true edge is less than a true edge threshold. The true edge check can be determined for a top horizontal subblock boundary as abs(CM-i,N-i - DM-I,O) and for a left vertical subblock boundary as abs(CM-i,N-i - BO,N-I).

[0144] The method as described in other embodiments can be determined to be used for a horizontal subblock boundary and / or a vertical subblock boundary when a criterion based on determining spatial activity for at least one of the reference subblocks centered at the default / initial MV is lower than a threshold. In another embodiment the reference subblocks can be identified by the candidate MV. If the spatial activity is lower than a threshold (e.g., there is a possibility of having an unreliable determination of motion), then implement one of the above embodiments, otherwise do not.

[0145] FIG. 9 is a flowchart illustrating process 900, according to some embodiments, for obtaining a current block of motion compensated predicted samples, e.g., obtaining an interprediction block for an input block. On the encoder side, the inter-prediction block and the input block are used to obtain a residual block (the residual block represents a difference between the prediction block and the input block). On the encoder and decoder side, the inter-prediction block is added to a reconstructed residual block to derive a reconstructed block.

[0146] Process 900 may begin with step s902.g

[0147] Step s902 comprises performing a process for selecting a motion vector (e.g., apair of motion vector offsets (e.g., horizontal component (x) and vertical component (y)) from a set of candidate motion vectors (e.g., a set of offset tuples), the set of candidate motion vectors comprising a first candidate motion vector and a second candidate motion vector.

[0148] Step s904 comprises using the selected motion vector to produce the block of motion compensated prediction samples. Step s904 uses a well-known method of producing the motion compensated prediction samples. For example, if we denote one of the motion compensated prediction samples as “A(x,y),” the step s904 may comprise calculating: A(x,y) = (refO(x,y)+refl(x,y)+l)»l, where refO and refl are reference blocks derived as: refO(x,y) = refPO(xPic+x+MvX+OffsetX,yPic+y+MvY+OffsetY), refl(x,y) = refPl(xPic+x+MvX-OffsetX,yPic+y+MvY-OffsetY), x and y is positions within the current block, xPic and yPic is in picture coordinates and positions the current block (e.g., positions the current block in a picture or another block),MvX and MvY are the horizontal and vertical components of a default motion vector,OffsetX is the horizontal component of the candidate MV offset,OffsetY is the vertical component of the candidate MV offset,MvX+OffsetX. MvY+OffsetY, MvX-OffsetX, MvY-OffsetY are in integer sample accuracy could be in fractional sample accuracy but to keep complexity low fractional sample accuracy is preferably rounded to the integer sample accuracy, otherwise interpolation of samples in the reference picture is needed to obtain the motion compensated prediction samples reft) and / or refl, if for example motion (mv+offset) is with 1 / 16 fractional accuracy, integer accuracy can be obtained by mvFrac » 4, so 16 in 1 / 16 accuracy becomes 1 in integer accuracy.) refPO is a first reference picture, and refPl is a second reference picture.

[0149] In another example, A(x,y)= (refO(x,y)*a+refl(x,y)*b+r)»s , where, as an example, a=l, b=7, r=4, and s=3. The specific weighting, a,b,r and s can for example come from BCW.

[0150] FIG. 10 illustrates functional components of encoder 702 according to some embodiments. It should be noted that encoders may be implemented differently so implementation other than this specific example can be used. Encoder 702 employs a subtractor 241 to produce a residual block which is the difference in sample values between an input block and a prediction block (i.e., the output of a selector 251, which is either an inter-prediction block output by an inter predictor 250 (a.k.a., motion compensator) or an intra-prediction block output by an intra predictor 249). Then a forward transform 242 is performed on the residual block to produce a transformed block comprising transform coefficients. A quantization unit 243 quantizes the transform coefficients based on a quantization parameter (QP) value (e.g., a QP value obtained based on a picture QP value for the picture in which the input block is a part and a block specific QP offset value for the input block), thereby producing quantized transform coefficients which are then encoded into the bitstream by encoder 244 (e.g., an entropy encoder) and the bitstream with the encoded transform coefficients is output from encoder 702. Next, encoder 702 uses the quantized transform coefficients to produce a reconstructed block. This is done by first applying inverse quantization 245 and inverse transform 246 to the transform coefficients to produce a reconstructed residual block and using an adder 247 to add the prediction block to the reconstructed residual block, thereby producing the reconstructed block, which is stored in the reconstruction picture buffer (RPB) 266. Loop filtering by a loop filter (LF) stage 267 is applied and the final decoded picture is stored in a decoded picture buffer (DPB) 268, where it can then be used by the inter predictor 250 to produce an inter-prediction block for the next picture to be processed. LF stage 267 may include three sub-stages: i) a deblocking filter, ii) a sample adaptive offset (SAO) filter, and iii) an Adaptive Loop Filter (ALF).

[0151] FIG. 11 illustrates functional components of decoder 704 according to some embodiments. It should be noted that decoder 704 may be implemented differently so implementations other than this specific example can be used. Decoder 704 includes a decodermodule 361 (e.g., an entropy decoder) that decodes from the bitstream quantized transform coefficient values of a block. Decoder 704 also includes a reconstruction stage 398 in which the quantized transform coefficient values are subject to an inverse quantization process 362 and inverse transform process 363 to produce a residual block. This residual block is input to adder 364 that adds the residual block and a prediction block output from selector 390 to form a reconstructed block. Selector 390 either selects to output an inter-prediction block or an intra-prediction block. The reconstructed block is stored in a RPB 365. The inter-prediction block is generated by the inter-prediction module 350 and the intra-prediction block is generated by the intra prediction module 369. Following the reconstruction stage 398, a loop filter stage 367 applies loop filtering and the final decoded picture may be stored in a decoded picture buffer (DPB) 368 and output to image processor 105. Pictures are stored in the DPB for two primary reasons: 1) to wait for picture output and 2) to be used for reference when decoding future pictures.

[0152] FIG. 12 is a block diagram of an apparatus 1200 for implementing encoder 702 and / or decoder 704, according to some embodiments. When apparatus 1200 implements encoder 702, apparatus 1200 may be referred to as an encoder apparatus, and when apparatus 1200 implements decoder 704, apparatus 1200 may be referred to as a decoder apparatus. As shown in FIG. 12, apparatus 1200 may comprise: processing circuitry (PC) 1202, which may include one or more processors (P) 1255 (e.g., one or more general purpose microprocessors and / or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (i.e., apparatus 1200 may be a distributed computing apparatus); at least one network interface 1248 (e.g., a physical interface or air interface) comprising a transmitter (Tx) 1245 and a receiver (Rx) 1247 for enabling apparatus 1200 to transmit data to and receive data from other nodes connected to a network 100 (e.g., an Internet Protocol (IP) network) to which network interface 1248 is connected (physically or wirelessly) (e.g., network interface 1248 may be coupled to an antenna arrangement comprising one or more antennas for enabling apparatus 1200 to wirelessly transmit / receive data); and a storage unit (a.k.a., “data storage system”) 1208, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. Inembodiments where PC 1202 includes a programmable processor, a computer readable storage medium (CRSM) 1242 may be provided. CRSM 1242 may store a computer program (CP) 1243 comprising computer readable instructions (CRI) 1244. CRSM 1242 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 1244 of computer program 1243 is configured such that when executed by PC 1202, the CRI causes apparatus 1200 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, apparatus 1200 may be configured to perform steps described herein without the need for code. That is, for example, PC 1202 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and / or software.

[0153] Summary of Various Embodiments (as used in the embodiments below, the term “block” is intended to be read broadly to mean a block or a subblock.)Al. A method for obtaining a current block of motion compensated predicted samples (e.g., an inter-prediction block for use in obtaining a reconstructed block by adding the interprediction block and a reconstructed residual block), the method comprising: performing a process for selecting a motion vector (e.g., a motion vector offset) from a set of candidate motion vectors (e.g., a set of offsets), the set of candidate motion vectors comprising a first candidate motion vector and a second candidate motion vector; and using the selected motion vector to produce the block of motion compensated prediction samples (e.g., to produce an inter-prediction block), wherein the process for selecting of a motion vector from the set of candidate motion vectors comprises calculating a first misalignment distortion value (denoted CM Distl) for the first candidate motion vector, and the calculating of the first misalignment distortion value for the first candidate motion vector comprises: calculating a boundary difference value using n and c and calculating the first misalignment distortion value using the boundary difference value, wheren is a sample value that is from a first neighboring block neighboring the current block and that is on a boundary between the first neighboring block and the current block, and c is a first candidate motion compensated prediction sample value that was determined using the first candidate motion vector and that is located in the current block on the boundary between the first neighboring block and the current block; and / or calculating a first difference (dl) between nl and n2; calculating a second difference (d2) between cl and c2; calculating a third difference (d3) between dl and d2 (e.g., d3 = dl - d2, or d3 = d2 - dl); and calculating the first misalignment distortion value using d3, where nl and n2 are sample values that are from the first neighboring block and that are on the boundary between the first neighboring block and the current block, cl is a first candidate motion compensated prediction sample value that was determined using the first candidate motion vector and that is located in the current block on the boundary between the first neighboring block and the current block, and c2 is a second candidate motion compensated prediction sample value that was determined using the first candidate motion vector and that is located in the current block on the boundary between the first neighboring block and the current block.A2. The method of embodiment Al, wherein calculating the boundary difference value using n and c comprises calculating n-c or calculating c-n.A3. The method of embodiment Al, wherein calculating the boundary difference value using n and c comprises calculating n2 - 2*n + c, where n2 is a sample value that is from the first neighboring block neighboring the current block and that is not on the boundary between the first neighboring block and the current block.A4. The method of any one of embodiments A1-A4, whereinthe set of candidate motion vectors, MVs, comprises a set of candidate MV offsets, selecting a motion vector for the current block comprises selecting, from the set of candidate MV offsets, an MV offset for the current block.A5. The method of any one of embodiments A1-A4, wherein the process for selecting of a motion vector from the set of candidate motion vectors further comprises: calculating a first distortion value (e.g., the T-Distortion value) using the first misalignment distortion value (CM Distl); calculating a second distortion value for the second candidate motion vector; comparing the first distortion value with the second distortion value to determine which is lower.A6. The method of embodiment A5, wherein the first distortion value, T-Distortion-1, for the first candidate MV is a function of i) a current block distortion value (denoted “CurrentDistortion”) and ii) the first misalignment distortion value, CM Distl.A7. The method of embodiment A6, wherein T-Distortion- 1= CurrentDistortion + CM Distl.A8. The method of embodiment A7, wherein CM Distl is a function of: a vertical misalignment distortion value, vMD, and / or a horizontal misalignment distortion value, hMD (e.g., CM_Dist = SF(vMD + hMD) or CM_Dist = SFl*vMD + SF2*hMD), where SF, SF1, and SF2 are preconfigured factors).A9. The method of embodiment A8, whereinCM Distl is a function of hMD, hMD is function of hMD a and / or hMD b, hMD a is a function of a first horizontal misalignment distortion value (hMDl) and a second horizontal misalignment distortion value (hMD2), andhMD b is a boundary difference value.A10. The method of embodiment A9, further comprising determining CM Distl, wherein determining CM Distl comprises: determining whether a first condition is satisfied (e.g., determining whether a spatial activity value is less than a threshold or determining whether hMD a is less than a threshold), and as a result of determining that the first condition is satisfied, setting hMD equal to hMD a or setting hMD equal to hMD b or setting hMD equal to hMD b + hMD a.Al l. The method of embodiment A9 or A10, wherein hMD a is a function of hMDl, hMD2, hMD3, and hMD4, wherehMD_b is a function of i) abs(ii) absQXo^ Bi,N-2-2*Bi,N-i + Ai,o)), andA is a set of candidate motion compensated prediction sample values that were generated using the first candidate motion vector, andB is a block above the current block and the dimensions of the blocks are MxN.A12. The method of any one of embodiments A8-A11, whereinCM Distl is a function of vMD, vMD is function of vMD a and / or vMD_b, vMD a is a function of a first vertical misalignment distortion value (vMDl) and a second vertical misalignment distortion value (vMD2), and vMD b is a boundary difference value.A13. The method of embodiment A12, further comprising determining CM Distl, wherein determining CM Distl comprises: determining whether a second condition is satisfied and, as a result of determining that the second condition is satisfied, setting vMD equal to vMD a or setting vMD equal to vMD b or setting vMD equal to vMD_b + vMD a.A14. The method of embodiment A12 or A13, wherein vMD a is a function of vMDl, vMD2, vMD3, and vMD4, wherevMD_b is a function of i) abs(^ C^)1(DM-i,i - Ao,i)) or ii)Ao,i)), andA is a set of candidate motion compensated prediction sample values that were generated using the first candidate motion vector, andD is a block to the left of the current block and the dimensions of the blocks are MxN.A15. The method of any one of embodiments A5-A14, wherein calculating the second distortion value for the second candidate motion vector comprises determining whether the second distortion value should be determined using a second misalignment distortion value (CM_Dist2), determining whether the second distortion value should be determined using a second misalignment distortion value (CM_Dist2) comprises: determining whether a spatial activity value (e.g., proxy value) for a first reference block satisfies a first criterion (e.g., determine whether the spatial activity is less than a first threshold); determining whether a first absolute difference between i) a first component of the second candidate MV and ii) a corresponding component of a MV of a firstneighboring block (e.g., either left neighbour or above neighbour) satisfies a second criterion (e.g., determine whether the absolute difference is greater than a second threshold); and / or determining whether a boundary between the current block and a neighboring block is a true edge.Al 6. The method of embodiment Al 5, wherein determining whether the second distortion value should be determined using a second misalignment distortion value (CM_Dist2) comprises: determining whether a first absolute difference between i) a first component of the second candidate MV and ii) a corresponding component of a MV of a first neighboring block (e.g., either left neighbour or above neighbour) satisfies a second criterion (e.g., determine whether the absolute difference is greater than a second threshold); and determining whether a second absolute difference between i) the first component of the second candidate MV and ii) a corresponding component of a MV of a second neighboring block (e.g., left if first was neighbour or above if first was left) satisfies the second criterion (e.g., determine whether the absolute difference is greater than a second threshold), wherein if either the first absolute difference or the second absolute difference satisfies the second criterion, then it is determined that the second distortion value should be determined using a second misalignment distortion value.A17. The method of embodiment Al 1 or A14, wherein A is derived based on CPB1 and / or CPB2, where CPB1 is a first candidate prediction block (e.g., a candidate inter-prediction block) derived using the first candidate MV and CPB2 is a second candidate prediction block derived using the first candidate MV.A18. The method of embodiment A17, whereinA is equal to the average of CPB 1 and CPB2,A is equal to a weighted average of CPB1 and CPB2,A is equal to CPB1, orA is equal to CPB2.Bl. A computer program (1243) comprising instructions (1244) which when executed by processing circuitry (1202) cause the processing circuitry (1202) to perform the method of any one of the above embodiments.B2. A carrier containing the computer program of embodiment Bl, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (1242).Cl. An apparatus (1200) for encoding a picture, the apparatus configured to perform a method comprising:C2. The apparatus of embodiment Cl, wherein the apparatus is further configured to perform the method of any one of embodiments A2-A18.DI. An apparatus (1200), the apparatus comprising: a memory (1242); and processing circuitry (1202) coupled to the memory (1242), wherein the apparatus (1200) is configured to perform the method of any one of embodiments A1-A18.

[0154] Conclusion

[0155] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0156] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, itis contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

[0157] References[1] K. Andersson, R, Yu, “AHG10: Encoder MV selections and DMVR revisited”, JVET- AD0045, Antalya Turkey, April 2023.

Claims

CLAIMS1. A method for obtaining a current block of motion compensated predicted samples, the method comprising: selecting a motion vector from a set of candidate motion vectors, the set of candidate motion vectors comprising a first candidate motion vector and a second candidate motion vector; and using the selected motion vector to produce the block of motion compensated prediction samples, wherein the selecting of a motion vector from the set of candidate motion vectors comprises calculating a first misalignment distortion value, CM Distl, for the first candidate motion vector, and the calculating of the first misalignment distortion value for the first candidate motion vector comprises: calculating a boundary difference value using n and c and calculating the first misalignment distortion value using the boundary difference value, where n is a sample value that is from a first neighboring block neighboring the current block and that is on a boundary between the first neighboring block and the current block, and c is a first candidate motion compensated prediction sample value that was determined using the first candidate motion vector and that is located in the current block on the boundary between the first neighboring block and the current block; and / or calculating a first difference, dl, between nl and n2; calculating a second difference, d2, between cl and c2; calculating a third difference, d3, between dl and d2; and calculating the first misalignment distortion value using d3, where nl and n2 are sample values that are from the first neighboring block and that are on the boundary between the first neighboring block and the current block, cl is a first candidate motion compensated prediction sample value that was determined using the first candidate motion vector and that is located in thecurrent block on the boundary between the first neighboring block and the current block, and c2 is a second candidate motion compensated prediction sample value that was determined using the first candidate motion vector and that is located in the current block on the boundary between the first neighboring block and the current block.

2. The method of claim 1, wherein calculating the boundary difference value using n and c comprises calculating n-c or calculating c-n.

3. The method of claim 1, wherein calculating the boundary difference value using n and c comprises calculating n2 - 2*n + c, where n2 is a sample value that is from the first neighboring block neighboring the current block and that is not on the boundary between the first neighboring block and the current block.

4. The method of any one of claims 1-3, wherein the set of candidate motion vectors, MVs, comprises a set of candidate MV offsets, selecting a motion vector for the current block comprises selecting, from the set of candidate MV offsets, an MV offset for the current block.

5. The method of any one of claims 1-4, wherein the process for selecting of a motion vector from the set of candidate motion vectors further comprises: calculating a first distortion value, T-Distortion-1, using the first misalignment distortion value, CM_Distl; calculating a second distortion value for the second candidate motion vector; comparing the first distortion value with the second distortion value to determine which is lower.

6. The method of claim 5, wherein the first distortion value, T-Distortion-1, for the first candidate MV is a function of i) a current block distortion value, “CurrentDistortion”, and ii) the first misalignment distortion value, CM Distl.

7. The method of claim 6, wherein T-Distortion-1= CurrentDistortion + CM Distl.

8. The method of claim 7, wherein CM Distl is a function of: a vertical misalignment distortion value, vMD, and / or a horizontal misalignment distortion value, hMD, CM_Dist = SF(vMD + hMD) or CM_Dist = SFl*vMD + SF2*hMD, where SF, SF1, and SF2 are preconfigured factors.

9. The method of claim 8, whereinCM Distl is a function of hMD, hMD is function of hMD a and / or hMD b, hMD a is a function of a first horizontal misalignment distortion value, hMDl, and a second horizontal misalignment distortion value, hMD2, and hMD b is a boundary difference value.

10. The method of claim 9, further comprising determining CM Distl, wherein determining CM Distl comprises: determining whether a first condition is satisfied, and as a result of determining that the first condition is satisfied, setting hMD equal to hMD a or setting hMD equal to hMD b or setting hMD equal to hMD b + hMD a.

11. The method of claim 9 or 10, wherein hMD a is a function of hMDl, hMD2, hMD3, and hMD4, wherehMD_b is a function of i) abs(ii) absAi,o)), andA is a set of candidate motion compensated prediction sample values that were generated using the first candidate motion vector, andB is a block above the current block and the dimensions of the blocks are MxN.

12. The method of any one of claims 8-11, whereinCM Distl is a function of vMD, vMD is function of vMD a and / or vMD_b, vMD a is a function of a first vertical misalignment distortion value, vMDl, and a second vertical misalignment distortion value, vMD2, and vMD b is a boundary difference value.

13. The method of claim 12, further comprising determining CM Distl, wherein determining CM Distl comprises: determining whether a second condition is satisfied and, as a result of determining that the second condition is satisfied, setting vMD equal to vMD a or setting vMD equal to vMD b or setting vMD equal to vMD_b + vMD a.

14. The method of claim 12 or 13, wherein vMD a is a function of vMDl, vMD2, vMD3, and vMD4, wherevMD_b is a function of i) abs(^ C^)1(DM-i,i - Ao,i)) or ii) abs ^^C DM-2,I -2*DM-I,I - Ao,i)), andA is a set of candidate motion compensated prediction sample values that were generated using the first candidate motion vector, andD is a block to the left of the current block and the dimensions of the blocks are MxN.

15. The method of any one of claims 5-14, wherein calculating the second distortion value for the second candidate motion vector comprises determining whether the second distortion value should be determined using a second misalignment distortion value, CM_Dist2, determining whether the second distortion value should be determined using a second misalignment distortion value, CM_Dist2, comprises: determining whether a spatial activity value for a first reference block satisfies a first criterion; determining whether a first absolute difference between i) a first component of the second candidate MV and ii) a corresponding component of a MV of a first neighboring block satisfies a second criterion; and / or determining whether a boundary between the current block and a neighboring block is a true edge.

16. The method of claim 15, wherein determining whether the second distortion value should be determined using a second misalignment distortion value, CM_Dist2, comprises: determining whether a first absolute difference between i) a first component of the second candidate MV and ii) a corresponding component of a MV of a first neighboring block satisfies a second criterion; and determining whether a second absolute difference between i) the first component of the second candidate MV and ii) a corresponding component of a MV of a second neighboring block satisfies the second criterion, wherein if either the first absolute difference or the second absolute difference satisfies the second criterion, then it is determined that the second distortion value should be determined using a second misalignment distortion value.

17. The method of claim 11 or 14, wherein A is derived based on CPB1 and / or CPB2, where CPB1 is a first candidate prediction block derived using the first candidate MV and CPB2 is a second candidate prediction block derived using the first candidate MV.

18. The method of claim 17, whereinA is equal to the average of CPB 1 and CPB2,A is equal to a weighted average of CPB1 and CPB2,A is equal to CPB1, orA is equal to CPB 2.

19. A computer program (1243) comprising instructions (1244) which when executed by processing circuitry (1202) cause the processing circuitry (1202) to perform the method of any one of the above claims.

20. A carrier containing the computer program of claim 19, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (1242).

21. An apparatus configured to perform the method of any one of claims 1-18.

Images