Obtaining motion compensated prediction samples

Abstract

A method for obtaining motion compensated predicted samples. In one embodiment, the method includes obtaining, for a first sample location (e.g., [x1,y1]) within a current block of a current picture, a first motion vector. Obtaining the first motion vector comprises selecting an affine motion model from a set of two or more affine motion models and deriving the first motion vector using the selected affine motion model, the set of two or more affine motion models comprises a first affine motion model and a second affine motion model, the first affine motion model is a first M-parameter affine motion model, where M is a positive whole number, and the second affine motion model is a second M-parameter affine motion model. The method also includes obtaining a motion compensated predicted sample using the first motion vector.

Description

OBTAINING MOTION COMPENSATED PREDICTION SAMPLESTECHNICAL FIELD

[0001] This disclosure relates to methods for obtaining motion compensated predicted samples, corresponding computer programs, corresponding carriers, and corresponding apparatuses.BACKGROUND

[0002] WC and ECM

[0003] Versatile Video Coding (VVC) is a block-based video codec standardized by ITU-T and MPEG. Enhanced Coding Model (ECM) is an exploratory codec which is currently under development. The aim of ECM is to demonstrate and try providing evidence of video coding capabilities beyond VVC.

[0004] Video and Picture

[0005] A video (a.k.a., “video sequence”) comprises of a series of pictures (a.k.a., frames or images). In WC, each picture is identified with a picture order count (POC) value. The POC value also represents the display order of the picture. A picture with a smaller POC value is displayed before another picture with a larger POC value.

[0006] Components

[0007] Each component can be described as a two-dimensional rectangular array of sample values. It is common that each picture consists of three components: one luma component Y, where the sample values are luma values, and two chroma components Cb and Cr, where the sample values are chroma values.

[0008] It is also common that the dimensions of the chroma components are smaller than the luma components by a factor of two in each dimension. For example, the size of the luma component of an HD picture would be 1920x1080, and the chroma components would each have the dimension of 960x540. Components are sometimes referred to as color components.

[0009] Coding Unit and Coding Block

[0010] A block is a two-dimensional (2D) array of sample values (or “samples” for short). A block may be divided into two or more blocks (a.k.a., subblocks), where each subblock is a 2D array of samples, which is also known as a “matrix” of samples. In video coding, each component of a picture is split into blocks and the coded video bitstream consists of a series of coded blocks. It is common in video coding that pictures are split into units that cover a specific area of the picture.

[0011] Each unit consists of all blocks from all components that make up that specific area of the picture and each block belongs fully to one unit. The Coding Unit (CU) in WC is an example of a unit. In WC, the CUs may be split recursively to smaller CUs. The CU at the top level is referred to as the coding tree unit (CTU).

[0012] A CU usually contains three coding blocks, i.e., one coding block for luma and two coding blocks for chroma. The size of luma coding block is same as the CU.

[0013] In WC, the CUs can have size of 4x4 up to 128x128. In the current ECM, the CUs can have size of 4x4 up to 256x256.

[0014] Parameter sets, slice headers, and picture headers

[0015] WC specifies three types of parameter sets: the picture parameter set (PPS), the sequence parameter set (SPS), and the video parameter set (VPS). The PPS contains data that is common for a whole picture, the SPS contains data that is common for a coded layer video sequence (CLVS), and the VPS contains data that is common for multiple coded layer video sequences (CLVSs), e.g., data for multiple layers in the bitstream.

[0016] The concept of slices divides the picture into independently coded slices, where decoding of one slice in a picture is independent of other slices of the same picture. Each slice has a slice header comprising syntax elements. Decoded slice header values from these syntax elements are used when decoding the slice.

[0017] In WC, a coded picture contains a picture header. The picture header contains parameters that are common for all slices of the coded picture.

[0018] Intra prediction

[0019] In intra prediction, also known as spatial prediction, a block is predicted using previous decoded blocks within the same picture. The samples from the previously decoded blocks within the same picture are used to predict the samples inside the current block. A picture consisting of only intra-predicted blocks is referred to as an intra picture.

[0020] Inter prediction

[0021] In inter prediction, also known as temporal prediction, blocks of the current picture are predicted using blocks from previously decoded pictures (these blocks are referred to as reference blocks). The samples from the reference blocks in the previously decoded pictures are used to predict the samples inside the current block. A picture that comprises one or more inter-predicted blocks is referred to as an inter-picture. The previous decoded pictures used for inter prediction are referred to as reference pictures.

[0022] The location of a referenced block inside a reference picture is indicated using a vector (i.e., a set of values) (which is referred to as “motion vector (MV)”). Each MV consists of two values: an x value (a.k.a., x component) and y value (a.k.a., y component) which represents the displacements between current block and the referenced block in x or y dimension. The value of a component may have a resolution finer than an integer position. When that is the case, a filtering (typically interpolation) is done to calculate values used for prediction.

[0023] FIG. 1 shows an example of an MV for the current block. The example MV consist of a horizontal value of 2 and a vertical value of 1.

[0024] An inter picture may use several reference pictures. The reference pictures are usually put into two reference picture lists, L0 and LI. The reference pictures that are output before the current picture are typically the first pictures in L0. The reference pictures that are output after the current picture are typically the first pictures in LI .

[0025] FIG. 2 shows an example of two different prediction types: uni-prediction and biprediction. Inter predicted blocks can use uni-prediction or bi-prediction. A uni-predicted block uses one MV (e.g., MV-a in FIG. 2) to predict from one reference picture, either using L0 or LI. Bi-prediction uses a pair of motion vectors (denoted Bi-MV), to predict from a first referencepicture, such as, for example, reference picture 0 in FIG. 2, and a second reference picture, such as, for example, reference picture 1 in FIG. 2. More specifically, the Bi-MV consists of a first MV (MV-b) and a second MV (MV-c), where the first MV is used to predict from the first reference picture and the second MV is used to predict from the second reference picture.

[0026] Picture coding type (Low delay picture and non-low delay picture)

[0027] A low delay picture is a picture that has all its reference pictures displayed before the picture. In other words, for a low delay picture, all its reference pictures have smaller POC values than the current POC.

[0028] A non-low delay picture is a picture that has at least one of its reference pictures displayed after the picture. In other words, a non-low delay picture has at least one reference picture with a larger POC value than the current POC.

[0029] Fractional MVs, Interpolation filter and MV rounding

[0030] The value of the MV’ s x or y component may correspond to a sample position which has finer granularity than integer (sample) position. Those positions are also referred to as fractional (sample) positions. In WC and current ECM, the MV can be at 1 / 16 sample position.

[0031] FIG. 3 depicts several fractional positions in the horizontal (x-) dimension. The square blocks represent integer positions. The circles represent 1 / 16-position. For example, MV = (4, 10) means the x component is at 4 / 16 position, the y component is at 10 / 16 position.

[0032] In video coding, an MV rounding process is sometimes used to convert an MV at one position to another target position. One example of rounding is to round a fractional MV position to the nearest integer position.

[0033] When an MV is at a fractional position, filtering (typically interpolation) is done to calculate the sample values at those positions. In WC, the length (number of filter taps) of the interpolation filter for luma component is 8, as shown in TABLE 1 below. In ECM, the length of the interpolation filter for luma component has been increased to 12.TABLE 1

[0034] Residual, transform and quantization

[0035] A residual block is generated by computing the difference between samples of a source block (a.k.a., “input block”), which contains original samples, and samples of a prediction block (a.k.a., “residual block”) (e.g., an inter-prediction block or an intra-prediction block). To remove further redundancy, the difference is then typically compressed by a spatial transform,thereby producing transform coefficients. The transform coefficients are then quantized based on a quantization parameter (QP) to control the fidelity of the residual block and thus also the bitrate required to compress the block. A coded block flag (CBF) is used to indicate if there are any non-zero quantized transform coefficients. All coding parameters are then entropy coded at the encoder and decoded at the decoder. If the coded block flag is one, a reconstructed block can then be derived by inverse quantization and inverse transformation of the quantized transform coefficients and then add that to the prediction block. If the coded block flag is zero, the reconstructed block is identical to the prediction block.

[0036] Inter prediction information / Motion information

[0037] For an inter block inside an inter picture in WC, its inter prediction information consists of the following three elements:

[0038] (1) a reference picture list flag (RefPicListFlag) which signals which reference picture list is used for the block (when the value of the flag is equal to 0, it means only L0 is used for predicting the current block, when the value of the flag is equal to 1, it means only LI is used for predicting the current block, and when the value of the flag is equal to 2, it means both L0 and LI are used for predicting the current block);

[0039] (2) a reference picture index (RefPicIdx) per reference picture list used (the index signals which reference picture inside the reference list to be used for predicting the current block), and

[0040] (3) a motion vector (MV) per reference picture used, which signals the position inside the reference picture that is used for predicting the current block.

[0041] The inter prediction information is also referred to as motion information. The decoder stores the motion information for each inter block. In other words, an inter block maintains its own motion information.

[0042] Encoder Decision and Rate Distortion (RD) Cost

[0043] In practice, for an encoder to decide the best prediction mode for a current block, the encoder would evaluate all the possible prediction modes for the current block and select the prediction mode that yields the smallest Rate-Distortion (RD) cost.

[0044] The RD cost is calculated as D + A. * Rate. D (Distortion) measures the difference between the reconstructed block and the corresponding source block. One commonly used metric for calculating D is the sum of squared error SSE— PB (X< y))2, where PA andPB are the sample values in the two blocks A and B respectively. Rate is usually an estimation of the bits to be spent on encoding the mode, and A is a trade-off parameter between Rate and D.

[0045] Motion Information Signaling

[0046] WC and ECM include several methods for implicit signaling of motion information for each block, including the merge method and the subblock merge method. A common motivation behind the implicit methods is to inherit or reuse motion information from neighboring coded blocks. This often works in practice due to spatial correlation of close-by blocks, i.e., the fact that nearby blocks often behave similarly.

[0047] Merge (block merge) Method and Merge Mode

[0048] The merge method derives a set of motion information from previously decoded blocks and use the derived motion information for generating the samples of the entire block. The merge method is sometimes referred to as the block merge method.

[0049] The merge method first generates a list of motion information candidates. The list is also referred to as the merge list. The candidates are derived from previously coded blocks. These previously coded blocks can be spatially adjacent neighboring blocks or temporal collocated blocks relative to the current block.

[0050] FIG. 4 shows the spatial neighboring blocks: left (L), top (T), top-right (TR), leftbottom (LB) and top-left (TL).

[0051] The merge list construction process usually checks the previously coded blocks in a predefined order, for example: T, L, TR, LB, then TL. For each previously coded block being checked, if this previously coded block is inter coded and its motion information has no duplicates in the list, then the motion information of this previously coded block is added to the merge list.

[0052] After the merge list is generated, one of the candidates inside the list is used toderive the motion information of the current block. The candidate selection process is done on the encoder side. An encoder would select a best candidate from the list and encode an index (merge index) in the bitstream to signal to a decoder. The decoder receives the index, it follows the same merge list derivation process as the encoder and uses the index to retrieve the correct candidate. The blocks that use the block merge method are sometimes referred to as blocks in merge mode.

[0053] In the current ECM, non-adjacent spatial blocks are also considered as sources of motion information during the merge list construction.

[0054] FIG. 5 shows some examples (marked with NA1, NA2, and NA3) of those non- adjacent spatial blocks.

[0055] Subblock Merge Method

[0056] WC and ECM also include the subblock merge method. It splits a current block into a number of subblocks and allows each subblock to have its own motion information.

[0057] FIG. 6 shows an example of a current block and its subblocks. Each subblock maintains its own motion information. It should be noted that the subblocks are all rectangular. In WC, only one so called collocated reference picture can be indicated, and the motion is temporally predicted from a corresponding position in the collocated picture (either from L0 or LI). ECM supports two collocated pictures to be indicated, one from L0 and one from LI. ECM also supports that the corresponding position in the collocated picture is offset vertically and / or horizontally with some variation.

[0058] Overlapped Block Motion Compensation (OBMC)

[0059] OBMC is a tool included in ECM which operates at the block boundaries or subblock boundaries of a current inter block. OBMC blends the current block or subblock’s prediction samples, P CUR, (generated using the current associated motion information) with another set of prediction samples, P_NB, which are generated using motion information from a neighboring block or a neighboring subblock at the current block or subblock boundaries. The OBMC blending process takes weighted average of P CUR and P_NB to produce a set of OBMC modified prediction samples, P OBMC. In other words, the set P CUR or P_NB eachassociate with a respective weighting factor, and the set of OBMC modified prediction samples P OBMC is derived as wl * P_CUR + w2 * P_NB (where wl + w2 = 1). The OBMC may give better prediction for samples that are close to the block or subblock boundary.

[0060] Bi-Directional Optical Flow (BDOF)

[0061] BDOF is a tool included in VVC and the current ECM that can be used to refine prediction samples that are generated from a Bi-MV. BDOF relies on optical flow estimation to derive a pair of refinement parameters (vx, vy) which can be further used to refine the prediction samples.

[0062] Decoder-side Motion Vector Refinement (DMVR)

[0063] DMVR is a tool included in VVC and the current ECM to refine motion vectors for a Bi-MV. DMVR operates on subblock level, usually 16x16. Different from BDOF, which relies on optical flow estimation, DMVR relies on bilateral matching of two reference blocks to refine the Bi-MV. The matching is based on SAD (sum of absolute differences). The DMVR searches within a window around the Bi-MV to find whether there exists another Bi-MV (Bi- MV’) that gives a better match between the L0 reference block and the LI reference block. If so, the Bi-MV’ is further used instead for generating the prediction samples of the current block. After that a sub-pixel adjustment is made based on the SAD costs around the motion with least SAD to determine subblock motion with sub-pixel accuracy.

[0064] Multi-pass DMVR

[0065] In ECM (enhanced compression beyond WC), DMVR has been further evolved to use multi-pass optimization, first bilateral block matching in a search area to find the refinement of the merge motion that gives best match, then bilateral 16x16 subblock matching, then bi-directional optical flow on 8x8 subblocks to refine motion further. The multi-pass DMVR also allows for modification of only one of the bi-predictive motions. It is also allowed to use DMVR when BCW (other weightings than just average) is used.

[0066] Template matching (TM)

[0067] In ECM block or subblock motion can also be refined by matching a template outside the current block with a corresponding template on the reference picture. The search islimited to a small range to find a better motion without signaling additional motion information.This can also be used cascaded with the bilateral matching in multi-pass DMVR.

[0068] Affine Motion Compensation

[0069] Generally affine motion compensation can be described by a motion vector (MV) at horizontal position x (mvx) and vertical position y (mvy), when the origin is at x = 0 and y = 0, using 6 model parameters a to f as: mvx= ax + by + eEquation 1 mvy = ex + dy + f

[0070] The parameters e and f correspond to the translational motion vector components. The parameters a to d corresponds to the non-translational model parameters to achieve, rotation, zoom, deformation etc.

[0071] A 6-parameter model can be restricted to a 4-parameter model by setting a = d and b = — c as: mvx= ax — cy + eEquation 2 mvy= ex + ay + f

[0072] In WC and ECM, the 6-parameter affine motion model is given by three control point motion vectors located at the top-left (MV0), the top-right (MV1), and at bottom-left (MV2), of the current block as in Equation 3, and a 4-parameter model given by the control point motion vectors located at the top-left (MV0) and the top-right (MV1) as in Equation 4. The origin of the affine motion model is at MV0, e.g., located at the top-left of the current block (x = 0,y = 0). The motion for a subblock within the current block (e.g., a 4x4 block within the current block of 16x16) is given by the center position of the subblock in x and y coordinates and the width of the current block (W) as in Equation 4 for the 4-parameter model, or both the width (W) and height (H) of the current block in Equation 3 for the 6-parameter model. mvr-mvcr, mv? r-mvn rmvx=lxwOXX +2xhy + mvOxmv1 v- mv0 vmv2v-mvovEquation 3 mvy1 =— W —x+ — n u — y +mv0yJEquation 4

[0073] FIG. 7A illustrates two control point motion vectors for the 4-parameter model (i.e., motion vectors MVO and MV1) and FIG. 7B illustrates three control point motion vectors for the 6-parameter model (i.e., motion vectors MVO, MV1, and MV2). MVO = {mvox, mvoy}, MV1 = {mvix, mviy}, and MV2 = {mv2x, mv2y}.

[0074] The non-translational part of the 4-parameter motion model can be described by a matrix A4:-mylx-mvoxmvoy-mvly-W WA4 = mvly-mvoymvlx-mvoxEquation 5WThe non-translational part of the 6-parameter motion model can be described by a matrix A6:Equation 6

[0075] The use of affine motion is selected by an affine block control flag and the model to be used is indicated by the affine type flag (4- or 6-parameter model). The selected motion model is then used to predict samples of subblocks of sample size 4x4 in WC and 4x4 or 1x1 (pixel wise) in ECM.

[0076] To avoid division as part of the determination of a motion vector at a x,y position inside the block according to Equations 3 and 4, shifts are typically used such that the divisions are removed, and respective differences use left shifts to be upscaled to the maximal block dimension, e.g., in ECM, 256 (2A8). So a difference that had a division by W is left shifted by 8- log2(W) (same as multiplied by 2A(8-log2(W)), and a difference that had a division by H is left shifted by 8-log2(H) (same as multiplied by 2A(8-log2(H)). The motion vector at the origin, e.g., MVO is left shifted by 8 before being added to the matrix multiplication. Then finally the motion vector is right shifted by 8 to obtain the correct motion vector.

[0077] Equation 4 can be described with shifts instead of divisions as:where LW is equal to (N-log2(W)) and R is equal to 1«(N-1), where N=8 when the max block size is 256. Throughout this disclosure, « is the bit left-shift operator and » is the bit rightshift operator.

[0078] Equation 3 can be described with shifts instead of divisions as:where LH is equal to (N-log2(H)).

[0079] The predicted control point motion vectors (mvpi) for the 4- or 6-parameter model can be updated by motion vector differences (mvdi), which are signalled in the bitstream, if they do not fit the motion model of the current block. For the 4-parameter model only mvdO and mvdl may be updated, and for the 6-parameter modal all three mvd’s may be updated.MV0 = mvpO + mvdOMV1 = mvpl + mvdO + mvdl Equation 7MV2 = mvp2 + mvdO + mvd2

[0080] Where mvp is given by one of several candidate control point motion vectors given by previously encoded affine models (inherited candidates) or neighboring motion vectors (constructed candidates), the selected candidate is indicated with an index (the index is signalled in the bitstream) to a list of length 2 in WC. In the case the list is not full, stuffing candidates are inserted. A stuffing candidate is derived by setting all control point motion vectors to one of the motion vectors from a predefined set. The predefined set consist of mvp2, mvpl, mvpO, TMPV (temporally predicted motion vector), or a zero motion vector. Inherited candidates could be both covering 4- and 6-parameter motion models, derived by two respectively three controlpoint motion vectors, and the current blocks control vectors are derived to fit the motion model of the selected inherited candidate. Constructed candidates come from blocks not using affine mode but the motion vectors from locations near the locations of the control points of the current block is used.

[0081] Alternatively, to signal mvp index and mvd’s control points for the current block, an index to a merge candidate list can instead be used and signalled in the bitstream. The merge candidates include control point vectors of a previously coded block. In this case no further correction of the control point motion vectors is used. A candidate can be an inherited candidate or a constructed candidate similarly to how mvp candidates are derived, but in this case the list is of length 5 in VVC. After adding candidates and the list is still not full, candidates with zero control point motion vectors are inserted.

[0082] In ECM it also exists an affine DMVR process. In that case the DMVR process is initialized according to an affine motion model (4-parameter or 6-parameter model). Then a refinement of respective subblock motion vectors initialized according to the affine motion model is performed. The bilateral cost for the block after DMVR is then compared with the bilateral cost of using refined control point vectors for affine motion compensation of the block. The bilateral cost is typically SAD (sum of absolute differences) or some other similarity metric between the two predictions of the block, the respective parts of the bi-prediction. The refined control point motion vectors are derived from the refined subblock motion vectors via a linear regression to correspond to the overall best affine motion. The one with least cost, between prediction of samples using DMVR refined subblock motions or affine motion compensation using DMVR refined control points are used as the final motion compensated bi-prediction of the block.SUMMARY

[0083] Certain challenges presently exist. For example, when the 4-parameter motion model is selected, then the motion vector to be used on a subblock or pixel basis within the current block is derived based on control point motion vectors located in the top-left and the topright of the block (i.e., motion vectors MV0 and MV1 as shown in FIG. 7A). When the 6- parameter motion model is selected, the motion model is derived based on three control pointmotion vectors located at the top-left, the top-right, and the bottom-left (i.e., motion vectors MVO, MV1, and MV2, as shown in FIG. 7B). The 6-parameter motion model can describe all kind of motions, which is fine as long as that is needed. When it’s not needed, then the 6- parameter motion model can be suboptimal (it requires more bits to use the 6-parameter motion model).

[0084] The 4-parameter motion model can describe rotations and scaling’s as long as they are symmetric along the x and y axis. However, the 4-parameter motion model is currently missing a representation of an affine motion model only given by control point motion vectors on the left side, such as the top-left and the bottom-left control point motion vectors, which can be suboptimal (i.e., cost more bits) if the current block’s motion model follows those two control point motion vectors closer than the top two control point motion vectors currently used. Furthermore, the current block’s motion model may only follow one of the dimensions of one or more of the two or the three control point motion vectors but not the other. This will also result in a suboptimal representation of the current motion model.

[0085] One reason for suboptimality is that, to get the desired control point motion vectors for a block, the control point motion vectors are predicted from previously coded motion vectors or motion models (from neighboring blocks) and the further away the predicted control point motion vectors are from the desired control point motion vectors the more bits will be required to modify them by adding control point motion vector differences (mvd’s).

[0086] Another reason for suboptimality is that, for algorithms that determine motion on the decoder side (for example affine DMVR), the algorithms may not search around the most adequate control point motion vectors or dimensions of control point motion vectors and thus not find the most accurate motion to use for affine motion compensation.

[0087] Accordingly, in one aspect, there is provided a method for obtaining motion compensated predicted samples. In one embodiment, the method includes obtaining, for a first sample location (e.g., [xl,yl]) within a current block of a current picture, a first motion vector. Obtaining the first motion vector comprises selecting an affine motion model from a set of two or more affine motion models and deriving the first motion vector using the selected affine motion model. The set of two or more affine motion models comprises a first affine motionmodel and a second affine motion model. The first affine motion model is a first M-parameter affine motion model, where M is a positive whole number, and the second affine motion model is a second M-parameter affine motion model. The method also includes obtaining a motion compensated predicted sample using the first motion vector.

[0088] In another embodiment the method for obtaining motion compensated predicted samples includes selecting a motion vector from a set of two or more motion vectors. The set of two or more motion vectors comprises a first motion vector derived using a first affine motion model and a first sample location (e.g., [xl, yl]), and a second motion vector derived using a second affine motion model and the first sample location. The first affine motion model is a first M-parameter affine motion model, where M is a positive whole number, and the second affine motion model is a second M-parameter affine motion model. The method also includes obtaining a motion compensated predicted sample using the selected motion vector.

[0089] In another aspect, there is provided a computer program comprising instructions which when executed by processing circuitry cause the processing circuitry to perform the method of any one of the above embodiments.

[0090] In a different aspect, there is provided a carrier containing the computer program of the above embodiment, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium.

[0091] In another aspect, there is provided an apparatus for obtaining motion compensated predicted samples. The apparatus comprises processing circuitry and at least one network interface. The apparatus is configured to perform any of the methods disclosed herein.

[0092] An advantage of some embodiments disclosed herein is that, based on one or more criterions, a simplified motion model can be selected from a set of two or more simplified motion models and the selected motion model can be used as part of affine motion compensation. This can be useful when, for example, the current block’s motion field, which are the motions you get when you apply the motion model at various positions inside the current block, is related more to motions at the left side of the current block than the above side of the current block. One example of use of this method together with DMVR refinement is comparedto ECM-13.0 for the random access configuration of JVET CTC. In TABLE 2, results for several classes with different resolutions is shown (a negative number means that a lower bitrate can be achieved at equal peak signal to noise ratio (PSNR)).TABLE 2: Objective benefit from embodiment 10 and 11 compared to ECM-13.0.BRIEF DESCRIPTION OF THE DRAWINGS

[0093] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0094] FIG. 1 shows an example of a motion vector (MV).

[0095] FIG. 2 illustrates uni-inter prediction and bi-inter prediction.

[0096] FIG. 3 depicts several fractional positions in the horizontal (x-) dimension.

[0097] FIG. 4 shows spatial neighboring blocks.

[0098] FIG. 5 shows some examples of non-adjacent spatial blocks.

[0099] FIG. 6 shows an example of a current block and its subblocks.

[0100] FIG. 7A illustrates two control point motion vectors.

[0101] FIG. 7B illustrates three control point motion vectors.

[0102] FIG. 8 illustrates a system according to some embodiments.

[0103] FIG. 9 is a schematic block diagram of an encoder according to an embodiment.

[0104] FIG. 10 is a schematic block diagram of a decoder according to an embodiment.

[0105] FIG. 11 is a flowchart illustrating a process according to some embodiments.

[0106] FIG. 12 is a flowchart illustrating a process according to some embodiments.

[0107] FIG. 13 is a block diagram of an apparatus according to some embodiments.DETAILED DESCRIPTION

[0108] FIG. 8 illustrates a system 800 according to an embodiment. System 800 includes an encoder 802 and a decoder 804, wherein, in the example shown, encoder 802 is in communication with decoder 804 via a network 110 (e.g., the Internet or other network). Encoder 802 encodes a source video sequence 801 (e.g., encodes blocks of units of pictures of video sequence 801) into a bitstream comprising an encoded video sequence (e.g., encoded blocks) and transmits the bitstream to decoder 804 via network 110. In some embodiments, encoder 802 is not in communication with decoder 804, and, in such an embodiment, rather than transmitting bitstream to decoder 804, the bitstream is stored in a data storage unit 890 and decoder 803 can retrieve the bitstream from data storage unit 890. Decoder 804 decodes the pictures included in the encoded video sequence to produce video data for display and / or for further image processing (e.g., a machine vision task). Accordingly, decoder 804 may be part of a device 803 having an image processor 805 and / or a display 806. The image processor 805 may perform machine vision tasks on the decoded pictures. The device 803 may be a mobile device, a set-top device, a head-mounted display, or any other device.

[0109] FIG. 9 illustrates functional components of encoder 802 according to some embodiments. It should be noted that encoders may be implemented differently, so implementations other than this specific example can be used. Encoder 802 employs a subtractor 241 to produce a residual block which is the difference in sample values between an input block and a prediction block (i.e., the output of a selector 251, which is either an interprediction block output by an inter predictor 250 (a.k.a., motion compensator) or an intraprediction block output by an intra predictor 249). Then a forward transform 242 is performed on the residual block to produce a transformed block comprising transform coefficients. Aquantization unit 243 quantizes the transform coefficients based on a quantization parameter (QP) value (e.g., a QP value obtained based on a picture QP value for the picture in which the input block is a part and a block specific QP offset value for the input block), thereby producing quantized transform coefficients which are then encoded into the bitstream by encoder 244 (e.g., an entropy encoder) and the bitstream with the encoded transform coefficients is output from encoder 802. Next, encoder 802 uses the quantized transform coefficients to produce a reconstructed block. This is done by first applying inverse quantization 245 and inverse transform 246 to the transform coefficients to produce a reconstructed residual block and using an adder 247 to add the prediction block to the reconstructed residual block, thereby producing the reconstructed block, which is stored in the reconstruction picture buffer (RPB) 266. Loop filtering by a loop filter (LF) stage 267 is applied and the final decoded picture is stored in a decoded picture buffer (DPB) 268, where it can then be used by the inter predictor 250 to produce an inter-prediction block for the next picture to be processed. LF stage 267 may include three sub-stages: i) a deblocking filter, ii) a sample adaptive offset (SAO) filter, and iii) an Adaptive Loop Filter (ALF).

[0110] FIG. 10 illustrates functional components of decoder 804 according to some embodiments. It should be noted that decoder 804 may be implemented differently, so implementations other than this specific example can be used. Decoder 804 includes a decoder module 361 (e.g., an entropy decoder) that decodes from the bitstream quantized transform coefficient values of a block. Decoder 804 also includes a reconstruction stage 398 in which the quantized transform coefficient values are subject to an inverse quantization process 362 and inverse transform process 363 to produce a residual block. This residual block is input to adder 364 that adds the residual block and a prediction block output from selector 390 to form a reconstructed block. Selector 390 either selects to output an inter-prediction block or an intra-prediction block. The reconstructed block is stored in a RPB 365. The inter-prediction block is generated by the inter-prediction module 350 and the intra-prediction block is generated by the intra prediction module 369. Following the reconstruction stage 398, a loop filter stage 367 applies loop filtering and the final decoded picture may be stored in a decoded picture buffer (DPB) 368 and output to image processor 105. Pictures are stored in the DPB fortwo primary reasons: 1) to wait for picture output and 2) to be used for reference when decoding future pictures.

[0111] As noted above, in WC and ECM, control point motion vectors are used to determine two affine motion models of the current block. One model is based on 6-parameters, which is the most capable version. The other model is based on 4-parameters, which is less capable. As further noted above, in some situations, these conventional motion models are suboptimal in terms of requiring more bits. Accordingly, to combat this issue, this disclosure describes, in one embodiment, enabling the possibility to use at least two restrictions of the 6- parameter affine motion model to enable more efficient representation of a simplified motion model. The embodiments can also be described as using at least one criterion for selecting between possible restrictions of the 6-parameter affine model to derive alternative simplified models. The embodiments described herein can be used at encoder 802 and / or decoder 804. The embodiments can be used for both blocks and subblocks. From here forward the term block should be interpreted broadly to encompass any subblocks, because a subblock is itself a block.

[0112] In other words, this disclosure enables selection of a motion model from a set of at least two alternative simplified motion models based on restrictions of the 6-parameter motion model such that the selected model that fits the current block will use less bits than would otherwise be used using the conventional approach. The selected simplified motion model is then used as part of affine motion compensation to make a prediction of at least one sample of the current block.

[0113] Embodiment 1 describes using two control point motion vectors for the derivation of a 4-parameter affine motion model from only the left side of the current block.

[0114] Embodiment 2 describes alternative restrictions of the 6-parameter model to obtain a simplified motion model.

[0115] Embodiment 3 describes how to enable alternative restrictions of a 6-parameter model by means of using control point motion vectors to obtain a simplified motion model based on a criterion.

[0116] Embodiment 4 provides examples of how the embodiments can be used for prediction of control point motion vectors without further refinement, or for prediction of control point motion vectors that are refined by control point motion vector differences.

[0117] Embodiments 5 to 9 provide example criteria for the selection of the motion model. The selection can either be implicit or explicit.

[0118] Embodiment 10 describes using the embodiments in combination of affine prediction and decoder side motion derivation (DMVR), where the initial motion vectors for the DMVR search are determined according to the invention. In the same context, the method can also be used for prediction of samples after refinement of control point motion vectors, see Embodiment 11.

[0119] Embodiment 1

[0120] In this embodiment, an alternative 4-parameter motion model is defined as shown below in Equation 8.Equation 8

[0121] This alternative 4-parameter motion model is based on control point motion vectors MV0 and MV2, i.e., the two control point motion vectors located at the left of the current block as shown in FIG. 7B. Equation 8 can also be described in terms of shifts instead of using division as follows:where LH=(N-log2(H)) and N=8 when the max block size is 256, and R is a rounding factor equal to (1«(N-1)).

[0122] The motion model in this case can be described by the matrix A4H as:Equation 9where x is a horizontal coordinate and y is a vertical coordinate. H is the height of the block. The motion model is then used for affine motion compensation of the current block.

[0123] This model can also be described more generally by a restriction of the parameters of the 6-parameter model, where a = d and c = — b, as:(mvY= dx + by + e i (mvy= —bx + i dy + i f f Equation 10

[0124] While the 4-parameter affine model is derived based on the control point motion vectors located at the left side of the current block (top-left (MVO) and bottom-left (MV2)), one could derive the same model by using the control point motion vectors located at the top-left (MVO) and top-right (MV1) after updating the control point motion vector at the top-right (MV1) according to the 4-parameter affine model, setting y = 0 and x = W in Equation 8, which then results in Equation 11 : Equation 11

[0125] Then the 4-parameter model based on control point motion vectors located at the top-left (MVO) and the top-right (MV1 from Equation 11) of the current block can be used to get the same result, see Equation 4. Also, a 6-parameter model based on control point motion vectors at the top-left (MVO), the bottom-left (MV2) and the top-right (MV1 from Equation 11) of the current block can be used to give same results, see Equation 3.

[0126] The derived motion model is used to derive at least one motion vector for a position inside the current block. That motion vector is then used to derive at least one predicted sample of the current block.

[0127] In case bi-prediction is used, there can be two motion models, one for each reference picture. In this case, the respectively derived motion model is used to derive at least one motion vector for a position inside the current block based on one of the motion models, and at least one motion vector for the same position inside the current block but based on the other motion model. The motion vectors are then used to derive at least one bi-predicted sample.

[0128] Embodiment 2

[0129] This embodiment describes different ways to produce a 4-parameter motion model based on a 6-parameter motion model. The reduction of the number of model parameters can be described with respect to the parameters of the 6-parameter model shown above in Equation 1.

[0130] One way to produce a simplified motion model using 4-parameters instead of the six parameters shown in Equation 1 is to set a = d and c = — b. Another way to achieve a simplified motion model using 4-parameters is to set d = a and b = — c (this is the conventional 4-parameter affine model shown above in Equation 2).

[0131] Another way of restricting the number of parameters is to set some parameters to 0, or to basically remove those parameters. This can be performed by setting two of the parameters that are dependent on position coordinate x or y to 0, also leading to 4-parameter models. For example setting b = 0 and c = 0 would make the horizontal motion to depend on x but not y, and the vertical motion only to depend on y but not x, alternatively one can set a = 0 and c = 0 such that both the horizontal and the vertical motion depend on y but not on x, or alternatively set b = 0 and d = 0 such that both the horizontal and the vertical motion depend on x but not on y.

[0132] The purpose of allowing the possibility for more than one restriction of a 6- parameter motion model is to be able to get alternative variations of the motion vector depending on positions inside the current block compared to only using both horizontal and vertical motion of all three control point motion vectors, or only using control point motion vectors from the top of the current block.

[0133] Embodiment 3

[0134] This embodiment describes different ways to produce from a 6-parameter motion model that itself is based on three control point motion vectors (see, e.g., Equation 3) a 4- parameter motion model based on, in one example, two control point motion vectors, and in another example based on the three control point motion vectors.

[0135] In one variant, two of the three control point motion vectors are used to produce the 4-parameter motion model. In one example, the two control point motion vectors that are used are the control point motion vectors located at the left side of the current block (top-left (MVO) and bottom-left (MV2)). In another example, the two control point motion vectors that are used are the control point motion vectors located at the top of the current block (top-left (MVO) and top-right (MV1)). This is to select between a restriction to define a 4-parameter motion model based on two control point motion vectors on the left side, along the y dimension, as in Equation 8 in embodiment 1, or based on two control point motion vectors at the top side, the ‘normal’ 4-parameter motion model, along the x dimension, as in Equation 4.

[0136] For the case of simplified motion model based on three control point motion vectors (MVO, MV2, and MV2), the criteria can select between the restrictions to use a 4- parameter motion model where mv2x= mvOxand mvly= mvOyEquation 12awhich is the same as: mylx-mvoxmvx= x + mvWOxmv2y-mv0yEquation 12b mvy = y + mvHOyor as another 4-parameter motion model where mvlx= mvOxand mv2y= mvOyEquation 13awhich is the same as:

[0137] For the case of two control point motion vectors, the criteria can select a restriction to use 4-parameter motion model where mv2x= mvOxand mv2y= mvOyor as another 4-parameter motion model where mvlx= mvOxand mvly= mvOymvx= Ox + - - - y + mv riOxmv2y-mv0yEquation 15a mvy= Ox 1 - - -y + mvOywhich is the same as:Equation 15b

[0138] The purpose of allowing the possibility to use of more than one restriction of a 6- parameter motion model is to be able to use different pairs of two control point motion vectors, or different combinations of three control point vectors, to get alternative variations of the motion vector dependent on positions inside the current block.

[0139] Embodiment 4

[0140] The control point motion vectors described in other embodiments can originate from a prediction of control point motion vectors based on candidates of previously used control point motion vectors or other motion vectors or affine motion models of neighboring blocks. An index to select candidate(s) from the list of candidates is signalled in the bitstream. The predicted control point motion vectors can also be refined by adding motion vector differences to them to obtain the desired control point motion vectors. The alternative simplified motion models can beused as a candidate in control point motion vector prediction. Either for prediction before coding motion vector differences, or without coding motion vector differences (merge mode).

[0141] In the case of prediction of control point motion vectors of the current block at least one of the alternative simplified motion models can be used as a candidate. This can either be used as part of prediction before computing motion vector differences, or as part of prediction when motion vector differences are not computed (affine merge).

[0142] An example is when motion vector differences are added to the predicted control point motion vectors to get the desired control point motion vectors, and the affine type corresponds to 4 (using two control point motion vectors). In this case, smaller motion vector differences may be needed if control point motion vectors on the left side can be selected to be used sometimes (MVO and MV2) since the predicted control point motion vector at the bottomleft of the current block (mvp2) may be closer to a desired control point motion vector (MV2) than the predicted control point motion vector at the top-right of the current block (mvpl) is compared to the desired MV1. In case only predicted control point motion vectors exist on the top side of the current block one can still create a predicted control point motion vector at the bottom-left of the current block (mvp2) using Equation 4 with x=0, y=H, MV0=mvp0, and MVl=mvpl. Similarly, in case only predicted control point motion vectors exist on the left side of the current block, one can still create a predicted control point motion vector at the top-right of the current block (mvpl) using Equation 11 with MV0=mvp0 and MV2=mvp2.

[0143] A criterion can determine if the control point motion vectors are predicted on the left side of the current block (MVO and MV2) or on the top side of the current block (MVO and MV1). When the criterion determines that control point motion vectors are predicted for control point motion vectors on the left side of the current block, the motion vector differences are added to the predicted control point motion vectors on the left side, mvpO and mvp2, to derive the control point motion vectors MVO and MV2:MVO = mvpO + mvdOMV2 = mvp2 + mvdO + mvd2.

[0144] The motion vector differences are signalled in the bitstream (mvdO and mvd2). Then MVO and MV2 can be used as in Equation 8 for affine motion compensation of the current block. Alternatively, MV 1 can be derived from MVO and MV2 as in Equation 11. Then, MVO and MV1 can be used, for example as in Equation 3, for affine motion compensation of the current block.

[0145] Otherwise, when the criterion determines that control point motion vectors are predicted for the top side of the current block (MVO and MV 1), the motion vector differences are added to the predicted control point motion vectors mvpO and mvpl to form MVO and MV 1:MVO = mvpO + mvdOMV1 = mvpl + mvdO + mvdl .

[0146] The motion vector differences are signalled in the bitstream (mvdO and mvdl). The control point motion vectors MVO and MV1 are then used for affine motion compensation, for example as in Equation 3.

[0147] Another example is when motion vector differences are added to the predicted control point motion vectors (mvpO, mvpl, mvp3) to get the desired control point motion vectors (MVO, MV1, MV2), and the affine type corresponds to 6 (using three control point motion vectors). A first criterion can determine if the control point motion vectors are predicted for all dimensions, or if it is determined for only one dimension. In case it is determined for only one dimension, a second criterion can determine between at least two alternative restrictions. For example, if the restriction is according to Equation 12 (vertical component of MV1 is same as vertical component of MVO, horizontal component of MV1 is adjusted, horizontal component of MV2 is same as horizontal component of MVO, vertical component of MV2 is adjusted) the following equations for the mvd’s apply: mvOy = mvpOy + mvdOy mvOx = mvpOx + mvdOx mvly = mvpOy + mvdOymvlx = mvplx + mvdOx + mvdlx mv2y = mvp2y + mvdOy + mvd2y mv2x = mvpOx + mvdOx, where the motion vector difference components mvdOx, mvdOy, mvdlx, and mvd2y, are signalled in the bitstream, or if the restriction is according to Equation 13 (horizontal component of MV1 is same as horizontal component of MVO, vertical component of MV1 is adjusted, vertical component of MV2 is same as vertical component of MVO, horizontal component of MV2 is adjusted): mvOy = mvpOy + mvdOy mvOx = mvpOx + mvdOx mvlx = mvpOx + mvdOx mvly = mvply + mvdOy + mvdly mv2x = mvp2x + mvdOx + mvd2x mv2y = mvpOy + mvdOy

[0148] where the motion vector difference components mvdOx, mvdOy, mvdly, and mvd2x, are signalled in the bitstream. The control point motion vectors MVO, MV1, and MV2, are then used for affine motion compensation, for example as in Equation 3.One example of adding an alternative candidate to the list of candidate control point motion vectors based on a criterion, that can be used for prediction of control point motion vectors both when motion vector differences are used (to get desired control point motion vectors) and not used (merge), is to add at least one candidate of control point motion vectors according to Equations 8, 12B , 13B, 14B, or 15B.

[0149] Embodiment 5

[0150] In this embodiment, a criterion for the selection of an alternative motion model is determined with the aim to get a motion model that is as different from the motion model given by a 6-parameter model. Similarity can be expressed as difference in model parameters, or differences in obtained motion vectors using the model parameters.

[0151] For example, the set of two control point motion vectors located at the left side of the current block (top-left and bottom-left), or the set of two control point vectors located at the top of the current block (top-left and top-right), for the derivation of a motion model (e.g. 4- parameter motion model) is decided based on the set of control point motion vectors that can give a motion model that is most different from a motion model derived from control point motion vectors located both on the left side and the top side of the current block (e.g., a 6- parameter motion model).

[0152] One reason is to avoid having a 4-parameter motion model that gives the same motion field as a 6-parameter motion model.

[0153] For example, a first motion vector (MVA) is calculated using Equation 14B (i.e., a first 4-parameter model based on two control point motion vectors (MV0 and MV1)), a second motion vector (MVB) is calculated using Equation 15B (i.e., a second 4-parameter model based on two control point motion vectors (MV0 and MV2)), and a third motion vector (MVc) is calculated using Equation 3 (i.e., a 6-parameter model based on three control point motion vectors (MV0, MV1, and MV2)). Then a first difference (DI) is calculated, where the first difference is a difference between MVA and MVc, and a second difference (D2) is calculated, where the second difference is a difference between MVB and MVc. Then DI is compared to D2, and if DI is lower than D2, then MVA will be used to derive at least one predicted sample of the current block, otherwise MVB will be used to derive at least one predicted sample of the current block.

[0154] Similarly, when a 4-parameter model based on two control point motion vectors gives the same motion model as a 6-parameter model, it can be of interest to exploit 4-parameter motion models that use three control point motion vectors to get alternatives to the 6-parameter motion model.

[0155] For example, a first motion vector (MVA) is calculated using Equation 12B (i.e., a first 4-parameter model based on three control point motion vectors (MVO, MV1, and MV2)), a second motion vector (MVB) is calculated using Equation 13B (i.e., a second 4-parameter model based on the three control point motion vectors), and a third motion vector (MVc) is calculated using Equation 3 (i.e., a 6-parameter model based on the three control point motion vectors). Then a first difference (DI) is calculated, where the first difference is a difference between MVA and MVc, and a second difference (D2) is calculated, where the second difference is a difference between MVB and MVc. Then DI is compared to D2, and if DI is lower than D2, then MVA will be used to derive at least one predicted sample of the current block, otherwise MVB will be used to derive at least one predicted sample of the current block.

[0156] As yet another example, four motion vectors are calculated using Equations 12B, 13B, 14B, and 15B, respectively, and the resulting motion vector that is more different than the motion vector calculated using Equation 3 is then used to derive at least one predicted sample of the current block.

[0157] As yet another example, seven motion vectors are calculated using Equations 4, 8, 11, 12B, 13B, 14B, and 15B, respectively, and the resulting motion vector that is more different than the motion vector calculated using Equation 3 is then used to derive at least one predicted sample of the current block.

[0158] Embodiment 6

[0159] In this embodiment the criterion for the selection of an alternative motion model is based on a comparison of the width to the height of the current block.

[0160] In the case of two control point motion vectors, for example, the control point motion vectors at the left side of the current block (top-left and the bottom-left) are selected when the width is greater than the height, otherwise the control point motion vectors at the top of the current block is used (top-left and top-right). The reason for this is to get a 4-parameter motion model that gives a motion field that is most different from the motion field given by a 6- parameter motion model. The motion field can become more extrapolated since the motionvectors at the locations to be predicted according to the affine motion model can become further away from the given control point motion vectors.

[0161] In an alternative embodiment, the control point motion vectors at the top-left and the bottom-left are selected when the height is greater than the width, otherwise the control point motion vectors at the top-left and the top-right are used. The reason for this could be to get a motion field that can be “less extrapolated”. But at the same time, it will be more similar to the motion field from the 6-parameter motion model (using three control point motion vectors).

[0162] For the case of using three control point motion vectors, the control point motion vectors according to Equation 14B can be selected when the width is greater than the height, and the control point motion vectors in Equation 15B can be selected when the height is greater than the width, and otherwise the unrestricted three control point motion vectors are selected.

[0163] In an alternative embodiment, the control point motion vectors according to Equation 15B can be selected when the width is greater than the height, and the control point motion vectors in Equation 14B can be selected when the height is greater than the width, and otherwise the unrestricted three control point motion vectors are selected.

[0164] Embodiment 7

[0165] In this embodiment, the criterion for the selection of an alternative motion model is based on the magnitude of one or more parameters of the motion model.

[0166] For example, the selection of the set of two control point motion vectors at the left side of the current block (top-left and bottom-left), or the set of control point vectors at the top of the current block (top-left and top-right), to use for the determination of a 4-parameter motion model is based on the difference in magnitude between the respective control point motion vectors and the control point motion vector at the origin (MV0). The 4-parameter model with greatest difference in magnitude between control point motion vectors is selected. The difference in magnitude between control point motion vectors of the alternative 4-parameter motion model can be defined as leftMag = abs(mv2x- mv0x)+ abs(mv2y- mvOy), and for the “normal” 4-parameter motion model as aboveMag = abs(mvlx- mv0x)+ abs(mvly- mvOy). A 4-parameter motion model based on the control point motion vectors left of the current block (topleft and bottom-left) is selected when aboveMag < leftMag.

[0167] An alternative variant is to use the actual affine motion model parameters that are derived for respective sets of control points and compute the magnitude for those instead of only the motion vector difference, e.g., use the respective A matrix. In this case, the magnitude of the alternative 4-parameter model is leftMag =abs(A4Hoi)+abs(A4Hn), and for the “normal” 4- parameter model it is aboveMag=abs(A4oo)+abs(A4io). Then the alternative 4-parameter model is selected if aboveMag < leftMag. One could also include all four elements of the respective matrix in the calculations of the magnitudes, but since the remaining elements are equal for both matrices the result would be the same.

[0168] Embodiment 8

[0169] In this embodiment the criterion for the selection of alternative motion model is indicated by a flag which encoded in the bitstream. The flag is present for respective blocks in the picture that use affine motion compensation. In one variant, the flag is only encoded / decoded when the affine type is equal to a 4-parameter model. In another variant, the flag is additionally or alternatively encoded / decoded when the affine type is equal to a 6-parameter model.

[0170] For example, in the case two control point motion vectors are used, the flag can indicate the selection of the set of two control point motion vectors at the left side of the current block (top-left and bottom-left), or the set of control point vectors at the top of the current block (top-left and top-right), to use for the derivation of a 4-parameter motion model.

[0171] In case three control point vectors are used, the flag can indicate the selection of restricted set of dimensions of three control point motion vectors, or unrestricted control point motion vectors. When a restricted set of dimensions is used, a flag can indicate which restrictions to select from, at least between two of those given in Equations 8, 12B, 13B, 14B, or 15B.

[0172] Embodiment 9

[0173] In this embodiment, the flags in Embodiment 8 are derived on the decoder side.

[0174] For example, the selection of the two control point motion vectors at the left side of the current block (top-left and bottom-left), or the two of control point vectors at the top of thecurrent block (top-left and top-right), to use for the determination of a 4-parameter motion model is decided by template matching or bilateral matching (DMVR). The set of control point motion vectors that give least template matching cost or bilateral matching cost (according to a similarity metric) is selected to derive the 4-parameter motion model.

[0175] As an example for bilateral matching using cost as SAD (sum of absolute differences) as similarity metric between a block of samples predicted according to a first affine motion compensation from one reference picture, and a block of samples predicted according to second affine or no affine motion compensation from another picture, where the first affine motion compensation is according to a 4-parameter motion model derived from control point motion vectors on the left side of the current block, is compared to the SAD given by a first affine motion compensation according to a 4-parameter motion model derived from control point motion vectors on the top side of the current block.

[0176] As an example for template matching using cost as SAD (sum of absolute differences) as similarity metric between neighboring samples above the current block and samples predicted according to affine motion compensation using corresponding samples above the current reference block in the reference picture, where the affine motion model is based on control point motion vectors above the current block, are compared with the SAD between neighboring samples left of the current block and samples predicted according to affine motion compensation using corresponding samples left of the current reference block based on affine motion compensation, where the affine motion model is derived from control point motion vectors on the left side of the current block.

[0177] Embodiment 10

[0178] In this embodiment, one or several of the above embodiments are used in combination with affine DMVR.

[0179] The initial motion vector for the DMVR search is determined based on restrictions of the 6-parameter model control point motion vectors based on one or several criteria as in above embodiments.

[0180] For example, when the affine type is equal to 4-parameter model (based on two control point motion vectors), the criterion is that the width of the block is greater than the height, then the control point motion vectors at the left side is selected, otherwise the control point motion vectors at the right side is selected, to derive the motion model for the current block.

[0181] The control point motion vectors are then used according to Equation 8 or Equation 4 (or corresponding division free version with shifts), to determine initial motion vectors for each direction (forward respective backward) of the bi-prediction centered spatially on respectively subblock for the DMVR search.

[0182] Embodiment 11

[0183] In addition to Embodiment 10, based on one or several criteria as in above embodiments, the refined control point motion vectors based on alternative restrictions of the 6- parameter model of the current block are used to derive the affine motion model for prediction of samples of the current block.

[0184] For example, when the affine type is 4-parameter model (based on two control point motion vectors), the criterion is that the width of the block is greater than the height, then the refined control point motion vectors at the left side is selected, otherwise the refined control point motion vectors at the top side is selected, to derive the motion model for the current block.

[0185] FIG. 11 is a flowchart illustrating process 1100, according to some embodiments, for obtaining motion compensated predicted samples. Process 1100 may begin in step si 102. Step si 102 comprises obtaining, for a first sample location (e.g., [xl,yl]) within a current block of a current picture, a first motion vector. Obtaining the first motion vector comprises selecting an affine motion model from a set of two or more affine motion models and deriving the first motion vector using the selected affine motion model, the set of two or more affine motion models comprises a first affine motion model and a second affine motion model, the first affine motion model is a first M-parameter affine motion model, where M is a positive whole number, and the second affine motion model is a second M-parameter affine motion model. Step si 104 comprises obtaining a motion compensated predicted sample using the first motion vector.

[0186] FIG. 12 is a flowchart illustrating process 1200, according to some embodiments, for obtaining motion compensated predicted samples. Process 1200 may begin in step sl202. Step si 202 comprises selecting a motion vector from a set of two or more motion vectors. The set of two or more motion vectors comprises a first motion vector derived using a first affine motion model and a first sample location (e.g., [xl, yl]), and a second motion vector derived using a second affine motion model and the first sample location. The first affine motion model is a first M-parameter affine motion model, where M is a positive whole number, and the second affine motion model is a second M-parameter affine motion model. Step si 204 comprises obtaining a motion compensated predicted sample using the selected motion vector.

[0187] FIG. 13 is a block diagram of an apparatus 1300 for implementing encoder 802 and / or decoder 804, according to some embodiments. When apparatus 1300 implements encoder 802, apparatus 1300 may be referred to as an encoder apparatus, and when apparatus 1300 implements decoder 804, apparatus 1300 may be referred to as a decoder apparatus. As shown in FIG. 13, apparatus 1300 may comprise: processing circuitry (PC) 1302, which may include one or more processors (P) 1355 (e.g., one or more general purpose microprocessors and / or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (e.g., apparatus 1300 may be a distributed, cloud computing system comprising two or more computers or a monolithic computing system consisting of a single computer); at least one network interface 1348 (e.g., a physical interface or air interface) comprising a transmitter (Tx) 1345 and a receiver (Rx) 1347 for enabling apparatus 1300 to transmit data to and receive data from other nodes connected to a network 100 (e.g., an Internet Protocol (IP) network) to which network interface 1348 is connected (physically or wirelessly) (e.g., network interface 1348 may be coupled to an antenna arrangement comprising one or more antennas for enabling apparatus 1300 to wirelessly transmit / receive data); and a storage unit (a.k.a., “data storage system”) 1308, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where PC 1302 includes a programmable processor, a computer readable storage medium (CRSM) 1342 may be provided. CRSM 1342 may store a computer program (CP) 1343 comprising computer readable instructions(CRI) 1344. CRSM 1342 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 1344 of computer program 1343 is configured such that when executed by PC 1302, the CRI causes apparatus 1300 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, apparatus 1300 may be configured to perform steps described herein without the need for code. That is, for example, PC 1302 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and / or software.

[0188] Alternative affine motion model for affine DMVR

[0189] Affine motion compensation is a useful coding tool to represent non-translational motion. Affine motion compensation is used together with several other tools in ECM such as GPM, MMVD and DMVR. Common to the usage of affine models in ECM is that they can be represented by 2 or 4 non-translational parameters in addition to the two translational parameters. The representation of the 4-parameter affine model is based on motion vectors located at two control points at the corners in the top of the current block. The 6-parameter model is also represented by the two control point motion vectors at the top of the current block but also a control point motion vector located at the bottom-left corner of the current block. However, ECM lacks a representation based on two control point motion vectors located at the corners at the left side of the current block. This may be suboptimal if the current block’s motion model follows those two closer than the two control point motion vectors located at the top of the current block. In this contribution the initial motion vectors of affine DMVR are derived from control point motion vectors on the left side of the current block when the width of the current block is higher than the height of the current block and the affine type is equal to the 4-parameter model. This approach is also applied on refined control point motion vectors. This can create a motion model that is more different from the 6-parameter model than using control point motion vectors from the top side of the current block. In comparison to ECM- 13 we can get a small but consistent objective benefit for all classes and on average -0.02%.

[0190] Affine motion compensation is a useful tool in ECM and it has been combined with several other tools such as GPM, MMVD and DMVR. In ECM the motion model is defined by motion vectors located at control points at the corners on the left and top side of the current block. The most general one is based on three control point potion vectors located at the top-left, the top-right and the bottom-left of the current block. There exists also a less general model that can handle zoom and rotation but not more complex affine motion as it is based on two control point motion vectors located at the top-left and the top-right of the current block. ECM currently lacks a representation of the 4-parameter model only given by control point motion vectors on the left side.

[0191] When the affine motion type indicates the 4-parameter model, the width is compared with the height of the block and when the width is greater than the height an alternative 4-parameter model based on control point vectors on the left side is derived as:Otherwise, the motion model is derived as in ECM.

[0192] A reason for deploying the alternative 4-parameter model when the width is greater than the height is that the motion field will become more different to the motion field that can be represented by the 6-parameter model since the distance to the used control point motion vectors will be longer than when control point motion vectors at the top corners of the current block is used.

[0193] The method is used for affine DMVR to derive initial motion vectors before DMVR search and also on refined control point motion vectors to derive an affine prediction.

[0194] The results are compared against ECM-13 and gcc 12.2.0 was used. The numbers for class A2 are predicted based on the anchor results three QPs are missing. The encoding and decoding time is not accurate except for class C and D.

[0195] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0196] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

Claims

CLAIMS1. A method (1100) for obtaining motion compensated predicted samples, the method comprising: obtaining (si 102), for a first sample location within a current block of a current picture, a first motion vector; and obtaining (si 104) a motion compensated predicted sample using the first motion vector, wherein obtaining the first motion vector comprises selecting an affine motion model from a set of two or more affine motion models and deriving the first motion vector using the selected affine motion model, the set of two or more affine motion models comprises a first affine motion model and a second affine motion model, the first affine motion model is a first M-parameter affine motion model, where M is a positive whole number, and the second affine motion model is a second M-parameter affine motion model.

2. The method of claim 1 , wherein the first M-parameter affine motion model is a first simplified version of a 6- parameter affine motion model, the second M-parameter affine motion model is a second simplified version of the 6- parameter affine motion model.

3. The method of claim 1 or 2, wherein the first M-parameter affine motion model is a first 4-parameter affine motion model, and the second M-parameter affine motion model is a second 4-parameter affine motion model.

4. The method of any one of claims 1 -3, wherein the current block has a width, W, and a height, H, andselecting an affine motion model from the set of two or more affine motion models comprises selecting an affine motion model from the set of two or more affine motion models based on the width and the height of the current block.

5. The method of claim 4, wherein selecting an affine motion model from the set of two or more affine motion models based on the width and the height of the current block comprises: determining that the width is greater than the height and, as a result of determining that the width is greater than the height, selecting the first M-parameter affine motion model, or determining that the width is less than the height and, as a result of determining that the width is less than the height, selecting the second M-parameter affine motion model.

6. The method of any one of claims 1-3, wherein selecting an affine motion model from the set of two or more affine motion models comprises selecting an affine motion model based on information obtained from a bitstream comprising encoded sample values that are used to obtain the motion compensated predicted sample.

7. The method of any one of claims 1-3, wherein selecting an affine motion model from the set of two or more affine motion models comprises selecting an affine motion model based on a template matching or a bilateral matching.

8. The method of any one of claims 5-7, wherein the first M-parameter affine motion model is defined as:where:H is the height of the current block,W is the width of the current block, mvox is a first component of a control point motion vector located at a top left corner of the current block, mvoy is a second component of the control point motion vector located at the top left corner of the current block, mvix is a first component of a control point motion vector located at a top right corner of the current block, mviy is a second component of the control point motion vector located at the top right corner of the current block, mv2x is a first component of a control point motion vector located at a bottom left corner of the current block, and mv2y is a second component of the control point motion vector located at the bottom left corner of the current block.

9. The method of any one of claims 1-8, wherein obtaining the first motion vector further comprises determining that an M-parameter affine motion model should be used to generate the motion vector, andthe step of selecting an affine motion model from a set of two or more affine motion models is performed as a result of determining that an M-parameter affine modion model should be used to generate the motion vector.

10. The method of any one of claims 1-9, wherein the method further comprises obtaining, for a second sample location within the current picture, a second motion vector, and obtaining the second motion vector comprises deriving the second motion vector using the selected affine motion model that was used to derive the first motion vector.

11. The method of any one of claims 1-9, wherein the method further comprises selecting a motion vector from a set of two or more motion vectors, wherein the set of two or more motion vectors comprises a motion vector derived using the first affine motion model and a second sample location, and a motion vector derived using the second affine motion model and the second sample location, and selecting the affine motion model from the set of two or more affine motion models is performed after selecting the motion vector from the set of two or more motion vectors and comprises choosing the affine motion model that was used to derive the selected motion vector.

12. The method of any one of claims 1-11, wherein the first motion vector is an affine motion vector.

13. The method of claim 10, wherein the second motion vector is a control point motion vector.

14. The method of claim 13, wherein obtaining a motion compensated predicted sample using the first motion vector comprises obtaining an affine motion vector using the control point motion vector, and obtaining the motion compensated predicted sample using the affine motion vector.

15. The method of claim 14, wherein obtaining the affine motion vector using the control point motion vector comprises obtaining the affine motion vector using the control point motion vector and an affine motion model.

16. The method of any one of claims 1-15, wherein the method further comprises using the first motion vector to derive an initial motion vector for a Decoder-side Motion Vector Refinement, DMVR, search.

17. The method of any one of claims 1-15, wherein the method further comprises determining control point motion vectors based on an alternative restriction of an affine motion model.

18. The method of any one of claims 1-15 or 17, wherein the method further comprises determining control point motion vectors based on an alternative restriction of an affine motion model using predicted control point motion vectors, and adding corresponding motion vector differences to the predicted control point motion vectors to obtain the control point motion vectors.

19. A method (1200) for obtaining motion compensated predicted samples, the method comprising: selecting (si 202) a motion vector from a set of two or more motion vectors, obtaining (si 204) a motion compensated predicted sample using the selected motion vector, wherein the set of two or more motion vectors comprises a first motion vector derived using a first affine motion model and a first sample location, and a second motion vector derived using a second affine motion model and the first sample location, the first affine motion model is a first M-parameter affine motion model, where M is a positive whole number, and the second affine motion model is a second M-parameter affine motion model.

20. The method of claim 19, wherein the method further comprises deriving a third motion vector using a third affine motion model, the third affine motion model is an N parameter affine motion model, where N is a positive whole number and N is greater than M, and selecting a motion vector from the set of two or more motion vectors comprises selecting a motion vector from a set of two or more affine motion vectors based on i) a difference between the first motion vector and the third motion vector and ii) a difference between the second motion vector and the third motion vector.

21. The method of claim 20, wherein selecting a motion vector from the set of two or more motion vectors comprises selecting the first motion vector as a result of determining that the difference between the first motion vector and the third motion vector is greater than the difference between the second motion vector and the third motion vector.

22. The method of any one of claims 19-21, wherein the first sample location is at the bottom-right corner of a current block.

23. The method of any one of claims 19-22, wherein the method further comprises using the affine motion model that was used to derive the selected motion vector to calculate a fourth motion vector for another sample location.

24. The method of claim 19, wherein the method further comprises deriving a third motion vector using a third affine motion model, the third affine motion model is an N parameter affine motion model, where N is a positive whole number and N is greater than M, and selecting a motion vector from the set of two or more motion vectors comprises selecting a motion vector from a set of two or more affine motion vectors based on i) a difference between a first predicted sample obtained using the first motion vector and a third predicted sampleobtained using the third motion vector and ii) a difference between a second predicted sample obtained using the second motion vector and the third predicted sample.

25. The method of claim 20, wherein selecting a motion vector from the set of two or more motion vectors comprises selecting the first motion vector as a result of determining that at least the difference between the first predicted sample and the third predicted sample is greater than the difference between the second predicted sample and the third predicted sample.

26. A computer program (1343) comprising instructions (1344) which when executed by processing circuitry (1302) cause the processing circuitry (1302) to perform the method of any one of claims 1-25.

27. A carrier containing the computer program of claim 26, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (1342).

28. An apparatus (1300) for obtaining motion compensated predicted samples, the apparatus comprising processing circuitry (1302) and at least one network interface (1348), the apparatus being configured to perform the method of any one of claims 1-25.

Images