Video image data encoding / decoding

Abstract

The present application relates to a method for decoding video images, the decoding being performed by motion-compensated temporal bidirectional prediction for an inter-coded block, the motion-compensated temporal bidirectional prediction using two reference video images in two separate reference video image lists and two affine motion fields defined by at least two control point motion vectors, the control point motion vectors being represented as CPMVs, at least two control point motion vectors being associated with each reference image, the refined CPMVs being obtained as output of a first step (171) of the following or, alternatively, as output of a second step (172) after the first step (171): In the first step (171), for each CPMV, bilateral matching is performed on a block centered on the CPMV to derive at least two CPMVs refined with integer precision, and a CPMV set including the unrefined CPMV and / or the CPMV refined with integer precision is selected to generate an overall predicted block with the minimum bilateral matching cost; - in said second step (172), for each successive CPMV in the selected CPMV set associated with the block, refine each CPMV with sub-sample precision to minimize the bilateral matching cost of said block; The second step (172) is bypassed based on a comparison of the bilateral matching cost associated with the CPMV performed in the first step with a threshold.

Description

Cross-Citation of Related Applications

[0001] This disclosure claims priority to and the benefit of European Patent Application No. 23305027.7, filed January 9, 2023, the entire contents of which are incorporated herein by reference. [Technical Field]

[0002] This application relates generally to encoding and decoding video images, and in particular, but not exclusively, to inter-prediction of video image blocks. [Background technology]

[0003] This section is intended to introduce the reader to aspects of the art, which may be related to at least one exemplary embodiment described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to better understand the various aspects of the present application. As such, these statements should be read in this light, and not as admissions of related art.

[0004] State-of-the-art video compression systems, such as HEVC (ISO / IEC 23008-2 High Efficiency Video Coding, ITU-T Recommendation H.265, https: / / www.itu.int / rec / T-REC-H.265-202108-P / en) or VVC (ISO / IEC 23090-3 Versatile Video Coding, ITU-T Recommendation H.266, https: / / www.itu.int / rec / T-REC-H.266-202008-I / en), provide low-level and high-level image partitions to divide a video image into image regions called coding tree units (CTUs). For HEVC, the size of a coding tree unit (CTU) is usually between 16x16 pixels and 64x64 pixels, while for VVC, the size of a coding tree unit (CTU) may be 32x32, 64x64, or 128x128 pixels.

[0005] The CTU division of the video image forms a grid of fixed-size CTUs, or a CTU grid, whose top and left boundaries spatially overlap the top and left boundaries of the video image. The CTU grid represents a spatial partition of the video image.

[0006] In VVC and HEVC, the CTU sizes (CTU width and CTU height) of all CTUs in a CTU grid are equal to the same default CTU size (default CTU width CTU DW and default CTU height CTU DH). For example, the default CTU size (default CTU height, default CTU width) may be 128 (CTU DW = CTU DH = 128). The default CTU size (height, width) is coded into the bitstream, for example, at the sequence level in the sequence parameter set (SPS).

[0007] The spatial position of a CTU within the CTU grid is determined based on the CTU address ctuAddr, which defines the spatial position of the CTU from the origin of the upper left corner of the CTU. As shown in Figure 1, the CTU address can define the spatial position from the upper left corner of a higher-level spatial structure S that contains the CTU.

[0008] A coding tree is associated with each CTU and determines the tree partitioning of the CTU.

[0009] As shown in Figure 1, in HEVC, a coding tree is a quadtree division of CTUs, and each leaf is called a coding unit (CU). The spatial location of a CU in a video image is defined by a CU index cuIdx, which indicates the spatial location from the upper left corner of the CTU. A CU is spatially divided into one or more prediction units (PUs). The spatial location of a PU in a video image VP is defined by a PU index puIdx, which defines the spatial location from the upper left corner of the CTU, and the spatial locations of the divided PU elements are defined by a PU partition index puPartIdx, which defines the spatial location from the upper left corner of the PU. Each PU is assigned some intra or inter prediction data.

[0010] The intra or inter coding mode is assigned at the CU level, which means that the same intra / inter coding mode is assigned to each PU of a CU even though the prediction parameters differ for different PUs.

[0011] Based on a quad-tree called a transform tree, a CU can be spatially divided into one or more transform units (TUs). A transform unit is a leaf of the transform tree. The spatial location of a TU in a video image is defined by a TU index tuIdx, which defines the spatial location from the top-left corner of the CU. Each TU is assigned several transformation parameters. A transform type is assigned at the TU level, and a 2D independent transform is performed at the TU level during encoding or decoding of an image block.

[0012] The PU partition types present in HEVC are shown in Figure 2. They include square partitions (2Nx2N and NxN), which are the only partitions used for both intra-predicted and inter-predicted CUs, symmetric non-square partitions (2NxN, Nx2N, used only for inter-predicted CUs), and asymmetric partitions (used only for inter-predicted CUs). For example, PU type 2NxnU represents an asymmetric horizontal partition of a PU, with the smaller partition located at the top of the PU. In another example, PU type 2NxnL represents an asymmetric horizontal partition of a PU, with the smaller partition located at the top of the PU.

[0013] As shown in Figure 3, in VVC, the coding tree starts with a root node (i.e., CTU). Then, a quadtree (or quadtree) partition divides the root node into four nodes corresponding to four equally sized sub-blocks (solid lines). Subsequently, the leaves of the quadtree (or quadtree) can be further divided by so-called multi-type trees, which relate to binary or ternary partitioning in one of four partitioning modes shown in Figure 4. These partitioning types are the vertical and horizontal binary partitioning modes, denoted SBTV and SBTH, and the vertical and horizontal ternary partitioning modes SPTTV and STTH.

[0014] In the case of a joint coding tree in which the luma and chroma components are shared, the leaves of the coding tree of a CTU are CUs.

[0015] Contrary to HEVC, in VVC, in most cases, CUs, PUs, and TUs have the same size, which means that except for some specific coding modes, coding units are generally not divided into PUs or TUs.

[0016] 5 and 6 provide an overview of the video encoding / decoding methods used in current video standard compression systems (e.g., HEVC or VVC).

[0017] FIG. 5 shows an exemplary block diagram of steps of a method 100 for encoding a video image VP according to the related art.

[0018] In step 110, the video image VP is divided into sample blocks and partition information data is signaled in the bitstream. Each block contains samples of one component of the video image VP. Thus, these blocks contain samples of each component that defines the video image VP.

[0019] For example, in HEVC, an image is divided into coding tree units (CTUs). Each CTU can be further divided by quadtree partitioning, where each leaf of the quadtree represents a coding unit (CU). The partition information data may then include data describing the CTUs and the quadtree subdivision of each CTU.

[0020] Each sample block (block for short) may then be a CU (if the CU contains a single PU) or a PU of a CU.

[0021] Each block is coded along a coding loop (also called "intra-loop") using either intra or inter prediction mode.

[0022] Intra prediction (step 120) uses intra prediction data to predict the current block using intra prediction blocks based on previously coded, decoded, and reconstructed samples located around the current block (usually at the top and left of the current block). Intra prediction is performed in the spatial domain.

[0023] In inter prediction mode, motion estimation (step 130) and motion compensation (135) are performed. Motion estimation searches for candidate reference blocks as good predictors of the current block in one or more reference video images that predictively encode the current video image. For example, a good predictor of the current block is a predictor similar to the current block. The output of motion estimation step 130 is inter prediction data, which includes motion information associated with the current block (usually one or more motion vectors and one or more reference video image indexes) and other information for obtaining the same prediction block on the encoding / decoding side. Subsequently, motion compensation (step 135) obtains a prediction block using the motion vector(s) and reference video image index(es) determined in motion estimation step 130. Essentially, a block belonging to the selected reference video image and pointed to by the motion vector can be used as a prediction block for the current block. Also, since motion vectors are expressed as fractions of integer pixel positions (known as sub-pixel accurate motion vector representation), motion compensation typically involves spatial interpolation of several reconstructed samples of a reference video image to calculate a prediction block.

[0024] The prediction information data is signaled in the bitstream. The prediction information may include the prediction mode (intra, inter, or skip), intra / inter prediction data, and any other information for obtaining the same predicted CU at the decoding side.

[0025] The method 100 selects a prediction mode (intra or inter prediction mode) by optimizing the rate-distortion trade-off, for example by considering the encoding of a prediction residual block calculated by subtracting a candidate prediction block from the current block, and the signaling of prediction information data required to determine the candidate prediction block at the decoding side.

[0026] Usually, the best prediction mode is given as the prediction mode of the best coding mode p* for the current block as follows:

number

[0027] where P is the set of all candidate coding modes for the current block, p is a candidate coding mode in the set, and RD cost (p) is the rate-distortion cost of candidate coding mode p, which is usually denoted as:

number

[0028] D(p) is the distortion between the current block and the reconstructed block obtained by encoding / decoding the current block using candidate coding mode p, R(p) is the rate cost associated with encoding the current block using coding mode p, and λ is a Lagrangian parameter representing the rate constraint for encoding the current block, which is generally calculated based on the quantization parameter for encoding the current block.

[0029] Typically, the current block is coded based on a prediction residual block PR. More precisely, the prediction residual block PR is calculated, for example by subtracting the best prediction block from the current block. Then, the prediction residual block PR is transformed (step 140), for example using a DCT (Discrete Cosine Transform) or DST (Discrete Sine Transform) type transform or any other suitable transform, and the obtained transform coefficient block is quantized (step 150).

[0030] In a variant, the method 100 can skip the transformation step 140, skipping the so-called coding mode by transformation, and apply quantization (step 150) directly to the prediction residual block PR.

[0031] The quantized transform coefficient block (or quantized prediction residual block) is entropy coded into a bitstream (step 160).

[0032] As part of the encoding loop, the quantized transform coefficient block (or quantized residual block) is then inverse quantized (step 170) and inverse transformed (180) (or not) to produce a decoded prediction residual block. The decoded prediction residual block and the prediction block are then combined, typically summed, to provide a reconstructed block.

[0033] Additionally, other information data may be entropy coded in step 160, such as for coding the current block of video image VP.

[0034] An in-loop filter (step 190) can be applied to the reconstructed image (including the reconstructed blocks) to reduce artifacts. After all image blocks have been reconstructed, a loop filter can be applied. For example, these include a deblocking filter, a sample self-adaptive offset (SAO), or a self-adaptive loop filter.

[0035] The reconstructed block or the filtered reconstructed block is formed as a reference picture, which can be stored in a decoded picture buffer (DPB) so that it can be used as the next current block of the video picture VP or the coded reference picture of the next coded video picture.

[0036] FIG. 6 shows an exemplary block diagram of steps of a method 200 for decoding a video image VP according to the related art.

[0037] In step 210, a bitstream of coded video image data is entropy decoded to obtain partition information data, prediction information data, and quantized transform coefficient blocks (or quantized residual blocks), for example, generated according to method 100.

[0038] The current block of the video image VP can also be decoded from the bitstream by entropy decoding other information data.

[0039] In step 220, the reconstructed image is divided into current blocks based on the division information. Each current block is entropy decoded from the bitstream along a decoding loop (also called "in-loop"). Each decoded current block is a quantized transform coefficient block or a quantized prediction residual block.

[0040] In step 230, the current block is inverse quantized and possibly inverse transformed (step 240) to obtain a decoded prediction residual block.

[0041] On the other hand, the prediction information data is used to predict the current block, by which intra prediction (step 250) or motion compensated temporal prediction (step 260) is performed to obtain a predicted block. The prediction process performed on the decoding side is the same as the prediction process performed on the encoding side.

[0042] The decoded prediction residual block and the prediction block are then merged, typically summed, which provides the reconstructed block.

[0043] In step 270, an in-loop filter can be applied to the reconstructed image (including the reconstructed block), and the reconstructed block or the filtered reconstructed block can be formed as a reference image, which can be stored in the decoded picture buffer (DPB) described above (FIG. 5).

[0044] In step 130 / 135 of Fig. 5 or step 260 of Fig. 6, an inter-predicted block is defined from inter-prediction data associated with a current block (CU or PU of a CU) of a video image, which inter-prediction data contains motion information that can be represented (encoded) based on the so-called whole-block-based AMVP mode (self-adaptive motion vector prediction) or the so-called whole-block-based merge mode.

[0045] In HEVC, in whole block-based AMVP mode, motion information for defining an inter-prediction block is represented by up to two reference video picture indices, each associated with up to two reference video picture lists (usually represented as L0 and L1). The reference video picture reference list L0 includes at least one reference video picture, and the reference video picture reference list L1 includes at least one reference video picture. Each reference video picture index is a temporal prediction for the current block. The motion information further includes up to two motion vectors, each associated with a reference video picture index in one of the two reference video picture lists. Each motion vector is predictively coded and signaled to the bitstream, i.e., one motion vector difference MVd is derived from the motion vector, one AMVP (self-adaptive motion vector predictor) candidate is selected from the whole block-based AMVP candidate list (built on the coding and decoding sides), and the MVd is signaled to the bitstream. The index of the AMVP candidate selected from the whole block based AMVP candidate list is also signaled in the bitstream.

[0046] FIG. 7 shows an illustrative example for constructing a whole block-based AMVP candidate list, which is used to define inter-predicted blocks for a current block of a current video picture.

[0047] The whole-block-based AMVP candidate list may include two spatial MVP (motion vector predictor) candidates derived from the current video image. The first spatial MVP candidate is derived from motion information associated with the current block's inter-predicted block and, if present, is located at neighboring positions A0 and A1 to the left of the current block. The second MVP candidate is derived from motion information associated with the current block's inter-predicted block and, if present, is located at neighboring positions B0, B1, and B2 on the top of the current block. A redundancy check is then performed among the derived spatial MVP candidates, i.e., duplicate derived MVP candidates are discarded. The whole-block-based AMVP candidate list may further include a temporal MVP candidate, derived from motion information associated with a co-located block (if present) at spatial location H in the reference video image or at spatial location C in other cases. The temporal MVP candidates are scaled based on the temporal distance between the current video image and the reference video image. Finally, if the whole block-based AMVP candidate list contains less than two MVP candidates, pad the whole block-based AMVP candidate list with a zero motion vector.

[0048] In HEVC, in whole block-based merge mode, motion information for defining inter-predicted blocks is represented by merge indexes of a whole block-based MVP candidate list. Each merge index points to motion predictor information that indicates which MVP to use to derive motion information. The motion information is represented by one unidirectional or bidirectional temporal prediction type, up to two reference video picture indexes, and up to two motion vectors, and each motion vector is associated with a reference video picture index in one of two reference video picture lists (L0 or L1).

[0049] No information other than the merge index is signaled. This means that the motion vector of the current block is equal to the motion vector of the whole-block-based MVP candidate indicated by the merge index. Therefore, contrary to the whole-block-based merge mode, the MVd and the reference picture index are not signaled in the bitstream. Only the index of the merge candidate selected from the whole-block-based merge candidate list is signaled in the bitstream.

[0050] Therefore, contrary to AMVP mode, in merge mode, MVd and reference pictures are not signaled: only the merge index is signaled in the bitstream.

[0051] The whole-block-based merge MVP candidate list may include five spatial MVP candidates, which are derived from the current video image shown in Figure 7. The first spatial MVP candidate is derived from motion information associated with the inter-predicted block and, if present, is located in the left neighboring position A1; the second spatial MVP candidate is derived from motion information associated with the inter-predicted block and, if present, is located in the upper neighboring position B1; the third spatial MVP candidate is derived from motion information associated with the inter-predicted block and, if present, is located in the upper right neighboring position B0; the fourth spatial MVP candidate is derived from motion information associated with the inter-predicted block and, if present, is located in the lower left neighboring position A0; and the fifth spatial MVP candidate is derived from the motion information associated with the inter-predicted block and, if present, is located in the left neighboring position B2. Then, a redundancy check is performed among the derived spatial MVPs, i.e., duplicate derived MVP candidates are discarded. The whole block-based merge candidate list may further include temporal MVP candidates, called TMVP candidates, which are derived from motion information associated with a co-located block (if present) located at position H or central spatial position "C" of the reference video image. Then, a redundancy check is performed among the derived spatial MVPs, i.e., duplicate derived MVP candidates are discarded. Finally, when using bidirectional temporal prediction type, if the whole block-based merge candidate list contains less than five MVP candidates, a merge candidate is added to the whole block-based merge candidate list. The merge candidate is derived from the motion information of one MVP candidate associated with one reference video image list and present in the whole block-based merge candidate list, and the motion information corresponds to another MVP candidate associated with another reference video image list and present in the whole block-based merge candidate list. Finally, if the whole block-based merge candidate list is still not full (five merge candidates), the whole block-based merge candidate list is filled with zero motion vectors.

[0052] Coding motion information according to VVC provides a richer representation of motion information than HEVC.

[0053] In VVC, motion information can be represented (coded) according to a whole block-based inter prediction mode that provides whole block-based motion representation, or a whole sub-block-based inter prediction mode that provides whole sub-block-based motion representation.

[0054] In VVC, whole block-based motion representation can be provided according to the whole block-based AMVP mode or the whole block-based merge mode.

[0055] In VVC, in whole block-based AMVP mode, the motion information for defining inter-predicted blocks is expressed similarly to the whole block-based AMVP mode of HEVC. The whole block-based AMVP candidate list can include spatial MVP candidates and temporal MVP candidates (if present), as in HEVC. The whole block-based AMVP candidate list can further include four additional HMVP (history-based motion vector prediction) candidates (if present). Finally, if the whole block-based AMVP candidate list contains fewer than two MVP candidates, the whole block-based AMVP candidate list is padded with zero motion vectors.

[0056] HMVP candidates are derived from previously coded MVPs associated with neighboring or non-neighboring blocks relative to the current block. To this end, a table of HMVP candidates is maintained in both the encoder and decoder and updated in real time as a first-in, first-out (FIFO) buffer of MVPs. There are up to five HMVP candidates in the table. After coding a block, the table is updated by adding the associated motion information as a new HMVP candidate to the end of the table. A FIFO rule is applied to manage the table, and in addition to the basic FIFO mechanism, redundant candidates in the HMVP table are removed first instead of the first candidate. To enable parallel processing, the table is reset for each CTU row.

[0057] In addition to whole-block-based AMVP candidate lists, compared to HEVC, the whole-block-based AMVP mode can employ the following new tools:

[0058] SMVD (Symmetric Motion Vector Differential): For bidirectional blocks, SMVD involves setting the MVd associated with one reference picture list for the current block equal to the opposite value of the MVd associated with another reference picture list for the current block. Furthermore, the reference video pictures used in SMVD mode are derived by the decoder according to some predefined rules. SMVD reduces the rate cost for coding the MVd and reference picture index information and can be selected at the block level.

[0059] -AMVR (Self-adaptive Motion Vector Resolution). The AMVR tool allows signaling MVd at 1 / 4 pixel, 1 / 2 pixel, full pixel, or 4 pixel luma sample resolution. This also allows saving bits in coding the MVd information. In AMVR, the resolution of the motion vectors is selected at the block level.

[0060] - BCW (Coding Unit Weighted Bidirectional Prediction) enables bidirectional prediction of blocks with unequal weights and signals at the block level.

[0061] Finally, the internal motion vector representation is achieved with 1 / 16 luma sample precision instead of HEVC's 1 / 4 luma sample precision.

[0062] In VVC, in the whole block-based merge mode, the motion information for defining inter-prediction blocks is expressed in the same way as in the whole block-based merge mode of HEVC.

[0063] The whole block based merge candidate list is different from the whole block based merge candidate list used in HEVC.

[0064] In VVC, a whole block based merge candidate list can be constructed for MMVD (merge mode with MV difference) mode, GPM (geometric partition mode) mode, or CIIP (combined intra / inter prediction) mode.

[0065] The MMVD mode allows for coding limited motion vector differentials (MVd) on top of selected merge candidates to represent motion information associated with inter-predicted blocks. As shown in Figure 8, MMVD coding is limited to four vector directions and eight magnitude values, from 1 / 4 luma sample to 32 luma samples. Because the MMVD mode provides a moderate level of accuracy, it makes a moderate trade-off between rate cost and MV (motion vector) accuracy for signaling motion information.

[0066] As shown in Figure 9, GPM refers to partitioning the current block PG into two motion partitions MV0 and MV1 along a straight line, where each motion vector MV0 and MV1 points to a block P0, P1 of a reference video image in one of the two reference video image lists.

[0067] Geometric partitions can be non-rectangular or rectangular. In the rectangular case, asymmetric splitting avoids redundancy through CU-level binary splitting. A CU-level flag signals the use of GPM for specific whole-block-based merge modes.

[0068] The orientation and position of the partition line relative to the center of the current block is signaled by a dedicated GPM index at the CU level. The geometric partition mode supports each possible CU size w × h = 2 m ×2 n,m,n∈{3…6} (excluding 8×64 and 64×8) supports a total of 64 partitions. For example, Figure 10 shows various partition lines that can be realized for different partition line angles at different position offsets from the center of the CU.

[0069] Each part of a geometric partition in the current block PG is inter-predicted using its own motion. Only unidirectional prediction is allowed for each partition, i.e., each part has only one motion vector and one reference video picture index. The unidirectional inter-prediction constraint is applied to ensure that only two motion-compensated predictions are needed for each current block PG, similar to regular bidirectional inter-prediction.

[0070] The motion vectors MV0 and MV1 of each partition are derived from the two merge indexes of each partition, respectively.

[0071] After each part of the geometric partition is predicted, a hybrid process with self-adaptive weights is used to adjust the sample values ​​along the edges of the geometric partition. This is the prediction signal for the entire current block PG, and as with other whole-block-based prediction modes, the transform and quantization processes are applied to the entire current block PG (rather than to each partition).

[0072] CIIP refers to combining inter-prediction signals and intra-prediction signals to predict the current block. In CIIP mode, the inter-prediction signal is derived using the same inter-prediction process as applied in whole-block-based merge mode, while the intra-prediction signal is derived according to the normal intra-prediction process in planar mode. Then, the intra-prediction signal and the inter-prediction signal are combined using a weighted average, where the weight value is calculated based on the coding modes of the upper and left neighboring blocks:

number

[0073] In summary, in VVC, the whole block-based merge candidate list can contain spatial MVP candidates similar to those in HEVC, except that the first two candidates are swapped: candidate B1 is considered before candidate A1 during construction of the whole block-based merge candidate list. The whole block-based merge candidate list can also contain TMVP candidates similar to those in HEVC, and HMVP candidates in VVC AMVP mode. Multiple HMVP candidates are inserted into the whole block-based merge candidate list until the merge candidate list reaches the maximum number of MVP candidates allowed (minus one). The whole block-based merge candidate list can also contain up to one pairwise average candidate, calculated as follows: consider the first two merge candidates present in the whole block-based merge candidate list and average their motion vectors. This averaging is calculated separately for each reference video picture list L0 and L1. Thus, if both MVPs are bidirectional, the motion vectors associated with list L0 and list L1 are averaged. If only one motion vector exists in the reference video picture list, it is used to form the pairwise candidate. Finally, if the merge candidate list does not reach the maximum number of allowed MVP candidates, we pad the whole block based merge candidate list with zero motion vectors.

[0074] In VVC, sub-block-based inter prediction basically means dividing the current block into sub-blocks, usually 4x4 or 8x8 sub-blocks of luma samples, and the sub-block-based motion representation for defining the inter prediction block is represented by the motion information associated with all the sub-blocks.

[0075] In HEVC, only the translational motion model is applied to motion-compensated temporal prediction, which cannot capture some motions, such as zoom-in, zoom-out, rotation, perspective transformation, and irregular motion.

[0076] To solve this problem, VVC can use a sub-block-based affine motion mode at the CU level. As shown in Figure 11, the affine motion field of the current block is defined by the translational motion vectors of two control points (four-parameter affine motion model) or three control points (six-parameter affine motion model). In Figure 11, the translational motion vectors v0, v1, and v2 of the control points located at the corners of the current block are used to derive the affine motion field of the current block.

[0077] For a four-parameter affine motion model, the motion vector (mv x ,mv y ) is derived as follows:

number

[0078] For a 6-parameter affine motion model, the motion vector at sample position (x,y) in a 4x4 sub-block of the current block is derived as follows:

number

number

[0079] To derive a motion vector for each sub-block (usually a luma sub-block) of the current block, the motion vector of the center sample of each sub-block is calculated according to Equation 1 or 2, as shown in Figure 12, and rounded to 1 / 16 decimal precision. Then, a motion compensation interpolation filter is applied to generate a prediction for each sub-block using the derived motion vector, and inter prediction is performed for the current block. The sub-block size for the chrominance component is also 4x4. The MV of a 4x4 chrominance sub-block is calculated as the average of the MVs of the luma sub-blocks in the upper left and lower right corners of the collocated 8x8 luma region.

[0080] Hereinafter, the translational motion vector of a control point will be referred to as a CPMV (control point motion vector).

[0081] The sub-block-based motion representation of blocks for defining inter-predicted blocks may be provided according to a sub-block-based affine AMVP mode or a sub-block-based affine merge mode.

[0082] The sub-block-based affine merge mode can be applied to current blocks whose width and height are both equal to or greater than 8. It uses an affine merge candidate list of up to 5 sub-block-based motion candidates. The sub-block-based motion candidates are CPMVP candidates based on motion information associated with spatially neighboring blocks. The index of the sub-block-based motion candidate selected from the affine merge candidate list is signaled in the bitstream. Finally, if the affine merge candidate list is less than (5 CPMVP candidates), it pads the affine merge candidate list with zero motion vectors.

[0083] The CPMVP candidate may be a so-called inherited CPMVP candidate, which is derived by extrapolating the affine motion models (CPMVs) of neighboring blocks, or a so-called constructed CPMVP, which is a CPMVP candidate derived by combining the translational MVs of neighboring sub-blocks.

[0084] In VVC, there are up to two inherited CPMVP candidates, which are derived by extrapolating the affine motion model of the left neighboring block and the affine motion model of the upper neighboring block. For a left CPMVP candidate, if an affine motion model exists for the left neighbor A0, the left CPMVP candidate is extrapolated from the CPMV of said affine motion model. Otherwise, the left CPMVP candidate is extrapolated from the CPMV of the affine motion model of the left neighbor A1 (if present). For an upper CPMVP candidate, if an affine motion model exists for the upper neighbor B0, the upper CPMVP candidate is extrapolated from the CPMV of said affine motion model. Otherwise, if an affine motion model exists for the upper neighbor B1, the upper CPMVP candidate is extrapolated from the CPMV of said affine motion model. Otherwise, the CPMVP candidate is extrapolated from the CPMV of the affine motion model of the left neighbor B2 (if present).

[0085] Extrapolating the affine motion models of neighboring blocks means deriving the inherited CPMVP candidates by combining the CPMVs of said affine motion models.

[0086] As shown in Figure 13, if there is an affine motion model for the lower-left neighboring sub-block A, we obtain the translational motion vectors v2, v3, and v4 of the affine motion model for the upper-left corner, upper-right corner, and lower-left corner of the neighboring block containing sub-block A. If sub-block A is coded with the above four-parameter affine model, the left CPMVP candidate of the current block is inherited (calculated) by combining the translational motion vectors v2 and v3 according to the following two equations:

number

[0087] If sub-block A is coded with the above six-parameter affine model, the three CPMVs of the current block are calculated by combining the translational motion vectors v2, v3 and v4 according to the following equations:

number

[0088] In VVC, the constructed affine candidates are derived by combining the non-affine MVs of neighboring blocks. The MVs of control points are derived from spatially and temporally neighboring sub-blocks.

[0089] FIG. 14 shows an example of deriving constructed CPMVP candidates by combining translational MVs of neighboring blocks.

[0090] CPMV k(k = 1, 2, 3, 4) represents the translation MV associated with the kth control point of the current block. For CPMV1, if a translation MV of neighboring sub-block B2 exists, CPMV1 is equal to said translation MV. Otherwise, if a translation MV of neighboring sub-block B3 exists, CPMV1 is equal to said translation MV. Otherwise, if a translation MV of neighboring sub-block A2 exists, CPMV1 is equal to said translation MV. For CPMV2, if a translation MV of neighboring sub-block B1 exists, CPMV2 is equal to said translation MV. Otherwise, if a translation MV of neighboring sub-block B0 exists, CPMV2 is equal to said translation MV. For CPMV3, if a translation MV of neighboring sub-block A1 exists, CPMV3 is equal to said translation MV. Otherwise, if a translation MV of neighboring sub-block A0 exists, CPMV3 is equal to said translation MV. For CPMV4, if there is a translational MV of a collocated neighboring sub-block in the reference video image at position T, then CPMV4 is equal to said translational MV.

[0091] The constructed CPMVP candidate is derived by combining CPMVk as follows: if CPMV1, CPMV2, and CPMV3 exist, then derive the constructed CPMVP candidate by combining {CPMV1, CPMV2, CPMV3}. Otherwise, if CPMV1, CPMV2, and CPMV4 exist, then derive the constructed CPMVP candidate by combining {CPMV1, CPMV2, CPMV4}. Otherwise, if CPMV1, CPMV3, and CPMV4 exist, then derive the constructed CPMVP candidate by combining {CPMV1, CPMV3, CPMV4}. Otherwise, if CPMV2, CPMV3, and CPMV4 exist, then derive the constructed CPMVP candidate by combining {CPMV2, CPMV3, CPMV4}. Otherwise, if CPMV1 and CPMV2 exist, then derive the constructed CPMVP candidate by combining {CPMV1, CPMV2}. Otherwise, if CPMV1 and CPMV3 exist, derive the constructed CPMVP candidate by combining {CPMV1, CPMV3}.

[0092] In addition, CPMV k point to different reference video images, the CPMV k The combination of control points MV associated with is discarded.

[0093] Three CPMVs k Combining these results in a six-parameter affine motion model (Equation 2), and combining two CPMVs results in a four-parameter affine motion model (Equation 1).

[0094] The sub-block-based affine merge model uses an affine merge candidate list of CPMVP candidates. A merge index mergeIdx indicates the CPMVP candidate for deriving motion information for the current block. The best CPMVP candidate in the affine merge candidate list for the current block is determined as the candidate that minimizes the rate-distortion cost of coding the given block.

[0095] The affine merge candidate list can include SbTMVP (subblock-based temporal motion vector prediction) candidates and / or inherited CPMVP candidates and / or constructed CPMVP candidates, as described above. Finally, if the affine merge candidate list does not reach the maximum number of allowed candidates, pad the affine merge candidate list with zero motion vectors.

[0096] SbTMVP candidates can be derived as follows: Similar to TMVP candidates, SbTMVP candidates are derived from the motion information of collocated sub-blocks (if present) in the reference video image. SbTMVP candidates differ from TMVP candidates in two main ways:

[0097] -TMVP predicts motion at the CU level, while SbTMVP predicts motion at the sub-block level (sub-CU level).

[0098] -TMVP obtains temporal motion vectors from co-located blocks in the reference video image, while SbTMVP applies motion shift before obtaining temporal motion information from the reference video image, and the motion shift is obtained from the motion vector of one of the spatially adjacent sub-blocks of the current block.

[0099] Sub-block-based affine AMVP mode can be applied to a current block whose width and height in VVC are both equal to or greater than 16. In sub-block-based affine AMVP mode, the CPMV difference MVd between the MVs of control points located at the corners of the current block and their CPMVPs (control point motion vector predictors) is signaled in the bitstream. Finally, a CPMVP index pointing to a CPMVP candidate in the affine AMVP candidate list is also signaled in the bitstream.

[0100] The CPMVP for predicting the MV of the control points located at the corners of the current block is derived from an affine AMVP candidate list containing two CPMVP candidates.

[0101] An affine AMVP candidate list can include inherited and / or constructed CPMVP candidates.

[0102] In ECM (“Algorithm description of Enhanced Compression Model 7 (ECM7)”, M. Coban, F. Le Leannec, R.-L. Liao, K. Naser, J. Strom, L. Zhang, document JVET-AB2025, ITU-TSG16WP3 and ISO / IEC JTC 1 / SC29 Joint Video Experts Group (JVET), 28th Meeting, October 21-28, 2022, Germany, https: / / jvet-experts.org / doc_end_user / documents / 28_Mainz / wg11 / JVET-AB2025-v1.zip), the motion information for defining inter-prediction blocks can also be represented according to the so-called bilateral matching AMVP merge mode (BM-AMVP merge mode).

[0103] BM-AMVP merge mode is allowed if at least one reference video picture is available in the reference video picture list (along one so-called inter direction) of the video picture containing the current block and at least one reference video picture is available in another reference video picture list (along another inter direction) of this video picture. In BM-AMVP merge mode, the motion information for defining the inter-predicted block includes first motion information associated with a reference picture in the reference picture list and second motion information associated with a reference picture in another reference picture list.

[0104] In the BM-AMVP merge mode defined in the ECM, the first motion information is a first whole block-based motion vector (MV) associated with a reference image in a reference image list associated with the inter direction, and the second motion information is a second whole block-based MV associated with a reference image in another reference image list associated with the other inter direction.

[0105] The first whole block-based MV is represented as described above in relation to the whole block-based AMVP mode in VVC, and is therefore derived from the whole block-based motion vector predictor (MVP) obtained by the whole block-based AMVP mode, the signaled reference picture index, and the signaled motion vector differential.

[0106] The second whole block-based MV is expressed as described above in relation to the whole block-based merge mode, and is therefore derived from the whole block-based motion vector predictor (MVP) obtained by the whole block-based merge mode.

[0107] The block-wide-based MVP for deriving the second block-wide-based MVP (i.e., the mergeIdx index in the block-wide-based merge MVP candidate list pointed to by the refListMerge index) is derived from the following bilateral matching method given by the following equation that minimizes the bilateral matching cost Bmcost:

number

[0108] For each MVP candidate in the block-wide merge MVP candidate list, calculate the bilateral matching cost using the MVP candidate and MvpIdx[refListAmvp]. The MVP candidate with the smallest cost is selected as the block-wide MVP for deriving the second block-wide MVP.

[0109] In VVC, to improve the accuracy of the MV in merge mode, a bilateral matching (BM)-based decoder-side motion vector refinement method (called the DMVR method) is applied to the current block, starting from the motion vector derived by the whole-block-based AMVP mode (MV0 in FIG. 15) and the second whole-block-based MV derived by the whole-block-based merge mode (MV1 in FIG. 15). The principle is as follows: In bidirectional prediction operation, two initial motion vectors MV0 and MV1 pointing to reference blocks of reference video images in reference video image lists L0 and L1 are enclosed, and refined motion vectors MV0' and MV1' are derived by minimizing the bilateral matching cost between the two reference blocks (minimizing the motion vector difference Mdiff). Essentially, a local search is used to derive the MV refinement offset MV_offset. The local search typically applies an nxn (n is an integer) square search mode, looping horizontally through the search range [-sHor, sHor] and vertically through the search range [-sVer, sVer]. The bilateral matching cost (BM cost) is calculated as the sum of mvDistanceCost and sadCost, where mvDistanceCost represents the coding cost of the motion vector and increases with the magnitude of the motion vector, and sadCost is a metric that provides the difference between two reference blocks. For example, sadCost is the sum of the absolute difference (SAD) of the samples of the two reference blocks. If the BM cost of the center point of the nxn search mode has the minimum cost, the local search ends. Otherwise, the current minimum-cost search point becomes the new center point of the nxn search mode, and the search for the minimum cost continues until the end of the search range is reached.

[0110] In DVMR, the search points surround the initial MV, and the refinement offset of the MV follows the MV difference mirror rule. In other words, any point (represented by a candidate MV pair (MV0, MV1)) checked by DVMR follows the following two formulas:

number

[0111] The DMVR search includes an integer sample offset search step (first step of DMVR) followed by a fractional refinement step (second step of DMVR). The refinement search range is two integer luma samples from the initial MV.

[0112] The refined MVs derived by the DMVR method are used to generate inter-prediction samples and also for temporal motion vector prediction for future image coding, while the original MVs are used for the deblocking process and also for spatial motion vector prediction for future CU coding.

[0113] In VVC, a full search of 25 points is applied to the first step of the DMVR method. First, calculate the BM cost of the initial MV pair. If the BM cost of the first MV pair is lower than the threshold, terminate the first step of the DMVR method. Otherwise, calculate and check the BM costs of the remaining 24 points in raster scan order. Select the search point with the smallest BM cost as the output of the first step of the DMVR method.

[0114] To reduce the penalty associated with the uncertainty of DMVR refinement, we propose to prioritize the original MV during the DMVR process. The SAD between the reference blocks pointed to by the initial MV candidates is reduced by 1 / 4 of the SAD value.

[0115] To reduce computational complexity, the second step of the DMVR method is derived using a parametric error surface formula instead of an additional search using SAD comparison. Based on the output of the first step of the DMVR method, fractional sample refinement is conditionally invoked. If the first step of the DMVR method terminates at the center with the smallest SAD in the first or second iteration of the first step of the DMVR method, fractional sample refinement (the second step of the DMVR method) is further applied.

[0116] In sub-pixel offset estimation based on parametric error surface, the BM cost at the center position and the BM costs at the four neighboring positions from the center are used to fit a 2D parabolic error surface equation of the following form:

number

[0117] where (x min ,y min ) corresponds to the fractional position with the smallest BM cost, and C corresponds to the smallest value of the BM cost. By solving the above equation using the BM cost values ​​of the five search points, (x min ,y min ) is calculated as follows:

number

[0118] In VVC, the application of DMVR is limited and applies only to CUs that meet one specific DMVR condition.

[0119] The DMVR condition is met if the CU is coded using one of the following coding modes and features: -CU level merge mode with bidirectional prediction MV. One reference image is located in the past and another in the future relative to the current image. The distance from the two reference images to the current image (i.e., the POC difference) is the same. - Both of the reference images are short-term reference images. -CU has more than 64 luma samples. - Both CU height and CU width are 8 luma samples or greater. -BCW weight index indicates equal weights. - Do not perform weighted prediction (WP) on the current block, and allow weighted prediction to be applied to motion compensated prediction by combining a multiplicative weighting factor and an additive offset. (“Weighted prediction in the H.264 / MPEG AVC video coding standard”, JM Boyce, May 2004, 10.1109 / iscas.2004.1328865). -Do not use CIIP mode for the current block.

[0120] In VVC, the MV resolution is 1 / 16 luma sample. An 8-tap interpolation filter is used to interpolate samples at fractional positions. In DMVR, search points with integer sample offsets surround the initial fractional pixels of the MV, and therefore, samples at those fractional positions must be interpolated to perform the DMVR search process. To reduce computational complexity, a bilinear interpolation filter is used to generate fractional samples for the search process in DMVR. Another important effect is that by using a bilinear filter with a search range of 2 samples, DVMR does not access more reference samples than a conventional motion compensation process. After obtaining the refined MV using the DMVR search process, a conventional 8-tap interpolation filter is applied to generate the final prediction. Because the conventional MC process does not access more reference samples, samples that are not required in the interpolation process based on the original MV but are required in the interpolation process based on the refined MV are padded from the available samples.

[0121] If the width and / or height of a CU is greater than 16 luma samples, the CU is further divided into sub-blocks with width and / or height equal to 16 luma samples. The maximum unit size used for the DMVR search process is limited to 16x16.

[0122] In ECM, DMVR is applied to affine merge coded blocks when the DMVR condition is met. This is the so-called affine DMVR. The first step of affine DMVR (integer sample offset search step) is applied to the translational part of the affine motion (Equation 3), so that if a candidate satisfies the DMVR condition, a translational MV offset is added to all CPMVs of this candidate in the affine merge list. The MV offset is derived by minimizing the BM cost, which is the same as normal DMVR.

[0123] The affine DMVR consists of a 3x3 square search mode (8 search points) to loop through a search range set to [-3,3] to find the optimal integer MV refinement offset (the first step of the affine DMVR).

[0124] Figure 16 shows an example of the scheme and search point order used in the affine DMVR method. The cross x represents the initial position of the search, and positions 0,...,7 represent all eight positions of the search points.

[0125] Next, we perform fractional refinement (the second step of affine DMVR) around the optimal integer position, and finally perform error surface estimation to find the optimal MV refinement offset with an accuracy of 1 / 16.

[0126] In affine DMVR, there is no bias towards the original MV, and all search points are evaluated.

[0127] To further improve the affine DMVR, a CPMV refinement for affine DMVR was submitted in JVET contribution JVET-AB0178 (https: / / jvet-experts.org / doc_end_user / documents / 28_Mainz / wg11 / JVET-AB2028-v1.zip), which allows applying different MV refinement offsets to different CPMVs.

[0128] The method described in JVET-AB0178 can be applied to merge candidates in affine merge mode and affine MMVD mode.

[0129] Figure 17 shows a schematic block diagram of the steps of the CPMV refinement method described in JVET-AB0178.

[0130] The CPMV refinement method follows the affine DMVR method as described in the related art.

[0131] In a first step 171, for each control point motion vector initCpMvLX[cpIdx], cpIdx=0...numCpMv-1, where numCpMv is the number of CPMVs for the current affine coding block, the method performs bilateral matching on blocks centered on the control point to derive a refined CPMV bmRefinedCpMvLx[cpIdx].

[0132] Next, we loop over the combinations of initCpMvLX[cpIdx] and bmRefinedCpMvLx[cpIdx] to derive the optimal CPMV set that minimizes the BM cost of the current block.

[0133] In the second step 172 (optional), the CPMVs in the optimal CPMV set are iteratively refined to further minimize the BM cost of the current block according to the principles of the second step of affine DMVR. In each iteration, a single CPMV is refined while the others are fixed.

[0134] The method of Figure 17 is preferred because it improves the compression efficiency of modern video coding.

[0135] However, the drawback of this method is that the trade-off obtained between improved compression efficiency and increased encoder and decoder complexity is not very attractive, since this method implies a significant increase in encoder and decoder complexity.

[0136] The objective of the present invention is to improve the trade-off between increased compression efficiency and encoding time overhead of the CPMV refinement method submitted in JVET-AB0178.

[0137] At least one embodiment of the present application is designed according to the above. Summary of the Invention [Problem to be solved by the invention]

[0138] The following presents a summary of at least one embodiment in order to provide a basic understanding of some aspects of the present application. This summary is not an exhaustive overview of the embodiments, and is not intended to identify key or critical elements of the embodiments. The following summary presents only some aspects of at least one embodiment in a simplified form as a prelude to the more detailed description provided elsewhere herein.

[0139] According to a first aspect of the present application, there is provided a method for decoding video images, wherein the decoding is performed by motion-compensated temporal bidirectional prediction for an inter-coded block, wherein the motion-compensated temporal bidirectional prediction uses two reference video images in two separate reference video image lists and two affine motion fields defined by at least two control point motion vectors, the control point motion vectors being represented as CPMVs, and at least two control point motion vectors are associated with each reference image, and a refined CPMV is obtained as the output of a first step below or, optionally, as the output of a second step after the first step.

[0140] In the first step, for each CPMV, perform bilateral matching on a block centered on the CPMV to derive at least two CPMVs refined with integer precision, and select a CPMV set including the unrefined CPMV and / or the CPMV refined with integer precision to generate an overall predicted block with the minimum bilateral matching cost.

[0141] In the second step, for each successive CPMV in the selected CPMV set associated with the block, refine each CPMV with sub-sample precision to minimize the bilateral matching cost of the block.

[0142] The second step is bypassed based on a comparison of the bilateral matching cost associated with the CPMV performed in the first step with a threshold.

[0143] In one embodiment, if the bilateral matching cost associated with the CPMV performed in the first step is lower than a first threshold, the second step is bypassed for the CPMV.

[0144] In one embodiment, if the bilateral matching cost associated with the CPMV performed in the first step is higher than a second threshold, the second step is bypassed for the CPMV.

[0145] In one embodiment, the first threshold and / or the second threshold are fixed or adapted to the prediction unit size.

[0146] In one embodiment, two affine motion fields are defined by a first CPMV, a second CPMV, and a third CPMV, and if the bilateral matching cost associated with the first CPMV or the second CPMV satisfies a bypass condition, the second step is bypassed for the third CPMV.

[0147] In one embodiment, the bypass condition is satisfied if the bilateral cost associated with the first CPMV or the second CPMV performed in the second step is higher than the bilateral matching cost associated with the first CPMV or the second CPMV performed in the first step.

[0148] In one embodiment, a bypass condition is met if the bilateral matching cost associated with the first CPMV or the second CPMV performed in the second step is lower than the bilateral matching cost associated with the first CPMV or the second CPMV performed in the first step, and the reduction in the bilateral cost is lower than a third threshold.

[0149] In one embodiment, the first and second steps are disabled for certain block shapes.

[0150] In one embodiment, if the width or height of the inter-coded block is higher than a first value when said width and height are equal, the first and second steps are disabled.

[0151] In one embodiment, the first and second steps are disabled if the width or height of the inter-coded block is higher than a second value where the width and height are different.

[0152] In one embodiment, block shapes are defined based on the width or height of the inter-coded blocks, and at least one block shape that disables the first and second steps is signaled in the bitstream.

[0153] In one embodiment, the at least one block shape that overrides the first and second steps is signaled by signaling a maximum and minimum block width and / or height.

[0154] In one embodiment, the at least one block shape that disables the first and second steps is signaled by signaling a minimum and / or maximum block ratio between a maximum value of block width and block height and a minimum value of block width and block height.

[0155] According to a second aspect of the present application, there is provided an apparatus including components for carrying out one of the methods according to the first aspect of the present application.

[0156] According to a third aspect of the present application, there is provided a computer program product comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of the first aspect of the present application.

[0157] According to a fourth aspect of the present application, there is provided a non-transitory storage medium containing program code instructions for performing the method of the first aspect of the present application.

[0158] According to a fifth aspect of the present application, there is provided an electronic device including a processor and a memory for storing instructions executable by the processor, the processor being configured to perform the method of the first aspect of the present application.

[0159] Specific features of at least one of the embodiments, as well as other objects, advantages, features, and applications of said at least one of the embodiments, will become apparent from the following description of examples taken in conjunction with the drawings. [Brief explanation of the drawings]

[0160] Reference will now be made, by way of example, to the drawings illustrating embodiments of the present application, in which: FIG. 1 shows an example of a coding tree unit according to HEVC. FIG. 2 shows an example of partitioning a coding unit according to HEVC into prediction units. FIG. 3 shows an example of CTU division related to VVC. FIG. 4 shows an example of a partition mode supported in a multi-type tree partition according to VVC. FIG. 5 shows a schematic block diagram of the steps of a method 100 for encoding a video image VP according to the related art. FIG. 6 shows a schematic block diagram of the steps of a method 200 for decoding a video image VP according to the related art. FIG. 7 shows an illustrative example of constructing a whole block-based AMVP candidate list, which is used to define inter-predicted blocks for a current block of a current video picture. FIG. 8 shows an example of constrained motion vector differentials according to the related art. FIG. 9 shows an example of performing geometric partitioning on blocks of a video image according to the related art. FIG. 10 shows an example of a partition of a geometric block according to the related art. FIG. 11 shows an example of a representation of an affine motion field of a video image block according to the related art. FIG. 12 shows an example of motion vectors associated with sub-blocks of a video image block according to the related art. FIG. 13 shows an example of deriving an affine motion model from translational motion vectors for control points located at the corners of a video image block according to the related art. FIG. 14 shows an example of deriving a CPMVP candidate constructed by combining translational MVs of neighboring blocks according to the related art. FIG. 15 is a diagram showing an example of a refinement method for bilateral matching according to the related art. FIG. 16 shows an example of the method and search point order used in affine DMVR. Figure 17 shows a schematic block diagram of the steps of the CPMV refinement method described in JVET-AB0178. FIG. 18 shows the location of search points according to the related art. FIG. 19 illustrates a simplified block diagram of an example system in which various aspects and embodiments may be implemented. Similar or identical elements are referenced with the same accompanying drawing marks. DETAILED DESCRIPTION OF THE INVENTION

[0161] At least one example of an embodiment will be described below with reference to the drawings, and at least one example of an embodiment will be illustrated. However, the embodiments may be implemented in various alternative forms and should not be construed as being limited to the examples set forth herein. Accordingly, it should be understood that the invention is not intended to be limited to the particular forms disclosed. On the contrary, the present application is intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the present application.

[0162] Various aspects relate, at least on the one hand, to the encoding and decoding of video images, on the other hand, generally to the provision or transmission of encoded bitstreams, and on the other hand, to the reception / access of decoded bitstreams.

[0163] At least one embodiment is described in terms of encoding / decoding a single video image, but as will be explained below, this extends to encoding / decoding multiple video images (a sequence of images), with each video image being encoded / decoded sequentially.

[0164] Also, for example, at least one embodiment may be compatible with AVC (ISO / IEC 14496-10 Advanced Video Coding for generic audio-visual services, ITU-T Recommendation H.264, https: / / www.itu.int / rec / T-REC-H.264-202108-P / en), EVC (ISO / IEC 23094-1 Essential video coding), HEVC (ISO / IEC 23008-2 High Efficiency Video Coding, ITU-T Recommendation H.265, https: / / www.itu.int / rec / T-REC-H.265-202108-P / en), or VVC (ISO / IEC 23090-3 Versatile Video Coding, ITU-T Recommendation The present invention may be applied to other standards or recommendations, such as, but not limited to, MPEG standards such as H.266 (https: / / www.itu.int / rec / T-REC-H.266-202008-I / en), including AV1 (AOMedia Video 1, http: / / aomedia.org / av1 / specification / ). At least one embodiment may be applied to existing or future developments and extensions of such standards and recommendations. Unless otherwise stated or technically excluded, various aspects described herein may be used alone or in combination.

[0165] A pixel corresponds to the smallest display unit on the screen and can consist of one or more light sources (one for a monochrome screen, three or more for a color screen).

[0166] A video image, also called a frame or image frame, includes at least one component (also called an image component or channel) determined by a particular image / video format that specifies all information associated with pixel values ​​and all information that can be used to display and / or decode video image data associated with said video image by a display unit and / or any other device.

[0167] A video image includes at least one component, which is usually represented in the form of a sample array.

[0168] A monochrome video image contains a single component, while a color video image may contain three components.

[0169] For example, if the image / video format is the well-known (Y,Cb,Cr) format, a color video image may include one luma (or glow) component and two chrominance components, while if the image / video format is the well-known (R,G,B) format, a color video image may include three color components (one each for red, green, and blue).

[0170] Each component of a video image may contain a number of samples relative to the number of pixels of the screen on which the video image is displayed. Alternatively, the number of samples contained in a component may be a multiple (or fraction) of the number of samples contained in another component of the same video image.

[0171] For example, if a video format includes one luma component and two chrominance components (e.g., a (Y,Cb,Cr) format), the chrominance components may include half the number of samples in width and / or height as the luma component, depending on the color format considered.

[0172] A sample is the smallest visual information unit of a video screen. Sample values ​​may be, for example, luma or chrominance values, or color values ​​in (R,G,B) format.

[0173] A pixel value is the value of a pixel on the screen. For monochrome video images, a pixel value can be represented by a single sample, but for color video images, a pixel value can be represented by multiple co-located samples. The co-located samples associated with a pixel refer to the samples that correspond to the pixel's location on the screen.

[0174] A video image is typically viewed as a collection of pixel values, with each pixel represented by at least one sample.

[0175] A block of a video image is a collection of samples of one component of the video image: if the image / video format is the well-known (Y,Cb,Cr) format, it can be considered a block of at least one luma sample or a block of at least one chrominance sample, or if the image / video format is the well-known (R,G,B) format, it can be considered a block of at least one color sample.

[0176] At least one embodiment is not limited to a particular image / video format.

[0177] In general, this application relates to various embodiments for reducing the complexity of the CPMV refinement method of JVET-AB0178 while preserving as much as possible the compression efficiency obtained in JVET-AB0178.

[0178] In the first embodiment, the MV / CPMV refinement search is optimized by systematically storing the affine DMVR results (i.e., the obtained optimal MV and minimum BM cost (the output of the first or second step of the affine DMVR method)).

[0179] In ECM, the affine DMVR search algorithm is processed, sometimes overlapping several times, to improve compression efficiency by selecting an MV that provides a BM cost in the second step of the affine DMVR and steps 171 and 172. In the case of overlapping several times, the second step of the affine DMVR method can be skipped if the previous iteration did not provide a better solution.

[0180] When using several iterations of the affine DMVR search method, the optimal results (optimal CPMV and optimal BM cost) obtained in the previous several iterations are used to update the initial MV and optimal BM cost.

[0181] Assume position 0 is the optimal position. Figure 18 shows the positions that will be evaluated in the next iteration. The positions corresponding to the grey rectangles are positions evaluated in the previous iteration. The positions corresponding to the hashed rectangles are positions evaluated in the previous iteration and will be reused in the current iteration. Positions 0, 1, 2, 7, and 6 will be evaluated in the new iteration.

[0182] In ECM, if multiple iterations of the refinement process are triggered, the results of the iteration (best MV / CPMV and minimum distortion) are stored (saved) only if the iteration is the first one. By doing so, when this refinement is combined with other algorithms, the other subsequent algorithms will not be initialized with the best identified MV / CPMV or use the minimum realized BM cost.

[0183] According to a first embodiment, after each iteration of the first or second step of the affine DMVR method, the results (optimal MV and minimum BM cost) are systematically stored.

[0184] In a second embodiment, the search strategy can be divided into two parts.

[0185] The primary positions may be, for example, left, top, right and bottom, i.e. positions {1,3,5,7} in FIG. The corner locations may be, for example, the top left corner, the top right corner, the bottom right corner, and the bottom left corner, i.e., locations {0,2,4,6} in FIG.

[0186] According to a second embodiment, the primary location is evaluated first, and the next corner location is evaluated based on the results of the neighboring locations of said corner location.

[0187] In a first modification of the second embodiment, if none of the neighboring positions of a corner position is the current optimum position, evaluation of the neighboring positions of the corner position is skipped.

[0188] For example, the evaluation of position 0 (corner position) can be processed based on the results of position 1, position 7, and initial position x (adjacent position of corner position 0) in Figure 16. If position 1, position 7, and initial position x are not the current optimal positions of the iteration, the probability that position 0 is the optimal position is low. In this case, it is beneficial to skip the evaluation of position 0 to save execution time.

[0189] In a second variation of the second embodiment, a corner position is evaluated if the BM costs of two adjacent positions of the corner position are within a predetermined range of the current optimal BM cost.

[0190] This range can be defined from 0 to the optimum distortion weighted by a factor strictly greater than one.

[0191] For example, consider that position 7 (adjacent to corner position 0) is the optimal position. Using the first variant of the second embodiment, skip positions 2 and 4 and evaluate positions 1, 7, and x. However, if the BM cost of position 1 is much higher than the current optimal BM cost (i.e., the BM cost of position 7), the probability that position 0 is the optimal position is low. If the BM costs of positions 1, 7, and x are all within a predetermined range (not low), the probability that position 0 is the optimal position is high.

[0192] In a third embodiment, the optimal MV that provides the smallest BM cost between the two reference video image lists L0 and L1 is selected. The refinement of motion vector candidates is processed for a given list (corresponding to one of the reference video image lists (either L0 or L1)) (the second step of affine DMVR), i.e., only one MV is refined. As described above in conjunction with Figure 16, the refinement offset of the MV follows the conventional scheme and search point order used by affine DMVR.

[0193] At the encoder side, the two reference video picture lists MV are sequentially and independently refined to identify and select the best prediction mode.

[0194] To measure the BM cost, a reference video picture list prediction needs to be available.

[0195] In the prior art, for a given PU, distortion is evaluated (e.g., evaluated by SAD) according to sub-blocks (from the PU size to 4x4 blocks), and two reference video picture list predictions are calculated. A sub-block is a group of consecutive 4x4 blocks (at least one) that share the same MV.

[0196] To optimize such evaluation for distortion (e.g., by SAD evaluation), the prediction of the unrefined reference video picture list remains constant / fixed and can be calculated only once if this prediction is intelligently stored for a given PU. For example, for the first defined MV refine offset, two predictions are calculated, but for the next MV refine offset, only the prediction for the already refined video picture list is calculated, saving half the prediction calculations.

[0197] Furthermore, instead of calculating distortion at the sub-block level (e.g., by SAD evaluation), distortion can be calculated at the PU level. In the prior art, distortion between dual-reference video picture list predictions (e.g., evaluated by SAD) is calculated at the sub-block level and summed across all sub-blocks to obtain distortion at the PU level (e.g., evaluated by SAD). Then, predictions can be generated at the sub-block level and distortion can be measured at the PU level. This helps reduce the number of function calls (one call per PU instead of one call per sub-block) and use more efficient intrinsics, thereby reducing execution time when processing larger pixel surfaces.

[0198] In a fourth embodiment, step 172 is bypassed for a CPMV based on a comparison of the bilateral matching cost associated with the CPMV performed in the first step 171 with a threshold.

[0199] This fourth embodiment avoids spending time evaluating merge candidates, thereby reducing the runtime consumption of the affine DMVR method, as described with respect to FIG. 17, since step 172 is the most runtime-consuming step of the affine DMVR method.

[0200] In a first modification of the fourth embodiment, if the BM cost performed for a CPMV in step 171 is lower than a threshold TH1, step 172 is bypassed for said CPMV.

[0201] After that, CPMV is partially optimized (refinement).

[0202] This early termination assumes that the BM cost (distortion) is already low and that the refinement has a clearly better rate-distortion tradeoff than the current merge candidate.

[0203] In a second variation of the fourth embodiment, if the BM cost performed for a CPMV in step 171 is higher than a threshold TH2, step 172 is bypassed for said CPMV.

[0204] After that, CPMV is partially optimized (refinement).

[0205] This early termination assumes that the BM cost (distortion) is sufficiently high and that the merge candidates are not sufficiently competitive.

[0206] In a third variant of the fourth embodiment, the thresholds TH1 and / or TH2 are adapted to the PU size or are fixed.

[0207] During the iterative refinement (steps 171 and 172), the CPMVs are successively refined to improve compression efficiency. The affine DMVR method attempts to refine the first CPMV, then the second CPMV, and finally, in the case of a six-parameter affine model, the third CPMV. The refinements are detailed, i.e., every CPMV is refined regardless of the results of the refinement of the previous CPMV.

[0208] In a third variation of the fourth embodiment, if the BM cost (distortion) associated with the first CPMV or the second CPMV satisfies the bypass condition, the second step 172 is bypassed for the third CPMV.

[0209] In one example of the third variant of the fourth embodiment, if the BM cost associated with the first CPMV or the second CPMV performed in the second step 172 is higher than the BM cost associated with the first CPMV or the second CPMV performed in the first step 171, the bypass condition is met.

[0210] In another third variant of the fourth embodiment, if the BM cost associated with the first CPMV or the second CPMV performed in the second step 172 is lower than the BM associated with the first CPMV or the second CPMV performed in the first step 171, and the reduction in the bilateral cost is lower than a third threshold TH3, the bypass condition is met.

[0211] In a fifth embodiment, the affine DMVR method is disabled for certain block (CU) shapes.

[0212] In the prior art, affine DMVR is performed for each CU whose CU height and CU width are equal to or greater than 8 luma samples. However, the affine DMVR method can provide different execution time / compression efficiency tradeoffs for every CU shape. Therefore, for example, if the execution time / compression efficiency tradeoff is not satisfactory, it is worth considering disabling affine DMVR for a specific CU shape. By identifying CU shapes for which affine DMVR performance is poor, affine DMVR can be disabled to save execution time.

[0213] This fifth embodiment is even more preferable because when bypassing the affine DMVR for a CU with a particular shape, the CU syntax signaled for this CU is also bypassed, which can further reduce the bitrate.

[0214] In a first variant of the fifth embodiment, the affine DMVR method (steps 171 and 172) can be disabled if the width or height of the inter-coded block (CU) is higher than a first value (e.g., 128) when the width and height are equal.

[0215] In a second variant of the fifth embodiment, the affine DMVR method (steps 171 and 172) can be disabled if the width or height of the inter-coded block (CU) is higher than a second value (e.g., 64) when the width and height are different.

[0216] In the first and second variants of the fifth embodiment, the shape of the CU for which affine DMVR is disabled (width and height of the inter-coded CU) may be signaled in the bitstream to be replicated at the decoder side.

[0217] In a first embodiment of said variant, at least one particular CU shape that overrides the affine DMVR (steps 171 and 172) can be signaled by signaling a maximum and / or minimum CU size (i.e., a maximum and minimum CU width and / or height, e.g., expressed as a number of pixels).

[0218] In a second embodiment of the variant, the at least one block shape that disables the affine DMVR (steps 171 and 172) can be signaled by signaling a minimum and / or maximum CU ratio between the maximum CU width and block height and the minimum CU width and CU height.

[0219] In the prior art, for each inter mode, neighboring blocks, temporal information, and combined spatial and temporal information are used to generate a list of candidates, as explained in the introduction of this application.

[0220] For the affine DMVR method, available spatial and temporal information is used to generate a merge candidate list. The merge candidate list is not optimal, and the same merge candidate can be duplicated in the merge candidate list.

[0221] For several reasons, overlapping merge candidates are suboptimal.

[0222] First, overlapping merge candidates are evaluated by the DMVR method like merge candidates, but are not selected because they provide lower compression efficiency, which is due to their higher merge index compared to merge candidates with lower merge index. Then, evaluating overlapping merge candidates increases the execution time of the DMVR method while not enhancing compression efficiency.

[0223] Second, the presence of a duplicate merge candidate in a merge candidate list degrades the rate-distortion performance of the merge candidate following the duplicate merge candidate, because it unnecessarily increases the merge index associated with the subsequent merge candidate.

[0224] Third, if the merge candidate list is finite, discard some potential merge candidates if the maximum size of the list is reached and there are useless duplicate merge candidates in the merge candidate list.

[0225] To overcome this drawback, the sixth embodiment checks whether a new merge candidate to be added to the merge candidate list is already present in the merge candidate list.

[0226] The sixth embodiment helps to limit the runtime overhead of the DMVE method while providing better compaction performance compared to a merge candidate list that includes at least one overlapping merge candidate.

[0227] In a variation of the sixth embodiment, the reference video image index of the new merge candidate is compared with the reference video image index of the merge candidate in the merge candidate list, and the MVs of the two reference video image lists (L0 and L1) are compared. In the case of an affine block, all CPMVs of the two lists are compared. More precisely, each affine merge index points to motion prediction information that indicates which CPMV is used to derive motion information. The motion information is represented by one temporal prediction type (unidirectional or bidirectional), up to two reference video image indexes, and up to three control point motion vectors (CPMVs), where each CPMV is associated with a reference video image index of one of the two reference video image lists (L0 or L1). Next, when comparing two affine merge candidates, the motion information of the two lists (L0 and / or L1) is compared. For lists, this involves comparing the reference video image indexes and CPMVs.

[0228] If the reference index and MV / CPMV are exactly the same, the potential candidate is identified as a duplicate.

[0229] A variation of the sixth embodiment discards only merge candidates that are exact duplicates. It may be extended by discarding potential candidates whose MV / CPMV are too close to candidates in one list.

[0230] For example, if all the differences between the MV / CPMV of a potential candidate and the MV / CPMV of one of the merge candidates in the list are below a threshold, discard this potential candidate.

[0231] The comparison includes comparing the reference indexes, calculating the MV / CPMV difference, and comparing the MV / CPMV difference with a threshold. In this example, the purpose is to compare the motion information of two candidates. It differs from the previous description in that the criterion in CPMV is expanded, thereby discarding more candidates. It now includes similar CPMV of the candidates in the list, i.e., discarding candidates whose CPMV difference between all two merge candidates is lower than the threshold.

[0232] If the reference indexes are exactly the same and the MV / CPMV difference is lower than the threshold TH4, the potential candidate is not added to the merge candidate list. The threshold TH4 may be fixed or self-adaptive, e.g., adjusted according to the PU size.

[0233] For the DMVR method, at the encoder side, all merge candidates in the merge candidate list are refined, i.e., the final merge candidate may differ from the merge candidates in the list.

[0234] Thus, a merge candidate in the list can be matched to an affine DMVR refined merge candidate from the list. More precisely, the encoder runs the affine DMVR algorithm on all affine merge candidates in the list. As described below, the affine DMVR refines the CPMV of the affine merge candidate, thereby changing the CPMV. The Nth refined affine merge candidate in the list is then likely to obtain motion information (reference picture index and CPMV) that is the same as or similar (i.e., close in CPMV) to an affine merge candidate with a merge index higher than N.

[0235] In this case, applying the DMVR method to the matching merge candidates is inefficient. Therefore, at the encoder side (which requires evaluating each merge candidate), it is possible to bypass the DMVR refinement of a merge candidate if the merge candidate is similar to a DMVR-refined merge candidate. A strategy similar to the one described above can be used to discard the merge candidate DMVR refinement.

[0236] FIG. 19 illustrates an exemplary schematic block diagram of a system 600 for implementing various aspects and embodiments.

[0237] System 600 may be implemented as one or more devices including various components described below. In various embodiments, system 600 may be configured to implement one or more aspects described herein.

[0238] Examples of devices that may comprise all or part of system 600 may include a personal computer, laptop, smartphone, tablet, digital multimedia set-top box, digital television receiver, personal video recording system, connected consumer electronics, connected vehicle and associated processing system, head-mounted display device (HMD, see-through glasses), projector (beamer), “cave” (a system including multiple displays), server, video encoder, video decoder, post-processor that processes output from a video decoder, pre-processor that provides input to a video encoder, web server, video server (e.g., broadcast server, video broadcast server, or network server), still or video camera, encoding or decoding chip, or any other communications device. The elements of system 600, singly or in combination, may be implemented within a single integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 600 may be distributed across multiple ICs and / or discrete components. In various embodiments, system 600 may be communicatively coupled to other similar systems or other electronic devices, for example, via a communications bus or dedicated input and / or output ports.

[0239] System 600 may include at least one processor 610 configured to execute loaded instructions to implement aspects described herein. Processor 610 may include embedded memory, input / output interfaces, and various other circuits known in the art. System 600 may also include at least one memory 620 (e.g., volatile and / or nonvolatile memory devices). System 600 may also include a storage device 640 including nonvolatile and / or volatile memory, including, but not limited to, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash, magnetic disk drives, and / or optical disk drives. By way of non-limiting example, storage device 640 may include an internal storage device, an attached storage device, and / or a network-accessible storage device.

[0240] The system 600 may include an encoder / decoder module 630 configured to, for example, process data to provide encoded / decoded video image data. The encoder / decoder module 630 may include its own processor and memory. The encoder / decoder module 630 may represent a module or modules that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include either one or both of an encoding module and a decoding module. The encoder / decoder module 630 may also be implemented as a separate element of the system 600 or may be incorporated within the processor 610 as a combination of hardware and software known to those skilled in the art.

[0241] Program code loaded into the processor 610 or the encoder / decoder 630 to perform aspects described herein may be stored in the storage device 640 and loaded into the memory 620 for execution by the processor 610. According to various embodiments, during the execution of processes described herein, one or more of the processor 610, the memory 620, the storage device 640, and the encoder / decoder module 630 may store one or more of a variety of items, including, but not limited to, video image data, information data used in encoding / decoding the video image data, bitstreams, matrices, variables, equations, mathematical expressions, operations, and intermediate or final results of arithmetic logic processing.

[0242] In some embodiments, memory within the processor 610 and / or the encoder / decoder module 630 may be used to store instructions and provide working memory for processes performed during encoding or decoding.

[0243] However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 610 or the encoder / decoder module 630) may be used for one or more of these functions. The external memory may be the memory 620 and / or the storage device 640, and may be, for example, dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory may be used to store, for example, the operating system of a television. In at least one exemplary embodiment, high-speed external dynamic volatile memory such as RAM may be utilized as working memory used for video encoding and decoding operations, for example, MPEG-2 Part 2 (also known as ITU-T Recommendation H.262 and ISO / IEC 13818-2, and also referred to as MPEG-2 Video), AVC, HEVC, EVC, VVC, AV1, etc.

[0244] As shown in block 690, inputs to the elements of system 600 can be provided via various input devices. Such input devices may include, but are not limited to, (i) an RF section capable of receiving RF signals transmitted wirelessly, such as by a broadcast device, (ii) a combined input terminal, (iii) a USB input terminal, (iv) an HDMI input terminal, and (v) a bus such as a CAN (Controller Area Network), CAN FD (Controller Area Network Flexible Data Rate), FlexRay (ISO 17458), or Ethernet (ISO / IEC 802-3) bus when the present application is implemented in the automotive field.

[0245] In various embodiments, the input devices of block 690 may have associated corresponding input processing elements, as known in the art. For example, the RF section may be associated with elements necessary to (i) select a desired frequency (also referred to as selecting a signal or limiting a signal to a frequency band), (ii) downconvert the selected signal, (iii) bandlimit again to a narrower frequency band to select a signal frequency band, referred to as a channel in particular embodiments (for example), (iv) demodulate the downconverted, bandlimited signal, (v) perform error correction, and (vi) demultiplex to select a desired data packet stream. The RF section of various embodiments may include one or more elements for performing these functions, such as a frequency selector, a signal selector, a bandlimiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may also include a tuner to perform various functions, such as downconverting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency near baseband) or to baseband.

[0246] In one set-top box embodiment, the RF section and associated input processing elements can receive RF signals transmitted over a wired (e.g., cable) medium, and the RF section can filter, downconvert, and filter again to perform frequency selection to a frequency band of interest.

[0247] In various embodiments, the order of these (and other) elements may be rearranged, some of these elements may be removed, and / or other elements that perform similar or different functions may be added.

[0248] Adding elements includes inserting elements between conventional elements, such as amplifiers and analog-to-digital converters. In various embodiments, the RF may include an antenna.

[0249] The USB and / or HDMI terminals may also include corresponding interface processors to connect system 600 to other electronic devices via USB and / or HDMI connections. Note that, when desired, aspects of input processing (e.g., Reed-Solomon error correction) can be implemented in a separate input processing IC or within processor 610. Similarly, when desired, aspects of USB or HDMI interface processing can be implemented in a separate interface IC or within processor 610. The demodulated, error corrected, and demultiplexed streams can be provided to various processing elements, including processor 610 and encoder / decoder 630, operating in combination with memory and storage elements to process the data streams for display on an output device, when desired.

[0250] The various elements of system 600 may be provided within a unified housing, within which any suitable connection arrangement 690 may be used to interconnect and transmit data between the various elements, for example, using internal buses (including I2C buses), wiring, and printed circuit boards as known in the art.

[0251] System 600 may include a communication interface 650 to enable communication with other devices over a communication channel 651. Communication interface 650 may include, but is not limited to, a transceiver for transmitting and receiving data over communication channel 651. Communication interface 650 may include, but is not limited to, a modem or a network card, and communication channel 651 may be implemented within a wired and / or wireless medium, for example.

[0252] In various embodiments, a Wi-Fi network, such as IEEE 802.11, can be used to stream data to system 600. The Wi-Fi signal in these embodiments can be received via a communication channel 651 suitable for Wi-Fi communication and a communication interface 650. The communication channel 651 in these embodiments can typically be connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications.

[0253] Another embodiment may provide streamed data to the system 600 using a set-top box, which delivers the data via an HDMI connection in the input block 690.

[0254] Other embodiments may provide streamed data to the system 600 using the RF connection of the input block 690 .

[0255] The streamed data can be used by the system 600 as a method for signaling information. The signaling information may include information such as the number of pixels in the bitstream B and / or the video image and / or any encoding / decoding configuration parameters.

[0256] It should be noted that signaling can be achieved in various ways, for example, in various embodiments, one or more syntax elements, flags, etc. can be used to signal information to a corresponding decoder.

[0257] System 600 can provide output signals to various output devices, including a display 661, speakers 671, and other peripheral devices 681. In various example embodiments, other peripheral devices 681 can include one or more of a separate DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of system 600.

[0258] In various embodiments, control signals may be communicated between the system 600 and the display 661, speaker 671, or other peripheral device 681 using AV.Link (Audio / Video Link), CEC (Consumer Electronics Control), or other communication protocol signaling that enables device-to-device control, with or without a user.

[0259] Output devices can be communicatively connected to the system 600 via dedicated connections by corresponding interfaces 660, 670 and 680.

[0260] Optionally, an output device can be connected to system 600 using communication channel 651 via communication interface 650. Display 661 and speakers 671 may be combined into a single unit along with other components of system 600 in an electronic device (e.g., a television).

[0261] In various embodiments, the display interface 660 can include a display driver, such as a timing controller (TCon) chip.

[0262] For example, if the RF portion of input 690 is part of a separate set-top box, display 661 and speakers 671 may optionally be configured separately from one or more of the other components. In various embodiments where display 661 and speakers 671 may be external components, the output signal may be provided via a dedicated output connection (including, for example, an HDMI port, a USB port, or a COMP output terminal).

[0263] 1-19 illustrate various methods, each of which includes one or more steps or actions to achieve the described method. Unless a specific order of steps or actions is required for the precise operation of the method, the order and / or use of specific steps and / or actions can be modified or combined.

[0264] While some examples have been described with reference to block diagrams and / or operational flowcharts, each block represents a circuit element, module, or portion containing one or more executable instruction codes for implementing a specified logic function(s). It should be noted that in other embodiments, the function(s) shown in the blocks may not occur in the order shown. For example, depending on the functionality involved, two blocks shown one after the other may actually be executed in parallel, or the blocks may be executed in the reverse order.

[0265] Each embodiment and aspect described herein may be embodied as, for example, a method or process, an apparatus, a computer program, a data flow, a bitstream, or a signal. Even if discussed only in the context of a single type of embodiment (e.g., discussed only as a method), embodiments of the discussed features may be embodied in other forms (e.g., an apparatus or a computer program).

[0266] The methods may be implemented in, for example, a processor, which generally refers to a processing device including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device, etc. Processors further include communication devices.

[0267] Additionally, methods may be implemented with instructions executed by a processor, and such instructions (and / or data values ​​produced by embodiments) may be stored on a computer-readable storage medium. A computer-readable storage medium may take the form of a computer-readable program product embodied in one or more computer-readable media and having computer-readable program code embodied therein and executable by a computer. Given their inherent ability to store information thereon and retrieve information provided thereby, computer-readable storage media as used herein may be considered non-transitory storage media. A computer-readable storage medium may be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Although the following provides more specific examples of computer-readable storage media to which embodiments of the present application may be applied, it should be understood that these are merely exemplary and not an exhaustive list, as would be readily apparent to one skilled in the art: portable computer floppy disks, hard disks, read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0268] The instructions may create an application tangibly embodied on a processor-readable medium.

[0269] For example, instructions may reside in hardware, firmware, software, or a combination thereof. For example, instructions may be found in an operating system, a standalone application, or a combination of both. A processor may therefore be characterized as, for example, a device configured to perform a process or a device that includes a processor-readable medium (e.g., a storage device) having instructions for performing a process. Also, in addition to or in place of instructions, the processor-readable medium may store data values ​​produced by an embodiment.

[0270] The device can be implemented in, for example, appropriate hardware, software, and firmware. Examples of such devices include personal computers, laptop computers, smartphones, tablets, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected consumer electronics products, head-mounted displays (HMDs, see-through glasses), projectors, "caves" (systems including multiple displays), servers, video encoders, video decoders, post-processors that process output from video decoders, pre-processors that provide input to video encoders, web servers, set-top boxes, and any other device for processing video images or other communication devices. Note that the device may be mobile and may be mounted in a moving vehicle.

[0271] The computer software may be implemented in the processor 610, in hardware, or in a combination of hardware and software. By way of non-limiting example, each embodiment may be implemented in one or more integrated circuits. The memory 620 may be of any type suitable for the technological environment and may be implemented in any suitable data storage technology (by way of non-limiting example, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memories, and removable memories). By way of non-limiting example, the processor 610 may be of any type suitable for the technological environment and may cover one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0272] An embodiment of the present application also provides an electronic device comprising a processor and a memory for storing instructions executable by the processor, the processor being configured to perform a method according to any of the above examples.

[0273] As will be apparent to those skilled in the art, embodiments can generate a signal formatted to carry information, for example, to be stored or transmitted. The information may include instructions for performing a method or data generated by one of the described embodiments. For example, a signal may be formatted to carry a bit stream of the described examples. The signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or baseband. Formatting may include encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, the signal can be transmitted over a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0274] The terms used herein are for the purpose of describing particular embodiments and are not intended to be limiting. Unless the context dictates otherwise, the singular forms "a," "an," and "the" as used herein are intended to include the plural. Furthermore, as used herein, the terms "comprises" and / or "comprising" may indicate the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof. Furthermore, when an element is said to be "responsive to," "connected," or "associated with" another element, it may be directly responsive to or connected to the other element, or intermediate elements may be present. In contrast, when an element is said to be "directly responsive to," "directly connected to," or "directly associated with" another element, there may be no intermediate elements present.

[0275] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," the use of any one of the symbols / terms " / ," "and / or," and "at least one" is intended to cover the selection of the first listed alternative (A), the selection of the second listed alternative (B), or the selection of two alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to cover the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, B, and C). This can be extended to any number of listed items, as would be apparent to one skilled in the art.

[0276] Various numerical values ​​may be used in this application, and the specific values ​​are exemplary and the described aspects are not limited to these specific values.

[0277] It should be noted that terms such as "first," "second," etc. may be used to describe various elements herein, but are not limited to these terms. These terms are used only to distinguish one element from another. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element, without departing from the teachings of the present application. No ordering is implied between a first element and a second element.

[0278] References to "one example" or "example" or "one embodiment" or "embodiment" and other variations are often used to convey that a particular feature, structure, characteristic, etc. (as described in connection with the example / embodiment) is included in at least one example / embodiment. Thus, the appearance of the terms "in one example" or "in an example" or "in one embodiment" or "in an embodiment" and any other variations appearing in various places in this application does not necessarily refer to all examples.

[0279] References to "one example" or "embodiment" or "one embodiment" or "embodiment" and other variations are often used to convey that a particular feature, structure, characteristic, etc. (as described in connection with the example / embodiment) is included in at least one example / embodiment. Thus, the phrases "according to an example / example / embodiment" or "in an example / example / embodiment" appearing in various places in this application do not necessarily refer to the same example / example / embodiment, nor are single or alternative examples / examples / embodiments necessarily mutually exclusive of other examples / examples / embodiments.

[0280] The reference numerals of the drawings appearing in the claims are for illustrative purposes only and do not limit the scope of the claims. The embodiments / examples and variants of the present application may be used in any combination or sub-combination, even if not explicitly described.

[0281] It is understood that when a figure is shown as a flow chart, a block diagram of the corresponding apparatus is also provided. Similarly, it is understood that when a figure is shown as a block diagram, a flow chart of the corresponding method / process is also provided.

[0282] Some figures include arrows that indicate the primary direction of communication in a communication path, however, it should be understood that communication can occur in the opposite direction to the illustrated arrow.

[0283] Various embodiments relate to decoding. As used herein, "decoding" may cover all or part of a process, for example, performed on received video images (which may include a received bitstream encoding one or more video images) to produce a final output suitable for display or further processing in the reconstructed video domain. In various embodiments, such a process may include one or more of the processes typically performed by a decoder. In various embodiments, for example, such a process may alternatively include a process performed by the decoder of various embodiments described herein.

[0284] As a further example, in one embodiment, "decoding" may refer to only inverse quantization, in one embodiment, "decoding" may refer to entropy decoding, in another embodiment, "decoding" may refer to differential decoding, and in another embodiment, "decoding" may refer to a combination of inverse quantization, entropy decoding, and differential decoding. Based on the specific context described, those skilled in the art will be able to understand whether the term "decoding process" refers to a subset of operations or to the broader decoding process.

[0285] Various embodiments relate to encoding. Similar to the discussion of "decoding" above, "encoding," as used herein, can include all or part of the processes performed on input video images to generate an output bitstream. In various embodiments, this type of process includes one or more of the processes typically performed by an encoder. In various embodiments, this process also includes, or alternatively includes, the processes performed by the encoder of various embodiments described herein.

[0286] As a further example, in one embodiment, "encoding" may refer only to quantization, in one embodiment, "encoding" may refer only to entropy coding, in another embodiment, "encoding" may refer only to differential coding, and in another embodiment, "encoding" may refer to a combination of quantization, differential coding, and entropy coding. Based on the context specifically described, it will be clear and well understood by those skilled in the art whether the term "encoding process" refers specifically to a subset of operations or to a broader encoding process.

[0287] Also, references are made herein to various "obtaining" of information, which may include, for example, one or more of: estimating information, calculating information, predicting information, or retrieving information from memory, processing information, moving information, copying information, deleting information, calculating information, determining information, predicting information, or estimating information.

[0288] Also, throughout this application, references are made to "receiving" various pieces of information. Receiving information may include, for example, one or more of: accessing the information or receiving the information from a communications network.

[0289] Additionally, as used herein, the term "signal" specifically refers to instructing a corresponding decoder to do something specific. For example, in some embodiments, an encoder signals specific information, such as coded parameters or encoded video image data. This allows the encoder and decoder to use the same parameters in some embodiments. Thus, for example, the encoder can transmit specific parameters to the decoder (explicit signaling), allowing the decoder to use the same specific parameters. Conversely, if the decoder has specific parameters and other parameters, signaling that does not require transmission (indirect signaling) can be used to inform the decoder and facilitate selection of the specific parameters. Various embodiments achieve bit savings by avoiding transmission of any actual functions. It should be understood that multiple signaling methods can be implemented. For example, in various embodiments, one or more grammatical elements, flags, etc., can be used to transmit information to a corresponding decoder. While the above relates to the verb form of the word "signal," the word "signal" may also be used as a noun in this specification.

[0290] While several embodiments have been described above, it should be understood that various modifications are possible. For example, elements of different embodiments can be combined, supplemented, modified, or deleted to produce other embodiments. Moreover, those skilled in the art will recognize that other structures and processes can be substituted for the disclosed structures and processes, thereby producing embodiments that perform essentially the same function(s) in essentially the same way(s) to achieve at least essentially the same result(s) as the disclosed embodiments. Accordingly, these and other embodiments are contemplated by this application.

Claims

1. 1. A method for decoding a video image, comprising: the decoding is performed by temporal bi-directional prediction with motion compensation for inter-coded blocks; The motion compensated temporal bidirectional prediction includes: two reference video images in two separate reference video image lists; two affine motion fields defined by at least two control point motion vectors, the control point motion vectors being expressed as CPMVs, the at least two control point motion vectors being associated with each reference image; The refined CPMV is obtained as the output of a first step (171) of the following, or alternatively as the output of a second step (172) after said first step (171), - in said first step (171), for each CPMV, performing bilateral matching on a block centered on said CPMV to derive at least two CPMVs refined with integer precision, and selecting a CPMV set including unrefined CPMVs and / or CPMVs refined with integer precision to generate an overall predicted block with the minimum bilateral matching cost; - in said second step (172), for each successive CPMV in the selected CPMV set associated with the block, refine each CPMV with sub-sample precision to minimize the bilateral matching cost of said block; bypassing the second step (172) based on a comparison of the bilateral matching cost associated with the CPMV performed in the first step with a threshold value; A method for decoding a video image.

2. If the bilateral matching cost associated with the CPMV performed in the first step (171) is lower than a first threshold (TH1), bypassing the second step (172) for the CPMV; 2. The method of decoding a video image according to claim 1.

3. If the bilateral matching cost associated with the CPMV performed in the first step (171) is higher than a second threshold (TH2), bypassing the second step (172) for the CPMV; 2. The method of decoding a video image according to claim 1.

4. the first threshold (TH1) and / or the second threshold (TH2) are fixed or adapted to the prediction unit size; A method for decoding video images according to claim 2 or 3.

5. the two affine motion fields are defined by a first CPMV, a second CPMV, and a third CPMV, and if a bilateral matching cost associated with the first CPMV or the second CPMV satisfies a bypass condition, bypassing the second step for the third CPMV; A method for decoding a video image according to any one of claims 1 to 4.

6. If the bilateral cost associated with the first CPMV or the second CPMV performed in the second step (172) is higher than the bilateral matching cost associated with the first CPMV or the second CPMV performed in the first step (171), a bypass condition is met. A method for decoding video images according to claim 5.

7. a bypass condition is met if the bilateral matching cost associated with the first CPMV or the second CPMV performed in the second step (172) is lower than the bilateral matching cost associated with the first CPMV or the second CPMV performed in the first step (171), and the reduction in the bilateral cost is lower than a third threshold (TH3); A method for decoding video images according to claim 5.

8. Disabling the first step and the second step for a specific block shape; A method for decoding video images according to any one of claims 1 to 7.

9. Disabling the first step and the second step if the width or height of the inter-coded block is higher than a first value when the width and height are equal.

9. A method for decoding video images according to claim 8.

10. Disabling the first and second steps if the width or height of the inter-coded block is higher than a second value when the width and height are different. A method for decoding video images according to claim 8 or 9.

11. defining block shapes based on the width or height of the inter-coded blocks and signaling at least one block shape in the bitstream that disables the first step and the second step; A method for decoding video images according to any of claims 8 to 10.

12. the at least one block shape that overrides the first and second steps is signaled by signaling a maximum and a minimum block width and / or height. A method for decoding video images according to claim 11.

13. the at least one block shape that disables the first and second steps is signaled by signaling a minimum and / or maximum block ratio between a maximum value of a block width and a maximum value of a block height and a minimum value of a block width and a minimum value of a block height. A method for decoding video images according to claim 11.

14. comprising components for carrying out one of the methods according to any one of claims 1 to 13, Device.

15. A computer program product comprising instructions, which when executed by one or more processors, cause said one or more processors to perform a method according to any of claims 1 to 13. Computer program products.

16. comprising program code instructions for carrying out the method according to any one of claims 1 to 13, Non-transitory storage media.

17. 1. An electronic device comprising: a processor; a memory for storing instructions executable by said processor; The processor is configured to perform the method according to any one of claims 1 to 13. Electronic devices.

Images