Video image data encoding / decoding

Abstract

The present application relates to a method for decoding video images, the decoding being performed by motion-compensated temporal bidirectional prediction for an inter-coded block, the motion-compensated temporal bidirectional prediction using two reference video images in two separate reference video image lists and two affine motion fields defined by at least two control point motion vectors, the control point motion vectors being represented as CPMVs, at least two control point motion vectors being associated with each reference image, the refined CPMVs being obtained as output of a first step (171) of the following or, alternatively, as output of a second step (172) after the first step (171): In the first step (171), for each CPMV, bilateral matching is performed on a block centered on the CPMV to derive at least two CPMVs refined with integer precision, and a CPMV set including the unrefined CPMV and / or the CPMV refined with integer precision is selected to generate an overall predicted block with the minimum bilateral matching cost; - in said second step (172), for each successive CPMV in the selected CPMV set associated with the block, refine each CPMV with sub-sample precision to minimize the bilateral matching cost of said block; The second step (172) is bypassed based on a comparison of the bilateral matching cost associated with the CPMV performed in the first step with a threshold.

Description

Cross-citation of related applications

[0001] This disclosure claims priority and interest in European Patent Application No. 23305027.7, filed on 9 January 2023, all of which are incorporated herein by reference. [Technical Field]

[0002] This application generally relates to the encoding and decoding of video images. In particular, the art of this application relates to, but is not limited to, the interpretation of video image blocks. [Background technology]

[0003] This section aims to introduce to the reader various aspects of the art, which relate to each aspect of at least one exemplary embodiment of the present application described below and / or for which protection is claimed. This description is intended to provide the reader with background information to better understand the various aspects of the present application. Therefore, these aspects do not constitute endorsements of related technology and should be read in the above-mentioned context.

[0004] State-of-the-art video compression systems such as HEVC (ISO / IEC 23008-2 High Efficiency Video Coding, ITU-T Recommendation H.265, https: / / www.itu.int / rec / T-REC-H.265-202108-P / en) or VVC (ISO / IEC 23090-3 Versatile Video Coding, ITU-T Recommendation H.266, https: / / www.itu.int / rec / T-REC-H.266-202008-I / en) provide low-level and high-level image partitions to divide the video image into image regions called coding tree units (CTUs). In the case of HEVC, the size of the coding tree unit (CTU) is typically between 16x16 pixels and 64x64 pixels, while in the case of VVC, the size of the coding tree unit (CTU) may be 32x32, 64x64, or 128x128 pixels.

[0005] The CTU partitioning of a video image forms a grid of fixed-size CTUs, or CTU grids, where the top and left boundaries of this grid spatially overlap with the top and left boundaries of the video image. The CTU grid represents the spatial partitions of the video image.

[0006] In VVC and HEVC, the CTU size (CTU width and CTU height) of all CTUs in a CTU grid is equal to the same default CTU size (default CTU width CTU DW and default CTU height CTU DH). For example, the default CTU size (default CTU height, default CTU width) may be 128 (CTU DW = CTU DH = 128). The default CTU size (height, width) is encoded into a bitstream, which is then encoded into a sequence level in a sequence parameter set (SPS), for example.

[0007] The spatial position of a CTU within a CTU grid is determined based on the CTU address ctuAddr, which defines the spatial position of the CTU from the origin at its upper-left corner. As shown in Figure 1, the CTU address can define the spatial position of the CTU from the upper-left corner of the higher-level spatial structure S containing the CTU.

[0008] The encoding tree is associated with each CTU and determines the tree partitioning of the CTU.

[0009] As shown in Figure 1, in HEVC, the coding tree is a quadtree partition of CTUs, and each leaf is called a coding unit (CU). The spatial location of a CU in a video image is defined by the CU index cuIdx, which indicates the spatial location from the upper-left corner of the CTU. A CU is spatially divided into one or more prediction units (PUs). The spatial location of a PU in a video image VP is defined by the PU index puIdx, which defines the spatial location from the upper-left corner of the CTU, and the spatial location of the elements of a partitioned PU is defined by the PU partition index puPartIdx, which defines the spatial location from the upper-left corner of the PU. Each PU is assigned some intra or inter-prediction data.

[0010] The intra- or inter-coding mode is assigned at the CU level. This means that the same intra / inter-coding mode is assigned to each PU of the CU, even though the prediction parameters differ among the PUs.

[0011] Based on a quadtree called a transformation tree, a CU can be spatially divided into one or more transformation units (TUs). A transformation unit is a leaf of the transformation tree. The spatial position of a TU in a video image is defined by the TU index tuIdx, which defines the spatial position from the upper-left corner of the CU. Several transformation parameters are assigned to each TU. A transformation type is assigned at the TU level, and a 2D single transformation is performed at the TU level during the encoding or decoding period of the image block.

[0012] The PU partition types present in HEVC are shown in Figure 2. They include square partitions (2N×2N and N×N), which are the only partitions used in both intra-predictive CUs and inter-predictive CUs; ​​symmetric non-square partitions (2N×N, N×2N, used only in inter-predictive CUs); and asymmetric partitions (used only in inter-predictive CUs). For example, PU type 2N×nU represents an asymmetric horizontal partition of the PU, with the smaller partition located at the top of the PU. In another example, PU type 2N×nL represents an asymmetric horizontal partition of the PU, with the smaller partition located at the top of the PU.

[0013] As shown in Figure 3, in VVC, the coding tree begins with the root node (i.e., CTU). The quadtree (or quad-tree) partition then divides the root node into four nodes corresponding to four subblocks of the same size (solid lines). Subsequently, the leaves of the quadtree (or quad-tree) can be further partitioned by so-called multi-type trees, which relate to one of the four partitioning modes shown in Figure 4, either binary or ternary. These partitioning types are the vertical and horizontal binary partitioning modes denoted as SBTV and SBTH, and the vertical and horizontal ternary partitioning modes SPTTV and STTH.

[0014] In the case of a co-encoded tree where the luma and chroma components are shared, the leaf of the CTU's encoding tree is the CU.

[0015] In contrast to HEVC, in VVC, most of the time, the CU, PU, and TU have the same size, which means that, except for some specific coding modes, the coding unit is generally not divided into a PU or a TU.

[0016] FIG. 5 and FIG. 6 provide an overview of the video encoding / decoding method used in current video standard compression systems (e.g., HEVC or VVC).

[0017] FIG. 5 shows an exemplary block diagram of the steps of a method 100 for encoding a video image VP based on related techniques.

[0018] In step 110, the video image VP is divided into sample blocks, and the division information data is signaled to the bitstream. Each block contains samples of one component of the video image VP. Thus, these blocks contain samples of each component that defines the video image VP.

[0019] For example, in HEVC, an image is divided into coding tree units (CTUs). Each CTU can be further subdivided by quadtree partitioning, where each leaf of the quadtree represents a coding unit (CU). And the division information data may include data describing the CTU and the quadtree subdivision of each CTU.

[0020] Thereafter, each sample block (abbreviated as block) may be a CU (if the CU contains a single PU) or a PU of a CU.

[0021] Each block is encoded along the encoding loop (also called "inside the loop") using an intra or inter prediction mode.

[0022] Intra-prediction (step 120) uses intra-prediction data. Intra-prediction predicts the current block using intra-prediction blocks based on already encoded, decoded, and reconstructed samples located around the current block (typically at the top and left of the current block). Intra-prediction is performed in the spatial domain.

[0023] In inter-prediction mode, motion estimation (step 130) and motion compensation (135) are performed. Motion estimation searches for candidate reference blocks as good predictors of the current block in one or more reference video images that predictively encode the current video image. For example, a good predictor of the current block is a predictor similar to the current block. The output of motion estimation step 130 is inter-prediction data that includes motion information associated with the current block (usually one or more motion vectors and one or more reference video image indices) and other information for obtaining the same predicted block on the encoding / decoding side. Subsequently, motion compensation (step 135) obtains a predicted block using the (one or more) motion vectors and (one or more) reference video image indices determined in motion estimation step 130. Basically, any block belonging to the selected reference video images and pointed to by the motion vectors is available as a predicted block for the current block. Furthermore, because motion vectors are represented as fractions of integer pixel positions (known as sub-pixel precision motion vector representation), motion compensation typically involves spatial interpolation of several reconstructed samples of a reference video image to compute predicted blocks.

[0024] Prediction information data is signaled to the bitstream. The prediction information may include the prediction mode (intra, inter, or skip), intra / inter prediction data, and any other information necessary to obtain the same prediction CU on the decoding side.

[0025] Method 100 selects a prediction mode (intra or inter-prediction mode) by optimizing the rate-distortion trade-off, for example, by considering the encoding of prediction residual blocks calculated by subtracting candidate prediction blocks from the current block, and the signaling of prediction information data necessary to determine the candidate prediction blocks on the decoding side.

[0026] Typically, the best prediction mode is given as the prediction mode of the best coding mode p* for the current block, as follows:

number

[0027] Here, P is the set of all candidate coding modes in the current block, p is the candidate coding mode in that set, and RD cost (p) is the rate distortion cost of the candidate coding mode p, and is usually expressed as follows:

number

[0028] D(p) is the distortion between the current block and the reconstructed block obtained by encoding / decoding the current block using the candidate encoding mode p; R(p) is the rate cost associated with encoding the current block using encoding mode p; and λ is a Lagrangian parameter representing the rate constraint for encoding the current block, which is generally calculated based on the quantization parameter for encoding the current block.

[0029] Typically, the current block is encoded based on the predicted residual block PR. More precisely, for example, the predicted residual block PR is calculated by subtracting the best predicted block from the current block. Then, the predicted residual block PR is transformed using, for example, a DCT (Discrete Cosine Transform) or DST (Discrete Sine Transform) type transformation or any other suitable transformation (step 140), and the obtained transformed coefficient block is quantized (step 150).

[0030] In a modified version, method 100 can skip the transformation step 140, skip the encoding mode by so-called transformation, and directly apply quantization (step 150) to the predicted residual block PR.

[0031] The quantized transformation coefficient block (or quantized prediction residual block) is entropy-encoded into a bitstream (step 160).

[0032] Next, as part of the coding loop, the quantized transformation coefficient block (or quantized residual block) is subjected to inverse quantization (step 170) and inverse transformation (180) (or not inversely transformed) to generate a decoded predicted residual block. The decoded predicted residual block and the predicted block are then combined and usually summed, which provides the reconstructed block.

[0033] Furthermore, other informational data may be entropically encoded in step 160 to encode the current block of the video image VP.

[0034] Artifacts can be reduced by applying an intra-loop filter (step 190) to the reconstructed image (including the reconstructed blocks). After all image blocks have been reconstructed, a loop filter can be applied. These include, for example, a deblocking filter, a sample self-adaptive offset (SAO), or a self-adaptive loop filter.

[0035] The reconstructed block or filtered reconstructed block is formed as a reference image, which can be stored in a decoded image buffer (DPB), so that it can be used as the next current block of the video image VP or as the encoded reference image of the next video image to be encoded.

[0036] Figure 6 shows an illustrative block diagram of the steps of method 200 for decoding a video image VP based on related technology.

[0037] In step 210, the bitstream of encoded video image data is entropy-decoded to obtain segmentation information data, prediction information data, and quantized transformation coefficient blocks (or quantized residual blocks). For example, this bitstream is generated based on method 100.

[0038] By entropy decoding other information data, it is also possible to decode the current block of the video image VP from the bitstream.

[0039] In step 220, the reconstructed image is divided into current blocks based on the division information. Each current block is entropy-decoded from the bitstream along the decoding loop (also called "in the loop"). Each decoded current block is either a quantized transformation coefficient block or a quantized prediction residual block.

[0040] In step 230, the current block is dequantized and, if applicable, the inverse transform (step 240) is performed to obtain the decoded predicted residual block.

[0041] On the other hand, the prediction information data is used to predict the current block. The predicted block is obtained by its intra-prediction (step 250) or motion-compensated time prediction (step 260). The prediction process performed on the decoding side is the same as the prediction process performed on the encoding side.

[0042] Next, the decoded prediction residual blocks and prediction blocks are merged and usually summed, which provides the blocks for reconstruction.

[0043] In step 270, the intra-loop filter is applicable to the reconstructed image (including the reconstructed blocks), and the reconstructed blocks or filtered reconstructed blocks are formed as a reference image, which can be stored in the decoded image buffer (DPB) described above (Figure 5).

[0044] In steps 130 / 135 of Figure 5 or step 260 of Figure 6, an inter-prediction block is defined from inter-prediction data associated with the current block (CU or PU of CU) of the video image. This inter-prediction data contains motion information that can be displayed (encoded) based on a so-called whole-block-based AMVP mode (self-adaptive motion vector prediction) or a so-called whole-block-based merge mode.

[0045] In HEVC, in block-based AMVP mode, motion information for defining an interprediction block is represented by up to two reference video image indices, each associated with up to two reference video image lists (typically represented as L0 and L1). Reference video image list L0 contains at least one reference video image, and reference video image list L1 contains at least one reference video image. Each reference video image index is a time prediction for the current block. The motion information further includes up to two motion vectors, each motion vector associated with one reference video image index from the two reference video image lists. Each motion vector is predictively encoded and signaled to the bitstream; that is, one motion vector difference MVd is derived from the motion vectors, one AMVP (self-adaptive motion vector predictor) candidate is selected from a block-based AMVP candidate list (constructed on the encoding and decoding side), and the MVd is signaled to the bitstream. The index of the AMVP candidate selected from the block-based AMVP candidate list is similarly signaled in the bitstream.

[0046] Figure 7 shows a descriptive example for constructing a block-based AMVP candidate list, which is used to define the interpretation blocks of the current block of the current video image.

[0047] The block-based AMVP candidate list may include two spatial MVP (motion vector predictor) candidates derived from the current video image. The first spatial MVP candidate is derived from motion information associated with the interprediction block of the current block and, if present, is located at adjacent positions A0, A1 on the left side of the current block. The second MVP candidate is derived from motion information associated with the interprediction block of the current block and, if present, is located at adjacent positions B0, B1, and B2 at the top of the current block. A redundancy check is then performed among the derived spatial MVP candidates, i.e., duplicate derived MVP candidates are discarded. The block-based AMVP candidate list may further include a temporal MVP candidate, which is derived from motion information associated with a co-located block (if present) at spatial position H or, in other cases, spatial position C in the reference video image. The temporal MVP candidate is scaled based on the temporal distance between the current video image and the reference video image. Finally, if the block-based AMVP candidate list contains fewer than two MVP candidates, embed the block-based AMVP candidate list with a zero-motion vector.

[0048] In HEVC, in block-based merge mode, motion information for defining interprediction blocks is represented by merge indices in a block-based MVP candidate list. Each merge index points to motion predictor information that indicates which MVP to use to derive motion information. Motion information is represented by one unidirectional or bidirectional time prediction type, up to two reference video image indices, and up to two motion vectors, with each motion vector associated with one of two reference video image lists (L0 or L1).

[0049] No information other than the merge index is signaled. This means that the motion vector of the current block is equal to the motion vector of the block-based MVP candidate indicated by the merge index. Therefore, contrary to the block-based merge mode, the MVd and reference image index are not signaled to the bitstream. In the bitstream, only the index of the merge candidate selected from the block-based merge candidate list is signaled.

[0050] Therefore, unlike AMVP mode, in merge mode, the MVd and reference image are not signaled. Only the merge index is signaled to the bitstream.

[0051] The block-wide-based merge MVP candidate list may contain five spatial MVP candidates, which are derived from the current video image shown in Figure 7. The first spatial MVP candidate is derived from motion information associated with the interprediction block and, if present, is located at the left adjacent position A1; the second spatial MVP candidate is derived from motion information associated with the interprediction block and, if present, is located at the upper adjacent position B1; the third spatial MVP candidate is derived from motion information associated with the interprediction block and, if present, is located at the upper right adjacent position B0; the fourth spatial MVP candidate is derived from motion information associated with the interprediction block and, if present, is located at the lower left adjacent position A0; and the fifth spatial MVP candidate is derived from the above motion associated with the interprediction block and, if present, is located at the left adjacent position B2. A redundancy check is then performed among the derived spatial MVPs, i.e., duplicate derived MVP candidates are discarded. The block-based merge candidate list may further include time MVP candidates called TMVP candidates, which are derived from motion information associated with a collated block (if any) located at position H of the reference video image or central spatial position "C". Then, a redundancy check is performed among the derived spatial MVPs, i.e., duplicate derived MVP candidates are discarded. Finally, when using the bidirectional time prediction type, if the block-based merge candidate list contains fewer than five MVP candidates, a merge candidate is added to the block-based merge candidate list. The merge candidate is derived from the motion information of one MVP candidate associated with one reference video image list and present in the block-based merge candidate list, and the motion information corresponds to another MVP candidate associated with another reference video image list and present in the block-based merge candidate list. Finally, if the block-based merge candidate list is still not filled (with five merge candidates), the block-based merge candidate list is filled with zero motion vectors.

[0052] By coding motion information according to VVC, a richer representation of motion information is provided than with HEVC.

[0053] In VVC, motion information can be represented (coded) according to either a block-based interprediction mode that provides a block-based motion representation, or a sub-block-based interprediction mode that provides a sub-block-based motion representation.

[0054] VVC can provide block-based motion representation according to either block-based AMVP mode or block-based merge mode.

[0055] In VVC, in block-based AMVP mode, motion information for defining interprediction blocks is represented similarly to HEVC's block-based AMVP mode. The block-based AMVP candidate list can include spatial MVP candidates and temporal MVP candidates (if any), as in HEVC. The block-based AMVP candidate list can further include four additional HMVP (history-based motion vector prediction) candidates (if any). Finally, if the block-based AMVP candidate list contains fewer than two MVP candidates, it is padded with zero motion vectors.

[0056] HMVP candidates are derived from previously coded MVPs associated with adjacent or non-adjacent blocks to the current block. Therefore, a table of HMVP candidates is maintained in both the encoder and decoder, updated in real time as a first-in, first-out (FIFO) buffer for MVPs. The table can contain up to five HMVP candidates. After coding a block, the table is updated by adding the relevant motion information to the end of the table as a new HMVP candidate. FIFO rules are applied to manage the table, and in addition to the basic FIFO mechanism, redundant candidates in the HMVP table are removed first instead of the first candidate. To enable parallel processing, the table is reset after each CTU row.

[0057] In addition to the block-based AMVP candidate list, compared to HEVC, the block-based AMVP mode allows for the use of the following new tools:

[0058] -SMVD (Symmetric Motion Vector Difference): SMVD involves setting the MVd associated with one reference image list for the current block to the opposite value of the MVd associated with another reference image list for the current block, for bidirectional blocks. Furthermore, the reference video images used in SMVD mode are derived by the decoder according to several predefined rules. SMVD can be selected at the block level, reducing the rate cost of coding the MVd and reference image index information.

[0059] - AMVR (Self-Adaptive Motion Vector Resolution). The AMVR tool allows signaling MVds at 1 / 4 pixel, 1 / 2 pixel, full pixel, or 4-pixel lumasample resolution. This also allows saving bits in coding MVd information. In AMVR, the resolution of the motion vectors is selected at the block level.

[0060] -BCW (Coding Unit Weighted Bidirectional Prediction) enables bidirectional prediction of blocks with uneven weighting and signals at the block level.

[0061] Finally, the internal motion vector representation is implemented with 1 / 16 the luma sample accuracy, instead of the 1 / 4 luma sample accuracy of HEVC.

[0062] In VVC, motion information for defining interpredictive blocks in block-based merge mode is represented similarly to HEVC's block-based merge mode.

[0063] The block-based merge candidate list is different from the block-based merge candidate list used in HEVC.

[0064] VVC allows you to build a block-based merge candidate list for MMVD (Merge Mode with MV Difference), GPM (Geometric Partition Mode), or CIIP (Intra / Inter Prediction Combination) modes.

[0065] The MMVD mode allows coding a limited motion vector difference (MVd) at the top of selected merge candidates to represent motion information associated with interprediction blocks. As shown in Figure 8, MMVD coding is limited to four vector directions and eight magnitude values, from 1 / 4 luma samples to 32 luma samples. Because the MMVD mode provides a moderate level of accuracy, there is a moderate trade-off between rate cost and MV (motion vector) accuracy when signaling motion information.

[0066] As shown in Figure 9, GPM refers to partitioning the current block PG into two motion partitions MV0 and MV1 along a straight line. Each motion vector MV0 and MV1 points to block P0 and P1 of one of the two reference video image lists.

[0067] Geometric partitions can be non-rectangular or rectangular. For rectangular partitions, redundancy is avoided by CU-level binary partitioning through asymmetrical division. The use of GPM is signaled by a CU-level flag as a specific block-based merge mode.

[0068] The direction and position of the partition line relative to the center of the current block are signaled at the CU level by a dedicated GPM index. Geometric partition mode is available for each possible CU size w×h=2 m ×2 nm,n∈{3…6}(excluding 8×64 and 64×8) supports a total of 64 partitions. For example, Figure 10 shows various partition lines that can be realized for various partition line angles at different positional offsets from the center of the CU.

[0069] Each part of the geometric partition within the current block PG performs interpretation using its own motion. Only one-way prediction is allowed for each partition, i.e., each part has only one motion vector and one reference video image index. The constraint of one-way interpretation is applied to ensure that, as with normal bidirectional interpretation, only two motion-compensated predictions are required for each current block PG.

[0070] The motion vectors MV0 and MV1 for each partition are derived from the two merge indices of each partition, respectively.

[0071] After each part of the geometric partition is predicted, a hybrid process with self-adaptive weights is used to adjust the sample values ​​along the edges of the geometric partition. This is the predicted signal for the entire current block PG, and, as with other whole-block-based prediction modes, the transform and quantization processes are applied to the entire current block PG (rather than to each partition).

[0072] CIIP refers to predicting the current block by combining the inter-prediction signal and the intra-prediction signal. In CIIP mode, the inter-prediction signal is derived using the same inter-prediction process applied to the whole-block-based merge mode, while the intra-prediction signal is derived according to the normal intra-prediction process in planar mode. The intra-prediction and inter-prediction signals are then combined using a weighted average, with weight values ​​calculated based on the coding modes of the upper and left-side neighboring blocks.

number

[0073] In summary, in VVC, the block-based merge candidate list can include spatial MVP candidates similar to HEVC, except that the first two candidates are swapped: candidate B1 is given preference over candidate A1 during the construction of the block-based merge candidate list. The block-based merge candidate list can further include TMVP candidates similar to HEVC, and HMVP candidates in VVC AMVP mode. Multiple HMVP candidates are inserted into the block-based merge candidate list so that the merge candidate list reaches the maximum number of MVP candidates allowed (minus 1). The block-based merge candidate list can further include up to one pairwise average candidate calculated as follows: the first two merge candidates present in the block-based merge candidate list are considered, and their motion vectors are averaged. This averaging is calculated separately for each reference video image list L0 and L1. Thus, if both MVPs are bidirectional, the motion vectors associated with lists L0 and L1 are averaged. If only one motion vector exists in the reference video image list, it is adopted to form a pairwise candidate. Finally, if the merge candidate list does not reach the maximum number of MVP candidates allowed, pad the block-based merge candidate list with a zero-motion vector.

[0074] In VVC, subblock-based interpretation essentially means dividing the current block into subblocks, typically 4x4 or 8x8 luma samples, and the subblock-based motion representation for defining the interpretation block is represented by the motion information associated with all subblocks.

[0075] In HEVC, only the translational motion model is applied to the temporal prediction with motion compensation. Such translational motion cannot capture some motions such as zoom-in, zoom-out, rotation, motion by perspective transformation, and irregular motion.

[0076] To solve this problem, in VVC, a sub-block-based affine motion mode can be used at the CU level. As shown in FIG. 11, the affine motion field of the current block is defined by the translational motion vectors of two control points (4-parameter affine motion model) or three control points (6-parameter affine motion model). In FIG. 11, the translational motion vectors v0, v1, v2 of the control points located at the corners of the current block are used to derive the affine motion field of the current block.

[0077] For the 4-parameter affine motion model, the motion vector (mv x , mv y ) at the sample position (x, y) within the current block is derived as follows:

Equation

[0078] For the 6-parameter affine motion model, the motion vector at the sample position (x, y) in the 4×4 sub-block of the current block is derived as follows:

Equation

number

[0079] To derive the motion vectors of each subblock (usually luma subblocks) in the current block, the motion vector of the center sample of each subblock is calculated according to Equation 1 or Equation 2, as shown in Figure 12, and rounded to 1 / 16 decimal precision. Then, a motion compensation interpolation filter is applied to generate predictions for each subblock using the derived motion vectors, and interpretation is performed for the current block. The subblock size in the chromaticity component is also 4x4. The MV of the 4x4 chromaticity subblock is calculated as the average of the MVs of the luma subblocks at the top-left and bottom-right corners of the collated 8x8 luma region.

[0080] In the following, the translational motion vector of a control point will be referred to as the CPMV (Control Point Motion Vector).

[0081] A subblock-based motion representation of a block for defining an interpretation block can be provided according to a subblock-based affine AMVP mode or a subblock-based affine merge mode.

[0082] The subblock-based affine merge mode can be applied to the current block if both its width and height are 8 or greater. It uses an affine merge candidate list of up to five subblock-based motion candidates. Subblock-based motion candidates are CPMVP candidates based on motion information associated with spatially adjacent blocks. The indices of the subblock-based motion candidates selected from the affine merge candidate list are signaled to the bitstream. Finally, if the affine merge candidate list is less than (five CPMVP candidates), it is padded with zero motion vectors.

[0083] The CPMVP candidate may be a so-called inherited CPMVP candidate derived by extrapolating the affine motion model (CPMV) of adjacent blocks, or a so-called constructed CPMVP, which is a CPMVP candidate derived by combining the translational MVs of adjacent subblocks.

[0084] In VVC, there are at most two inherited CPMVP candidates, which are derived by extrapolating the affine motion model of the left adjacent block and the affine motion model of the upper adjacent block. For a left CPMVP candidate, if an affine motion model exists for the left adjacent block A0, the left CPMVP candidate is extrapolated from the CPMV of the affine motion model. Otherwise, the left CPMVP candidate is extrapolated from the CPMV of the affine motion model of the left adjacent block A1 (if it exists). For an upper CPMVP candidate, if an affine motion model exists for the upper adjacent block B0, the upper CPMVP candidate is extrapolated from the CPMV of the affine motion model. Otherwise, if an affine motion model exists for the upper adjacent block B1, the upper CPMVP candidate is extrapolated from the CPMV of the affine motion model. Otherwise, the CPMVP candidate is extrapolated from the CPMV of the affine motion model of the left adjacent block B2 (if it exists).

[0085] Extrapolating the affine motion model of an adjacent block means deriving a candidate for an inherited CPMVP by combining the CPMV of the aforementioned affine motion model.

[0086] As shown in Figure 13, if an affine motion model exists for the lower left adjacent subblock A, the translational motion vectors v2, v3, and v4 of the affine motion model are obtained for the upper left, upper right, and lower left corners of the adjacent block containing subblock A. If subblock A is coded with the above 4-parameter affine model, the left CPMVP candidate of the current block is inherited (calculated) by combining the translational motion vectors v2 and v3 according to the following two equations.

number

[0087] If subblock A is coded with the above 6-parameter affine model, the three CPMVs of the current block are calculated by combining the translational motion vectors v2, v3, and v4 according to the following equation.

number

[0088] In VVC, the constructed affine candidate is derived by combining the non-affine MVs of adjacent blocks. The MV of the control point is derived from the spatially and temporally adjacent subblocks.

[0089] Figure 14 shows an example of deriving a candidate CPMVP by combining the translational MVs of adjacent blocks.

[0090] CPMV k(k = 1, 2, 3, 4) represents the translation MV associated with the k-th control point of the current block. For CPMV1, if a translation MV of the adjacent subblock B2 exists, CPMV1 is equal to the translation MV. Otherwise, if a translation MV of the adjacent subblock B3 exists, CPMV1 is equal to the translation MV. Otherwise, if a translation MV of the adjacent subblock A2 exists, CPMV1 is equal to the translation MV. For CPMV2, if a translation MV of the adjacent subblock B1 exists, CPMV2 is equal to the translation MV. Otherwise, if a translation MV of the adjacent subblock B0 exists, CPMV2 is equal to the translation MV. For CPMV3, if a translation MV of the adjacent subblock A1 exists, CPMV3 is equal to the translation MV. Otherwise, if a translation MV of the adjacent subblock A0 exists, CPMV3 is equal to the translation MV. For CPMV4, if a translational MV of a collated adjacent subblock exists in the reference video image at position T, then CPMV4 is equal to the said translational MV.

[0091] The constructed CPMVP candidates are derived by combining CPMVk as follows: If CPMV1, CPMV2, and CPMV3 exist, the constructed CPMVP candidates are derived by combining {CPMV1, CPMV2, CPMV3}. Otherwise, if CPMV1, CPMV2, and CPMV4 exist, the constructed CPMVP candidates are derived by combining {CPMV1, CPMV2, CPMV4}. Otherwise, if CPMV1, CPMV3, and CPMV4 exist, the constructed CPMVP candidates are derived by combining {CPMV1, CPMV3, CPMV4}. Otherwise, if CPMV2, CPMV3, and CPMV4 exist, the constructed CPMVP candidates are derived by combining {CPMV2, CPMV3, CPMV4}. Otherwise, if CPMV1 and CPMV2 exist, the constructed CPMVP candidates are derived by combining {CPMV1, CPMV2}. Otherwise, if CPMV1 and CPMV3 exist, the constructed CPMVP candidates are derived by combining {CPMV1,CPMV3}.

[0092] Note CPMV k To avoid motion scaling occurring when different reference video images are pointed to, the CPMV k The combination of control points MV associated with this is discarded.

[0093] 3 CPMV k Combining these results in a 6-parameter affine motion model (Equation 2), and combining the two CPMVs results in a 4-parameter affine motion model (Equation 1).

[0094] The subblock-based affine merge model uses a list of affine merge candidates for CPMVP candidates. The merge index mergeIdx indicates the CPMVP candidate for deriving motion information for the current block. The optimal CPMVP candidate in the affine merge candidate list is determined for the current block, and this candidate minimizes the rate distortion cost for coding a given block.

[0095] The affine merge candidate list can include SbTMVP (Subblock-Based Temporal Motion Vector Prediction) candidates and / or inherited CPMVP candidates and / or constructed CPMVP candidates, as described above. Finally, if the affine merge candidate list does not reach the maximum number of allowed candidates, it is padded with zero motion vectors.

[0096] SbTMVP candidates can be derived as follows: Similar to TMVP candidates, SbTMVP candidates are derived from the motion information of collated subblocks (if any) in the reference video image. SbTMVP candidates differ from TMVP candidates in the following two main ways:

[0097] -TMVP predicts movement at the CU level, while SbTMVP predicts movement at the subblock level (subCU level).

[0098] -TMVP obtains temporal motion vectors from colocated blocks in the reference video image, whereas SbTMVP applies a motion shift before obtaining temporal motion information from the reference video image, and the motion shift is obtained from the motion vector of any of the spatially adjacent subblocks of the current block.

[0099] The subblock-based affine AMVP mode can be applied to the current block in the VVC where both the width and height are 16 or greater. In subblock-based affine AMVP mode, the CPMV difference MVd between the MV of the control points located at the corners of the current block and their CPMVPs (control point motion vector predictors) is signaled to the bitstream. Finally, the CPMVP index pointing to the CPMVP candidate in the affine AMVP candidate list is also signaled to the bitstream.

[0100] The CPMVP for predicting the MV of a control point located at the corner of the current block is derived from an affine AMVP candidate list that includes two CPMVP candidates.

[0101] The Affine AMVP candidate list can include inherited CPMVP candidates and / or constructed CPMVP candidates.

[0102] In ECM ("Algorithm description of Enhanced Compression Model 7 (ECM7)", M. Coban, F. Le Leannec, R.-L. Liao, K. Naser, J. Strom, L. Zhang, document JVET-AB2025, ITU-TSG16WP3 and ISO / IEC JTC 1 / SC29 Joint Video Expert Group (JVET), 28th meeting held in Germany from 21 to 28 October 2022, https: / / jvet-experts.org / doc_end_user / documents / 28_Mainz / wg11 / JVET-AB2025-v1.zip), motion information for defining interprediction blocks can also be represented according to the so-called bilateral matching AMVP merge mode (BM-AMVP merge mode).

[0103] The BM-AMVP merge mode is permitted if at least one reference video image is available in the reference video image list (along one inter-direction) for the video image containing the current block, and at least one reference video image is available in another reference video image list (along another inter-direction) for this video image. In BM-AMVP merge mode, the motion information for defining the inter-prediction block includes first motion information associated with the reference image in the reference image list and second motion information associated with the reference image in the other reference image list.

[0104] In the BM-AMVP merge mode defined in ECM, the first motion information is a first block-based motion vector (MV) associated with a reference image in a reference image list associated with the interdirection, and the second motion information is a second block-based MV associated with a reference image in another reference image list associated with the other interdirection.

[0105] The first block-based MV is expressed as described above in relation to the block-based AMVP mode in VVC, and is therefore derived from the block-based motion vector predictor (MVP), signaled reference image index, and signaled motion vector difference obtained by the block-based AMVP mode.

[0106] The second block-based MV is expressed as described above in relation to the block-based merge mode and is therefore derived from the block-based motion vector predictor (MVP) obtained by the block-based merge mode.

[0107] The block-based MVP for deriving the second block-based MVP (i.e., the mergeIdx index in the block-based merge MVP candidate list indicated by the refListMerge index) is derived from the following bilateral matching method, given by the following formula that minimizes the bilateral matching cost Bmcost:

number

[0108] For each MVP candidate in the block-based merge MVP candidate list, the bilateral matching cost is calculated using the MVP candidate and MvpIdx[refListAmvp]. The MVP candidate with the minimum cost is selected as the block-based MVP for deriving the second block-based MV.

[0109] In VVC, to improve the accuracy of the merge mode MV, a bilateral matching (BM)-based decoder-side motion vector refinement method (called the DMVR method) is applied to the current block, starting with the motion vector derived by the block-based AMVP mode (MV0 in Figure 15) and the second block-based MV derived by the block-based merge mode (MV1 in Figure 15). The principle is as follows: In bidirectional prediction operation, the refined motion vectors MV0' and MV1' are derived by enclosing the two initial motion vectors MV0 and MV1, which point to the reference blocks of the reference video images in reference video image lists L0 and L1, and minimizing the bilateral matching cost between the two reference blocks (minimizing the motion vector difference Mdiff). Basically, the local search is performed to derive the MV refinement offset MV_offset. Local search typically applies an n×n (where n is an integer) square search mode, looping horizontally through the search range [-sHor,sHor] and vertically through the search range [-sVer,sVer]. The bilateral matching cost (BM cost) is calculated as the sum of mvDistanceCost and sadCost, where mvDistanceCost represents the coding cost of the motion vector, increasing with the magnitude of the motion vector, and sadCost is a metric that provides the difference between two reference blocks. For example, sadCost is the sum of the absolute difference (SAD) of samples from the two reference blocks. If the BM cost of the center point of the n×n search mode has the minimum cost, the local search terminates. Otherwise, the current minimum cost search point becomes the new center point of the n×n search mode, and the minimum cost search continues until the end of the search range is reached.

[0110] In DVMR, the search point encloses the initial MV, and the refinement offset of the MV follows the MV difference mirror rule. In other words, any point checked by DMVR (represented by a candidate MV pair (MV0, MV1)) follows the following two equations:

number

[0111] The DMVR search consists of an integer sample offset search step (the first step of the DMVR) followed by a fractional refinement step (the second step of the DMVR). The refinement search range is two integer Luma samples from the initial MV.

[0112] The refined MV derived by the DMVR method is used to generate interprediction samples and also to predict the temporal motion vector for future image coding. Meanwhile, the original MV is used in the deblocking process and also to predict the spatial motion vector for future CU coding.

[0113] In VVC, a full search of 25 points is applied to the first step of the DMVR method. First, the BM cost of the initial MV pair is calculated. If the BM cost of the first MV pair is below a threshold, the first step of the DMVR method is terminated. Otherwise, the BM costs of the remaining 24 points are calculated and checked in raster scan order. The search point with the minimum BM cost is selected as the output of the first step of the DMVR method.

[0114] To mitigate the penalties associated with the uncertainty of DMVR refinement, we propose prioritizing the original MV during the DMVR process. The SAD between reference blocks indicated by the initial MV candidate will be reduced by 1 / 4 of the SAD value.

[0115] To reduce computational complexity, the second step of the DMVR method is derived using a parametric error surface formula instead of additional search with SAD comparison. Based on the output of the first step of the DMVR method, fractional sample refinement is conditionally invoked. In the first or second iteration of the first step of the DMVR method, if the first step of the DMVR method terminates at a center with the smallest SAD, fractional sample refinement (the second step of the DMVR method) is further applied.

[0116] In subpixel offset estimation based on parametric error surfaces, the central position BM cost and the BM costs at four adjacent positions from the center are used to fit a two-dimensional parabolic error surface equation of the following form:

number

[0117] Here, (x min ,y min ) corresponds to the fractional position with the minimum BM cost, and C corresponds to the minimum value of the BM cost. By solving the above equation using the BM cost values ​​of the five search points, (x min ,y min ) is calculated as follows:

number

[0118] In VVC, the application of DMVRs is restricted and applies only to CUs that meet one specific DMVR condition.

[0119] The DMVR condition is met if the CU is coded using any of the following coding modes and features: - CU level merge mode using bidirectional predictive MV. - One reference image is positioned in the past relative to the current image, and another reference image is positioned in the future. - The distance from the two reference images to the current image (i.e., the POC difference) is the same. -Both reference images are short-term reference images. -CU has more than 64 luma samples. - Both the height and width of the CU are 8 luma samples or more. -BCW weight index indicates equal weights. - Do not perform weighted prediction (WP) on the current block; instead, allow weighted prediction to be applied to motion-compensated prediction using a combination of multiplicative weight coefficients and additive offsets. (“Weighted prediction in the H.264 / MPEG AVC video coding standard”, JM Boyce, May 2004, 10.1109 / iscas.2004.1328865). - Do not use CIIP mode for the current block.

[0120] In VVC, the resolution of the MV is 1 / 16 of a luma sample. An 8-tap interpolation filter is used to interpolate samples at fractional positions. In DMVR, search points with integer sample offsets enclose the initial fractional pixels in the MV, and therefore, interpolation is required for samples at those fractional positions in order to perform the DMVR search process. To reduce computational complexity, a bilinear interpolation filter is used to generate fractional samples for the search process in DMVR. Another important effect is that by using a bilinear filter with a search range of 2 samples, DVMR accesses fewer reference samples compared to the normal motion compensation process. After obtaining the refined MV using the DMVR search process, the normal 8-tap interpolation filter is applied to generate the final prediction. In the normal MC (motion compensation) process, samples that are not needed in the interpolation process based on the original MV but are needed in the interpolation process based on the refined MV are padded from the available samples because fewer reference samples are accessed.

[0121] If the width and / or height of a CU is greater than 16 luma samples, the CU is further divided into subblocks whose width and / or height are equal to 16 luma samples. The maximum unit size used for the DMVR search process is limited to 16x16.

[0122] In ECM, when the DMVR condition is met, the DMVR is applied to the affine merge coded block. This is the so-called affine DMVR. The first step of the affine DMVR (the integer sample offset search step) is applied to the translational portion of the affine motion (Equation 3) so that if the candidate satisfies the DMVR condition, a translational MV offset is added to all CPMVs of this candidate in the affine merge list. The MV offset is derived by minimizing the BM cost, which is the same as in a normal DMVR.

[0123] Affine DMVR consists of a 3x3 square search mode (8 search points) to loop through a search range set to [-3,3] in order to find the optimal integer MV refinement offset (the first step of affine DMVR).

[0124] Figure 16 shows an example of the scheme and search point sequence used in the affine DMVR method. The cross x represents the initial search position, and positions 0, ..., 7 represent all eight search points.

[0125] Next, fractional refinement (the second step of affine DMVR) is performed by enclosing the optimal integer position, and finally error surface estimation is performed to find the optimal MV refinement offset that estimates and has an accuracy of 1 / 16.

[0126] Affin DMVR does not have a bias towards original music videos, and all aspects of exploration are evaluated.

[0127] To further improve affine DMVR, a CPMV refinement of affine DMVR was submitted in JVET submission JVET-AB0178 (https: / / jvet-experts.org / doc_end_user / documents / 28_Mainz / wg11 / JVET-AB2028-v1.zip), allowing different MV refinement offsets to be applied to different CPMVs.

[0128] The method described in JVET-AB0178 can be applied to merge candidates in affine merge mode and affine MMVD mode.

[0129] Figure 17 schematically shows a block diagram of the steps of the CPMV refinement method described in JVET-AB0178.

[0130] The CPMV refinement method follows the affine DMVR method as described in the related technologies.

[0131] In the first step 171, for each control point motion vector initCpMvLX[cpIdx], cpIdx = 0...numCpMv-1, where numCpMv is the number of CPMVs in the current affine coding block. This method performs bilateral matching on the control point-centered block to derive the refined CPMV bmRefinedCpMvLx[cpIdx].

[0132] Next, we loop through the combinations of initCpMvLX[cpIdx] and bmRefinedCpMvLx[cpIdx] to derive the optimal CPMV set that minimizes the BM cost of the current block.

[0133] In the second step 172 (optional), following the principle of the second step of affine DMVR, the CPMVs in the optimal set of CPMVs are iteratively refined to further minimize the BM cost of the current block. In each iteration, a single CPMV is refined, while the other CPMVs are fixed.

[0134] The method shown in Figure 17 is preferred because it improves the compression efficiency of the latest video coding.

[0135] However, the drawback of this method is that the trade-off between improved compression efficiency and increased encoder and decoder complexity is not very attractive, as this method means a significant increase in encoder and decoder complexity.

[0136] The object of the present invention is to improve the trade-off between increased compression efficiency and encoding time overhead in the CPMV refinement method submitted in JVET-AB0178.

[0137] At least one embodiment of the present application is designed in accordance with the above. [Overview of the project] [Problems that the invention aims to solve]

[0138] To provide a basic understanding of some aspects of this application, a summary of at least one embodiment is provided below. This summary is not an exhaustive overview of the embodiments, nor is it intended to identify essential or important elements of the embodiments. The following summary presents only a few aspects of at least one embodiment in a simplified form, serving as a prelude to the more detailed descriptions provided elsewhere in this specification.

[0139] According to a first aspect of the present invention, a method for decoding a video image is provided, wherein the decoding is performed by temporal bidirectional prediction with motion compensation for an encoded block, the temporal bidirectional prediction with motion compensation using two reference video images in two separate reference video image lists and two affine motion fields defined by at least two control point motion vectors, the control point motion vectors being represented as CPMVs, at least two control point motion vectors associated with each reference image, and the refined CPMV is obtained as the output of a first step, or optionally as the output of a second step following the first step.

[0140] -In the first step described above, for each CPMV, bilateral matching is performed on the block centered on that CPMV to derive at least two integer-precision refined CPMVs, and a set of CPMVs including the unrefined CPMV and / or integer-precision refined CPMVs is selected to generate an overall predicted block with the minimum bilateral matching cost.

[0141] - In the second step, for each consecutive CPMV in the selected set of CPMVs associated with the block, each CPMV is refined to subsample accuracy to minimize the bilateral matching cost of the block.

[0142] Based on the comparison between the bilateral matching cost associated with the CPMV performed in the first step and the threshold, the second step is bypassed.

[0143] In one embodiment, if the bilateral matching cost associated with the CPMV performed in the first step is lower than a first threshold, the second step is bypassed for the CPMV.

[0144] In one embodiment, if the bilateral matching cost associated with the CPMV performed in the first step is higher than a second threshold, the second step is bypassed for the CPMV.

[0145] In one embodiment, the first threshold and / or the second threshold are fixed or adapted to the prediction unit size.

[0146] In one embodiment, two affine motion fields are defined by a first CPMV, a second CPMV, and a third CPMV, and the second step is bypassed for the third CPMV if the bilateral matching cost associated with the first or second CPMV satisfies the bypass condition.

[0147] In one embodiment, the bypass condition is met if the bilateral cost associated with the first or second CPMV performed in the second step is higher than the bilateral matching cost associated with the first or second CPMV performed in the first step.

[0148] In one embodiment, the bypass condition is met if the bilateral matching cost associated with the first or second CPMV performed in the second step is lower than the bilateral matching cost associated with the first or second CPMV performed in the first step, and the reduction in the bilateral cost is lower than a third threshold.

[0149] In one embodiment, the first and second steps are disabled for a specific block shape.

[0150] In one embodiment, if the width or height of the block to be intercoded is greater than a first value when the width and height are equal, the first and second steps are disabled.

[0151] In one embodiment, if the width or height of the block to be intercoded is greater than a second value when the width and height are different, the first and second steps are disabled.

[0152] In one embodiment, a block shape is defined based on the width or height of the interconnected block, and at least one block shape is signaled in the bitstream to invalidate the first and second steps.

[0153] In one embodiment, the at least one block shape that invalidates the first and second steps is signaled by signaling the maximum and minimum block width and / or height.

[0154] In one embodiment, the at least one block shape that invalidates the first and second steps is signaled by signaling the minimum and / or maximum block ratio between the maximum block width and block height and the minimum block width and block height.

[0155] A second aspect of the present application provides an apparatus including components for performing one of the methods according to the first aspect of the present application.

[0156] According to a third aspect of the present application, a computer program product including instructions is provided, and when the program is executed by one or more processors, one or more processors are made to execute the method of the first aspect of the present application.

[0157] A fourth aspect of the present application provides a non-temporary storage medium containing program code instructions for performing the method of the first aspect of the present application.

[0158] A fifth aspect of the present application provides an electronic device including a processor and a memory for storing instructions that can be executed by the processor. The processor is configured to perform the method of the first aspect of the present application.

[0159] The specific properties of at least one of the embodiments, and the other purposes, advantages, features, and applications of that at least one embodiment, will become apparent from the following description of the examples in conjunction with the drawings. [Brief explanation of the drawing]

[0160] The drawings illustrating embodiments of the present application are referenced here, Figure 1 shows an example of a coding tree unit related to HEVC. Figure 2 shows an example of partitioning the coding unit related to HEVC into a prediction unit. Figure 3 shows an example of CTU partitioning related to VVC. Figure 4 shows an example of a partitioning mode supported in a multi-type tree partition related to VVC. Figure 5 shows a schematic block diagram of the steps of method 100 for encoding a video image VP related to the relevant technology. Figure 6 shows a schematic block diagram of the steps of method 200 for decoding video image VP related to the relevant technology. Figure 7 shows an exemplary example of constructing a block-based AMVP candidate list, which is used to define the interpretation blocks of the current block in the current video image. Figure 8 shows an example of a constrained motion vector difference related to the relevant technology. Figure 9 shows an example of performing geometric partitioning on a block of video images related to the relevant technology. Figure 10 shows an example of a geometric block partition related to the relevant technology. Figure 11 shows an example of the representation of the affine motion field of a video image block related to the relevant technology. Figure 12 shows an example of motion vectors associated with subblocks of a video image block related to the relevant technology. Figure 13 shows an example of deriving an affine motion model from translational motion vectors for control points located at the corners of video image blocks related to the relevant technology. Figure 14 shows an example of deriving a CPMVP candidate constructed by combining the translational MVs of adjacent blocks related to the relevant technology. Figure 15 shows an example of a method for refining bilateral matching related to the technology. Figure 16 shows an example of the scheme and search point sequence used in affine DMVR. Figure 17 schematically shows a block diagram of the steps of the CPMV refinement method described in JVET-AB0178. Figure 18 shows the locations of search points related to the relevant technologies. Figure 19 shows a schematic block diagram of an example of a system in which various embodiments and examples are realized. Similar or identical elements are referenced by the same attached drawing mark. [Modes for carrying out the invention]

[0161] Hereinafter, with reference to the drawings, at least one example of the embodiments will be described and illustrated. However, the embodiments can be carried out in various alternative forms and should not be construed as being limited to the examples described herein. Accordingly, it should be understood that the present invention is not intended to be limited to any particular form in which the embodiments are disclosed. In contrast, this application is intended to cover all modifications, equivalents and alternatives that fall within the spirit and scope of this application.

[0162] In various embodiments, at least one relates to the encoding and decoding of video images, the other generally to the transmission of a provided or encoded bitstream, and the other to the reception / access of a decoded bitstream.

[0163] In at least one embodiment, the encoding / decoding of a single video image is described, but as will be discussed later, each video image is encoded / decoded sequentially, so the method can be extended to encoding / decoding of multiple video images (sequences of images).

[0164] Furthermore, for example, at least one example is AVC (ISO / IEC 14496-10 Advanced Video Coding for generic audio-visual services, ITU-T Recommendation H.264, https: / / www.itu.int / rec / T-REC-H.264-202108-P / en), EVC (ISO / IEC 23094-1 Essential video coding), HEVC (ISO / IEC 23008-2 High Efficiency Video Coding, ITU-T Recommendation H.265, https: / / www.itu.int / rec / T-REC-H.265-202108-P / en), VVC (ISO / IEC 23090-3 Versatile Video Coding, ITU-T Recommendation H.265) While not limited to MPEG standards such as H.266 (https: / / www.itu.int / rec / T-REC-H.266-202008-I / en), the invention may be applied to other standards or recommendations such as AV1 (AOMedia Video 1, http: / / aomedia.org / av1 / specification / ). At least one embodiment may be applied to prior or future developments and extensions of such standards and recommendations. Unless otherwise specified or technically excluded, the various embodiments described herein may be used individually or in combination.

[0165] A pixel corresponds to the smallest display unit on the screen and can consist of one or more light sources (one for a monochrome screen, or three or more for a color screen).

[0166] A video image, also called a frame or image frame, comprises at least one component (also called an image component or channel) determined by a particular image / video format, which specifies all the information associated with pixel values ​​that this particular image / video format provides, and all the information that can be used by a display unit and / or any other device to display and / or decode the video image data associated with the video image.

[0167] A video image typically contains at least one component, which is usually represented in the form of a sample array.

[0168] A monochrome video image contains a single component, while a color video image may contain three components.

[0169] For example, if the image / video format is the well-known (Y,Cb,Cr) format, a color video image may contain one luma (or glow) component and two chromaticity components. On the other hand, if the image / video format is the well-known (R,G,B) format, a color video image may contain three color components (one each for red, green, and blue).

[0170] Each component of a video image can contain a number of samples relative to the number of pixels on the screen in which the video image is displayed. In a modified example, the number of samples in a component may be a multiple (or fraction) of the number of samples in another component of the same video image.

[0171] For example, if a video format includes one luma component and two chromaticity components (e.g., a (Y,Cb,Cr) format), the chromaticity component may contain half the number of samples in width and / or height compared to the luma component, depending on the color format being considered.

[0172] A sample is the smallest visual information unit of a video screen. A sample value may be, for example, a luminance value, a chromaticity value, or a color value in (R,G,B) format.

[0173] A pixel value is the value of a pixel on a screen. For a monochrome video image, a pixel value can be represented by a single sample, but for a color video image, a pixel value can be represented by multiple collated samples. A collated sample associated with a pixel refers to a sample that corresponds to the pixel's position on the screen.

[0174] Typically, a video image is viewed as a set of pixel values, with each pixel represented by at least one sample.

[0175] A block of video images is a set of samples of one component of a video image. If the image / video format is a well-known (Y,Cb,Cr) format, a block of at least one luma sample or at least one chromaticity sample can be considered, or if the image / video format is a well-known (R,G,B) format, a block of at least one color sample can be considered.

[0176] At least one embodiment is not limited to a specific image / video format.

[0177] In general, this application relates to various embodiments for reducing the complexity of the CPMV refinement method of JVET-AB0178 while maintaining as much of the compression efficiency obtained with JVET-AB0178 as possible.

[0178] In the first embodiment, the MV / CPMV refinement search is optimized by systematically storing the results of affine DMVR (i.e., the obtained optimal MV and minimum BM cost (output of the first or second step of the affine DMVR method)).

[0179] In ECM, the affine DMVR search scheme is processed in the second step and steps 171 and 172 of the affine DMVR to improve compression efficiency by selecting an MV that provides a BM cost, sometimes repeating several times. If there is a repeat, the second step of the affine DMVR method can be skipped if the previous iteration did not provide a better solution.

[0180] When using several iterations of the affine DMVR search strategy, the initial MV and optimal BM cost are updated using the best results (optimal CPMV and optimal BM cost) obtained in the previous few iterations.

[0181] Assume that position 0 is the optimal position. Figure 18 shows the positions to be evaluated in the next iteration. The positions corresponding to the gray rectangles are the positions evaluated in the previous iteration. The positions corresponding to the hashed rectangles are the positions evaluated in the previous iteration and will be reused in the current iteration. Positions 0, 1, 2, 7, and 6 will be evaluated in the new iteration.

[0182] In ECM, if multiple iterations of the refinement process are triggered, the results of the iterations (optimal MV / CPMV and minimum strain) are stored only if the iteration is the first one. By handling it this way, when this refinement is combined with other algorithms, the other subsequent algorithms will not be initialized with the optimal identified MV / CPMV or will not use the minimum realized BM cost.

[0183] According to the first embodiment, after each iteration of the first or second step of the affine DMVR method, the results (optimal MV and minimum BM cost) are systematically stored.

[0184] In the second embodiment, the search strategy can be divided into two parts.

[0185] The primary positions may be, for example, left, top, right, and bottom, i.e., positions {1, 3, 5, 7} in Figure 16. The corner positions may be, for example, the upper left corner, the upper right corner, the lower right corner, and the lower left corner, i.e., the positions {0,2,4,6} in Figure 16.

[0186] According to the second embodiment, first the main position is evaluated, and then the next corner position is evaluated based on the results for the adjacent positions of the aforementioned corner positions.

[0187] In the first modification of the second embodiment, if none of the adjacent positions to the corner position are currently the optimal position, the evaluation of the adjacent positions to the corner position is skipped.

[0188] For example, the evaluation of position 0 (corner position) can be processed based on the results of positions 1, 7, and the initial position x (adjacent position to corner position 0) in Figure 16. If positions 1, 7, and the initial position x are not the current optimal positions for the iteration, the probability that position 0 is the optimal position is low. In this case, it is beneficial to skip the evaluation of position 0 to save execution time.

[0189] In the second modification of the second embodiment, the corner position is evaluated if the BM costs of two adjacent positions to the corner position are within a predetermined range of the current optimal BM cost.

[0190] This range can be defined up to the optimal distortion, weighted by coefficients that are strictly greater than 1, from 0 to 1.

[0191] For example, consider position 7 (adjacent to corner position 0) to be the optimal position. Using the first variation of the second embodiment, skip positions 2 and 4 and evaluate positions 1, 7, and x. However, if the BM cost of position 1 is much higher than the current optimal BM cost (i.e., the BM cost of position 7), the probability that position 0 is the optimal position is low. If the BM costs of positions 1, 7, and x are all within a given range (not low), the probability that position 0 is the optimal position is high.

[0192] In the third embodiment, the optimal MV that provides the minimum BM cost is selected between two reference video image lists L0 and L1. The motion vector candidates are refined for a given list (corresponding to one of the reference video image lists (or L0 or L1)) (the second step of affine DMVR), i.e., only one MV is refined. As described above in conjunction with Figure 16, the refined offset of the MV follows the conventional scheme and search point order used by affine DMVR.

[0193] On the encoder side, the two reference video image lists (MV) are sequentially refined individually to identify and select the optimal prediction mode.

[0194] To measure BM costs, a reference video image list prediction needs to be available.

[0195] In the prior art, for a given PU, distortion is evaluated according to subblocks (from the PU size down to 4x4 blocks) (e.g., evaluated by SAD), and two reference video image list predictions are computed. A subblock is a collection of at least one consecutive 4x4 blocks that share the same MV.

[0196] To optimize such evaluations for distortion (e.g., by SAD evaluation), the predictions for the unrefined reference video image list remain constant / fixed, and if these predictions are intelligently stored for a given PU, they can be calculated only once. For example, two predictions are calculated for the first defined MV refinement offset, but for the next MV refinement offset, prediction calculations are halved by calculating only the predictions for the already refined video image list.

[0197] Furthermore, instead of calculating distortion at the subblock level (e.g., by SAD evaluation), distortion can be calculated at the PU level. In conventional techniques, distortion between dual reference video image list predictions (e.g., evaluated by SAD) is calculated at the subblock level and summed across all subblocks to obtain distortion at the PU level (e.g., evaluated by SAD). Then, predictions can be generated at the subblock level and distortion measured at the PU level. This helps reduce the number of function calls (once per PU instead of once per subblock) and reduces execution time when processing larger pixel surfaces by using more efficient intrinsic functions.

[0198] In the fourth embodiment, step 172 is bypassed for the CPMV based on a comparison of the bilateral matching cost associated with the CPMV performed in the first step 171 with a threshold.

[0199] This fourth embodiment, as described with respect to Figure 17, avoids spending time evaluating merge candidates because step 172 is the most time-consuming step of the affine DMVR method, thereby reducing the execution time consumption of the affine DMVR method.

[0200] In the first modification of the fourth embodiment, if the BM cost performed on the CPMV in step 171 is lower than the threshold TH1, step 172 is bypassed for the CPMV.

[0201] After that, CPMV is partially optimized (refined).

[0202] This early termination assumes that the BM cost (distortion) is already low and that the refinement is clearly better than the rate-distortion trade-off of the current merge candidate.

[0203] In the second modification of the fourth embodiment, if the BM cost performed on the CPMV in step 171 is higher than the threshold TH2, step 172 is bypassed for the CPMV.

[0204] After that, CPMV is partially optimized (refined).

[0205] This early termination assumes that the BM costs (distortions) are quite high and that the merge candidates are not sufficiently competitive.

[0206] In the third modification of the fourth embodiment, the thresholds TH1 and / or TH2 are adapted to the PU size or are fixed.

[0207] During iterative refinement (steps 171 and 172), the CPMV is refined sequentially to improve compression efficiency. The affine DMVR method attempts to refine the first CPMV, then the second CPMV, and finally, if it is a 6-parameter affine model, the third CPMV. The refinement is processed in detail, i.e., all CPMVs are refined regardless of the results of the previous CPMV refinements.

[0208] In the third modification of the fourth embodiment, the second step 172 is bypassed for the third CPMV if the BM cost (strain) associated with the first or second CPMV satisfies the bypass condition.

[0209] In one example of a third modification of the fourth embodiment, the bypass condition is met if the BM cost associated with the first or second CPMV performed in the second step 172 is higher than the BM cost associated with the first or second CPMV performed in the first step 171.

[0210] In another third modification of the fourth embodiment, the bypass condition is met if the BM cost associated with the first or second CPMV performed in the second step 172 is lower than the BM associated with the first or second CPMV performed in the first step 171, and the reduction in the bilateral cost is lower than the third threshold TH3.

[0211] In the fifth embodiment, the affine DMVR method is disabled for a specific block (CU) shape.

[0212] Conventional techniques perform affine DMVR for each CU whose height and width are greater than or equal to eight luma samples. However, the affine DMVR method can offer different execution time / compression efficiency trade-offs for all CU shapes. Therefore, for example, if the execution time / compression efficiency trade-off is unsatisfactory, it is worth considering disabling affine DMVR for a particular CU shape. By identifying CU shapes where affine DMVR performs poorly, execution time can be saved by disabling affine DMVR.

[0213] This fifth embodiment is even more preferable because, when bypassing affine DMVR for CUs having a specific shape, the CU syntax signaled to that CU can also be bypassed, thereby further reducing the bitrate.

[0214] In the first modification of the fifth embodiment, the affine DMVR method (steps 171 and 172) can be disabled if the width or height of the intercoded block (CU) is greater than a first value (e.g., 128) when the width and height are equal.

[0215] In a second modification of the fifth embodiment, the affine DMVR method (steps 171 and 172) can be disabled if the width or height of the intercoded block (CU) is greater than a second value (e.g., 64) when the width and height are different.

[0216] In the first and second modifications of the fifth embodiment, the shape of the CU that disables the affine DMVR (the width and height of the intercoded CU) may be signaled to the bitstream so that it is replicated on the decoder side.

[0217] In the first embodiment of the above modification, at least one particular CU shape that disables the affine DMVR (steps 171 and 172) can be signaled by signaling the maximum and / or minimum CU size (i.e., the maximum and minimum CU width and / or height, expressed, for example, as the number of pixels).

[0218] In a second embodiment of the above modification, the at least one block shape that disables the affine DMVR (steps 171 and 172) can be signaled by signaling the minimum and / or maximum CU ratio between the maximum width and height of the CU and the minimum width and height of the CU.

[0219] In the prior art, for each intermode, a list of candidates is generated using adjacent blocks, temporal information, and combined spatial and temporal information, as described in the introduction of this application.

[0220] The affine DMVR method generates a merge candidate list using available spatial and temporal information. This merge candidate list is not optimal, and the same merge candidate may appear multiple times in the list.

[0221] For several reasons, duplicate merge candidates are suboptimal.

[0222] First, duplicate merge candidates are evaluated by the DMVR method like other merge candidates, but are not selected because they offer lower compression efficiency, and this is because their merge index is higher than that of merge candidates with a lower merge index. Subsequently, evaluating duplicate merge candidates increases the execution time of the DMVR method, while not expanding the compression efficiency.

[0223] Next, the presence of duplicate merge candidates in the merge candidate list degrades the rate distortion performance of subsequent merge candidates, which unnecessarily increases the merge index associated with the subsequent merge candidates.

[0224] Thirdly, if the merge candidate list is finite, and the maximum size of the list is reached, and there are unnecessary duplicate merge candidates in the list, some potential merge candidates are discarded.

[0225] To overcome these shortcomings, the sixth embodiment checks whether a new merge candidate to be added to the merge candidate list already exists in the merge candidate list.

[0226] The sixth embodiment helps limit the execution time overhead of the DMVE method while providing better compression performance compared to a merge candidate list that includes at least one overlapping merge candidate.

[0227] In a variation of the sixth embodiment, the reference video image index of the new merge candidate is compared to the reference video image index of the merge candidate in the merge candidate list, and the MVs of the two reference video image lists (L0 and L1) are compared. For affine blocks, all CPMVs of the two lists are compared. More precisely, each affine merge index points to motion prediction information that indicates which CPMV is used to derive motion information. The motion information is represented by one temporal prediction type, either unidirectional or bidirectional, up to two reference video image indices, and up to three control point motion vectors (CPMVs), with each CPMV associated with a reference video image index of one of the two reference video image lists (L0 or L1). Then, when comparing the two affine merge candidates, the motion information of the two lists (L0 and / or L1) is compared. For the lists, this involves a comparison of the reference video image index and CPMVs.

[0228] If the baseline index and MV / CPMV are exactly the same, potential candidates are identified as duplicates.

[0229] A variation of the sixth embodiment discards only merge candidates that are strictly overlapping. This may be extended by discarding potential candidates where MV / CPMV is too close to a candidate in one list.

[0230] For example, if all the differences between a potential candidate's MV / CPMV and the MV / CPMV of one of the merge candidates in the list are below a threshold, this potential candidate is discarded.

[0231] The comparison involves comparing the baseline index, calculating the MV / CPMV difference, and comparing the MV / CPMV difference to a threshold. In this example, the objective is to compare the motion information of two candidates. Unlike the previous explanation, the criterion in CPMV is expanded, thereby discarding further candidates. Now, it involves discarding candidates whose CPMV is similar to that of the candidates in the list, i.e., any CPMV difference between the two merge candidates is below the threshold.

[0232] If the reference indices are exactly the same and the MV / CPMV difference is lower than threshold TH4, the potential candidate is not added to the merge candidate list. Threshold TH4 may be fixed or self-adaptive, for example, adjusted self-adaptively based on PU size.

[0233] Regarding the DMVR method, on the encoder side, all merge candidates in the merge candidate list are refined, meaning the final merge candidate may differ from the merge candidates in the list.

[0234] Therefore, merge candidates in the list can be matched with merge candidates refined using affine DMVR from the list. More precisely, the encoder runs the affine DMVR algorithm on all affine merge candidates in the list. As described later, affine DMVR refines the CPMV of the affine merge candidates, thereby modifying the CPMV. The Nth refined affine merge candidate in the list may then have the same or similar (i.e., close CPMV) motion information (reference image index and CPMV) as an affine merge candidate with a merge index higher than N.

[0235] In this case, applying the DMVR method to the matched merge candidates is inefficient. Therefore, on the encoder side (where each merge candidate needs to be evaluated), if a merge candidate is similar to a DMVR-refined merge candidate, it may be possible to bypass the DMVR refinement of the merge candidate. The merge candidate DMVR refinement can be discarded using a strategy similar to that described above.

[0236] Figure 19 shows an exemplary schematic block diagram of System 600 that realizes various embodiments and examples.

[0237] System 600 may be incorporated as one or more devices including various components described below. In various embodiments, System 600 may be configured to realize one or more of the embodiments described herein.

[0238] Examples of devices constituting all or part of System 600 may include personal computers, laptops, smartphones, tablets, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, connected vehicles, and associated processing systems, head-mounted display devices (HMDs, X-ray glasses), projectors (beamers), "caves" (systems including multiple displays), servers, video encoders, video decoders, post-processors for processing output from video decoders, pre-processors for providing input to video encoders, web servers, video servers (e.g., broadcast servers, video broadcast servers, or network servers), still image or video cameras, encoding or decoding chips, or any other communication devices. The elements of System 600 can be implemented individually or in combination within a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of System 600 can be distributed across multiple ICs and / or discrete components. In various embodiments, System 600 can be communicably coupled to other similar systems or other electronic devices, for example, via a communication bus or dedicated input and / or output ports.

[0239] The system 600 may include at least one processor 610, which is configured to execute instructions loaded to realize each of the embodiments described herein. The processor 610 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 600 may include at least one memory 620 (e.g., a volatile memory device and / or a non-volatile memory device). The system 600 may include a storage device 640 containing non-volatile and / or volatile memory, including, but not limited to, electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, magnetic disk drives and / or optical disk drives. In non-limiting examples, the storage device 640 may include an internal storage device, an attached storage device and / or a network-accessible storage device.

[0240] The system 600 may include, for example, an encoder / decoder module 630 configured to provide encoded / decoded video image data by processing the data. The encoder / decoder module 630 may include its own processor and memory. The encoder / decoder module 630 can represent one or more modules that may be included in the device to perform encoding and / or decoding functions. As is well known, the device may include either one or both encoding and decoding modules. The encoder / decoder module 630 may also be implemented as a separate element of the system 600, or it may be incorporated into the processor 610 as a combination of hardware and software known to those skilled in the art.

[0241] Program code loaded into the processor 610 or encoder / decoder 630 to perform each embodiment described herein is stored in the storage device 640 and loaded into memory 620 to be executed by the processor 610. According to various embodiments, during the execution of the processes described herein, one or more of the processor 610, memory 620, storage device 640 and encoder / decoder module 630 may store one or more of various items. Such stored items include, but are not limited to, video image data, information data used for encoding / decoding video image data, bitstreams, matrices, variables, equations, formulas, operations and intermediate or final results of arithmetic logic processing.

[0242] In some embodiments, memory within the processor 610 and / or encoder / decoder module 630 may be used to store instructions and provide working memory for processing performed during encoding or decoding.

[0243] However, in other embodiments, memory outside the processing device (for example, the processing device may be a processor 610 or an encoder / decoder module 630) may be used for one or more of these functions. The external memory may be memory 620 and / or storage device 640, and may be, for example, dynamic volatile memory and / or non-volatile flash memory. In some embodiments, external non-volatile flash memory may be used to store, for example, the operating system of a television. In at least one exemplary embodiment, high-speed external dynamic volatile memory such as RAM may be used as working memory for video encoding and decoding operations such as, for example, the second part of MPEG-2 (also known as ITU-T Recommendation H.262 and ISO / IEC 13818-2, and also called MPEG-2 video), AVC, HEVC, EVC, VVC, AV1, etc.

[0244] As shown in block 690, inputs to the elements of system 600 can be provided via various input devices. Such input devices may include, but are not limited to, (i) an RF section capable of receiving RF signals transmitted wirelessly, for example, by a broadcasting device; (ii) a composite input terminal; (iii) a USB input terminal; (iv) an HDMI input terminal; and (v) a bus such as a CAN (Controller Area Network), CAN FD (Controller Area Network Flexible Data Rate), FlexRay (ISO 17458), or Ethernet (ISO / IEC 802-3) bus, as this application is implemented in the automotive field.

[0245] In various embodiments, the input device of block 690 may have corresponding input processing elements as known in the Art. For example, the RF portion may be associated with elements necessary to (i) select a desired frequency (also called selecting a signal or limiting a signal to a frequency band), (ii) downconvert the selected signal, (iii) again band-limit it to a narrower frequency band to select a signal frequency band called a channel in a particular embodiment (e.g.), (iv) demodulate the downconverted and band-limited signal, (v) perform error correction, and (vi) multiplex decouple in order to select a desired data packet stream. The RF portion of various embodiments may include one or more elements for performing these functions, for example, a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a multiplex decoupler. The RF portion may also include a tuner that performs various functions, such as downconverting the received signal to a lower frequency (e.g., an intermediate frequency or a frequency close to the baseband) or the baseband.

[0246] In one embodiment of a set-top box, the RF section and its associated input processing elements can receive an RF signal transmitted via a wired (e.g., cable) medium. The RF section can then perform frequency selection to the desired frequency band by filtering, downconverting, and re-filtering.

[0247] In various embodiments, the order of the above (and other) elements is rearranged, some of these elements are removed, and / or other elements that perform similar or different functions are added.

[0248] Adding elements includes inserting elements between conventional elements, such as amplifiers or analog or digital converters. In various embodiments, the RF may include an antenna.

[0249] Furthermore, the USB and / or HDMI terminals may include corresponding interface processors to connect the system 600 to other electronic devices via USB and / or HDMI connections. When necessary, each aspect of input processing (e.g., Reed-Solomon error correction) can be implemented within a separate input processing IC or within the processor 610. Similarly, when necessary, each aspect of USB or HDMI interface processing can be implemented within a separate interface IC or within the processor 610. Demodulation, error correction, and multiplexed streams can be provided to various processing elements, including, for example, the processor 610 and the encoder / decoder 630, which are operated in combination with memory and storage elements to process the data stream and display it on an output device when necessary.

[0250] Various elements of system 600 can be provided within an integrated housing. Within the integrated housing, appropriate connection arrangements 690 can be used, for example, to interconnect each type of element and transmit data between them using internal buses (including I2C buses), wiring, and printed circuit boards known in the art.

[0251] System 600 may include a communication interface 650 to communicate with other devices via a communication channel 651. The communication interface 650 may include, but is not limited to, a transceiver for transmitting and receiving data over the communication channel 651. The communication interface 650 includes, but is not limited to, a modem or a network card, and the communication channel 651 can be implemented, for example, within a wired and / or wireless medium.

[0252] In various embodiments, a Wi-Fi network such as IEEE802.11 can be used to stream data to system 600. The Wi-Fi signals of these embodiments can be received via a communication channel 651 and a communication interface 650 suitable for Wi-Fi communication. The communication channel 651 of these embodiments can typically be connected to an access point or a router, and the access point or the router enables streaming applications and other over-the-top communications by providing access to an external network including the Internet.

[0253] Other embodiments can provide streamed data to system 600 using a set-top box, and the set-top box distributes the data via the HDMI connection of input block 690.

[0254] Other embodiments can provide streamed data to system 600 using the RF connection of input block 690.

[0255] The streamed data can be used by system 600 as a method for signaling information. The signaling information may include information such as a bitstream B and / or the number of pixels of a video image and / or any encoding / decoding setting parameters.

[0256] Note that signaling can be realized in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. can be used to signal information to the corresponding decoder.

[0257] System 600 can provide output signals to various output devices including display 661, speaker 671, and other peripheral devices 681. In various examples of the embodiments, other peripheral devices 681 can include one or more of an independent DVR, disc player, stereo system, lighting system, and other devices that provide functions based on the output of system 600.

[0258] In various embodiments, the control signal can be used to communicate between system 600 and display 661, speaker 671, or other peripheral devices 681 using signaling of AV.Link (audio / video link), CEC (consumer electronics control), or other communication protocols that enable device-to-device control, with or without a user.

[0259] The output devices can be connected to system 600 via dedicated connections so as to be communicable via corresponding interfaces 660, 670, and 680.

[0260] Optionally, the output devices can be connected to system 600 using communication channel 651 via communication interface 650. Display 661 and speaker 671 may be combined into a single unit together with other components of system 600 in an electronic device (e.g., a television).

[0261] In various embodiments, display interface 660 can include a display driver such as a timing controller (TCon) chip.

[0262] For example, if the RF portion of the input terminal 690 is part of a separate set-top box, the display 661 and speaker 671 may be selectively configured separately from one or more of the other components. In various embodiments where the display 661 and speaker 671 are external components, output signals can be provided via dedicated output connections (including, for example, an HDMI port, a USB port, or a COMP output terminal).

[0263] Figures 1-19 illustrate various methods, each including one or more steps or actions to achieve the described method. Unless a specific sequence of steps or actions is required for the precise operation of the method, the sequence and / or use of certain steps and / or actions can be modified or combined.

[0264] While several examples of block diagrams and / or operation flowcharts have been described, each block represents a portion containing code for one or more executable instructions to implement a circuit element, module, or (one or more) specified logic function. Note that in other embodiments, the (one or more) functions shown in a block do not necessarily have to occur in the indicated order. For example, depending on the function involved, two blocks shown sequentially may actually be executed in basic parallel, or they may be executed in reverse order.

[0265] Each embodiment and aspect described herein can be implemented, for example, as a method or process, apparatus, computer program, data flow, bitstream, or signal. Even when considered only in the context of a single form of embodiment (for example, as a method only), embodiments of the features discussed can be implemented in other forms (for example, apparatus or computer program).

[0266] The method can be implemented, for example, in a processor, which typically refers to a processing device including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. The processor may further include a communication device.

[0267] Furthermore, the method can be implemented by instructions executed by a processor, and such instructions (and / or data values ​​generated by the embodiment) can be stored in a computer-readable storage medium. The computer-readable storage medium can be one or more computer-readable media, and can be in the form of a computer-readable program product having computer-executable computer-readable program code executed therein. Given its inherent ability to store information and to retrieve information provided thereby, the computer-readable storage medium used herein can be considered a non-temporary storage medium. The computer-readable storage medium may be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination thereof. The following provides more specific examples of computer-readable storage media to which the embodiments of this application can be applied, but it should be understood that these are illustrative and not exhaustive, as will be readily apparent to those skilled in the art: portable computer floppy disks, hard disks, read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0268] Instructions can generate applications that are tangibly executed on a processor-readable medium.

[0269] For example, instructions can reside in hardware, firmware, software, or a combination thereof. Instructions can be found, for example, in an operating system, a standalone application, or a combination of both. Therefore, a processor can be characterized as a device including, for example, a device configured to execute a process, and a processor-readable medium (e.g., a storage device) containing instructions for executing a process. In addition to or instead of instructions, the processor-readable medium can store data values ​​generated by the embodiment.

[0270] The device can be implemented, for example, with appropriate hardware, software, and firmware. Examples of such devices include personal computers, laptop computers, smartphones, tablets, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected consumer electronics, head-mounted displays (HMDs, X-ray glasses), projectors, "caves" (systems including multiple displays), servers, video encoders, video decoders, post-processors that process the output from the video decoder, pre-processors that provide input to the video encoder, web servers, set-top boxes, and any other devices for processing video images or other communication devices. The device is portable and can be mounted on a moving vehicle.

[0271] Computer software can be implemented by the processor 610, hardware, or a combination of hardware and software. In non-limiting examples, each embodiment can be implemented by one or more integrated circuits. Memory 620 may be of any type suitable for the technical environment and can be implemented by any suitable data storage technology (in non-limiting examples, e.g., optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and portable memory). In non-limiting examples, processor 610 may be of any type suitable for the technical environment and can cover one or more of microprocessors, general-purpose computers, dedicated computers, and processors based on multi-core architectures.

[0272] Embodiments of the present invention also provide an electronic device comprising a processor and a memory for storing instructions that can be executed by the processor. The processor is configured to perform the method described in any of the above embodiments.

[0273] As will be obvious to those skilled in the art, embodiments can generate signals formatted to carry information, for example, to be stored or transmitted. The information may include instructions for performing the method or data generated by one of the embodiments described. For example, a signal may be formatted to carry a bitstream of the embodiment described. This signal may be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband. The formatting may include encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, signals can be transmitted over a variety of different wired or wireless links. Signals can be stored on a processor-readable medium.

[0274] The terms used herein are for illustrative purposes only and not limiting to specific embodiments. Unless otherwise indicated by the context, the singular terms “one,” “one kind,” and “the / the foregoing” as used herein are intended to also include the plural. Furthermore, as used herein, the terms “include / contain” and / or “include / contain” may indicate the presence of a feature, integer, step, operation, element, and / or component described, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof. Furthermore, when one element is said to “respond to,” “connect to,” or “associated with” another element, it may directly respond to the other element, or it may be connected to the other element, or there may be an intermediate element. In contrast, when one element is said to “directly respond to,” or “directly connect to,” or “directly associated with” another element, there is no intermediate element.

[0275] For example, in the cases of "A / B," "A and / or B," and "at least one of A and B," the use of any one of the symbols / terms " / ," "and / or," and "at least one" is intended to cover the selection of the first listed option (A), or the selection of the second listed option (B), or the selection of two options (A and B). As further examples, in the cases of "A, B and / or C" and "at least one of A, B and C," such wording is intended to cover the selection of only the first listed option (A), or only the selection of the second listed option (B), or only the selection of the third listed option (C), or only the selection of the first and second listed options (A and B), or the selection of the first and third listed options (A and C), or only the selection of the second and third listed options (B and C), or all three options (A, B, and C). As will be apparent to those skilled in the art, this can thus be extended to the number of listed items.

[0276] In this application, various numerical values can be used. The specific values are exemplary, and each aspect described is not limited to these specific values.

[0277] Terms such as "first", "second", etc. can be used in this specification to describe various elements, but these elements are not limited to these terms. These terms are only used to distinguish one element from another. For example, unless departing from the teachings of this application, the first element can be called the second element, and similarly, the second element can also be called the first element. There is no implicit ordering indicated between the first element and the second element.

[0278] The citation of "one example" or "example" or "one embodiment" or "embodiment" and other variations is frequently used to convey that certain features, structures, characteristics, etc. (as described in the example / embodiment) are included in at least one example / embodiment. Therefore, the appearance of the terms "in one example" or "in an example" or "in one embodiment" or "in an embodiment" and any other variations in each part of this application does not necessarily refer to an example.

[0279] The citation of "one example" or "example" or "one embodiment" or "embodiment" and other variations is frequently used to convey that certain features, structures, characteristics, etc. (as described in the example / embodiment) are included in at least one example / embodiment. Therefore, the expressions "according to an example / embodiment" or "in an example / embodiment" that appear in each part of this application do not necessarily refer to the same example / embodiment, and individual or alternative examples / embodiments are not necessarily mutually exclusive of other examples / embodiments.

[0280] The reference numerals in the drawings appearing in the claims are used only for explanation and do not limit the scope of the claims. Despite the lack of specific explanation, the examples / embodiments and variations of this application can be used in any combination or partial combination.

[0281] Please understand that when a diagram is presented as a flowchart, a corresponding block diagram of the device is also provided. Similarly, please understand that when a diagram is presented as a block diagram, a corresponding flowchart of the method / process is also provided.

[0282] Some diagrams include arrows indicating the primary direction of communication within the communication path, but please understand that communication can also occur in the opposite direction to the arrows described.

[0283] Various embodiments relate to decoding. As used in this application, “decoding” can cover all or part of the process of a final output adapted for further processing in the displayed or reconstructed video region, performed, for example, on a received video image (which may include a bitstream received by encoding one or more video images). In various embodiments, such a process includes one or more of the processes typically performed by a decoder. In various embodiments, for example, such a process may alternatively include the processes performed by the decoders of the various embodiments described herein.

[0284] As further examples, in one embodiment, “decoding” may refer only to inverse quantization; in another embodiment, “decoding” may refer to entropy decoding; in yet another embodiment, “decoding” may refer to differential decoding; and in yet another embodiment, “decoding” may refer to a combination of inverse quantization, entropy decoding, and differential decoding. In the context specifically described, it should be clear to those skilled in the art whether the term “decoding process” refers to a subset of operations or to a broader decoding process.

[0285] Various embodiments relate to encoding. As with respect to “decoding” described above, “encoding” as used in this application may include all or part of the processes performed on an input video image to generate an output bitstream. In various embodiments, this type of process includes one or more of the processes typically performed by an encoder. In various embodiments, this process further includes, or selectively includes, the processes performed by the encoders of the various embodiments described herein.

[0286] As further examples, in one embodiment, “encoding” may refer only to quantization; in another embodiment, “encoding” may refer only to entropy coding; in yet another embodiment, “encoding” may refer only to differential coding; and in yet another embodiment, “encoding” may refer to a combination of quantization, differential coding, and entropy coding. Based on the context specifically described, it is clear and should be well understood by those skilled in the art whether the term “encoding process” refers to a specific subset of operations or to a broader encoding process.

[0287] Furthermore, this application refers to various forms of "acquisition" of information. Acquisition of information may include, for example, one or more of the following: estimation of information, calculation of information, prediction of information or retrieval of information from memory, processing of information, movement of information, copying of information, deletion of information, calculation of information, determination of information, prediction of information or estimation of information.

[0288] Furthermore, this application refers to the "reception" of various types of information. Reception of information may include, for example, one or more of the following: accessing information or receiving information from a communication network.

[0289] Furthermore, as used herein, the term “signal” specifically indicates that the corresponding decoder is instructing something particular. For example, in some embodiments, the encoder signals specific information, such as coded parameters or encoded video image data. This allows the same parameters to be used on both the encoder and decoder sides in the embodiments. Thus, for example, the encoder can transmit specific parameters to the decoder (exact signaling), thereby allowing the decoder to use the same specific parameters. Conversely, if the decoder has specific parameters and other parameters, signaling that does not require transmission (indirect signaling) facilitates informing the decoder and allowing it to select specific parameters. Bit saving is achieved in various embodiments to avoid the transmission of any actual functions. It should be understood that multiple signaling methods can be used. For example, in various embodiments, one or more grammatical elements, flags, etc., are used to transmit information to the corresponding decoder. The above relates to the verb form of the word “signaling,” but the word “signal” may also be used as a noun in this specification.

[0290] Although several embodiments have already been described, it should be understood that various modifications are possible. For example, other embodiments can be generated by combining, supplementing, modifying, or deleting elements from different embodiments. Also, as those skilled in the art will understand, other structures and processes can substitute for the disclosed structures and processes, and the embodiments thus generated will achieve at least one or more results that are essentially the same as the disclosed embodiments by performing essentially the same functions in essentially the same way. Accordingly, these and other embodiments are envisioned in this application.

Claims

1. A method for decoding video images, The decoding is performed by temporal bidirectional prediction with motion compensation for the interconnected blocks. The aforementioned bidirectional temporal prediction using motion compensation is Two reference video images in two separate reference video image lists, Using two affine motion fields defined by at least two control point motion vectors, wherein the control point motion vectors are represented as CPMV, and the at least two control point motion vectors are associated with each reference image, The refined CPMV is obtained as the output of the following first step (171), or as the output of the second step (172) following the first step (171). - In the first step (171) above, for each CPMV, bilateral matching is performed on the block centered on the CPMV to derive at least two CPMVs refined to integer precision, and a set of CPMVs including the unrefined CPMV and / or the CPMVs refined to integer precision is selected to generate an overall predicted block having the minimum bilateral matching cost, - In the second step (172) above, for each consecutive CPMV in the selected CPMV set associated with the block, each CPMV is refined to subsample accuracy to minimize the bilateral matching cost of the block. Based on the comparison between the bilateral matching cost associated with the CPMV performed in the first step and the threshold, the second step (172) is bypassed. How to decode video images.

2. If the bilateral matching cost associated with the CPMV performed in the first step (171) is lower than the first threshold (TH1), the second step (172) is bypassed for the CPMV. A method for decoding a video image according to claim 1.

3. If the bilateral matching cost associated with the CPMV performed in the first step (171) is higher than the second threshold (TH2), the second step (172) is bypassed for the CPMV. A method for decoding a video image according to claim 1.

4. The first threshold (TH1) and / or the second threshold (TH2) are fixed or adapted to the prediction unit size. A method for decoding a video image according to claim 2 or 3.

5. The two affine motion fields are defined by a first CPMV, a second CPMV, and a third CPMV, and the second step is bypassed for the third CPMV if the bilateral matching cost associated with the first CPMV or the second CPMV satisfies the bypass condition. A method for decoding a video image according to any one of claims 1 to 3.

6. If the bilateral matching cost associated with the first CPMV or the second CPMV performed in the second step (172) is higher than the bilateral matching cost associated with the first CPMV or the second CPMV performed in the first step (171), the bypass condition is met. A method for decoding a video image according to claim 5.

7. The bypass condition is met if the bilateral matching cost associated with the first CPMV or the second CPMV performed in the second step (172) is lower than the bilateral matching cost associated with the first CPMV or the second CPMV performed in the first step (171), and the reduction between the bilateral matching cost associated with the first CPMV or the second CPMV performed in the first step (171) and the bilateral matching cost associated with the first CPMV or the second CPMV performed in the second step (172) is lower than a third threshold (TH3). A method for decoding a video image according to claim 5.

8. For a specific block shape, the first and second steps are disabled. A method for decoding a video image according to any one of claims 1 to 3.

9. If the width or height of the block to be intercoded is greater than the first value when the width and height are equal, the first and second steps are disabled. A method for decoding a video image according to claim 8.

10. If the width or height of the block to be intercoded is greater than a second value when the width and height are different, the first and second steps are disabled. A method for decoding a video image according to claim 8.

11. Define a block shape based on the width or height of the intercoded block, and signal at least one block shape in the bitstream that invalidates the first step and the second step. A method for decoding a video image according to claim 8.

12. The at least one block shape that invalidates the first and second steps is signaled by signaling the maximum and minimum block width and / or height. A method for decoding a video image according to claim 11.

13. The at least one block shape that invalidates the first and second steps is signaled by signaling the minimum and / or maximum block ratio between the maximum block width and block height and the minimum block width and block height. A method for decoding a video image according to claim 11.

14. Includes a component for performing one of the methods described in any one of claims 1 to 3, Device.

15. A computer program, wherein if the computer program is executed by one or more processors, the one or more processors are made to perform the method according to any one of claims 1 to 3. Computer program.

16. The program code includes instructions for performing the method according to any one of claims 1 to 3. A non-temporary storage medium.

17. It is an electronic device, Processor and Includes a memory for storing instructions that can be executed by the processor, The processor is configured to perform the method described in any one of claims 1 to 3. Electronic devices.

Images