Method and device for decoding video image, as well as data media

Abstract

FIELD: video decoding.SUBSTANCE: invention relates to decoding a video image by means of temporal bidirectional prediction with motion compensation for a block with interframe coding using two reference video images from two separate lists of reference video images and two affine motion fields defined by at least two control point motion vectors (CPMV) associated with each reference image. The method includes stages of performing, for each CPMV, a two-way matching for a block centred on the CPMV in order to obtain at least two refined CPMVs with integer precision, and selecting a set of CPMVs that includes unrefined or refined CPMVs with integer precision in order to obtain a complete predicted block with a minimum cost of two-way matching; for each successive CPMV of the selected set of CPMVs associated with the block, refining the CPMVs with a sub-counting precision in order to minimize the cost of two-way matching of the block, wherein the refinement is skipped in accordance with the result of a comparison between the cost of two-way matching associated with the CPMV for which the two-way matching is performed and a threshold value.EFFECT: reduced complexity of CPMV refinement while maintaining maximum compression efficiency.11 cl, 19 dwg

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is the national phase entry in the Russian Federation of international application PCT / CN2023 / 099204, filed on June 8, 2023, which claims priority from European patent application No. 23305027.7, filed on January 9, 2023, and the contents of both of these applications are fully incorporated into this description by reference.

[0003] FIELD OF TECHNOLOGY

[0004] The present invention generally relates to video encoding and decoding. In particular, but not exclusively, the technical field of the present invention relates to a method and apparatus for decoding video images.

[0005] BACKGROUND TO THE INVENTION

[0006] According to the current state of video compression technology, the ECM (Enhanced Compression Model) applies decoder-side motion vector refinement (DMVR) to blocks encoded using affine fusion when the DMVR condition is met. This is called affine DMVR.

[0007] The paper JVET-AB0178 submitted by JVET (Joint Video Experts Team) (https: / / jvet-experts.org / doc_end_user / documents / 28_Mainz / wg11 / JVET-AB2028-v1.zip) proposes a CPMV refinement for affine DMVR to further improve affine DMVR, and different MV refinement bias can be applied to different CPMVs.

[0008] SUMMARY OF THE INVENTION

[0009] According to a first aspect of the present invention, a method is provided for decoding a video image by means of temporal bidirectional prediction with motion compensation for an inter-frame coded block using two reference video images from two separate lists of reference video images and two affine motion fields defined by at least two control point motion vectors (CPMVs) and associated with each reference image, wherein according to the method:

[0010] - In the first step, for each CPMV, two-way matching is performed for the block centered on the CPMV to obtain at least two integer-precision refined CPMVs, and a set of CPMVs including unrefined or integer-precision refined CPMVs is selected to obtain a complete predicted block with the minimum two-way matching cost,

[0011] - in the second step, for each consecutive CPMV of the selected CPMV set associated with the block, each CPMV is refined with a sub-count accuracy to minimize the cost of two-way matching of the specified block,

[0012] wherein the second step is skipped in accordance with the result of the comparison between the cost of the bilateral agreement related to the CPMV, which is performed in the first step, and the threshold value.

[0013] According to a second aspect of the present invention, a computer-readable medium is provided, comprising program code instructions, which, when executed by a processor of the device, cause the device to perform a method for decoding a video image by means of temporal bidirectional prediction with motion compensation for a block with inter-frame coding using two reference video images from two separate lists of reference video images and two affine motion fields defined by at least two control point motion vectors (CPMV) associated with each reference image, wherein the method comprises:

[0014] for each CPMV, perform two-way matching on the block centered on the CPMV to obtain at least two integer-precision refined CPMVs, and select a set of CPMVs including the unrefined or integer-precision refined CPMVs to obtain a complete predicted block with the minimum two-way matching cost,

[0015] For each consecutive CPMV of the selected CPMV set associated with the block, refine the CPMV with sub-count accuracy to minimize the cost of two-way block matching,

[0016] wherein the refinement is skipped in accordance with the result of the comparison between the cost of the bilateral agreement related to the CPMV for which the bilateral agreement is performed and the threshold value.

[0017] According to a third aspect of the present invention, a device is proposed for decoding a video image by means of temporal bidirectional prediction with motion compensation for a block with inter-frame coding using two reference video images from two separate lists of reference video images and two affine motion fields defined by at least two control point motion vectors (CPMVs) associated with each reference image, comprising a processor and a memory for storing instructions executable by the processor. The processor is configured to perform the following:

[0018] for each CPMV, perform two-way matching on a block centered on the CPMV to obtain at least two integer-precision refined CPMVs, and select a set of CPMVs including the unrefined or integer-precision refined CPMVs to obtain a complete predicted block with a minimum two-way matching cost,

[0019] For each consecutive CPMV of the selected CPMV set associated with the block, refine the CPMV with sub-count accuracy to minimize the cost of two-way block matching,

[0020] wherein the refinement is skipped in accordance with the result of the comparison between the cost of the bilateral agreement related to the CPMV for which the bilateral agreement is performed and the threshold value.

[0021] Specific characteristics of at least one of the embodiments, as well as other objectives, advantages, features and methods of using said at least one of the embodiments will become apparent from the following description.

[0022] BRIEF DESCRIPTION OF DRAWINGS

[0023] The following are references to the accompanying drawings, which show embodiments of the present invention:

[0024] Fig. 1 shows an example of a coding tree block according to HEVC;

[0025] Fig. 2 shows an example of dividing coding blocks into prediction blocks in accordance with HEVC;

[0026] Fig.3 shows an example of CTU division according to VVC;

[0027] Fig.4 shows examples of partition modes supported when partitioning a multi-type tree according to VVC;

[0028] Fig. 5 shows a flow chart of the steps of the method 100 for encoding a video image (VP) according to the prior art;

[0029] Fig. 6 shows a flow chart of the steps of the method 200 for decoding a video image (VP) according to the prior art;

[0030] Fig. 7 shows an example of constructing an AMVP candidate list based on whole blocks, used to determine an inter-prediction coded block for the current block of the current video image;

[0031] Fig. 8 shows an example of limited motion vector differences according to the prior art;

[0032] Fig. 9 shows an example of geometric partitioning of a video image block according to the prior art;

[0033] Fig. 10 shows examples of geometric partitioning of a block according to the prior art;

[0034] Fig. 11 shows an example of representing an affine motion field of a video image block according to the prior art;

[0035] Fig. 12 shows an example of motion vectors associated with sub-blocks of a video image block according to the prior art;

[0036] Fig. 13 shows examples of obtaining an affine model of motion from a translational motion vector relative to control points located in the corners of a video image block, according to the prior art;

[0037] Fig. 14 shows an example of obtaining designed CPMVP candidates by combining MV translational motion of adjacent blocks according to the prior art;

[0038] Fig. 15 shows an illustration of a method for refining bilateral agreement according to the prior art;

[0039] Fig. 16 shows an example of the scheme and order of search points used by the DMVR affine method;

[0040] Fig. 17 is a schematic block diagram of the steps of the CPMV refinement method described in JVET-AB0178;

[0041] Fig. 18 shows the positions of search points according to the prior art;

[0042] Fig. 19 is a block diagram of an example system in which various aspects and embodiments of the present invention are implemented.

[0043] Similar or identical items are designated by the same reference numbers.

[0044] DESCRIPTION OF EMBODIMENTS OF THE INVENTION

[0045] Below, at least one embodiment is described in more detail with reference to the accompanying drawings, which show examples of at least one embodiment. However, the embodiment may be implemented in many alternative forms and should not be construed as limited to the examples presented in this description. Accordingly, it should be understood that the embodiments are not limited to the specific forms disclosed. On the contrary, the present description covers all modifications, equivalents, and alternatives that fall within the spirit and scope of the present invention.

[0046] This section is intended to familiarize the reader with various aspects of the art that may be related to various aspects of at least one embodiment described and / or claimed below. It is believed that this discussion will be useful in providing the reader with background information to better understand the various aspects of the present invention. Accordingly, it should be understood that this information is to be considered in this light and not as an admission that it relates to the existing art.

[0047] Modern video compression systems such as HEVC (ISO / IEC 23008-2, High Efficiency Video Coding, ITU-T Recommendation H.265, https: / / www.itu.int / rec / T-REC-H.265-202108-P / en) or VVC (ISO / IEC 23090-3, Versatile Video Coding, ITU-T.266 Recommendation, https: / / www.itu.int / rec / T-REC-H.266-202008-I / en) provide low-level and high-level image partitioning to divide the video image into image areas, the so-called Coding-Tree Units (CTUs), the size of which can typically range from 16x16 to 64x64 pixels for HEVC and 32x32, 64x64 or 128x128 pixels for VVC.

[0048] Partitioning a video image into CTUs creates a grid of CTUs of a fixed size, namely a CTU grid whose top and left boundaries spatially coincide with the top and left boundaries of the video image. The CTU grid represents a spatial partition of the video image.

[0049] In the VVC and HEVC systems, the CTU size (CTU width and CTU height) of all CTUs in a CTU grid is the same default CTU size (Default CTU Width (CTU DW) and Default CTU Height (CTU DH). For example, the default CTU size (Default CTU Height, Default CTU Width) can be 128 (CTU DW=CTU DH=128). The default CTU size (height, width) is encoded in the bitstream, for example, at the sequence level in the Sequence Parameter Set (SPS).

[0050] The spatial position of a CTU in the CTU grid is determined based on the CTU address (ctuAddr, CTU address), which specifies the spatial position of the upper-left corner of the CTU relative to the coordinate origin. As shown in Fig. 1, the CTU address can specify the spatial position relative to the upper-left corner of a higher-level spatial structure S containing the CTU.

[0051] Each CTU has an associated coding tree, which defines the tree-like partitioning of the CTU.

[0052] As shown in Fig. 1, in the HEVC system, the coding tree is a partition of a CTU quadtree, each leaf of which is called a coding unit (CU). The spatial position of a CU in a video picture is determined by a CU index (cuIdx, CU index), which indicates the spatial position relative to the upper left corner of the CTU. A CU is spatially partitioned into one or more prediction units (PUs). The spatial position of a PU in a video picture (VP) is specified by a PU index (puIdx, PU index), which specifies the spatial position relative to the upper left corner of the CTU, and the spatial position of an element of the partitioned PU is determined by a PU partition index (puPartIdx, PU Partition Index), which specifies the spatial position relative to the upper left corner of the PU. Each PU is assigned some intra-frame or inter-frame prediction data.

[0053] The coding mode (intra-frame or inter-frame) is assigned at the CU level. This means that each PU in a CU is assigned the same intra-frame / inter-frame coding mode, although the prediction parameters vary from PU to PU.

[0054] A CU can also be spatially partitioned into one or more transform units (TUs) according to a quadtree called a transform tree. Transform units are the leaves of the transform tree. The spatial position of a TU in a video image is determined by the TU index (tuIdx, TU Index), which specifies the spatial position relative to the upper-left corner of the CU. Each TU is assigned certain transform parameters. The transform type is assigned at the TU level, and a separate two-dimensional transform is performed at the TU level during encoding or decoding of the image block.

[0055] The PU partition types used in the HEVC system are shown in Fig. 2. They include square partitions (2N x 2N and N x N), which are used only in CUs with intra- and inter-prediction, symmetric non-square partitions (2N x N, N x2 N, which are used only in CUs with inter-prediction), and asymmetric partitions (used only in CUs with inter-prediction). For example, a PU of the 2N x nU type means an asymmetric horizontal partition of the PU, in which the smaller partition is located at the top of the PU. According to another example, a PU of the 2N x nL type means an asymmetric horizontal partition of the PU, according to which the smaller partition is located at the top of the PU.

[0056] As shown in Fig. 3, in the VVC system, the coding tree starts from the root node, i.e., CTU. Next, the quadtree (or quaternary tree) divides the root node into 4 nodes, corresponding to 4 sub-blocks of the same size (solid lines). Then, the leaves of the quaternary tree (or quadtree) can be further partitioned using the so-called multi-type tree, which includes binary or ternary partitioning according to one of the 4 partitioning modes shown in Fig. 4. These partitioning types are vertical and horizontal binary partitioning modes, denoted as SBTV and SBTH, as well as vertical and horizontal ternary partitioning modes (SPTTV and STTH).

[0057] The leaves of the CTU coding tree are CUs in the case of a joint coding tree used by the luma and chroma components.

[0058] Unlike HEVC, in VVC system, in most cases, CU, PU and TU have the same size, which means that coding units are generally not divided into PU or TU except for some specific coding modes.

[0059] Figures 5 and 6 provide an overview of video signal encoding / decoding methods used in modern standard video signal compression systems such as HEVC or VVC.

[0060] Fig. 5 shows a flow chart of steps of a method 100 for encoding a video image (VP) according to the prior art.

[0061] In step 110, the video image (VP) is divided into blocks of samples, and the partition data is transmitted into the bitstream. Each block contains samples of one component of the video image (VP). Thus, the blocks contain samples of each component defining the video image (VP).

[0062] For example, in HEVC, an image is divided into coding tree units (CTUs). Each CTU can be further partitioned using a quadtree, in which each leaf is designated a coding unit (CU). The partition data can then include data describing the CTU and the subdivision of each CTU within the quadtree.

[0063] Each block of samples in a shortened block can represent either a CU (if a CU consists of a single PU) or a PU of a CU block.

[0064] Each block is encoded in a coding cycle, also called an "intra cycle", using intra-frame or inter-frame prediction mode.

[0065] Intra-prediction (step 120) uses intra-prediction data. Intra-prediction involves predicting the current block using an intra-prediction block based on already encoded, decoded, and reconstructed samples located around the current block, typically above and to the left of it. Intra-prediction is performed in the spatial domain.

[0066] In the inter-prediction mode, motion estimation (step 130) and motion compensation (step 135) are performed. During motion estimation, a candidate reference block that is a good predictor of the current block is searched for in one or more reference video images used for predictive coding of the current video image. For example, a good predictor of the current block is a block similar to the current block. The output of the motion estimation step 130 is inter-prediction data containing motion information (usually one or more motion vectors and one or more reference video image indices) associated with the current block, as well as other information used to obtain the same predicted block on the encoding / decoding side.Then, as a result of motion compensation (step 135), a predicted block is obtained using the motion vector(s) and the index(es) of the reference video image determined in the motion estimation step 130. Essentially, a block belonging to the selected reference video image and indicated by the motion vector can be used as the predicted block of the current block. Furthermore, since motion vectors are expressed in fractions of integer pixel positions (known as a sub-pixel motion vector representation), motion compensation typically involves spatial interpolation of some reconstructed samples of the reference video image to calculate the predicted block.

[0067] Prediction data is transmitted in the bitstream. The prediction information may include the prediction mode (intra-frame, inter-frame, or skip), intra-frame / inter-frame prediction data, and any other information used to obtain the same predicted CU on the decoding side.

[0068] According to the method 100, one prediction mode (an intra-frame or inter-frame prediction mode) is selected by optimizing a rate-distortion trade-off taking into account coding of a prediction residual block calculated, for example, by subtracting a predicted block candidate from the current block, and transmitting prediction data necessary for determining said predicted block candidate on the decoding side.

[0069] Normally, the best prediction mode is set as the prediction mode of the best coding mode p* for the current block as follows:

[0070]

[0071] where P is the set of all possible coding modes for the current block, p is a possible coding mode in this set, RD cost (p) is the rate-distortion cost for a possible coding mode p, usually expressed as:

[0072]

[0073] D(p) is the distortion between the current block and the reconstructed block obtained after encoding / decoding the current block using a candidate coding mode p, R(p) is the rate cost associated with encoding the current block using the coding mode p, and λ is the Lagrange parameter, which is the rate constraint for encoding the current block and is usually calculated based on the quantization parameter used to encode the current block.

[0074] The current block is typically encoded based on a Prediction Residual (PR) block. More specifically, the PR block is calculated, for example, by subtracting the best predicted block from the current block. The PR block is then transformed (step 140) using, for example, a DCT (Discrete Cosine Transform) or DST (Discrete Sinus Transform) transform, or any other suitable transform, after which the resulting transformed block of coefficients is quantized (step 150).

[0075] According to an embodiment, when performing the method 100, the transform step 140 may also be skipped and the quantization (step 150) may be applied directly to the residual prediction (PR) block in accordance with a so-called transform skip coding mode.

[0076] The block of quantized transform coefficients (or quantized residual prediction block) is entropy coded into a bit stream (step 160).

[0077] The quantized block of transform coefficients (or quantized residual block) is then dequantized (step 170) and inversely transformed (step 180) (or not) in a coding cycle, resulting in a decoded residual prediction block. The decoded residual prediction block and the predicted block are then combined, typically summed, to produce a reconstructed block.

[0078] Other data may also be entropy coded in step 160 to code the current video image (VP) block.

[0079] In-loop filters (step 190) can be applied to the reconstructed image (containing the reconstructed blocks) to reduce compression artifacts. In-loop filters can be applied after all image blocks have been reconstructed. For example, they include a deblocking filter, a sample adaptive offset (SAO) filter, or an adaptive loop filter.

[0080] The reconstructed blocks or filtered reconstructed blocks form a reference picture, which can be stored in the decoded picture buffer (DPB) so that it can be used as a reference picture for encoding the next current video picture (VP) block or the next video picture to be encoded.

[0081] Fig. 6 shows a flow chart of the steps of a method 200 for decoding a video image (VP) in accordance with the existing art.

[0082] In step 210, partition data, prediction data, and a quantized block of transform coefficients (or a quantized residual block) are obtained by entropy decoding the bitstream of encoded video image data. For example, this bitstream was generated according to method 100.

[0083] Other data may also be entropy decoded to decode from the bitstream of the current video picture (VP) block.

[0084] In step 220, the reconstructed image is divided into current blocks based on the partition information. Each current block is entropy decoded from the bitstream in a decoding loop, also called the "inner loop." Each decoded current block represents either a quantized block of transform coefficients or a quantized residual prediction block.

[0085] In step 230, the current block is dequantized and possibly inverse transformed (step 240) to obtain a decoded prediction residual block.

[0086] On the other hand, the prediction data is used to predict the current block. The predicted block is obtained through intra-frame prediction (step 250) or motion-compensated temporal prediction (step 260). The prediction process performed on the decoding side is identical to that performed on the encoding side.

[0087] The decoded residual prediction block and the predicted block are then combined, usually summed, to obtain a reconstructed block.

[0088] In step 270, in-loop filters may be applied to the reconstructed image (containing the reconstructed blocks), and the reconstructed blocks or filtered reconstructed blocks form a reference image, which may be stored in a decoded picture buffer (DPB), as discussed above (see Fig. 5).

[0089] In step 130 / 135 shown in Fig. 5 or step 260 shown in Fig. 6, an inter-prediction block is determined based on the inter-prediction data associated with the current block (CU or PU of the CU block) of the video image. This inter-prediction data contains motion information that can be represented (coded) in accordance with the so-called Adaptive Motion Vector Prediction (AMVP) mode based on whole blocks or the so-called fusion mode based on whole blocks.

[0090] In HEVC, in the block-based AMVP mode, the motion information used to determine a block with inter-picture prediction is represented by up to two reference video picture indices, associated with up to two reference video picture lists, respectively, typically denoted as L0 and L1. The reference video picture list L0 contains at least one reference video picture, and the reference video picture list L1 contains at least one reference video picture. Each reference video picture index is associated with the temporal prediction of the current block. The motion information also contains up to two motion vectors, each of which is associated with a reference video picture index from one of the two reference video picture lists.Each motion vector is predictively encoded and transmitted into the bitstream. This means that a single differential motion vector (MVd) value is derived from the motion vector and one AMVP (Adaptive Motion Vector Predictor) candidate selected from the whole-block AMVP candidate list (generated on the encoding and decoding side), and this MVd is transmitted into the bitstream. The index of the selected AMVP candidate from the whole-block AMVP candidate list is also transmitted in the bitstream.

[0091] Figure 7 illustrates an example of forming an AMVP candidate list based on whole blocks, used to determine a block with inter-frame prediction for the current block of the current video image.

[0092] A whole-block AMVP candidate list may include two spatial MVP (Motion Vector Predictor) candidates obtained from the current video image. The first spatial MVP candidate is generated based on the motion information associated with the inter-predicted blocks, if any, located to the left of the neighboring positions A0 and A1 of the current block, and the second spatial MVP candidate is generated based on the motion information associated with the inter-predicted blocks, if any, located at the neighboring positions B0, B1, and B2 of the current block above. A redundancy check is then performed between the obtained spatial MVP candidates, i.e., duplicate MVP candidates are discarded.The whole-block-based AMVP candidate list may additionally include a temporal MVP candidate derived from the motion information associated with the adjacent block, if any, in the reference video image at spatial position H or, otherwise, at spatial position C. The temporal MVP candidate is scaled according to the temporal distances between the current video image and the reference video image. Finally, if the whole-block-based AMVP candidate list contains fewer than two MVP candidates, the whole-block-based AMVP candidate list is padded with zero motion vectors.

[0093] In HEVC, in the whole-block fusion mode, the motion information used to determine a block for inter-picture prediction is represented by a single fusion index of a whole-block fusion MVP candidate list. Each fusion index references motion predictor information, which indicates which MVP is used to obtain the motion information. The motion information is represented by one type of unidirectional or bidirectional temporal prediction, a maximum of two reference video picture indices, and a maximum of two motion vectors, each associated with a reference video picture index of one of the two reference video picture lists (L0 or L1).

[0094] No information other than the merge index is transmitted. This means that the motion vector of the current block is set equal to the motion vector of the whole-block-based MVP candidate specified by the merge index. Thus, unlike the whole-block-based merge mode, neither the MVd nor the reference image index are transmitted in the bitstream. Only the index of the selected merge candidate from the whole-block-based merge candidate list is transmitted in the bitstream.

[0095] Thus, unlike AMVP mode, in merge mode neither the MVd nor the reference image are transmitted. Only the merge index is transmitted in the bitstream.

[0096] The MVP candidate list for whole block fusion may contain five spatial MVP candidates obtained from the current video image, as shown in Fig. 7.The first spatial MVP candidate is obtained from the motion information associated with the inter-predictable block, if it exists, located at the left neighboring position A1, the second spatial MVP candidate is obtained from the motion information associated with the inter-predictable block, if it exists, located at the upper neighboring position B1, the third spatial MVP candidate is obtained from the motion information associated with the inter-predictable block, if it exists, located at the upper right neighboring position B0, the fourth spatial MVP candidate is obtained from the motion information associated with the inter-predictable block, if it exists, located at the left lower neighboring position A0, and the fifth spatial MVP candidate is obtained from the motion information associated with the inter-predictable block, if it exists, located at the left neighboring position B2.Next, a redundancy check is performed between the resulting spatial MVPs, i.e., duplicate MVP candidates are discarded. The whole-block-based fusion candidate list may additionally include a temporal MVP candidate, called a TMVP (Temporal MVP) candidate, which is derived from the motion information associated with the adjacent block, if any, in the reference video image at position H or, otherwise, at the central spatial position C. Next, a redundancy check is performed between the resulting spatial MVPs, i.e., duplicate MVP candidates are discarded. Finally, when using bidirectional temporal prediction, if the whole-block-based fusion candidate list contains fewer than 5 MVP candidates, the merged candidate is added to the whole-block-based fusion candidate list.A merged candidate is formed based on the motion information associated with one reference video list and related to one MVP candidate already present in the whole-block-based fusion candidate list, with the motion information associated with another reference video list and related to another MVP candidate already present in the whole-block-based fusion candidate list. Finally, if the whole-block-based fusion candidate list is still incomplete (5 fusion candidates), the whole-block-based fusion candidate list is padded with zero motion vectors.

[0097] VVC motion information encoding provides a more complete representation of motion information than HEVC.

[0098] In VVC, motion information can be represented (coded) according to the whole-block-based inter-picture prediction mode, which provides motion representation on a whole-block basis, or the sub-block-based inter-picture prediction mode, which provides motion representation on a sub-block basis.

[0099] In VVC, whole-block-based motion representation can be provided either according to the whole-block-based AMVP mode or according to the whole-block-based fusion mode.

[0100] In VVC, in the whole-block AMVP mode, the motion information used to determine the inter-frame prediction block is represented in the same way as in the whole-block AMVP mode in HEVC. The whole-block AMVP candidate list may include spatial and temporal MVP candidates, as in HEVC, if they exist. The whole-block AMVP candidate list may also include four additional HMVP (History-Based Motion Vector Prediction) candidates, if they exist. Finally, if the whole-block AMVP candidate list contains fewer than two MVP candidates, the whole-block AMVP candidate list is padded with zero motion vectors.

[0101] HMVP candidates are formed from previously encoded MVPs associated with adjacent or non-adjacent blocks relative to the current block. For this purpose, a table of HMVP candidates is maintained on both the encoder and decoder sides and is updated on the fly as an MVP buffer operating on the First-In-First-Out (FIFO) principle. The table can contain up to five HMVP candidates. After encoding one block, the table is updated by appending the corresponding motion information to the end of the table as a new HMVP candidate. The table is managed using the FIFO rule, which, in addition to the basic FIFO mechanism, first removes a redundant candidate from the HMVP table instead of the first element. The table is reset at each CTU row to enable parallel processing.

[0102] In addition to the whole-block-based AMVP candidate list, the whole-block-based AMVP mode uses new tools compared to HEVC, which are listed below.

[0103] SMVD (Symmetric Motion Vector Difference): SMVD consists of setting the MVd values ​​associated with one reference video list for a bidirectional block to be equal to the opposite MVd values ​​associated with the other reference video list for the current block. Furthermore, the reference video images used in SMVD mode are generated by the decoder according to certain predefined rules. SMVD reduces the cost of coding MVd and reference picture index information and is selected at the block level.

[0104] AMVR (Adaptive Motion Vector Resolution). The AMVR tool allows MVDs to be transmitted with luminance sample resolutions of quarter-pixel, half-pixel, full-pixel, or 4 pixels. This also allows for bit savings when encoding MVD information. In AMVR, motion vector resolution is selected at the block level.

[0105] BCW (Bi-prediction with coding unit weights) enables bi-prediction of a block with unequal weights reported at the block level.

[0106] Finally, the internal representation of the motion vector is achieved with an accuracy of 1 / 16 of the brightness sample instead of the accuracy of 1 / 4 luminance samples in HEVC.

[0107] In VVC, in the whole block fusion mode, the motion information used to determine the inter-frame prediction block is represented as in HEVC whole block fusion mode.

[0108] The whole block-based merge candidate list is different from the whole block-based merge candidate list used in HEVC.

[0109] In VVC, a whole block-based merge candidate list can be created for MMVD (Merge Mode with MV Difference), GPM (Geometric Partitioning Mode), or CIP (Combined Infra / Inter Prediction) mode.

[0110] The MMVD mode enables encoding a limited motion vector difference (MVd) on top of a selected fusion candidate to represent motion information associated with an inter-predicted block. MMVD encoding is limited to 4 vector directions and 8 amplitude values, from 1 / 4 luminance samples to 32 luminance samples, as shown in Fig. 8. The MMVD mode provides an intermediate level of accuracy, and therefore an intermediate trade-off between the cost of speed and the accuracy of the MV (motion vector) for conveying motion information.

[0111] GPM consists of dividing the current block PG into two motion sections MV0 and MV1 along a straight line, as shown in Fig. 9. Each motion vector MV0 and MV1 points to the block P0, P1 of the reference video image of one of the two lists of reference video images.

[0112] The geometric partitioning can be either rectangular or non-rectangular. If the partitioning is rectangular, asymmetric partitioning is performed, avoiding the redundancy of binary partitioning at the CU level. The use of GPM is specified using a CU-level flag as a specific merge mode based on whole blocks.

[0113] The orientation and position of the split line relative to the center of the current block is specified at the CU level using the GPM index allocated for this purpose. A total of 64 partitions are supported in the geometric partitioning mode for each possible CU size, i.e., w×h=2^m×2^n, where m,n∈{3…6}, except for the 8 x 64 and 64 x 8 sizes. For example, Fig. 10 shows the different split lines that can be obtained for different position offsets from the CU center and for different split line angles.

[0114] Each geometric partition in the current PG block is predicted using inter-frame prediction using eigenmotion. For each partition, only unidirectional prediction is allowed, meaning each partition is characterized by one motion vector and one reference video image index. The unidirectional inter-frame prediction constraint is applied to ensure that only two motion-compensated predictions are required for each current PG block, similar to traditional bidirectional inter-frame prediction.

[0115] The motion vector MV0 and MV1 of each partition are output based on the two merge indices, respectively, for each partition.

[0116] After each geometric partition is predicted, the sample values ​​along the geometric partition boundary are adjusted using adaptive weight blending. This serves as the prediction signal for the entire current PG block, and the transform and quantization process is applied to the entire current PG block (rather than to each partition), as in other block-based prediction modes.

[0117] The CIIP mode combines the inter-frame prediction signal with the intra-frame prediction signal to predict the current block. The inter-frame prediction signal in the CIIP mode is generated using the same inter-frame prediction process as in the whole-block fusion mode; and the intra-frame prediction signal is generated using the conventional intra-frame prediction process in the planar mode. The inter-frame and intra-frame prediction signals are then combined using weighted averaging, where the weight is calculated depending on the coding modes of the adjacent blocks located above and to the left:

[0118]

[0119] Sum of weights W merge and W intra is equal to 4, and these weights are constant throughout the current block.

[0120] Finally, in VVC, the whole-block-based merge candidate list can contain spatial candidates similar to HEVC, except that the first two candidates are swapped: candidate B1 is considered before candidate A1 when constructing the whole-block-based merge candidate list. The whole-block-based merge candidate list can also include TMVPs, similar to HEVC, and HMVP candidates, as in the VVC AMVP mode. Multiple HMVP candidates are included in the whole-block-based merge candidate list so that the merge candidate list reaches the maximum allowed number of MVP candidates (minus 1). The whole-block-based merge candidate list can also contain up to 1 pairwise averaged candidate, calculated as follows: the first two merge candidates present in the whole-block-based merge candidate list are considered, and their motion vectors are averaged.This averaging is calculated separately for each list of reference videos, L0 and L1. Thus, if both MVPs are bidirectional, the motion vectors belonging to both lists, L0 and L1, are averaged. If only one motion vector is present in the list of reference videos, it is considered in its original form to form a paired candidate. Finally, if the list of fusion candidates does not reach the maximum allowed number of MVP candidates, then the list of whole-block fusion candidates is padded with zero motion vectors.

[0121] In VVC, sub-block-based inter-prediction mainly consists of dividing the current block into sub-blocks, usually 4x4 or 8x8 luminance sample sub-blocks, and the sub-block-based motion representation used to determine the inter-predicted block is represented by motion information associated with all the sub-blocks.

[0122] HEVC uses only a translational motion model for motion-compensated temporal prediction. This translational motion model is unable to detect certain types of motion, such as approaching, receding, rotating, perspective motion, and irregular motion.

[0123] To solve this problem, the affine motion mode based on sub-blocks at the CU level can be used in VVC. As shown in Fig. 11, the affine motion field of the current block is determined by the translational motion vectors of two control points (4-parameter affine motion model) or three control points (6-parameter affine motion model). In Fig. 11, the translational motion vectors ν0, ν1, ν2 of the control points located at the corners of the current block are used to obtain the affine motion field of the current block.

[0124] For the 4-parameter affine motion model, the vector (mν x , mν y) the movement at the location (x, y) of the current block's reference is determined as follows:

[0125]

[0126] where (mν 0x , mν 0y ) is the translational motion vector of the control point of the upper left corner, a {mν 1x , mν 1y ) - the motion vector of the control point of the upper right corner.

[0127] For the 6-parameter affine motion model, the motion vector at the (x, y) reference location in the 4x4 subblock of the current block is determined as follows:

[0128]

[0129] where (mν 0x , mν 0y ) is the vector of translational motion of the control point in the upper left corner, (mν 1x , mν 1y ) is the vector of translational motion of the control point in the upper right corner, and (mν 2х , mν 2y ) - the vector of translational motion of the control point in the lower left corner. Parameters represent parameters that are not parameters of translational motion (rotation, scaling).

[0130] To obtain the motion vector of each sub-block (usually the luminance sub-block) of the current block, the motion vector of the center sample of each sub-block is calculated according to Equations 1 or 2, as shown in Fig. 12, and rounded to an accuracy of 1 / 16. Then, motion compensation interpolation filters are applied to generate a prediction of each sub-block using the obtained motion vector, resulting in an inter-frame prediction of the current block. The sub-block size in chroma components is also 4×4. The MV value of a 4×4 chroma sub-block is calculated as the average of the MV values ​​of the upper-left and lower-right luminance sub-blocks in the combined 8×8 luminance region.

[0131] Hereinafter, the control point motion vector is called CPMV (Control Point Motion Vector).

[0132] The sub-block-based motion representation used to determine the inter-frame prediction block can be set according to the sub-block-based affine AMVP mode or the sub-block-based affine fusion mode.

[0133] Subblock-based affine fusion mode can be applied to the current block whose width and height are greater than or equal to 8. An affine fusion candidate list containing up to 5 subblock-based motion candidates is used. The subblock-based motion candidates are CPMVP candidates based on the motion information associated with adjacent spatial blocks. The index of the selected subblock-based motion candidate from the affine fusion candidate list is transmitted in the bitstream. Finally, if the affine fusion candidate list is not full (5 CPMVP candidates), this list is padded with zero motion vectors.

[0134] CPMVP candidates can be either so-called inherited CPMVP candidates, which are obtained by extrapolating the affine motion model (CPMV) of neighboring blocks, or so-called constructed CPMVP candidates, which are CPMVP candidates obtained by combining the translational motion MVs of neighboring sub-blocks.

[0135] There are at most two inherited CPMVP candidates in the VVC, which are derived by extrapolating the affine motion model of the left neighboring block and the affine motion model of the upstream neighboring block. For a left CPMVP candidate, if the affine motion model of the left neighboring block A0 exists, the left CPMVP candidate is extrapolated from the CPMV of this affine motion model. Otherwise, the left CPMVP candidate is extrapolated from the CPMV of the affine motion model of the left neighboring block A1, if it exists. For the above CPMVP candidate, if the affine motion model of the upstream neighboring block B0 exists, the above CPMVP candidate is extrapolated from the CPMV of this affine motion model. Otherwise, if the affine motion model of the upstream neighboring block B1 exists, the above CPMVP candidate is extrapolated from the CPMV of this affine motion model.Otherwise, the CPMVP candidate is extrapolated from the CPMV affine motion model of the left neighboring block B2, if it exists.

[0136] Extrapolation of affine motion models of neighboring blocks means obtaining inherited CPMVP candidates by merging CPMVs of the specified affine motion models.

[0137] As shown in Fig. 13, if there is an affine motion model of the adjacent lower left sub-block A, then the translational motion vectors v2, v3, and v4 of the said affine motion model, which refer to the upper left corner, upper right corner, and lower left corner of the adjacent block containing the sub-block A, are available. When encoding the sub-block A using the above affine model with 4 parameters, the left CPMVP candidate of the current block is inherited (calculated) by combining the translational motion vectors v2 and v3 according to the following two equations.

[0138]

[0139] where {X curr,Y curr ) - position of the current block, (X neighb ,Y neighb ) - position of the adjacent block and (W neighb ,H neighb ) - the size of the neighboring block.

[0140] When coding sub-block A using the above 6-parameter affine model, the three CPMVPs of the current block are calculated by combining the translational motion vectors v2, v3, and v4 according to the following equations.

[0141]

[0142] In VVC, constructed affine candidates are obtained by merging non-affine MVs of adjacent blocks. Checkpoint MVs are obtained from spatial and temporal adjacent subblocks.

[0143] Fig. 14 shows an example of obtaining constructed CPMVP candidates by merging the forward MVs of adjacent blocks.

[0144] CPMV k(k=1,2,3,4) represents the progressive MV associated with the k-th checkpoint of the current block. For CPMV1, if the progressive MV of the neighboring subblock B2 exists, CPMV1 is equal to the specified progressive MV. Otherwise, if the progressive MV of the neighboring subblock B3 exists, CPMV1 is equal to the specified progressive MV. Otherwise, if the progressive MV of the neighboring subblock A2 exists, CPMV1 is equal to the specified progressive MV. For CPMV2, if the progressive MV of the neighboring subblock B1 exists, CPMV2 is equal to the specified progressive MV. Otherwise, if the progressive MV of the neighboring subblock B0 exists, CPMV2 is equal to the specified progressive MV. For CPMV3, if the progressive MV of the neighboring subblock A1 exists, CPMV3 is equal to the specified progressive MV. Otherwise, if there is a translational MV of the adjacent subblock A0, CPMV3 is equal to the specified translational MV.For CPMV4, if there is a translational MV of a co-located adjacent sub-block in the reference video picture at position T, CPMV4 is equal to the specified translational MV.

[0145] The constructed CPMVP candidates are derived by concatenating CPMVk as follows: If CPMV1, CPMV2, and CPMV3 exist, the constructed CPMVP candidate is derived by concatenating {CPMV1, CPMV2, CPMV3}. Otherwise, if CPMV1, CPMV2, and CPMV4 exist, the constructed CPMVP candidate is derived by concatenating {CPMV1, CPMV2, CPMV4}. Otherwise, if CPMV1, CPMV3, and CPMV4 exist, the constructed CPMVP candidate is derived by concatenating {CPMV1, CPMV3, CPMV4}. Otherwise, if CPMV2, CPMV3, and CPMV4 exist, the constructed CPMVP candidate is derived by concatenating {CPMV2, CPMV3, CPMV4}. Otherwise, if CPMV1 and CPMV2 exist, then the constructed candidate CPMVP is derived by combining {CPMV1, CPMV2}. Otherwise, if CPMV1 and CPMV3 exist, then the constructed candidate CPMVP is derived by combining {CPMV1, CPMV3}.

[0146] To avoid motion scaling, if CPMVk point to different reference video images, the MV combinations of the control points associated with the specified CPMVk are discarded.

[0147] Combining 3 CPMVk results in a 6-parameter affine motion model (Equation 2), and combining 2 CPMV results in a 4-parameter affine motion model.

[0148] In subblock-based affine merge mode, an affine merge candidate list containing CPMVP candidates is used. The merge index mergeIdx specifies the CPMVP candidate used to obtain the current block's motion information. The best CPMVP candidate in the affine merge candidate list is determined for the current block as the candidate that minimizes the distortion-rate cost of encoding that block.

[0149] The affine merge candidate list may include the SbTMVP (Subblock-based Temporal Motion Vector Prediction) candidate and / or legacy CPMVP candidates and / or constructed CPMVP candidates, as discussed above. Finally, if the affine merge candidate list does not contain the maximum number of candidates, it is padded with zero motion vectors.

[0150] The SbTMVP candidate can be obtained as follows: Like the TMVP candidate, the SbTMVP candidate is derived from the motion information of the co-located sub-block, if any, in the reference video image. The SbTMVP candidate differs from the TMVP candidate in the following two main aspects:

[0151] - TMVP predicts motion at the CU level, and SbTMVP predicts motion at the sub-CU level;

[0152] - TMVP extracts temporal motion vectors from a co-located block in a reference video image, SbTMVP applies a motion shift before extracting temporal motion information from the reference video image, and the motion shift is obtained based on a motion vector from one of the neighboring spatial sub-blocks of the current block.

[0153] The sub-block-based affine AMVP mode can be applied to current blocks whose width and height are greater than or equal to 16 in VVC. In the sub-block-based affine AMVP mode, the CPMV (Mvd) differences between the MVs of the control points located at the corners of the current block and their Control Point Motion Vector Predictors (CPMVPs) are transmitted into the bitstream. Finally, the CPMVP index, which refers to a CPMVP candidate from the affine AMVP candidate list, is transmitted into the bitstream.

[0154] The CPMVPs used to predict the MV checkpoints located at the corners of the current block are derived from an affine AMVP candidate list consisting of two CPMVP candidates.

[0155] The affine AMVP candidate list may contain inherited CPMVP candidates and / or constructed CPMVP candidates.

[0156] In ECM ("Algorithm description of Enhanced Compression Model 7 (ECM 7)"), M. Coban, F. Le Léannec, R.-L. Liao, K. Naser, J. Ström, L. Zhang, document JVET-AB2025, Joint Video Experts Team (JVET), ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29, 28 еmeeting, Germany, 21-28 October 2022, https: / / jvet-experts.org / doc_end_user / documents / 28_Mainz / wg11 / JVET-AB2025-v1.zip) the motion information used to determine the inter-prediction block can also be represented according to the so-called Bilateral Matching AMVP (BM-AMVP) mode.

[0157] The BM-AMVP merging mode is allowed if at least one video picture is available in one reference picture list (in one so-called inter-prediction direction) of the video picture containing the current block, and at least one reference video picture is available in another reference picture list (in another inter-prediction direction) of the video picture. In the BM-AMVP merging mode, the motion information used to determine the inter-prediction block includes first motion information associated with a reference video picture from the reference video picture list and second motion information associated with a reference video picture from the other reference video picture list.

[0158] In the BM-AMVP fusion mode, as defined in the ECM, the first motion information is a first whole-block-based motion vector (MV) associated with a reference video picture from a reference video picture list associated with an inter-prediction direction, and the second motion information is a second whole-block-based MV associated with a reference video picture from another reference video picture list associated with another inter-prediction direction.

[0159] The first whole-block-based MV is represented as discussed above in connection with the whole-block-based AMVP mode in VVC, thus it is derived from the whole-block-based motion vector predictor (MVP), the signaled reference image index, and the signaled motion vector difference obtained using the whole-block-based AMVP mode.

[0160] The second whole-block-based MV is represented as discussed above in relation to the whole-block-based fusion mode in VVC, thus it is derived from the whole-block-based motion vector predictor (MVP) obtained by the whole-block-based fusion mode.

[0161] The whole-block-based MVP used to obtain the second whole-block-based MV (i.e., the index mergeIdx in the whole-block-based merge MVP candidate list specified by the index refListMerge) is derived based on the following two-way consensus method that minimizes the two-way consensus cost Bmcost defined as follows:

[0162]

[0163] where mIdx[refListMerge] is an index that points to a candidate MVP from the list of MVP candidates for whole-block-based merge specified by the index refListMerge, and MvpIdx[refListAmvp] is a whole-block-based MVP used to predict the first whole-block-based MV of the BM-AMVP merge mode.

[0164] For each MVP candidate in the whole-block-based MVP candidate list, the cost of a two-way consensus is calculated using the specified MVP candidate and MvpIdx[refListAmvp]. The MVP candidate with the lowest cost is selected as the whole-block-based MVP used to derive the second whole-block-based MV.

[0165] In VVC, in order to improve the accuracy of the MV vectors of the fusion mode, a method for refining the motion vector on the decoder side based on bilateral matching (BM), referred to as the DMVR (Decoder side Motion Vector Refinement) method, is applied to the current block, wherein the motion vector obtained in the AMVP mode based on whole blocks (MV0 in Fig. 15) and the second MV based on whole blocks (MV1 in Fig. 15) obtained in the fusion mode based on whole blocks are used as starting points. The principle is as follows. In the bidirectional prediction operation, refined motion vectors MV0' and MV1' are derived around two initial motion vectors MV0 and MV1 pointing to the reference blocks of reference videos from the reference video lists L0 and L1 by minimizing the cost of bilateral matching between the two reference blocks (minimizing the motion vector difference (Mdiff)).Essentially, local search consists of obtaining the MV refinement offset (MV offset). Local search typically uses an n×n square search pattern (n is an integer) to cycle through the search range [-sHor, sHor] in the horizontal direction and [-sVer, sVer] in the vertical direction. The Bilateral Matching (BM) cost is calculated as the sum of mvDistanceCost and sadCost, where mvDistanceCost is the motion vector encoding cost and increases as a function of the motion vector magnitude, and sadCost is a metric representing the differences between two reference blocks. For example, sadCost is the sum of the absolute difference (SAD) of the samples of the two reference blocks. If the BM cost at the center of the n×n search pattern reaches the minimum cost, the local search terminates.Otherwise, the current minimum-cost search point becomes the new center point of the n×n search pattern, and the minimum-cost search continues until the end of the search range is reached.

[0166] In DVMR, search points surround the original MV, and the MV refinement offset obeys the mirror image rule of the MV difference. In other words, any points checked by DMVR and specified by a candidate MV pair (MV0, MV1) obey the following two equations:

[0167]

[0168] where MV_offset represents the MV refinement offset between the original MV and the refined MV in one of the reference images.

[0169] The DMVR search includes an integer sample offset search step (the first DMVR step) followed by a fractional refinement step (the second DMVR step). The refinement search range is two integer luminance samples from the original MV.

[0170] The refined MV obtained by the DMVR method is used to generate inter-frame prediction samples and for temporal motion vector prediction for coding subsequent images. Meanwhile, the original MV is used in the deblocking process and for spatial motion vector prediction for coding subsequent CUs.

[0171] In VVC, the first step of the DMVR method uses a full search across 25 points. First, the VM cost of the original MV pair is calculated. If the VM cost of the original MV pair is less than the threshold, the first step of the DMVR method terminates. Otherwise, the VM costs of the remaining 24 points are calculated and checked in raster scanning order. The search point with the lowest VM cost is selected as the result of the first step of the DMVR method.

[0172] To reduce the uncertainty penalty for DMVR refinement, it is proposed to favor the original MV during the DMVR process. The SAD between the reference blocks referenced by the original MV candidates is reduced by 1 / 4 of the SAD value.

[0173] To reduce computational complexity, the second step of the DMVR method is implemented using a parametric error surface equation instead of an additional search by comparing with the SAD. Refinement of the fractional count is performed conditionally based on the result of the first step of the DMVR method. If the first step of the DMVR method terminates at the center with the smallest SAD in either the first or second iteration of the first step of the DMVR method, additional refinement of the fractional count is applied (the second step of the DMVR method).

[0174] In the parametric estimation of subpixel offsets based on the error surface, the VM cost of the center position and the VM costs at four adjacent positions from the center are used to fit a two-dimensional parabolic error surface equation of the following form:

[0175]

[0176] where (x min ,y min ) corresponds to the fractional position with the lowest VM cost, and C corresponds to the minimum VM cost value. By solving the above equations using the VM cost of five points, search for the value (x min ,y min ) are calculated as follows:

[0177]

[0178] Values ​​x min and y minare automatically limited to the range from -8 to 8, since all VM cost values ​​are positive and the smallest value is E(0,0). This corresponds to a half-pixel offset with an MV accuracy of 1 / 16 pixel in VVC. The calculated fractional values ​​(x min ,y min ) are added to the integer distance refinement MV to obtain a differential refinement MV with sub-pixel accuracy.

[0179] In VVC, the use of DMVR is limited and applies only to those CUs that meet one of the specific DMVR conditions.

[0180] The DMVR condition is met if the CUs are encoded using one of the following encoding modes and functions:

[0181] - CU level fusion mode with bidirectional MV prediction;

[0182] - one reference image is in the past and the other reference image is in the future relative to the current image;

[0183] - the distances (i.e. the difference in ROS) from the two reference images to the current image are the same;

[0184] - Both reference images are short-term reference images;

[0185] - CU consists of more than 64 brightness counts;

[0186] - the height and width of CU are greater than or equal to 8 brightness counts;

[0187] -BCW weight index indicates equal weight;

[0188] - Weighted Prediction (WP) is not enabled for the current block; Weighted Prediction allows a multiplicative weight and additive bias to be applied to motion-compensated prediction.

[0189] ("Weighted prediction in the H.264 / MPEG AVC video coding standard", JM Voce, May 2004, 10.1109 / iscas.2004.1328865).

[0190] - SIR mode is not used for the current block.

[0191] In VVC, the MV resolution is 1 / 16 of a luminance sample. Samples at fractional positions are interpolated using an 8-sample interpolation filter. In DMVR, search points surround the original fractional pixel MV with an integer sample offset, so samples at these fractional positions need to be interpolated for the DMVR search process. To reduce the computational complexity, a bilinear interpolation filter is used to generate fractional samples during the DMVR search process. Another important effect is that by using a bilinear filter over a search range of 2 samples, DVMR does not access more reference samples than the conventional motion compensation process. After the refined MV is obtained through the DMVR search process, a conventional 8-sample interpolation filter is applied to generate the final prediction.To avoid accessing more reference samples for the normal MC (Motion Compensation) process, samples that are not required for the original MV-based interpolation process but are required for the refined MV-based interpolation process are supplemented based on these available samples.

[0192] If the width and / or height of a CU exceeds 16 luminance samples, it is further divided into sub-blocks with a width and / or height equal to 16 luminance samples. The maximum block size for the DMVR search process is limited to 16x16.

[0193] In ECM, the DMVR method is applied to blocks encoded using affine merge when the DMVR condition is met. This is called affine DMVR. The first step of affine DMVR (the integer offset search step) is applied to the forward part of the affine move (Equation 3) such that the offset MV of the forward move is added to all CPMVs of the candidate in the affine merge list if the candidate satisfies the DMVR condition. The offset MV is obtained by minimizing the cost of VM, which is the same as the cost of conventional DMVR.

[0194] The affine DMVR method involves a 3x3 square pattern (8 search points) which is used to cycle through the search range set within [-3, 3] to find the best integer refinement offset of the MV (the first step of affine DMVR).

[0195] Figure 16 shows an example of the scheme and order of search points used in the affine DMVR method. The cross x denotes the initial search position, and positions 0,…,7 denote all eight search point positions.

[0196] Then fractional refinement (the second step of DMVR affine method) is performed around the best integer position, and finally, error surface estimation is performed to find the optimal MV refinement offset with 1 / 16 accuracy.

[0197] In affine DMVR, no preference is given to the original MV and all search points are evaluated.

[0198] The paper submitted to JVET, JVET-AB0178 (https: / / jvet-experts.org / doc_end_user / documents / 28_Mainz / wg11 / JVET-AB2028-v1.zip), proposes a CPMV refinement for affine DMVR to further improve affine DMVR, and different MV refinement bias can be applied to different CPMVs.

[0199] The method described in JVET-AB0178 can be applied to fusion candidates in affine fusion mode and affine MMVD mode.

[0200] Fig. 17 is a schematic flow chart showing the steps of executing the CPMV refinement method described in JVET-AB0178.

[0201] The method for refining CPMV corresponds to the DMVR affinity method described for the relevant technical field.

[0202] In the first step 171, for each vector initCpMvLX[cpIdx] of the checkpoint motion with cpIdx=0.numCpMv - 1, where numCpMv is the number of CPMVs of the current affine coding block, the method performs a two-way matching for the block centered on the checkpoints to obtain refined CPMVs (bmRefinedCpMvLx[cpIdx]).

[0203] Then the initCpMvLX[cpIdx] and bmRefinedCpMvLx[cpIdx] combinations are iterated over in a loop and the best set of CPMVs is selected that minimize the VM cost of the current block.

[0204] In the second optional step, 172 CPMVs from the best CPMV set are iteratively refined to further minimize the VM cost of the current block in accordance with the principles of the second step of the affine DMVR method. At each iteration, one CPMV is refined, and the others are fixed.

[0205] The advantage of the method shown in Fig.17 is that it improves the compression efficiency of modern video codecs.

[0206] However, the disadvantage is that the resulting trade-off between increasing compression efficiency and increasing the complexity of the encoder and decoder is not so advantageous, since the method involves a significant increase in the complexity of the codec.

[0207] The purpose of the present invention is to improve the tradeoff between increasing compression efficiency and encoding time cost when performing the CPMV refinement method proposed in JVET-AB0178.

[0208] At least one embodiment of the present invention is developed taking into account the above.

[0209] At least one aspect generally relates to encoding and decoding a video image, another aspect generally relates to transmitting a provided or encoded bitstream, and another aspect relates to receiving / accessing a decoded bitstream.

[0210] At least one embodiment is described for encoding / decoding a video image, but extends to encoding / decoding video images (sequences of images), since each video image is sequentially encoded / decoded, as described below.

[0211] Furthermore, at least one embodiment is not limited to MPEG standards such as AVC (ISO / IEC 14496-10 Advanced Video Coding for generic audio-visual services, ITU-T Recommendation H.264, https: / / www.itu.int / rec / T-REC-H.264-202108-P / en), EVC (ISO / IEC 23094-1 Essential video coding), HEVC (ISO / IEC 23008-2 High Efficiency Video Coding, ITU-T Recommendation H.265, https: / / www.itu.int / rec / T-REC-H.265-202108-P / en), VVC (ISO / IEC 23090-3 Versatile Video Coding, Recommendation, ITU-T H.266, https: / / www.itu.int / rec / T-REC-H.266-202008-I / en), but can be applied to other standards and recommendations, such as AV1 (AOMedia Video 1, http: / / aomedia.org / av1 / specification / ).At least one embodiment may be applied to existing standards and recommendations, or those being developed for future use, as well as to extensions thereof. Unless otherwise noted or technically feasible, the aspects described herein may be used individually or in combination.

[0212] A pixel corresponds to the smallest unit of display on a screen, which can consist of one or more light sources (1 for a monochrome screen or 3 or more for color screens).

[0213] A video image, also called a frame or image frame, contains at least one component (also called an image component or channel) defined by a specific image / video format that indicates all the information related to the pixel values ​​and all the information that can be used by a display device and / or any other device to display and / or decode the video image data related to the video image.

[0214] A video image contains at least one component, typically expressed in the form of an array of samples.

[0215] A monochrome video image consists of one component, while a color video image may consist of three components.

[0216] For example, a color video image may contain a luminance (or luminance) component and two chrominance components if the image / video format is a well-known format (Y, C, Cr), or may contain three color components (one for red, one for green, and one for blue) if the image / video format is a well-known format (R, G, B).

[0217] Each video image component may contain a number of samples corresponding to the number of pixels on the screen on which the video image is to be displayed. In some embodiments, the number of samples included in a component may be a multiple (or fractional) of the number of samples included in another component of the same video image.

[0218] For example, in the case of a video format that includes a luminance component and two chrominance components, as in the (Y, Cb, Cr) format, depending on the color format in question, the chrominance component may contain half the number of samples in width and / or height relative to the luminance component.

[0219] A sample is the smallest unit of visual information of a component that makes up a video image. A sample value can be, for example, a luminance value, a chroma value, or a color value in the format (R, G, B).

[0220] A pixel value is the value of a pixel on the screen. A pixel value can be represented by a single sample for a monochrome video image and multiple adjacent samples for a color video image. The adjacent samples associated with a pixel are the samples corresponding to the pixel's position on the screen.

[0221] A video image is typically thought of as a set of pixel values, with each pixel represented by at least one sample.

[0222] A video image block is a set of samples of one component of a video image. It can be considered a block consisting of at least one luminance sample, or a block consisting of at least one chrominance sample if the image / video information format is a well-known format (Y, Cb, Cr), or a block consisting of at least one chrominance sample if the image / video information format is a well-known format (R, G, B).

[0223] At least one embodiment is not limited to a specific image / video information format.

[0224] In general, the present invention relates to various embodiments of a method for reducing the complexity of CPMV refinement performed according to JVET-AB0178 while maximizing the compression efficiency obtained using JVET-AB0178.

[0225] In one embodiment, the MV / CPMV refinement search is optimized by systematically storing the results of the affine DMVR, i.e., the obtained best MVs and the lowest cost VM (the result of the first or second step of the affine DMVR method).

[0226] In ECM, the second step of the affine DMVR and steps 171 and 172 process the affine DMVR search scheme, and it is sometimes repeated multiple times to improve compression efficiency by selecting MVs that provide the VM cost. When repeated multiple times, the second step of the affine DMVR method can be skipped if the previous iteration did not provide a better solution.

[0227] When using multiple iterations of the affine DMVR search scheme, the original MV and best VM cost are updated using the best results (best CPMV and best VM cost) obtained in the previous iterations.

[0228] Assume position 0 is the best. Figure 18 shows the positions that will be evaluated in the next iteration. The positions corresponding to the gray rectangles represent positions that were evaluated in the previous iteration. The positions corresponding to the shaded rectangles represent positions that were evaluated in the previous iteration and will be reused in the current iteration. Positions 0, 1, 2, 7, and 6 will be evaluated in the new iteration.

[0229] In ECM, when running multiple iterations of the refinement process, the iteration result (the best MV / CPMV and the lowest distortion) is recorded (saved) only if it is the first iteration. When processing in this way, if this refinement is combined with other algorithms, subsequent algorithms are not initialized using the best identified MV / CPMV or do not use the lowest achieved VM cost.

[0230] According to the first embodiment, the results (the best MV and the lowest VM cost) are systematically saved after each iteration of the first or second step of the DMVR affine method.

[0231] In the second embodiment, the search pattern can be divided into two types of positions:

[0232] - main positions, which can be located, for example, on the left, top, right and bottom, i.e. positions {1,3,5,7}, shown in Fig. 16;

[0233] - corner positions, which can be located, for example, at the top left, top right, bottom right and bottom left, i.e. positions {0,2,4,6}, shown in Fig. 16.

[0234] According to the second embodiment, the main positions are first evaluated, and the corner positions are evaluated according to the evaluation results of the adjacent corner positions.

[0235] In one first embodiment of the second embodiment, the evaluation of an adjacent position of a corner position is skipped if none of the adjacent positions of the given corner position is the current best position.

[0236] For example, the evaluation of position 0 (the corner position) can be performed according to the evaluation results of positions 1, 7, and the initial position x (the adjacent positions of the corner position 0), as shown in Fig. 16. The probability that position 0 is the best is small if neither position 1, nor position 7, nor the initial position x represent the best position of the current iteration. In this case, it is advantageous to skip the evaluation of position 0 to save execution time.

[0237] In the second embodiment of the second implementation, a corner position is valued if the VM values ​​of both adjacent positions of the corner position are within a specified range of the current best VM value.

[0238] The range can be set from 0 to the best distortion weighted by a coefficient strictly greater than one.

[0239] For example, suppose position 7 (adjacent to corner position 0) is the best. Using the first variant of the second implementation, positions 2 and 4 are skipped, and positions 1, 7, and x are evaluated. However, if the VM value of position 1 is significantly higher than the current best VM value, that is, the VM value of position 7, then the probability that position 0 is the best is low. The probability that position 0 is the best is higher if the VM values ​​of positions 1, 7, and x are within a given range (not low).

[0240] In the third embodiment, the best MV provides the lowest VM cost between both reference video lists L0 and L1. Motion vector candidate refinement (the second step of affine DMVR) is processed for a given list, i.e., only one MV from the corresponding reference video list (L0 or L1) is refined. The MV refinement bias follows the usual search point pattern and order used by affine DMVR, as discussed above in the description of Fig. 16.

[0241] On the encoder side, both MV reference video lists are sequentially and independently refined to identify and select the best prediction mode.

[0242] To measure the cost of VM, reference video list predictions must be available.

[0243] In the current state of the art, for a given PU, the distortion (e.g., estimated using SAD) is estimated for each sub-block (from the PU block size to a 4x4 block), and predictions are calculated for both lists of reference video images. A sub-block is a set of adjacent 4x4 blocks (at least one) that share the same MVs.

[0244] To optimize such distortion estimation (e.g., estimated using SAD), the reference video list prediction that is not refined remains unchanged / fixed and can be computed only once if the prediction is reasonably preserved for a given PU. For example, for the first defined MV refinement offsets, both predictions are computed, while for subsequent MV refinement offsets, only the predictions of the refined video list are computed, saving half the prediction computations.

[0245] Furthermore, instead of calculating the distortion (e.g., estimated using SAD) at the sub-block level, it can be calculated at the PU level. In the prior art, the distortion (e.g., estimated using SAD) between two predictions of a video image list is calculated for each sub-block and summed across all sub-blocks to obtain the distortion (e.g., estimated using SAD) at the PU level. Predictions can then be generated for each sub-block and the distortion can then be measured at the PU level. This helps reduce the number of function calls (one per PU instead of one per sub-block) and use more efficient internal functions to reduce runtime processing when processing large pixel surfaces.

[0246] In one embodiment of the fourth embodiment, step 172 is skipped for the CPMV in accordance with a comparison between the cost of the two-way negotiation associated with said CPMV, which is performed in the first step 171, and a threshold value.

[0247] In this fourth embodiment, no time is spent on evaluating merge candidates, and thus the execution time of the DMVR affine method is reduced, since step 172 is the longest in execution time in the DMVR affine method, as indicated with respect to Fig. 17.

[0248] In the first embodiment of the fourth embodiment, if the cost of the VM performed for the CPMV in step 171 is below the threshold value TH1, then step 172 for this CPMV is skipped.

[0249] Next, CPMV is partially optimized (being clarified).

[0250] This early completion assumes that the VM cost (distortion) is already low and the refinement significantly exceeds the current speed-distortion tradeoffs of the merge candidate.

[0251] In the second embodiment of the fourth embodiment, if the cost of the VM performed for the CPMV in step 171 exceeds the threshold value TH2, step 172 for this CPMV is skipped.

[0252] Next, CPMV is partially optimized (being clarified).

[0253] Such an early termination suggests that the VM cost (distortion) is high enough and the merger candidate is not competitive enough.

[0254] In the third embodiment of the fourth embodiment, the threshold values ​​TH1 and / or TH2 are either adapted to the PU size or are fixed.

[0255] During the iterative refinement process (steps 171 and 172), the CPMVs are sequentially refined to improve compression efficiency. Then, when running the affine DMVR method, an attempt is made to refine the first CPMV, then the second, and finally, in the case of a 6-parameter affine model, the third CPMV. Refinements are processed holistically, meaning all CPMVs are refined independently of the results of the previous CPMV refinement.

[0256] In the third embodiment of the fourth embodiment, the second step 172 is skipped for the third CPMV if the VM cost (distortion) associated with the first or second CPMV satisfies the skipping conditions.

[0257] In one embodiment of the third variant of the fourth embodiment, the skip condition is satisfied if the value of the VM associated with the first or second CPMV that is generated in the second step 172 exceeds the value of the VM associated with the first or second CPMV that is generated in the first step 171.

[0258] In another embodiment of the third variant of the fourth embodiment, the skip condition is met if the value of the VM associated with the first or second CPMV, which are generated in the second step 172, is lower than the value of the VM associated with the first or second CPMV, which are generated in the first step 171, and if the decrease in the bilateral values ​​is lower than the third threshold value of the TNZ.

[0259] According to the fifth embodiment, the DMVR affine method is prohibited for specific block shapes (CUs).

[0260] Currently, affine DMVR is performed for each CU whose height and width are greater than or equal to 8 luma samples. However, affine DMVR may provide different tradeoffs between execution time and compression efficiency for all CU shapes. Therefore, it is of interest to prohibit affine DMVR for specific CU shapes, for example, if the tradeoff between execution time and compression efficiency is unsatisfactory. By identifying CU shapes for which affine DMVR is ineffective, this affine DMVR can be prohibited to save execution time.

[0261] This fifth embodiment has another advantage because when omitting the affine DMVR for a CU having a specific shape, the CU syntax reported for this CU can also be omitted to further reduce the transmission bit rate.

[0262] In the first embodiment of the fifth embodiment, the affine DMVR method (steps 171 and 172) may be prohibited if the width or height of the block (CU) to be inter-frame coded exceeds a first value, for example 128, if the width and height are equal.

[0263] In the second embodiment of the fifth embodiment, the affine DMVR method (steps 171 and 172) may be prohibited if the width or height of the block (CU) to be inter-frame coded exceeds a second value, for example 64, if the width and height are different.

[0264] In the first and second embodiments of the fifth embodiment, the CU shapes (the width and height of the inter-frame coding CU) in which the affine DMVR is prohibited may be signaled in a bitstream for duplication on the decoder side.

[0265] In a first embodiment of this embodiment, at least one specific shape of the CU in which the affine DMVR is prohibited (steps 171 and 172) may be indicated by signaling a maximum and / or minimum size of the CU, i.e. a maximum and minimum width and / or height of the CU, expressed, for example, as a number of pixels.

[0266] In the second embodiment of this embodiment, at least one block shape in which the affine DMVR is prohibited (steps 171 and 172) may be indicated by signaling a minimum and / or maximum CU ratio between a maximum CU width and a block height and a minimum CU width and a CU height.

[0267] In the current state of the art, for each inter-frame prediction mode, a candidate list is generated using neighboring blocks, temporal information, and combined spatial and temporal information, as discussed above in the introduction to this description.

[0268] When using the affine DMVR method, a list of merge candidates is generated using available spatial and temporal information. The list of merge candidates is not optimal because it may contain duplicates of the same candidates.

[0269] Duplicate merge candidates are not optimal for several reasons.

[0270] First, a duplicate merge candidate is evaluated by the DMVR method as a merge candidate but is not selected because it results in lower compression efficiency due to a higher merge index compared to a merge candidate with a lower merge index. Evaluating duplicate merge candidates then increases the execution time of the DMVR method without improving compression.

[0271] Second, the presence of duplicate merge candidates in the merge candidate list reduces the speed-distortion metrics of the merge candidates following the duplicate merge candidates by needlessly increasing the merge indices associated with the subsequent merge candidates.

[0272] Third, if the merge candidate list is limited, several potential merge candidates are discarded when the maximum list size is reached, while useless duplicate merge candidates remain in the merge candidate list.

[0273] To overcome these shortcomings, in one of the six embodiments, it is checked whether a new merge candidate intended to be added to the merge candidate list is already present in the merge candidate list.

[0274] The sixth embodiment helps limit the overhead of executing the DMVE method, while providing higher compression performance compared to a merge candidate list containing at least one duplicate merge candidate.

[0275] In one embodiment of the sixth embodiment, the reference video index of the new merge candidate is compared with the reference video index of the merge candidates from the merge candidate list and the MVs of both reference video lists (L0 and L1). In the case of an affine block, all CPMVs of both lists are compared. More specifically, each affine merge index refers to predictor information that indicates which CPMVs are used to obtain motion information. The motion information is represented by one type of unidirectional or bidirectional temporal prediction, a maximum of two reference video indexes, and a maximum of three checkpoint motion vectors (CPMVs), each of which is associated with the reference video index of one of the two reference video lists (L0 or L1). Then, when comparing two affine merge candidates, the motion information of both lists (L0 and / or L1) is compared.For the list, this includes a comparison of the reference video image index and the CPMV vectors.

[0276] A potential candidate is identified as a duplicate if the reference indices and MV / CPMV are identical.

[0277] In the sixth implementation, only strictly duplicate merge candidates are discarded. This process could be improved by discarding candidates with MV / CPMV values ​​that are too close to a candidate from one of the lists.

[0278] For example, if all the differences between the MV / CPMV of a potential candidate and one merge candidate from the list are below the threshold, the potential candidate is discarded.

[0279] The comparison involves comparing reference indices, calculating the MV / CPMV difference, and comparing the MV / CPMV difference with a threshold. In this example, the objective is to compare the movement information of both candidates. This differs from the previously described variant in that the CPMV criterion has been expanded to allow for the rejection of additional candidates. This now involves rejecting candidates with CPMV values ​​similar to those in the list, i.e., all elements for which the CPMV difference between both merge candidates is below the threshold.

[0280] If the reference indices are identical and the difference between MV and CPMV is less than the TH4 threshold, the potential candidate is not added to the merge candidate list. The TH4 threshold can be fixed or adaptive, for example, based on the PU size.

[0281] In the DMVR method on the coder side, all merge candidates in the merge candidate list are refined, i.e., the final merge candidates may be different from the merge candidates in the list.

[0282] Therefore, it is possible that a merge candidate from the list matches a merge candidate refined by affine DMVR from the list. More precisely, the encoder runs the affine DMVR algorithm on all merge candidates from the list. As described below, affine DMVR refines the CPMV of an affine merge candidate and thus modifies the CPMV vectors. In this case, it is possible that the Nth refined affine merge candidate from the list, which has received motion information (the indices of the reference video image and CPMV), is equal to or close (i.e., the CPMVs are close) to a merge candidate with a merge index greater than N.

[0283] In this case, applying the DMVR method to the selected merge candidate is ineffective. However, on the coder side (which must evaluate each merge candidate), DMVR refinement can be skipped for a merge candidate if the merge candidate is similar to the merge candidate refined using DMVR. Similar strategies described above can be used to skip DMVR refinement for a merge candidate.

[0284] FIG. 19 is a block diagram illustrating an example of a system 600 in which various aspects and embodiments are implemented.

[0285] System 600 may be embedded in one or more devices including various components described below. According to various embodiments, system 600 may be configured to implement one or more aspects described in this description.

[0286] Examples of equipment that may form all or part of the system 600 include personal computers, laptop computers, smart phones, tablets, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, connected vehicles and related processing systems, head-mounted display devices (HMD, Head Mounted Display, viewing through glasses), projectors (multimedia projectors), "virtual reality environments" (systems containing multiple monitors), servers, video encoders, video decoders, post-processors that process the output signal from a video decoder, pre-processors that provide an input signal to a video encoder, web servers, video servers (e.g., a broadcast server, a video on demand server, or a web server), a camera or camcorder, an encoding or decoding chip, or any other communication devices.The elements of system 600, individual elements or a combination thereof, may be integrated as a single integrated circuit (IC), a plurality of ICs, and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 600 may be distributed across a plurality of ICs and / or discrete components. According to various embodiments, system 600 may be interconnected with other similar systems or with other electronic devices, such as via a communication bus or via dedicated input or output ports.

[0287] System 600 may comprise at least one processor 610 configured to execute downloadable instructions to implement, for example, the various aspects described herein. Processor 610 may comprise embedded memory, an input / output interface, and other various circuits known in the art. System 600 may comprise at least one memory 620 (e.g., volatile and / or non-volatile memory).The system 600 may comprise a memory device 640 that may include volatile and / or nonvolatile memory, including, but not limited to, electrically erasable programmable read-only memory (EEPROM, Electrically Erasable Programmable Read-Only Memory), read-only memory (ROM, Read-Only Memory), programmable read-only memory (PROM, Programmable Read-Only Memory), random access memory (RAM, Random Access Memory), dynamic random access memory (DRAM, Dynamic Random-Access Memory), static random access memory (SRAM, Static Random Access Memory), flash memory, magnetic and / or optical disk. The memory device 640 may comprise an internal memory device, a pluggable memory device, and / or a network storage device, and the memory devices are not limited to the examples given.

[0288] The system 600 may comprise an encoder / decoder module 630 configured, for example, to process data to provide encoded / decoded video image data, and the encoder / decoder module 630 may comprise its own processor and memory. The encoder / decoder module 630 may represent module(s) that may be included in a device to perform encoding and / or decoding functions. As is known, the device may comprise either one encoding or decoding module, or both modules. Furthermore, the encoder / decoder module 630 may be implemented as a separate element of the system 600 or may be integrated into the processor 610 as a combination of hardware and software, as is known to those skilled in the art.

[0289] The program code loaded into the processor 610 or into the encoder / decoder module 630 to perform the various aspects described herein may be stored in the memory 640 and subsequently loaded into the memory 620 for execution by the processor 610. According to various embodiments, one or more of the processors 610, the memory 620, the memory 640, and the encoder / decoder module 630 may store one or more different elements during the execution of the processes described herein. Such stored elements may include, but are not limited to, video images, information used to encode / decode video image data, bit streams, matrices, variables, and intermediate or final results of processing expressions, formulas, operations, and operational logic.

[0290] According to some embodiments, memory integrated into the processor 610 and / or the encoder / decoder module 630 may be used to store instructions and to provide working memory for processing that may be performed during the encoding or decoding process.

[0291] However, according to other embodiments, memory external to the processing device (e.g., the processing device may be either the processor 610 or the encoder / decoder module 630) may be used to perform one or more of these functions. The external memory may be the memory 620 and / or the storage device 640, such as dynamic volatile memory and / or non-volatile flash memory. According to some embodiments, the external non-volatile flash memory may be used to store the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory, such as RAM, can be used as working memory for encoding and decoding operations of a video signal, such as a signal of the MPEG-2 Part 2 standard (also known as ITU-T Recommendation H.262 and ISO / IEC 13818-2, also known as MPEG-2 Video), AVC, HEVC, EVC, VVC, AV1, etc.

[0292] Input data may be supplied to the elements of the system 600 through various input devices shown in block 690. Such input devices include, but are not limited to the given examples, (i) an RF unit that can receive an RF signal transmitted, for example, over an air interface using a radio broadcast network, (ii) a composite input terminal, (iii) a USB input terminal and / or (iv) an HDMI input terminal, (v) a bus such as CAN (Controller Area Network), CAN FD (Controller Area Network Flexible Data-Rate), FlexRay (ISO 17458) or an Ethernet bus (ISO / IEC 802-3), if the present invention is implemented in the automotive field.

[0293] According to various embodiments, the input devices of block 690 may comprise associated corresponding input processing elements known in the art. For example, the RF block may be associated with elements necessary for (i) selecting the desired frequency (also referred to as signal selection or signal band limiting to a frequency band), (ii) down-converting the selected signal, (iii) re-band limiting to a narrower frequency band to select (for example) the frequency band of the signal, which in certain embodiments may be referred to as a channel, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired stream of data packets.In various embodiments, the RF unit may comprise one or more elements for performing these functions, such as frequency selectors, signal selectors, range limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF unit may comprise a tuner that performs the various functions mentioned, including, for example, downconverting the received signal to a lower frequency (e.g., an intermediate frequency or a frequency close to the baseband) or to the baseband frequency.

[0294] According to one embodiment of a television set-top box, an RF unit and associated front-end processing elements can receive an RF signal transmitted over a wired medium (e.g., a cable). The RF unit can then perform frequency selection by filtering, down-conversion, and re-filtering to the desired frequency band.

[0295] In various embodiments, the order of the elements described above (as well as other elements) is changed, some of these elements are deleted, and / or other elements that perform similar or different functions are added.

[0296] Adding elements may include inserting additional elements, such as amplifiers and analog-to-digital converters, between existing elements. In various embodiments, the RF unit may include an antenna.

[0297] Furthermore, the USB and / or HDMI terminals may comprise corresponding interface processors for connecting the system 600 to other electronic devices via USB and / or HDMI connections. It should be taken into account that various aspects of input data processing, such as error correction using a Reed-Solomon code, may be implemented, for example, in separate input data processing ICs or in the processor 610, if necessary. Similarly, aspects of USB or HDMI interface data processing may be implemented, for example, in separate interface ICs or in the processor 610, if necessary. The demodulated and demultiplexed stream with corrected errors may be supplied to various processing elements, including, for example, the processor 610 and the encoder / decoder module 630, working in conjunction with memory and storage elements to process the data stream in order to correctly supply it to the output device.

[0298] The various components of the 600 system may be contained within an integrated housing. Within the integrated housing, the various components may be connected and communicate using a suitable interconnection scheme 690, such as an internal bus known in the art, including an I2C bus, a wired connection, and printed circuit boards.

[0299] System 600 may comprise a communication interface 650 that enables communication with other devices via a communication channel 651. Communication interface 650 may comprise, but is not limited to, a transceiver configured to transmit and receive data via communication channel 651. Communication interface 650 may comprise, but is not limited to, a modem or a network card, and communication channel 651 may be implemented, for example, as a wired and / or wireless medium.

[0300] Streaming data can be transmitted to system 600 in various embodiments using a Wi-Fi network, such as IEEE 802.11. In these embodiments, the Wi-Fi signal can be received via communication channel 651 and communication interface 650, which are adapted for Wi-Fi communication. Communication channel 651 according to these embodiments can typically be connected to an access point or router that provides access to external networks, including the Internet, to activate streaming applications and other upper-layer communication means.

[0301] According to other embodiments, streaming data may be supplied to the system 600 using a set-top box via an HDMI connection of the input unit 690.

[0302] According to other embodiments, streaming data may be supplied to system 600 using an RF connection of input unit 690.

[0303] Stream data may be used as a method for generating signaling information used by system 600. The signaling information may comprise a bit stream B and / or information such as the number of pixels of a video image and / or any encoding / decoding settings.

[0304] It should be noted that signaling can be performed in various ways. For example, in various embodiments, one or more syntax elements, flags, etc., may be used to convey signaling information to the corresponding decoder.

[0305] System 600 may provide an output signal to various output devices, including display 661, speakers 671, and other peripheral devices 681. Other peripheral devices 681 in various embodiments may include one or more stand-alone DVRs, a disc player, a stereo system, a lighting system, and other devices that perform a function based on the output of system 600.

[0306] According to various embodiments, control signals may be transmitted between the system 600 and the display 661, speakers 671, and other peripheral devices 681 using signaling such as AV.Link (Audio / Video Link), CEC (Consumer Electronics Control), or other communication protocols that allow control signals to be transmitted between devices with or without user intervention.

[0307] The output devices can be connected to communicate with the 600 system using dedicated connections through the corresponding interfaces 660, 670 and 680.

[0308] Alternatively, the output devices may be connected to the system 600 via the communication channel 651 through the communication interface 650. The display 661 and speakers 671 may be integrated into a single unit with other components of the system 600 in an electronic device, such as a television.

[0309] According to various embodiments, the display interface 660 may comprise a display driver, such as, for example, a timing controller (T Con) chip.

[0310] The display 661 and speaker 671 may alternatively be located separately from one or more other components, for example, if the RF unit at input 690 is part of a separate television set-top box. According to various embodiments in which the display 661 and speakers 671 may be external components, the output signal may be supplied via dedicated output connections, including, for example, HDMI ports, USB ports, or COMP outputs.

[0311] With reference to Figs. 1-19, various methods are described, and each method includes one or more steps or actions for performing the described method. If a specific order of performing the steps or actions is not required for the correct functioning of the method, the order and / or use of specific steps and / or actions may be changed or combined.

[0312] Some examples are described using flowcharts and / or flowcharts. Each block represents a circuit element, module, or code fragment that includes one or more executable instructions for implementing the specified logical function(s). It should also be noted that in other implementations, the function(s) specified in the blocks may be executed in a different order. For example, two blocks shown sequentially may actually be executed in parallel, or blocks may sometimes be executed in reverse order, depending on the functionality involved.

[0313] The described implementations and aspects may be implemented, for example, by a method or process, equipment, a computer program, a data stream, a bitstream, or a signal. Even if the discussion is conducted in the context of one form of implementation (e.g., only by means of a method), the implementation of the discussed functions may also be implemented in other forms (e.g., by means of equipment or a computer program).

[0314] The methods may be implemented, for example, by a processor, which is commonly referred to as a processing device and includes, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include communication devices.

[0315] Furthermore, the methods may be implemented using instructions executed by a processor, and such instructions (and / or data values ​​generated by the implementation) may be stored on a machine-readable medium. The machine-readable medium may be formed in the form of a machine-readable program product, implemented as one or more machine-readable media and containing machine-readable program code executable by a computer. The machine-readable medium used in this description may be considered as a storage medium with an inherent ability to store information, as well as with the ability to retrieve information stored thereon. The machine-readable medium may be, without limitation to the given examples, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, equipment or device, or any suitable combination of the above devices.It should be noted that in the following more specific examples, the computer-readable media to which embodiments of the present invention may be applicable are given merely by way of illustration and do not represent an exhaustive list, as should be understood by a person skilled in the art, and such media may include a laptop computer diskette, a hard disk, a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or flash memory, a compact disc accessible only for reading (CD-ROM, Compact Disc Read-Only Memory), an optical storage device, a magnetic storage device, or any suitable combination of these devices.

[0316] The instructions may form an application program physically located on the storage medium and readable by the processor.

[0317] Instructions may be embedded in, for example, hardware, firmware, software, or a combination of these. Instructions may form, for example, an operating system, a standalone application, or a combination of the two. A processor may thus be characterized, for example, as a device configured to execute a process and as a device comprising a machine-readable medium (such as a memory device) storing instructions for executing the process. Furthermore, the machine-readable medium may store, in addition to or instead of instructions, data values ​​generated by the implementation.

[0318] The equipment may be implemented, for example, as suitable hardware, software, and firmware. Examples of such equipment include personal computers, laptops, smartphones, tablets, digital multimedia television set-top boxes, digital television receivers, personal video recording systems, plug-in home appliances, head-mounted displays (HMDs, glasses-based displays), projectors (multimedia projectors), "virtual reality environments" (systems containing multiple monitors), servers, video encoders, video decoders, post-processors processing the output signal from the video decoder, pre-processors providing the input signal for the video encoder, web servers, television set-top boxes, and any other device for processing video images, or other communication devices. Obviously, the equipment may be mobile and even mounted on a vehicle.

[0319] The computer software may be implemented using the processor 610 or hardware, or a combination of hardware and software. For example, embodiments may also be implemented using one or more integrated circuits. The memory 620 may be any type of memory suitable for the technical environment and may be implemented using any suitable data storage technology, such as optical memory devices, semiconductor memory devices, magnetic memory devices, read-only memories, and erasable memory, without being limited to the given examples. The processor 610 may be any type of processor suitable for the technical environment and may include one or more microprocessors, general-purpose computers, specialized computers, and processors based on a multi-core architecture, without being limited to the given examples.

[0320] Embodiments of the present invention also include an electronic device comprising a processor and memory for storing instructions executable by the processor. The processor is configured to perform the method according to any of the embodiments described above.

[0321] As will be apparent to those skilled in the art, various signals formatted to carry information may be generated within the implementations, which may, for example, be stored or transmitted. The information may include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal may be formatted to carry a bit stream used in the described embodiment. Such a signal may be formatted, for example, as an electromagnetic wave (e.g., using a portion of the radio frequency spectrum) or as a baseband signal. Formatting may include, for example, encoding the data stream and modulating a carrier using the encoded data stream. The information carried by the signal may, for example, be analog or digital information. As is known, the signal may be transmitted over various wired or wireless communication lines.The signal can be stored on a processor-readable medium.

[0322] The terms used in the description of the disclosure of the present invention are used only for the purpose of describing particular embodiments and should not be construed as limiting the disclosure of the invention. The singular used in this description also implies the plural forms, unless the context clearly indicates otherwise. It should be understood that the terms "includes / comprises" and / or "including / comprising" when used in this description may indicate the presence of the designated entities, such as features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups of such entities. Furthermore, if an element is said to "react" to another element or "connect" to another element, it may directly react to the other element or be connected to it, or intermediate elements may be present.On the other hand, if an element is stated to be "directly responsive" to another element, or "directly connected" to, or "directly linked" to, another element, there are no intervening elements.

[0323] It should be noted that the use of any of the symbols / terms " / ", "and / or" and "at least", for example, in the cases of "A / B", "A and / or B" and "at least A or B" may imply only the selection of the first listed option (A) or only the selection of the second listed option (B), or the selection of both options (A and B). According to another example, in the cases of using the terms "A, B and / or C" and "at least A, B or C", such a phrase may imply only the selection of the first listed option (A), or only the selection of the second listed option (B), or only the selection of the third listed option (C), or only the selection of the first and second listed options (A and B), or only the selection of the first and third listed options (A and C), or only the selection of the second and third listed options (B and C), or the selection of all three options (A and B, and C). This rule, as is known to those skilled in the art, can be extended to any number of the listed elements.

[0324] Various numerical values ​​may be used in this description. Specific values ​​may, for example, be used for specific purposes, and the aspects described are not limited to these specific values.

[0325] It should be noted that although the terms "first," "second," etc., can be used to describe various elements, these elements are not limited to these terms. These terms are used only to distinguish one element from another. For example, without deviating from the scope of the disclosure of the present invention, the first element may also refer to the second element, and similarly, the second element may also refer to the first element. No ordering between the first and second elements is implied.

[0326] Reference to "one embodiment" or "an embodiment example," or "one implementation," or "an implementation," and other variations of these phrases is often used to convey information that a particular function, structure, characteristic, etc. (described in connection with the implementation / implementation) is included in at least one implementation / implementation. Therefore, the appearance of the phrase "in one embodiment" or "in an embodiment," or "in one implementation," or "in an implementation," and other variations appearing in various parts of the description does not necessarily indicate the same embodiment in each instance.

[0327] Similarly, the phrase "according to an implementation / example / implementation" or "in an implementation / example / implementation," and other variations of these phrases, are often used to convey the information that a particular function, structure, or characteristic (described in connection with a typical implementation / example / implementation) can be included in at least one implementation / example / implementation. Thus, the appearance of the phrase "according to an implementation / example / implementation" or "in an implementation / example / implementation" appearing in various passages of the description does not necessarily indicate the same implementation / example / implementation in each instance or that a separate or alternative implementation / example / implementation necessarily mutually excludes another implementation / example / implementation.

[0328] The reference numbers in the claims are for illustrative purposes only and are not intended to limit the scope of the invention. Although not expressly stated, embodiments / examples and variants may, in whole or in part, form any combination or part of such combinations.

[0329] If a drawing depicts an algorithm, it should be understood that it also represents a block diagram of the corresponding equipment. Similarly, if a drawing depicts a block diagram, it should be understood that it also represents an algorithm of the corresponding method / process.

[0330] Although some diagrams show arrows to indicate the primary direction of communication, it should be understood that communication can occur in the opposite direction to these arrows.

[0331] In various embodiments, decoding is mentioned. The term "decoding" as used in this description may encompass all or part of the processes performed, for example, on a received video image (including, possibly, a received bitstream in which one or more video images are encoded) to generate final output data suitable for display or for further processing in a reconstructed region of the video image. In various embodiments, such processes include one or more processes typically performed by a decoder. In various embodiments, such processes also or alternatively include processes, for example, performed by a decoder in the various implementations described herein.

[0332] According to other examples, in one embodiment, "decoding" may refer only to the inverse of quantization, in one embodiment, "decoding" may refer to entropy decoding, in another embodiment, "decoding" may refer only to differential decoding, and in yet another embodiment, "decoding" may refer to a combination of the inverse of quantization, entropy decoding, and differential decoding. Whether the phrase "decoding process" specifically refers to a subset of operations or generally to a broader decoding process should be clear from the context of the specific description, and this should be obvious to those skilled in the art.

[0333] In various embodiments, the term "coding" is used. Similar to the term "decoding" mentioned above, the term "coding" as used in this description may encompass all or part of the processes performed, for example, on an input video image to generate an output bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder. In various embodiments, such processes also or alternatively include processes performed by an encoder in the various implementations described herein.

[0334] According to other examples, in one embodiment, "coding" may mean only quantization, in one embodiment, "coding" may mean only entropy coding, in another embodiment, "coding" may mean only differential coding, and in yet another embodiment, "coding" may mean a combination of quantization, entropy coding, and differential coding. Whether the phrase "coding process" specifically refers to a subset of operations or generally to a broader coding process should be clear from the context of the specific descriptions, and this should be obvious to those skilled in the art.

[0335] Furthermore, this description may refer to "obtaining" various information. Obtaining information may, for example, include one or more of the following operations: evaluating information, calculating information, predicting information, retrieving information from memory, processing information, moving information, copying information, deleting information, or determining information.

[0336] Furthermore, this description may refer to "receiving" various types of information. Receiving information may represent, for example, one or more of the following operations: accessing information or receiving information from a network device.

[0337] Furthermore, in this description, the word "signal" also denotes, among other things, an indication of some information to the corresponding decoder. For example, in certain embodiments, the encoder signals specific information, such as an encoding parameter or encoded video image data. Thus, according to an embodiment, the same parameter can be used on both the encoder and decoder sides. For example, the encoder can transmit (explicit signaling) a specific parameter to the decoder so that the decoder can use this specific parameter. On the other hand, if the decoder already uses this specific parameter, as well as other parameters, then signaling can be used without transmitting (implicit signaling) simply to provide the decoder with information and select the parameter. By eliminating the transmission of some actual functions, bit savings are realized in various embodiments.It should be noted that signaling can be accomplished in a variety of ways. For example, in various embodiments, one or more syntactic elements, flags, etc., are used to convey signaling information to the appropriate decoder. While the verb "signal" is used in the preceding description, the word "signal" can also be used as a noun.

[0338] A number of implementations have been described above. However, it should be understood that various changes can be made. For example, elements in various implementations can be combined, supplemented, modified, or removed to create other implementations. Furthermore, one skilled in the art understands that other structures and processes can replace those disclosed within the framework of the present invention, and the resulting implementations perform at least substantially the same function in the same way to achieve at least substantially the same result as the disclosed implementations. Accordingly, the invention contemplates these and other implementations.

Claims

1. A method for decoding a video image by means of temporal bidirectional prediction with motion compensation for an inter-frame coded block using two reference video images from two separate lists of reference video images and two affine motion fields defined by at least two control point motion vectors (CPMVs) associated with each reference image, wherein the method comprises: for each CPMV, performing a two-way matching for a block centered on the CPMV to obtain at least two refined CPMVs with integer precision, and selecting a set of CPMVs including unrefined and / or refined CPMVs with integer precision to obtain a complete predicted block with a minimum cost of two-way matching, for each successive CPMV of the selected set of CPMVs associated with the block, refine the CPMVs with sub-count accuracy to minimize the cost of bilateral matching of the block, in this case, the clarification is skipped for the CPMV if the cost of the bilateral agreement related to the CPMV for which the bilateral agreement is performed is higher than the second threshold value.

2. The method according to claim 1, in which the refinement is omitted for the CPMV if the cost of the bilateral agreement related to the CPMV for which the bilateral agreement is performed is below the first threshold value.

3. The method according to claim 1 or 2, wherein the first threshold value and / or the second threshold value are fixed or adaptable to the size of the prediction block.

4. The method according to any one of paragraphs 1-3, in which two affine motion fields are defined by a first CPMV, a second CPMV and a third CPMV, wherein the refinement is skipped for the third CPMV if the cost of the bilateral agreement associated with the first CPMV or the second CPMV satisfies the skipping condition.

5. The method of claim 4, wherein the skip condition comprises one of the following: the cost of the bilateral negotiation associated with the first CPMV or the second CPMV for which the clarification is performed exceeds the cost of the bilateral negotiation associated with the first CPMV or the second CPMV for which the bilateral negotiation is performed; or the cost of bilateral approval associated with the first CPMV or the second CPMV for which the clarification is performed is less than the cost of bilateral approval associated with the first CPMV or the second CPMV for which the bilateral approval is performed, and the reduction in the said costs of bilateral approval is less than the third threshold value.

6. The method according to any of paragraphs 1-5, in which the said bilateral agreement, selection and clarification are prohibited for specific block shapes.

7. The method according to paragraph 6, in which the said bilateral agreement, selection and clarification are not performed if: the width or height of the block to be inter-frame coded is greater than the first value when the specified width and height are equal; or the width or height of the block to be inter-frame coded is greater than the second value when the specified width and height are different.

8. The method according to claim 6 or 7, wherein the block shape is defined in accordance with the width or height of the inter-frame coded block, and at least one block shape is signaled in the bitstream that prohibits the performance of said two-way matching, selection and refinement.

9. The method of claim 8, wherein said at least one block shape that prohibits said two-way matching, selection and refinement is signaled by signaling a maximum and minimum block width and / or a maximum and minimum block height; or the at least one block shape that prohibits the execution of said two-way matching, selection and refinement is signalled by signalling the minimum ratio between the width and height of the block and / or the maximum ratio between the width and height of the block.

10. A machine-readable medium containing instructions that, when executed by a processor of a device, cause the device to perform the method according to any one of paragraphs 1-9.

11. A device for decoding a video image by means of temporal bidirectional prediction with motion compensation for an inter-frame coded block using two reference video images from two separate lists of reference video images and two affine motion fields defined by at least two control point motion vectors (CPMVs) associated with each reference image, comprising: processor and memory for storing instructions executed by the processor; wherein the processor is configured to perform the method according to any one of paragraphs 1-9.

Images