Video coding method and device for mvd candidate number selection in amvp with sbtmvp mode

Abstract

A method and apparatus for video coding using MVD candidates are disclosed. According to the method, syntax related to a plurality of MVD candidates, one or more patterns of the plurality of MVD candidates, or both is signaled or parsed. A target MVD is determined for the current block from the plurality of MVD candidates. One or more reference blocks are determined according to the target MVD and one or more MVPs (motion vector predictors). An inter prediction candidate is generated for the current block based on the one or more reference blocks. The current block is encoded or decoded using coding information including the inter prediction candidate.

Description

[0001] Cross-referencing This invention is a non-provisional application of U.S. Provisional Patent Application No. 63 / 635,671 (filed April 18, 2024), and claims priority thereto. The aforementioned U.S. Provisional Patent Application is incorporated herein by reference in its entirety. Technical Field

[0002] This invention relates to video encoding and decoding systems. More specifically, this invention relates to transmitting motion vector difference (MVD) candidates to improve the encoding and decoding performance of video encoding and decoding systems. Background Technology

[0003] Versatile video coding (VVC) is the latest international video coding standard jointly developed by the ITU-T Video Coding Experts Group (VCEG) and the Joint Video Experts Group (JVET) of the ISO / IEC Moving Picture Experts Group (MPEG). This standard has been published as an ISO standard: ISO / IEC 23090-3:2021, Informationtechnology - Coded representation of immersive media - Part 3: Versatile Video Coding, released in February 2021. VVC improves coding and decoding efficiency by adding more coding and decoding tools to its predecessor, HEVC (High Efficiency Video Coding), and can handle various types of video sources, including 3D video signals.

[0004] Figure 1AAn exemplary adaptive inter / intra video coding system incorporating loop processing is described. For intra prediction 110, prediction data is derived from previously encoded / decoded video data in the current frame. For inter prediction 112, motion estimation (ME) is performed on the encoder side, and motion compensation (MC) is performed based on the result of ME to provide prediction data derived from other frames and motion data. Switch 114 selects either intra prediction 110 or inter prediction 112, and the selected prediction data is provided to adder 116 to form a prediction error, also known as a residual. The prediction error is then processed by transform (T) 118 and subsequent quantization (Q) 120. The residuals from transform and quantization are then encoded by entropy encoder 122 to be included in the video bitstream corresponding to the compressed video data. The bitstream associated with the transform coefficients is then packaged together with side information (e.g., motion and encoding / decoding modes associated with intra-frame and inter-frame prediction) and other information (e.g., parameters associated with loop filters applied to the underlying image area). The side information associated with intra-frame prediction 110, inter-frame prediction 112, and the intra-loop filter 130 is provided to the entropy encoder 122, such as... Figure 1A As shown. When using inter-frame prediction mode, one or more reference images must also be reconstructed at the encoder end. Therefore, the transformed and quantized residuals are processed by inverse quantization (IQ) 124 and inverse transform (IT) 126 to recover the residuals. Then, at reconstruction (REC) 128, the residuals are added back to the prediction data 136 to reconstruct the video data. The reconstructed video data can be stored in the reference image buffer 134 and used to predict other frames.

[0005] like Figure 1AAs shown, the input video data undergoes a series of processing steps in the encoding system. Due to these processing steps, the reconstructed video data from REC 128 may suffer various forms of degradation. Therefore, loop filters 130 are frequently applied to the reconstructed video data before it is stored in the reference picture buffer 134 to improve video quality. For example, deblocking filters (DF), Sample Adaptive Offset (SAO), and Adaptive Loop Filters (ALF) can be used. It may be necessary to incorporate loop filter information into the bitstream so that the decoder can correctly recover the required information. Therefore, loop filter information is also provided to the entropy encoder 122 for incorporation into the bitstream. Figure 1A In the process, the loop filter 130 is applied to the reconstructed video before the reconstructed samples are stored in the reference image buffer 134. Figure 1A The system described herein is intended to illustrate an exemplary architecture of a typical video encoder. It may correspond to a High Efficiency Video Codec (HEVC) system, VP8, VP9, ​​H.264, or VVC.

[0006] like Figure 1B As shown, the decoder can use similar or identical function blocks to the encoder, except for transform 118 and quantization 120, because the decoder only needs inverse quantization 124 and inverse transform 126. Instead of entropy encoder 122, the decoder uses entropy decoder 140 to decode the video bitstream into quantized transform coefficients and the required encoding / decoding information (e.g., ILPF information, intra-frame prediction information, and inter-frame prediction information). Intra-frame prediction 150 on the decoder side does not require mode search. Instead, the decoder only needs to generate intra-frame predictions based on the intra-frame prediction information received from entropy decoder 140. Furthermore, for inter-frame prediction, the decoder only needs to perform motion compensation (MC 152) 140 based on the inter-frame prediction information received from entropy decoder without motion estimation.

[0007] VVC Inter-Frame Prediction For each inter-frame prediction CU, motion parameters consist of motion vectors, reference picture indices, reference picture list usage indices, and additional information required for generating new VVC encoding / decoding features from inter-frame prediction samples. Motion parameters can be communicated explicitly or implicitly. When encoding / decoding a CU using skip mode, the CU is associated with a PU and has no significant residual coefficient, no encoded motion vector increment, or no reference picture index. A merge mode is specified, through which motion parameters for the current CU can be obtained from neighboring CUs, including spatial and temporal candidates and additional scheduling introduced in the VVC. The merge mode can be applied to any inter-frame prediction CU, not just skip mode. An alternative to the merge mode is to explicitly transmit motion parameters, where motion vectors, corresponding reference picture indices for each reference picture list, reference picture list usage flags, and other required information are explicitly communicated per CU.

[0008] Affine Motion Compensated Prediction In HEVC, only the translation motion model is applied to motion compensation prediction (MCP). However, in the real world, there are many types of motion, such as scaling, rotation, perspective motion, and other irregular motions. In VVC, block-based affine transformation motion compensation prediction is applied. Figure 2A As shown in -B, the affine motion field of the block is composed of Figure 2A Motion information of two control points (4 parameters) or Figure 2B The motion vectors (6 parameters) of the three control points are described.

[0009] For the 4-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as: (1) For the 6-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as: (2) Where (mv 0x , mv 0y The motion vector of the top-left control point, (mv 1x , mv 1y ) is the motion vector of the upper right control point, and (mv 2x , mv 2y) is the motion vector of the lower left control point.

[0010] To simplify motion compensation prediction, a block-based affine transformation prediction is applied. To derive the motion vector for each 4×4 brightness sub-block, the motion vector of the center sample of each sub-block is calculated according to the above equation, such as... Figure 3 As shown, the values ​​are rounded to 1 / 16 fractional precision. A motion-compensated interpolation filter is then applied to generate a prediction for each sub-block with the derived motion vector. The sub-block size for the chroma components is also set to 4×4. The MV of the 4×4 chroma sub-block is calculated as the average of the MVs of the top-left and bottom-right luminance sub-blocks in the juxtaposed 8x8 luminance region.

[0011] Similar to the inter-frame prediction for translational motion, there are two other inter-frame prediction modes for affine motion: affine merging mode and Advanced Motion Vector Prediction (AMVP) mode.

[0012] Affine merge prediction The AF_MERGE (Affine Merge) mode can be applied to CUs with a width and height greater than or equal to 8. In this mode, the current CU's CPMV (Control Point MV) is generated based on the motion information of spatially neighboring CUs. There can be a maximum of five CPMV prediction candidates, and the sending index indicates which one should be used for the current CU. The affine merge candidate list is constructed using the following three CPMV candidates: Inherited affine merging candidates inferred from the CPMV of neighboring CUs Affine merging candidate CPMVP derived using the translation MV of neighboring CUs Zero MV In VVC, there are at most two inherited affine candidates, derived from the affine motion model of neighboring blocks: one from the left neighboring CU and one from the top neighboring CU. Candidate blocks are as follows: Figure 4 As shown. For the left predictor of the current block 1010, the scan order is A0->A1, and for the upper predictor, the scan order is B0->B1->B2. Only the first inherited candidate of each side is selected. No pruning check is performed between two inherited candidates. When a neighboring affine CU is identified, its control point motion vector is used to derive the CPMVP candidate in the affine merge list of the current CU. Figure 5 As shown, if the lower left corner block A of the current block 510 is encoded and decoded in affine mode, then the motion vectors of the upper left, upper right, and lower left corners of CU 520 containing block A are obtained. , and When block A uses a 4-parameter affine model for encoding and decoding, according to , Calculate the two CPMVs of the current CU (i.e., and When block A uses a 6-parameter affine model encoding and decoding, according to , and Calculate the three CPMVs of the current CU.

[0013] The constructed affine candidate refers to the candidate constructed by combining the neighbor translational motion information of each control point. For example... Figure 6 As shown, the motion information of the control points is derived from the specified spatial and temporal adjacencies of the current block 610. CPMV k (k=1, 2, 3, 4) represents the k-th control point. For CPMV1, the blocks B2->B3->A2 are checked in order, and the MV of the first available block is used. For CPMV2, the blocks B1->B0 are checked in order, and for CPMV3, the blocks A1->A0 are checked in order. If a TMVP is available, it is used as CPMV4.

[0014] After obtaining the motion values ​​(MVs) of the four control points, affine merging candidates are constructed based on motion information. The following combinations of control point MVs are used for sequential construction: {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4},{CPMV2, CPMV3, CPMV4}, { CPMV1, CPMV2}, { CPMV1, CPMV3}. Combining three CPMVs constructs a 6-parameter affine merging candidate, while combining two CPMVs constructs a 4-parameter affine merging candidate. To avoid motion scaling, combinations of control point MVs are discarded if the reference indices of the control points are different.

[0015] After checking the affine merge candidates of inheritance and construction, if the list is still not full, zero MV is inserted at the end of the list.

[0016] Affine AMVP Prediction Affine AMVP mode can be applied to CUs with a width and height greater than or equal to 16. An affine flag at the CU level is sent in the bitstream to indicate whether affine AMVP mode is used, followed by another flag to indicate whether it is a 4-parameter or 6-parameter affine. In this mode, the difference between the current CU's CPVM and its predicted sub-CPVM is sent in the bitstream. The affine AMVP candidate list is of size 2 and is generated sequentially from the following four CPVM candidates: Inherited affine AMVP candidates inferred from the CPMV of neighboring CUs Affine AMVP candidate CPMVP derived using the translation MV of neighboring CUs Translation MV from neighboring CU Zero MVs The checking order for inherited affine AMVP candidates is the same as that for inherited affine merge candidates. The only difference is that, for AVMP candidates, only affine CUs with the same reference picture as those in the current block are considered. No pruning process is applied when inserting inherited affine motion predictors into the candidate list.

[0017] The constructed AMVP candidates are from Figure 6 The specified spatial proximity is shown in the diagram. The same checking order as in the affine merge candidate construction is used. Additionally, the reference picture index of neighboring blocks is checked. The first block in the checking order, which is inter-frame encoded and has the same reference picture as in the current CU, is used. This applies when the current CU is encoded and decoded in 4-parameter affine mode, and... and If all three CPMVs are available, they are added as candidates to the affine AMVP list. Otherwise, the constructed AMVP candidates are set to unavailable.

[0018] If, after inserting valid inherited and constructed affine AMVP candidates, the number of candidates in the affine AMVP list is still less than 2, then add them sequentially. , and As a translation MV, it is used to predict the MV of all control points of the current CU (if available). Finally, if the affine AMVP list is still not full, it is filled with zero MV.

[0019] Affine motion information storage In VVC, the CPMV of an affine CU is stored in a separate buffer. The stored CPMV is used only in affine merge mode and affine AMVP mode to generate inherited CPMVPs for the most recently encoded / decoded CU. Sub-block MVs derived from the CPMV are used for motion compensation, MV export from merge / AMVP lists in translation MVs, and deblocking filtering.

[0020] To avoid using the image line buffer for additional CPMV, affine motion data inherited from the CU above the CTU is treated differently compared to affine motion data inherited from neighboring CUs. If the candidate CU used for affine motion data inheritance is in the line of the upper CTU, the bottom left and bottom right sub-block MVs (instead of CPMVs) in the line buffer are used for affine motion vector predictor (MVP) derivation. Thus, CPMVs are only stored in the local buffer. If the candidate CU is a 6-parameter affine codec, the affine model is degraded to a 4-parameter model. Figure 7 As shown, along the top CTU boundary, the motion vectors of the lower left and lower right sub-blocks of the CU are used for affine inheritance of the CU in the bottom CTU. Figure 7 In the diagram, lines 710 and 712 represent the x and y coordinates of the image with the top-left corner at the origin (0, 0). Legend 720 illustrates the meaning of various motion vectors, where arrow 722 represents the CPMV used for affine inheritance in the local buff, arrow 724 represents the sub-block vector used for MC / merge / skip / AMVP / deblocking / TMVP in the local buff and for affine inheritance in the line buffer, and arrow 726 represents the sub-block vector used for MC / merge / skip / AMVP / deblocking / TMVP.

[0021] Affine mode optical flow prediction refinement (PROF) Compared to pixel-based motion compensation, sub-block-based affine motion compensation saves memory access bandwidth and reduces computational complexity, but at the cost of decreased prediction accuracy. To achieve finer granularity of motion compensation, prediction refinement with optical flow (PROF) is used to refine sub-block-based affine motion compensation predictions without increasing the memory access bandwidth used for motion compensation. In VVC, after performing sub-block-based affine motion compensation, the brightness prediction samples are refined by adding the difference derived from the optical flow equation. PROF is described in the following four steps: Step 1) Perform sub-block-based affine motion compensation to generate sub-block prediction I(i,j).

[0022] Step 2) Calculate the spatial gradient of the sub-block prediction using a 3-tap filter [-1, 0, 1] at each sample location. and The gradient calculation is exactly the same as the gradient calculation in BDOF.

[0023] , , In the equation above, shift1 controls the precision of the gradient. The sub-block (i.e., 4x4) prediction expands by one sample on each side for gradient computation. To avoid additional memory bandwidth and extra interpolation computation, those expanded samples on the expanded boundaries are copied from the nearest integer pixel position in the reference image.

[0024] Step 3) Brightness prediction refinement is calculated using the following optical flow equation.

[0025] (3) Where Δv(i,j) is the difference between the sample MV (denoted as v(i,j)) calculated for sample position (i,j) and the sub-block MV of the sub-block to which sample (i,j) belongs, such as... Figure 8 As shown. Δv(i,j) is quantized with a precision of 1 / 32 of the brightness sample. Figure 8 In the middle, sub-block 822 of CU 810 corresponds to the motion vector V SB (812) points to the reference sub-block of sub-block 820. Reference sub-block 822 represents the reference sub-block generated by the translational motion of block 820. Reference sub-block 824 corresponds to the reference sub-block with PROF. The motion vector of each pixel is thinned by Δv(i,j). For example, the thinned motion vector v(i,j) 814 of the top-left pixel of sub-block 820 is based on the sub-block MV V modified by Δv(i,j) 816.SB (812) Exported.

[0026] Since the affine model parameters and the position of the sample relative to the sub-block center remain unchanged between sub-blocks, Δv(i,j) can be calculated for the first sub-block and reused for other sub-blocks in the same CU. Let dx(i,j) and dy(i,j) represent the distance from the sample position (i,j) to the sub-block center. The horizontal and vertical offsets can be derived from the following equation: Δv(x,y) , .

[0027] To maintain accuracy, sub-blocks The center is calculated as ( ( W SB -1) / 2, (H) SB -1) / 2), where W SB and H SB These are the width and height of the sub-block, respectively.

[0028] For a 4-parameter affine model, .

[0029] For a 6-parameter affine model, .

[0030] in , , is the motion vector of the upper left, upper right, and lower left control points, and w and h are the width and height of the CU.

[0031] Step 4) Finally, the brightness prediction refinement ΔI(i,j) is added to the sub-block prediction I(i,j). The final prediction I' is generated as follows: .

[0032] PROF does not apply affine codecs to CUs in two cases: 1) All control points MV are the same, indicating that the CU only has translational motion; 2) The affine motion parameters are greater than the specified limits, because the sub-block-based affine MC is downgraded to the CU-based MC to avoid large memory access bandwidth requirements.

[0033] Fast encoding methods are applied to reduce the coding complexity of affine motion estimation with PROF. PROF is not applied in the affine motion estimation stage in the following two cases: a) If the CU is not the root block and its parent block has not chosen an affine mode as its optimal mode, then PROF is not applied because the probability of the current CU choosing an affine mode as its optimal mode is low; b) If the magnitudes of the four affine parameters (C, D, E, F) are all less than a predefined threshold and the current image is not a low-latency image, then PROF is not applied because the improvement introduced by PROF is small in this case. In this way, affine motion estimation with PROF can be accelerated.

[0034] Merge mode with MVD (MMVD) In addition to the merge mode, where implicitly derived motion information is directly used to generate prediction samples for the current codec unit, VVC introduces a merge mode with motion vector differences (MMVD). After sending the regular merge flag, the MMVD flag is signaled to specify whether the codec unit uses the MMVD mode.

[0035] In MMVD, after selecting a merge candidate, it is further refined through the transmitted MVD information. This further information includes a merge candidate flag, an index specifying the motion amplitude, and an index indicating the motion direction. In MMVD mode, one of the top two candidates in the merge list is selected as the MV basis. The MMVD candidate flag is transmitted to specify which one to use between the first and second merge candidates.

[0036] The distance index specifies motion amplitude information and indicates, for example... Figure 9 The predefined offsets of the starting points (912 and 922) of the L0 reference block 910 and L1 reference block 920 shown. Figure 9 In this context, offsets are added to the horizontal or vertical component of the starting MV, with different styles of small circles corresponding to different offsets from the center. The relationship between the distance index and the predefined offsets is specified in Table 1.

[0037] Table 1 – Relationship between Distance Index and Predefined Offset The direction index indicates the direction of MVD relative to the starting point. The direction index can represent the four directions shown in Table 2. It is important to note that the meaning of the MVD symbol may vary depending on the information of the starting MV. When the starting MV is a uni-prediction MV or a bi-directional prediction MV and both lists point to the same side of the current image (i.e., both reference POCs are greater than or less than the current image's POC), the symbols in Table 2 specify the sign of the MV offset added to the starting MV. When the starting MV is a bi-directional prediction MV and the two MVs point to different sides of the current image (i.e., one reference POC is greater than the current image's POC, and the other reference POC is less than the current image's POC), and the POC difference in list 0 is greater than the difference in list 1, the symbols in Table 2 specify the sign of the MV offset added to the list 0 MV component of the starting MV, and the signs of the list 1 MVs have the opposite values. Otherwise, if the POC difference in List 1 is greater than that in List 0, the sign in Table 2 specifies the sign of the MV offset of the List 1 MV component added to the starting MV, and the sign of the List 0 MV has the opposite value.

[0038] MVD is scaled based on the POC difference in each direction. If the POC differences in the two lists are the same, no scaling is needed. Otherwise, if the POC difference between the reference image in list 0 (L0) and the current image is greater than the POC difference between the image in list 1 (L1) and the current image, then the MVD of list 1 is scaled according to the ratio of td and tb, where td corresponds to the POC difference between the L0 reference image and the current image, and tb corresponds to the POC difference between the L1 reference image and the current image. If the POC difference between the L1 reference image and the current image is greater than the POC difference between the L0 reference image and the current image, then the MVD of list 0 is scaled in a similar manner. If the initial MV is a unidirectional prediction, the MVD is added to the available MVs.

[0039] Table 2. MV Offset Symbols Specifyed by Direction Index TM-based MMVD and Affine MMVD Reordering The MMVD offset is extended to both MMVD and affine MMVD modes. Additional refinement positions are added along a k×π / 8 diagonal angle (circle 1010), such as... Figure 10As shown, this increases the number of directions from 4 to 16. Next, based on the SAD cost between the template (one row above and one column to the left of the current block) and its reference, all possible MMVD refinement positions (16×6) for each base candidate at each refinement position are reordered. Finally, the top 1 / 8 refinement positions with the minimum template SAD cost are retained as available positions for MMVD index encoding and decoding. The MMVD index is binary-coded using Rice code with a parameter equal to 2. The affine MMVD reordering is extended, with additional refinement positions added along a k×π / 4 diagonal angle. After reordering, the top 1 / 2 refinement positions with the minimum template SAD cost are retained.

[0040] Before reordering, the top N motion candidates in the candidate list are used as the base candidates for both MMVD and affine MMVD. For MMVD, N equals 3, and for affine MMVD, N depends on the affine flag of the neighboring blocks [1, 3]. Two ways of adding MMVD offsets are allowed, including "bilateral" and "single-sided," depending on whether the offset of the other reference picture list is mirrored or set directly to zero. Which method is applied to a block depends on the TM cost.

[0041] JVET-W0097 EE2 related: Combinations of EE2-3.3, EE2-3.4, and EE2-3.5 At the JVET-V conference, several methods were proposed to improve the encoding and decoding efficiency of GPM (Geometric Partition Mode) in VVC. These methods were included in the exploratory experiment (EE2) for further evaluation, and their main technical aspects are described below: EE2-3.3 Regarding GPM with MMVD (GPM-MMVD): 1) Add an additional MVD to the existing GPM merge candidate; 2) The MVD is signaled in the same way as the MMVD in VVC (i.e., a distance index plus a direction index); 3) Signal two flags to control whether the MMVD is applied to each GPM segment.

[0042] EE2-3.4-3.5 Using template matching in GPM (GPM-TM): 1) Template matching is extended to GPM mode by refining the GPM motion vector based on the left and upper neighboring samples of the current encoder / decoder unit; 2) Template samples are selected according to the GPM segmentation direction; 3) A single flag is sent to jointly control whether template matching is applied to the motion vectors of two GPM segments.

[0043] This contribution proposes combining EE2-3.3, EE2-3.4, and EE2-3.5 to further improve the encoding and decoding efficiency of GPM mode. Specifically, in the proposed combination, the existing designs in EE2-3.3, EE2-3.4, and EE2-3.5 remain unchanged, while the following modifications are further applied to reconcile the two encoding and decoding tools: 1) GPM-MMVD and GPM-TM are exclusively enabled on a single GPM CU. This is done by first signaling the GPM-MMVD syntax. When both GPM-MMVD control flags are false (i.e., GPM-MMVD is disabled for both GPM partitions), the GPM-TM flag is signaled to indicate whether template matching should be applied to both GPM partitions. Otherwise (if at least one GPM-MMVD flag is true), the GPM-TM flag is inferred to be false.

[0044] 2) The GPM merging candidate list generation methods in EE2-3.3 and EE2-3.4-3.5 are directly combined. The method is to apply the MV pruning scheme in EE2-3.4-3.5 (where the MV pruning threshold is adjusted according to the current CU size) to replace the default MV pruning scheme applied in EE2-3.3. In addition, as in EE2-3.4-3.5, multiple zero MV are added until the GPM candidate list is completely filled.

[0045] JVET-AG0098 EE2-2.7: AMVP with SbTMVP mode In the proposed AMVP with SbTMVP (Sub-Block-Based Temporal Motion Vector Prediction) mode, the CU is divided into 4x4 sub-blocks, each sub-block deriving its own motion from the corresponding sub-block in the co-location image. One co-location image is used for non-low-latency images, while two co-location images are used for low-latency images. The corresponding sub-blocks are obtained using an MVP with Message MVD. The MVP is derived in the same way as the AMVP mode. The number of MVDs is determined based on the area percentage of blocks encoded and decoded at the same temporal layer in the previously encoded and decoded images, as follows: If the current image is the first encoded / decoded image in the time layer, then the number of MVDs is set to 8, such as... Figure 11A As shown, the hollow circle corresponds to the MVP with MVD equal to 0, and the solid circle corresponds to the position of the MVP with various MVD values.

[0046] Otherwise, if the area percentage of the proposed pattern is less than threshold1, the number of MVDs is set to 4, such as... Figure 11B As shown.

[0047] Otherwise, if the area percentage of the proposed pattern is less than threshold2, the number of MVDs is set to 8, such as... Figure 11A As shown.

[0048] Otherwise, the number of MVDs is set to 12, such as... Figure 11C As shown.

[0049] In the first test, threshold1 and threshold2 were set to 3% and 6%, respectively. In the second test, threshold1 and threshold2 were set to 4% and 7%, respectively.

[0050] When deriving motion for a sub-block, the reference image is fixed to the image with reference image index equal to 0. Then, temporal motion vector scaling is applied to the motion vector of the corresponding sub-block to obtain the sub-block's motion vector. When the corresponding sub-block in the co-location image is not inter-frame encoded, temporal motion information is considered unavailable, and the motion of the CU's central sub-block is used. When the motion of the CU's central sub-block is also unavailable, an MVP with communication MVD is used.

[0051] When the proposed mode is combined with other inter-frame prediction modes, LIC and MHP are not applied. OBMC mode is always enabled for the proposed mode. AMVR is enabled when the image resolution is greater than or equal to 3840*2160 luminance samples. When AMVR is enabled for AMVP with SbTMVP codec blocks, the MVD magnitude increases from {4, 8, 12}-pel to a larger value. In the first test, the MVD magnitude was {16, 32, 48}-pel when AMVR was enabled, while in the second test, the MVD magnitude was {16, 24, 32}-pel.

[0052] To reduce encoder complexity, the proposed mode is disabled for the current image when the POC distance between the co-located image and the current image is greater than threshold3. threshold3 is set to 2 and 4 in the first and second tests, respectively. Furthermore, in the second test, if the current optimal encoding / decoding mode is SbTMVP or the proposed AMVP and SbTMVP modes on the encoder, the AMVP affine ME process is skipped.

[0053] This invention discloses a method and apparatus for communicating the number and / or pattern of MVD candidates in AMVP mode. Summary of the Invention

[0054] A video encoding / decoding method and apparatus using MVD (Motion Vector Difference) candidates are disclosed. According to the method, input data associated with a current block is received, wherein the input data includes pixel data of the current block to be encoded at the encoder end or encoded data associated with the current block to be decoded at the decoder end, and the current block is encoded / decoded in an inter-frame prediction mode. A syntax associated with a plurality of MVD (Motion Vector Difference) candidates, one or more patterns of the plurality of MVD candidates, or both is communicated or parsed. A target MVD is determined for the current block from the plurality of MVD candidates. One or more reference blocks are determined based on the target MVD and one or more MVPs (Motion Vector Predictors). Inter-frame prediction candidates are generated for the current block based on the one or more reference blocks. The current block is encoded or decoded using encoding / decoding information including the inter-frame prediction candidates.

[0055] In one embodiment, the inter-frame prediction mode corresponds to the SbTMVP (Sub-block Temporal Motion Vector Prediction) mode. In one embodiment, SbTMVP candidates for the current block are generated using sub-block motion information from one or more co-location images derived for the sub-blocks of the current block, based on the target MVD and the one or more MVPs.

[0056] In one embodiment, one or more syntax elements are communicated or parsed in a picture header, slice header, PPS (picture parameter set), SPS (sequence parameter set), or a combination thereof.

[0057] In one embodiment, one or more syntax elements are communicated or parsed to indicate one or more patterns of the plurality of MVD candidates in the inter-frame prediction mode, and the inter-frame prediction mode includes one or more candidate modes using the plurality of MVD candidates. In one embodiment, the inter-frame prediction mode includes regular MMVD (Merge-and-Merge-with-Motion-Vector Difference) in merging modes, GPM (Geometry-Segmentation-with-MMVD), affine MMVD, AMVP (Advanced Motion Vector Prediction) mode with SbTMVP (Sub-Block-Based Temporal Motion Vector Prediction), or combinations thereof.

[0058] In one embodiment, the pattern of the plurality of MVD candidates includes the number of the plurality of MVD candidates or the position of the plurality of MVD candidates.

[0059] In one embodiment, the pattern of one or more of the plurality of MVD candidates is adaptively changed according to each codec block, each CTU (codec tree unit), or each slice.

[0060] In one embodiment, the one or more patterns of the plurality of MVD candidates correspond to one of the non-adjacent patterns used in other prediction patterns. Attached Figure Description

[0061] Figure 1A This demonstrates an example of an adaptive inter-frame / intra-frame video coding system that includes loop processing.

[0062] Figure 1B Show Figure 1A The corresponding decoder of the encoder.

[0063] Figure 2A An example of a block affine motion field described by motion information from two control points (4 parameters) is shown.

[0064] Figure 2B An example of a block affine motion field is shown, described by motion information from three control point motion vectors (6 parameters).

[0065] Figure 3 An example of block-based affine transformation prediction is shown, where the motion vector of each 4×4 luma subblock is derived from the control point motion vector.

[0066] Figure 4 The neighboring blocks used to derive VVC space merge candidates are shown.

[0067] Figure 5 An example of an inherited affine candidate derived from the control point motion vectors of neighboring blocks is shown.

[0068] Figure 6 This paper presents an example of constructing affine merging pattern candidates by combining translational motion information from each control point in spatial proximity and time.

[0069] Figure 7 This illustrates an example of using motion vectors in a combination method.

[0070] Figure 8 An example of affine motion compensation based on sub-blocks is shown, where the motion vectors of each pixel in a sub-block are derived by refining the motion vectors.

[0071] Figure 9 This example illustrates a merge mode (MMVD) with motion vector differences, where the distance index specifies motion amplitude information and indicates a predefined offset of the starting points of the L0 and L1 reference blocks.

[0072] Figure 10 Examples of additional directions along a diagonal angle of k×π / 8 are shown (positions are shown in dark).

[0073] Figure 11 shows possible motion vector paradigms for the proposed pattern, where Figure 11A It contains 8 motion vector difference values. Figure 11B It contains 4 motion vector difference values. Figure 11C It contains 12 motion vector difference values.

[0074] Figure 12 This shows an example of a candidate for merging non-adjacent spaces.

[0075] Figure 13A -B shows two search patterns for inheriting non-adjacent spatial proximity models.

[0076] Figure 14 A flowchart of an exemplary video encoding / decoding system is shown, which, according to an embodiment of the invention, communicates a syntax relating to the number of multiple motion vector difference candidates, the type of one or more motion vector difference candidates, or both. Detailed Implementation

[0077] It will be readily understood that the components of the present invention, as generally described and illustrated in the figures, can be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of embodiments of the systems and methods of the present invention, as illustrated in the figures, is not intended to limit the scope of the claimed invention, but is merely representative of selected embodiments of the invention. References throughout this specification to “one embodiment,” “an embodiment,” or similar language mean that a particular feature, structure, or characteristic described in relation to that embodiment may be included in at least one embodiment of the invention. Therefore, the phrases “in one embodiment” or “in a kind of embodiment” appearing in various places in this specification do not necessarily refer to the same embodiment.

[0078] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. However, those skilled in the art will recognize that the invention can be practiced without one or more specific details, or other methods, components, etc., can be used. In other instances, well-known structures or operations have not been shown or described in detail to avoid obscuring aspects of the invention. Embodiments of the invention will be best understood by referring to the accompanying drawings, in which similar parts are labeled with similar numerals throughout the figures. The following description is merely illustrative, illustrating only embodiments of certain selected devices and methods consistent with the invention claimed herein.

[0079] Several novel methods for selecting the number of MVD candidates in an AMVP with an SbTMVP pattern are revealed. In existing ECMs, the number of MVD candidates for the current image is determined by calculation of area percentages from previous images or by certain default settings. Existing MVD candidate counts can introduce latency or resolution issues because the number of MVD candidates for the current image can only be determined after statistical calculations are completed for the proposed AMVP with an SbTMVP pattern. In the proposed method, the number of MVD candidates can be determined by explicit communication in the image header, slice header, PPS (image parameter set), or SPS (sequence parameter set). Alternatively, in the proposed method, multiple different MVD candidate patterns can be selected by communication, each pattern containing the number and location (or coordinates) of MVD candidates. The pattern selection can be performed by block, by CTU, or by slice. Furthermore, a reordering mechanism for the AMVP with an SbTMVP pattern is proposed. Additionally, a unified MVD candidate pattern is proposed within the prediction pattern. Furthermore, for bidirectional prediction blocks, the pattern of MVD candidates can be selected from a single reference image or from multiple reference images. Bidirectional prediction blocks can select the pattern of their MVD candidates from either true bidirectional prediction or forward / backward prediction (low latency).

[0080] Example 1: Communication of the number and types of MVD candidates In one embodiment, the number of MVD candidates in an AMVP with the SbTMVP mode is determined directly based on explicit signals from the image header, slice header, PPS, or SPS.

[0081] In another embodiment, the MVD candidate pattern can be one of the non-adjacent patterns used in other prediction patterns. For example, such as Figure 12 and Figure 13A As shown in -B, non-adjacent patterns are used as MVD candidate patterns. MVD candidate patterns can include either the number of MVD candidates or the position of the MVD candidates.

[0082] In another embodiment, the MVD candidate pattern in an AMVP with SbTMVP mode is determined by signals from the image header, slice header, PPS, or SPS.

[0083] In another embodiment, the MVD candidate pattern in an AMVP with SbTMVP is adaptively varied based on each block, each CTU, or each slice (rather than each image). The MVD candidate pattern may include the number of MVD candidates or the location of the MVD candidates.

[0084] In another embodiment, one or more MVD candidate pattern selection syntax elements are signaled to indicate MVD candidate patterns in an inter-frame prediction mode, which includes one or more modes using MVD candidates (e.g., regular MMVD in merge mode, GPM with MMVD, affine MMVD, AMVP with SbTMVP mode). The pattern of MVD candidates may include the number of MVD candidates or the location of the MVD candidates.

[0085] Example 2: AMVP with SbTMVP and MVD candidate reordering In one embodiment, template-match-based reordering is performed in an AMVP with the SbTMVP pattern.

[0086] In another embodiment, boundary-match-based reordering is performed in an AMVP with the SbTMVP pattern.

[0087] In another embodiment, after reordering in an AMVP with SbTMVP mode, the design is shorter for the first N candidate codewords and longer for the last M candidate codewords, where N and M are integers greater than or equal to 0.

[0088] Example 3: Unification of MVD candidate patterns in inter-frame prediction mode In one embodiment, one or more MVD candidate patterns in the inter-frame prediction mode are uniform or identical (e.g., regular MMVD in merge mode, GPM with MMVD, affine MMVD, AMVP with SbTMVP mode).

[0089] In another embodiment, the number of one or more MVD candidates in the inter-frame prediction mode is uniform or the same (e.g., regular MMVD in merge mode, GPM with MMVD, affine MMVD, AMVP with SbTMVP mode).

[0090] In another embodiment, one or more MVD candidate locations in the inter-frame prediction mode are uniform or identical (regular MMVD in merge mode, GPM with MMVD, affine MMVD, AMVP with SbTMVP mode). When the MMVD candidate locations are identical, the pixel distances can be different or the same.

[0091] Example 4: MVD candidate selection pattern in bidirectional forecasting In one embodiment, for bidirectional prediction blocks, the MVD candidate pattern can be adaptively selected. For example, based on the POC distance, the reference image in L0 or L1 that is closer to the current image (i.e., with a smaller POC distance) can have its MVD candidate pattern determined by communication. Other reference images use predefined or default MVD candidate patterns.

[0092] In another embodiment, the MVD candidate pattern can be adaptively selected for bidirectional prediction blocks. For example, reference images located at lower time layers can have their MVD candidate patterns determined via communication, based on the time layer. Other reference images use predefined or default MVD candidate patterns.

[0093] In another embodiment, for bidirectional prediction blocks, the pattern of MVD candidates can be adaptively selected from multiple reference images.

[0094] In another embodiment, for bidirectional prediction blocks, when one reference image is forward prediction and the other is backward prediction (i.e., true bidirectional prediction), one or more reference images are allowed to communicate MVD candidate patterns. For reference images that do not use MVD candidate communication, predefined or default MVD candidate patterns are used.

[0095] In another embodiment, for bidirectional prediction blocks, when both reference images are forward or backward predictions, one or more reference images are allowed to communicate MVD candidate patterns. For reference images that do not use communicated MVD candidates, predefined or default MVD candidate patterns are used.

[0096] In another embodiment, for a bidirectional prediction block in true bidirectional prediction, if there is communication, an iterative MVD candidate pattern search is performed. For example, an MVD candidate pattern is searched in L0 while predefined MVD candidate patterns in L1 are maintained. Then, when searching for MVD candidate patterns in L1, the best MVD candidate pattern in L0 is used. Subsequently, the MVD candidate patterns in L0 can be updated while using the best searched MVD candidate pattern in L1. This process can continue until certain conditions are met to terminate the search, such as a threshold or a number of iterations.

[0097] Any of the methods proposed above can be implemented in the encoder and / or decoder. For example, any of the proposed methods can be implemented in the predictor-derived module of the encoder and / or the predictor-derived module of the decoder. Alternatively, any of the proposed methods can be implemented as a circuit connected to the predictor-derived module of the encoder and / or the predictor-derived module of the decoder to provide the information required by the predictor-derived module. References Figure 1Aand Figure 1B The exemplary encoder and decoder in the AMVP mode, any method of proposing the number and / or pattern of MVD candidates can be implemented in the inter-frame encoding / decoding module in the decoder (e.g., Figure 1B MC 152 in the encoder or the inter-frame codec module in the encoder (e.g., Figure 1A The proposed method is implemented in the inter-frame prediction (112). Furthermore, the entropy encoder 122 of the encoder and the entropy decoder 140 of the decoder are also used. The proposed method can also be implemented using executable software or firmware code stored on a medium, such as a hard disk or flash memory, for a central processing unit (CPU) or a programmable device (e.g., a digital signal processor (DSP) or a field-programmable gate array (FPGA)).

[0098] Figure 14 A flowchart of an exemplary video encoding / decoding system is shown, which, according to an embodiment of the invention, communicates a syntax associated with the number of multiple MVD (Motion Vector Difference) candidates, the patterns of one or more MVD candidates, or both. The steps shown in the flowchart can be implemented as program code executable on one or more processors (e.g., one or more CPUs) at the encoder end. The steps shown in the flowchart can also be implemented based on hardware, such as one or more electronic devices or processors arranged to perform the steps in the flowchart. According to the method, in step 1410, input data associated with the current block is received, wherein the input data includes pixel data of the current block to be encoded at the encoder end or encoded data associated with the current block to be decoded at the decoder end, and the current block is encoded / decoded in an inter-frame prediction mode. In step 1420, a syntax associated with the multiple MVD (Motion Vector Difference) candidates, the patterns of one or more MVD candidates, or both is communicated or parsed. In step 1430, a target MVD is determined for the current block from the multiple MVD candidates. In step 1440, one or more reference blocks are determined based on the target MVD and one or more MVPs (Motion Vector Predictors). In step 1450, inter-frame prediction candidates are generated for the current block based on the one or more reference blocks. In step 1460, the current block is encoded or decoded using encoding / decoding information including the inter-frame prediction candidate.

[0099] The flowchart shown is intended to illustrate examples of video encoding and decoding according to the present invention. Those skilled in the art can modify, rearrange, split, or combine each step to implement the invention without departing from its spirit. Specific syntax and semantics have been used in this disclosure to illustrate examples to implement embodiments of the invention. Those skilled in the art can implement the invention by substituting equivalent syntax and semantics without departing from its spirit.

[0100] The above description is provided to enable those skilled in the art to practice the invention in the context of a particular application and its requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments. Therefore, the invention is not intended to be limited to the specific embodiments shown and described, but is accorded the widest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details have been illustrated to provide a thorough understanding of the invention. However, those skilled in the art will understand that the invention can be practiced.

[0101] The embodiments of the invention described above can be implemented in various hardware, software code, or combinations thereof. For example, one embodiment of the invention may be one or more circuits integrated into a video compression chip or program code integrated into video compression software to perform the processes described herein. Embodiments of the invention may also be program code to be executed on a digital signal processor (DSP) to perform the processes described herein. The invention may also relate to numerous functions performed by a computer processor, digital signal processor, microprocessor, or field-programmable gate array (FPGA). These processors may be configured to perform specific tasks according to the invention by executing machine-readable software code or firmware code that defines specific methods embodied in the invention. The software code or firmware code may be developed in different programming languages ​​and different formats or styles. The software code may also be compiled for different target platforms. However, different code formats, styles, and languages ​​of the software code, as well as other ways of configuring the code to perform the tasks according to the invention, do not depart from the spirit and scope of the invention.

[0102] This invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples should be considered illustrative rather than restrictive in all respects. Therefore, the scope of the invention is indicated by the appended claims rather than by the foregoing description. All variations falling within the meaning and scope of the equivalents of the claims should be included within their scope.

Claims

1. A video encoding / decoding method, comprising: Receive input data related to the current block, wherein the input data includes pixel data of the current block to be encoded at the encoder end or encoded data related to the current block to be decoded at the decoder end, and the current block is encoded and decoded in an inter-frame prediction mode; The syntax associated with the number of multiple MVD (Motion Vector Difference) candidates, one or more patterns of the multiple MVD candidates, or both. Determine the target MVD for the current block from among the multiple MVD candidates; One or more reference blocks are determined based on the target MVD and one or more MVPs (Motion Vector Predictors); Based on the one or more reference blocks, generate inter-frame prediction candidates for the current block; Encode or decode the current block using encoding / decoding information that includes the inter-frame prediction candidate.

2. The method according to claim 1, wherein the inter-frame prediction mode corresponds to the SbTMVP (sub-block-based temporal motion vector prediction) mode.

3. The method of claim 2, wherein, based on the target MVD and the one or more MVPs, sub-block motion information from corresponding sub-blocks in one or more co-location images derived for the sub-blocks of the current block is used to generate SbTMVP candidates for the current block.

4. The method of claim 1, wherein one or more syntax elements are communicated or parsed in the image header, slice header, PPS (image parameter set), SPS (sequence parameter set), or a combination thereof.

5. The method of claim 1, wherein one or more syntax elements are communicated or parsed to indicate one or more patterns of the plurality of MVD candidates in the inter-frame prediction mode, and the inter-frame prediction mode includes one or more candidate modes using the plurality of MVD candidates.

6. The method of claim 5, wherein the inter-frame prediction mode includes conventional MMVD (merging mode with motion vector differences), GPM (geometric segmentation mode) with MMVD, affine MMVD, AMVP (advanced motion vector prediction) mode with SbTMVP (sub-block-based temporal motion vector prediction), or a combination thereof.

7. The method of claim 1, wherein the one or more patterns of the plurality of MVD candidates include the number of the plurality of MVD candidates or the position of the plurality of MVD candidates.

8. The method of claim 1, wherein the one or more patterns of the plurality of MVD candidates are adaptively changed according to each codec block, each CTU (codec tree unit), or each slice.

9. The method of claim 1, wherein the one or more patterns of the plurality of MVD candidates correspond to one of the non-adjacent patterns used in other prediction patterns.

10. A video encoding / decoding apparatus, the apparatus comprising one or more electronic circuits or processors arranged to: Receive input data related to the current block, wherein the input data includes pixel data of the current block to be encoded at the encoder end or encoded data related to the current block to be decoded at the decoder end, and the current block is encoded and decoded in an inter-frame prediction mode; The syntax associated with the number of multiple MVD (Motion Vector Difference) candidates, one or more patterns of the multiple MVD candidates, or both. Determine the target MVD for the current block from among the multiple MVD candidates; One or more reference blocks are determined based on the target MVD and one or more MVPs (Motion Vector Predictors); Based on the one or more reference blocks, generate inter-frame prediction candidates for the current block; Encode or decode the current block using encoding / decoding information that includes the inter-frame prediction candidate.

Images