System and method for signaling motion merging mode in video codec

Abstract

The present disclosure relates to a video encoding method, which includes: sending a regular merge identifier by signaling for a coding unit (CU); when the regular merge identifier is signaled as 1, it indicates that a regular merge mode or a merge mode with motion vector difference (MMVD) is used by the CU, and a single merge list is constructed for the CU, wherein the single merge list includes regular motion vector candidates and MMVD motion vector candidates selected by a regular merge index to indicate the candidates to be used, wherein the single merge list is constructed for both the regular merge mode and the MMVD; and when the regular merge identifier is signaled as zero, it indicates that the regular merge mode is not used by the CU, and further signaling a mode identifier to indicate that an associated merge-related mode is used when the constraint condition of the mode identifier is met.

Description

[0001] This application is a divisional application of the invention patent application with application number 201980087346.0 and invention name “System and method for signaling motion merging mode in video encoding and decoding”, which is the Chinese national phase application of the international patent application PCT / US2019 / 068977 filed on December 30, 2019. This international patent application is based on and claims priority to provisional application No. 62 / 787,230 filed on December 31, 2018. The entire contents of the application are incorporated herein by reference. Technical Field

[0002] The present application relates to video coding and compression. More specifically, the present application relates to a system and method for signaling motion merging mode in a video codec. Background Art

[0003] Various video codec techniques can be used to compress video data. Video coding is performed according to one or more video coding standards. For example, video coding standards include Versatile Video Codec (VVC), Joint Exploration Test Model (JEM), High Efficiency Video Codec (H.265 / HEVC), Advanced Video Codec (H.264 / AVC), Moving Picture Experts Group (MPEG) encoding, etc. Video coding typically utilizes a prediction method (e.g., inter-frame prediction, intra-frame prediction, etc.) that utilizes the redundancy present in a video image or sequence. An important goal of video coding technology is to compress video data into a form that uses a lower bit rate while avoiding or minimizing degradation of video quality. Summary of the invention

[0004] Examples of the present disclosure provide a method for improving the efficiency of semantic signaling of merge-related modes.

[0005] According to a first aspect of the present disclosure, a method for video encoding is provided, comprising: sending a regular merge identifier by signaling for a coding unit (CU), the coding unit being encoded as a regular merge mode and a merge-related mode; when the regular merge identifier is signaled as 1, it indicates that the regular merge mode or the merge mode with motion vector difference (MMVD) is used by the CU, and a single merge list is constructed for the CU, wherein the single merge list includes regular motion vector candidates and MMVD motion vector candidates, and the regular motion vector candidates and the MMVD motion vector candidates are selected by a regular merge index to indicate the candidates to be used, wherein the single merge list is constructed for both the regular merge mode and the MMVD; and when the regular merge identifier is signaled as zero, it indicates that the regular merge mode is not used by the CU, and further signaling a mode identifier to indicate that the associated merge-related mode is used when the constraint condition of the mode identifier is met.

[0006] According to a second aspect of the present disclosure, a computing device is provided, comprising: one or more processors; a memory coupled to the one or more processors; and a plurality of programs stored in the memory, wherein when the plurality of programs are executed by the one or more processors, the computing device performs the above-mentioned video encoding method.

[0007] According to a third aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, storing a plurality of programs, wherein the plurality of programs, when executed by one or more processing units of a computing device, enable the computing device to perform the above-mentioned video encoding method to generate a bit stream and store the bit stream in the non-transitory computer-readable storage medium.

[0008] According to a fourth aspect of the present disclosure, a computer program product is provided, comprising a plurality of program codes for execution by a computing device having one or more processing units, wherein the plurality of program codes, when executed by the one or more processing units, enable the computing device to perform the above-mentioned video encoding method.

[0009] According to a fifth aspect of the present disclosure, a method for storing a bit stream is provided, wherein the bit stream is generated according to the above-mentioned video encoding method.

[0010] It is to be understood that both the foregoing general description and the following detailed description are exemplary only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0012] Figure 1 is a block diagram of an encoder according to an example of the present disclosure.

[0013] Figure 2 is a block diagram of a decoder according to an example of the present disclosure.

[0014] Figure 3 is a flowchart illustrating a method for deriving constructed affine merge candidates according to an example of the present disclosure.

[0015] Figure 4 is a flow chart illustrating a method for determining whether an identification constraint is satisfied according to an example of the present disclosure.

[0016] Figure 5A is a diagram illustrating an MMVD search point according to an example of the present disclosure.

[0017] Figure 5B is a diagram illustrating an MMVD search point according to an example of the present disclosure.

[0018] Fig. 6A is a control point based affine motion model according to an example of the present disclosure.

[0019] Figure 6B is a control point based affine motion model according to an example of the present disclosure.

[0020] Figure 7 is a diagram illustrating an affine motion vector field (MVF) of each sub-block according to an example of the present disclosure.

[0021] Figure 8 is a diagram illustrating the position of an inherited affine motion predictor according to an example of the present disclosure.

[0022] Fig. 9 is a diagram illustrating inheritance of control point motion vectors according to an example of the present disclosure.

[0023] Fig.10 is a diagram illustrating positions of candidate orientations according to an example of the present disclosure.

[0024] Fig.11 is a diagram illustrating spatially neighboring blocks used by sub-block based temporal motion vector prediction (SbTMVP) according to an example of the present disclosure.

[0025] FIG. 12A is a diagram illustrating a sub-block based temporal motion vector prediction (SbTMVP) process according to an example of the present disclosure.

[0026] Fig. 12Bis a diagram illustrating a sub-block based temporal motion vector prediction (SbTMVP) process according to an example of the present disclosure.

[0027] Fig.13A is a diagram illustrating triangular partitioning according to an example of the present disclosure.

[0028] Fig. 13B is a diagram illustrating triangular partitioning according to an example of the present disclosure.

[0029] Fig.14 is a diagram illustrating a computing environment coupled with a user interface according to examples of the present disclosure. DETAILED DESCRIPTION

[0030] Reference will now be made in detail to example embodiments, examples of which are illustrated in the accompanying drawings. The following description refers to the accompanying drawings, in which the same numbers in different drawings represent the same or similar elements, unless otherwise indicated. The implementations set forth in the following description of the example embodiments do not represent all implementations consistent with the present disclosure. Instead, they are merely examples of apparatus and methods consistent with aspects of the present disclosure as recited in the appended claims.

[0031] The terms used in this disclosure are only used for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and the appended claims, the singular forms "a", "an", and "the" are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein is intended to represent and include any or all possible combinations of one or more associated listed items.

[0032] It should be understood that although the terms "first", "second", "third", etc. may be used herein to describe various information, the information should not be limited by these terms. These terms are only used to distinguish one category of information from another category of information. For example, without departing from the scope of the present disclosure, the first information may be considered as the second information; and similarly, the second information may also be considered as the first information. As used herein, depending on the context, the term "if" may be understood to mean "when" or "at" or "in response to a judgment".

[0033] Video codec system.

[0034] Conceptually, video codec standards are similar. For example, many use block-based processing and share similar video codec block diagrams to implement video compression.

[0035] In this embodiment of the present disclosure, several methods are proposed to improve the efficiency of semantic signaling of merge-related modes. It should be noted that the proposed methods can be applied independently or in combination.

[0036] Figure 1 A typical encoder 100 is shown. The encoder 100 has a video input 110, motion compensation 112, motion estimation 114, intra / inter mode decision 116, block predictor 140, adder 128, transform 130, quantization 132, prediction related information 142, intra prediction 118, picture buffer 120, inverse quantization 134, inverse transform 136, adder 126, memory 124, loop filter 122, entropy coding 138, and bitstream 144.

[0037] In an example embodiment of the encoder, a video frame is partitioned into blocks for processing. For each given video block, a prediction is formed based on either inter-frame prediction or intra-frame prediction. In inter-frame prediction, a predictor can be formed based on pixels from a previously reconstructed frame, through motion estimation and motion compensation. In intra-frame prediction, a predictor can be formed based on reconstructed pixels in the current frame. Through mode determination, the best predictor can be selected to predict the current block.

[0038] The prediction residual (i.e., the difference between the current block and its predictor) is sent to the transform module. The transform coefficients are then sent to the quantization module for entropy reduction. The quantized coefficients are fed to the entropy coding module to generate a compressed video bitstream. Figure 1 As shown in , prediction-related information from the inter-frame and / or intra-frame prediction modules (such as block partition information, motion vector, reference picture index and intra-frame prediction mode, etc.) also passes through the entropy coding module and is saved in the bitstream.

[0039] In the encoder, decoder related modules are also required in order to reconstruct the pixels for prediction purposes. First, the prediction residual is reconstructed by inverse quantization and inverse transformation. This reconstructed prediction residual is combined with the block predictor to generate the unfiltered reconstructed pixels of the current block.

[0040] In order to improve coding efficiency and visual quality, loop filters are usually used. For example, deblocking filters are available in AVC, HEVC, and the current VVC. In HEVC, an additional loop filter called SAO (sample adaptive offset) is defined to further improve coding efficiency. In the latest VVC, another loop filter called ALF (adaptive loop filter) is being actively studied, and this loop filter has a great chance of being included in the final standard.

[0041] Figure 2A block diagram of a typical decoder 200 is shown. The decoder 200 has a bitstream 210, entropy decoding 212, inverse quantization 214, inverse transform 216, adder 218, intra / inter mode selection 220, intra prediction 222, memory 230, loop filter 228, motion compensation 224, picture buffer 226, prediction related information 234 and video output 232.

[0042] In the decoder, the bitstream is first decoded by the entropy decoding module to derive the quantization coefficient levels and prediction related information. The quantized coefficient levels are then processed by the inverse quantization and inverse transform modules to obtain the reconstructed prediction residuals. Based on the decoded prediction information, a block predictor is formed by intra-frame prediction or motion compensation process. The unfiltered reconstructed pixels are obtained by summing the reconstructed prediction residuals and the block predictor. With the loop filter turned on, filtering operations are performed on these pixels to derive the final reconstructed video for output.

[0043] Figure 3 An example method for deriving constructed affine merge candidates according to the present disclosure is shown.

[0044] In step 310, a conventional merge flag for a coding unit (CU) is obtained from a decoder, the coding unit being encoded in a merge mode and a merge-dependent mode.

[0045] In step 312, when the normal merge flag is 1, indicating that the normal merge mode or the merge mode with motion vector difference (MMVD) is used by the CU, a motion vector merge list is constructed for the CU and a normal merge index is used to indicate the candidate to be used.

[0046] In step 314, when the normal merge flag is zero, it indicates that the normal merge mode is not used by the CU, and further receives a mode flag to indicate that the associated merge-related mode is used when the constraint condition of the mode flag is satisfied.

[0047] Figure 4 An example method for determining whether an identification constraint is satisfied according to the present disclosure is shown.

[0048] In step 410, a coding block is obtained from a decoder, wherein the coding block has a width and a height.

[0049] In step 412, the decoder determines whether the width of the coding block and the height of the coding block are both not equal to 4.

[0050] In step 414, it is determined by the decoder that the width of the coding block is not equal to 8 or the height of the coding block is not equal to 4.

[0051] In step 416 , it is determined by the decoder that the width of the coding block is not equal to 4 or the height of the coding block is not equal to 8.

[0052] In step 418, it is determined by the decoder that the normal merge flag is not set.

[0053] Versatile Video Codec (VVC).

[0054] At the 10th JVET meeting (April 10-20, 2018, San Diego, USA), JVET defined the first draft of the generic video codec (VVC) and the VVC Test Model 1 (VTM1) coding method. It was decided to include quadtrees with nested multi-type trees using binary and ternary splits coding block structure as the initial new coding features of VVC. Since then, the reference software VTM for implementing the coding method and the draft VVC decoding process has been developed during the JVET meeting.

[0055] The picture partition structure divides the input video into blocks called coding tree units (CTUs). CTUs are divided into coding units (CUs) using a quadtree with a nested multi-type tree structure, where leaf coding units (CUs) define regions that share the same prediction mode (e.g., intra or inter). In this document, the term "unit" defines an image region covering all components; the term "block" is used to define a region covering a specific component (e.g., luma), and when considering chroma sampling formats such as 4:2:0, the term "block" may differ in spatial location.

[0056] Extended merge mode in VVC.

[0057] In VTM3, a merge candidate list is constructed by including the following five types of candidates in order:

[0058] 1. Spatial MVP from spatial neighbor CU

[0059] 2. Time MVP from collocated CU

[0060] 3. History-based MVP from FIFO table

[0061] 4. Average MVP in pairs

[0062] 5. Zero MV.

[0063] The size of the merge list is signaled in the slice header, and the maximum allowed size of the merge list is 6 in VTM3. For each CU code in merge mode, the index of the best merge candidate is encoded using truncated unary binarization (TU). The first bin of the merge index is encoded using context, and bypass coding is used for other bins. In the following context of the present disclosure, the extended merge mode is also referred to as the normal merge mode, because its concept is the same as the merge mode used in HEVC.

[0064] Merged Mode with MVD (MMVD).

[0065] In addition to the merge mode in which the implicitly derived motion information is directly used for the generation of prediction samples for the current CU, a merge mode with motion vector difference (MMVD) is introduced in VVC. The MMVD flag is signaled immediately after the skip flag and the merge flag to specify whether the MMVD mode is used for the CU.

[0066] In MMVD, after the merge candidates are selected, the merge candidates are further refined by signaling MVD information. The further information includes a merge candidate identifier, an index to specify the magnitude of motion, and an index to indicate the direction of motion. In MMVD mode, one of the first two candidates in the merge list is selected as the MV basis. The merge candidate identifier is signaled to specify which one is used.

[0067] The distance index specifies the motion magnitude information and indicates a predefined offset from the starting point. As shown in Figure 5 (described below), the offset is added to the horizontal component or the vertical component of the starting MV. The relationship between the distance index and the predefined offset is specified in Table 1:

[0068] Table 1 - Relationship between distance index and predefined offset

[0069]

[0070] The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate four directions, as shown in Table 2. It should be noted that the meaning of the MVD symbol may change depending on the information of the starting MV. When the starting MV is a unidirectional prediction MV or a bidirectional prediction MV, and both lists point to the same side of the current picture (i.e., the POCs of the two reference pictures are both greater than the POC of the current picture, or both are less than the POC of the current picture), the symbol in Table 2 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectional prediction MV, and both MVs point to different sides of the current picture (i.e., the POC of one reference picture is greater than the POC of the current picture, and the POC of the other reference picture is less than the POC of the current picture), the symbol in Table 2 specifies the sign of the MV offset added to the list 0 MV component of the starting MV, and the sign of the list 1 MV has the opposite value.

[0071] Table 2 - Sign of MV offsets specified by direction index

[0072]

[0073]

[0074] Figure 5A A diagram illustrating MMVD search points for a first list (L0) of references according to the present disclosure is shown.

[0075] Figure 5B A diagram illustrating MMVD search points for a second list (L1) reference according to the present disclosure is shown.

[0076] Affine motion compensated prediction.

[0077] In HEVC, only the translational motion model is applied to motion compensation prediction (MCP). In the real world, there are many kinds of motion, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VTM3, block-based affine transformation motion compensation prediction is applied. Fig. 6A and 6B As shown in (and described below), the affine motion field of a block is described by the motion information of two control points (4 parameters) or three control point motion vectors (6 parameters).

[0078] Fig. 6A A control point based affine motion model for a 4-parameter affine model according to the present disclosure is shown.

[0079] Figure 6B A control point based affine motion model for a 6-parameter affine model according to the present disclosure is shown.

[0080] For the 4-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as:

[0081]

[0082] For the 6-parameter affine motion model, the motion vector at the sample position (x, y) in the block is derived as:

[0083]

[0084] Among them (m v0x ,m v0y ) is the motion vector of the upper left control point, (m v1x ,m v1y ) is the motion vector of the upper right control point, and (m v2x ,m v2y ) is the motion vector of the lower left control point.

[0085] To simplify motion compensated prediction, block-based affine transform prediction is applied. To derive the motion vector for each 4×4 luminance sub-block, the following formula is used to calculate Figure 7 The motion vector of the center sample of each sub-block shown in (described below) is obtained and the motion vector is rounded to 1 / 16 fractional precision. Then, a motion compensated interpolation filter is applied to generate a prediction for each sub-block using the derived motion vector. The sub-block size of the chroma component is also set to 4×4. The MV of a 4×4 chroma sub-block is calculated as the average of the MVs of the four corresponding 4×4 luminance sub-blocks.

[0086] Figure 7 The affine motion vector field (MVF) of each sub-block according to the present disclosure is shown.

[0087] As done for translational motion inter prediction, there are also two affine motion inter prediction modes: affine merge mode and affine AMVP mode.

[0088] Affine merging predictions.

[0089] AF_MERGE mode can be applied to CUs whose width and height are both greater than or equal to 8. In this mode, the CPMV of the current CU is generated based on the motion information of spatially neighboring CUs. There can be up to five CPMVP candidates, and the signaling index indicates which candidate to use for the current CU. The following three types of CPVM candidates are used to form the affine merge candidate list:

[0090] 6. Inherited affine merge candidates extrapolated from the CPMV of neighboring CUs

[0091] 7. The constructed affine merge candidate CPMVP derived using the translation MV of the neighbor CU

[0092] 8. Zero MV.

[0093] In VTM3, there are at most two inherited affine candidates derived from the affine motion models of neighboring blocks, one from the left neighboring CU and one from the top neighboring CU. Figure 8 -> A1 for the left predictor and B0->B1->B2 for the top predictor. Only the first inheritance candidate from each side is selected. No pruning check is performed between two inheritance candidates. When a neighboring affine CU is identified, its control point motion vectors are used to derive the CPMVP candidate in the affine merge list of the current CU. Fig. 9 As shown in (described below), if the lower left neighbor block A is encoded in affine mode, the motion vectors v2, v3 and v4 of the upper left corner, upper right corner and lower left corner of the CU containing block A are obtained. When block A is encoded using a 4-parameter affine model, two CPMVs of the current CU are calculated based on v2 and v3. In the case where block A is encoded using a 6-parameter affine model, three CPMVs of the current CU are calculated based on v2, v3 and v4.

[0094] Figure 8 The position of the inherited affine motion predictor according to the present disclosure is shown.

[0095] Fig. 9 Control point motion vector inheritance according to the present disclosure is shown.

[0096] Constructing affine candidates means constructing candidates by combining the translation motion information of the neighbors of each control point. The motion information of the control point is obtained from Fig.10 The specified spatial and temporal neighbors are derived from the CPMV (described below). k (k=1,2,3,4) represents the kth control point. For CPMV1, the B2->B3->A2 block is checked, and the MV of the first available block is used. For CPMV2, the B1->B0 block is checked, and for CPMV3, the A1->A0 block is checked. For TMVP, it is used as CPMV4 if it is available.

[0097] Fig.10 The locations of candidate orientations for a constructed affine merge pattern according to the present disclosure are shown.

[0098] After obtaining the MVs of the four control points, construct the affine merge candidates based on the corresponding motion information. The following combinations of the control point MVs are used to construct the following in order:

[0099] {CPMV1, CPMV2, CPMV3}, {CPMV1, CPMV2, CPMV4}, {CPMV1, CPMV3, CPMV4}, {CPMV2, CPMV3, CPMV4}, {CPMV1, CPMV2}, {CPMV1, CPMV3}.

[0100] Combinations of 3 CPMVs constitute 6-parameter affine merging candidates, and combinations of 2 CPMVs constitute 4-parameter affine merging candidates. To avoid the motion scaling process, relevant combinations of control point MVs are discarded if the reference indices of the control points are different.

[0101] After checking inherited affine merge candidates and constructed affine merge candidates, if the list is still not full, a zero MV is inserted at the end of the list.

[0102] Sub-block based temporal motion vector prediction (SbTMVP).

[0103] VTM supports the sub-block based temporal motion vector prediction (SbTMVP) method. Similar to the temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses the motion field in the collocated picture to improve the motion vector prediction and merge mode of the CU in the current picture. The same collocated picture used by TMVP is used for SbTVMP. SbTMVP differs from TMVP in the following two main aspects:

[0104] 1. TMVP predicts motion at the CU level, but SbTMVP predicts motion at the sub-CU level;

[0105] 2. Although TMVP obtains the temporal motion vector from a collocated block in a collocated picture (the collocated block is the bottom right or center block relative to the current CU), before obtaining the temporal motion information from the collocated picture, SbTMVP applies a motion shift, where the motion shift is obtained from the motion vector of one of the spatial neighboring blocks of the current CU.

[0106] SbTVMP process Fig.11 , Figure 12A and Fig. 12B As shown in Figure 1 (described below). SbTMVP predicts the motion vector of a sub-CU within the current CU in two steps. In the first step, check the sub-CUs in the order of A1, B1, B0, and A0. Fig.11Once the first spatial neighboring block with a motion vector that uses the collocated picture as its reference picture is identified, that motion vector is selected as the motion shift to be applied. If no such motion is identified from the spatial neighbors, the motion shift is set to (0,0).

[0107] Fig.11 Spatial neighboring blocks used by sub-block based temporal motion vector prediction (SbTMVP) are shown. SbTMVP is also known as alternative temporal motion vector prediction (ATMVP).

[0108] In the second step, the motion shift identified in step 1 is applied (i.e., added to the coordinates of the current block) to obtain sub-CU level motion information (motion vector and reference index) from the collocated picture, as shown in Figures 12A and 12B. The examples in Figures 12A and 12B assume that the motion shift is set to the motion of block A1. Then, for each sub-CU, the motion information of the sub-CU is derived using the motion information of its corresponding block in the collocated picture (the minimum motion grid covering the center sample). After the motion information of the collocated sub-CU is identified, it is converted into the motion vector and reference index of the current sub-CU in a manner similar to the TMVP process of HEVC, where temporal motion scaling is applied to align the reference picture of the temporal motion vector with those of the current CU.

[0109] 12A shows the SbTMVP process for collocated pictures in VVC when the sub-CU motion field is derived by applying motion shifts from spatial neighbors and scaling motion information from the corresponding collocated sub-CU.

[0110] Fig. 12B The SbTMVP process for the current picture in VVC is shown when the sub-CU motion field is derived by applying motion shifts from spatial neighbors and scaling the motion information from the corresponding collocated sub-CU.

[0111] In VTM3, a sub-block based merge list containing a combination of SbTVMP candidates and affine merge candidates is used for signaling of sub-block based merge mode. In the following context, sub-block merge mode is used. The SbTVMP mode is enabled / disabled by the sequence parameter set (SPS) flag. If the SbTMVP mode is enabled, the SbTMVP predictor is added as the first entry in the list of sub-block based merge candidates, followed by the affine merge candidates. The size of the sub-block based merge list is signaled in the SPS, and the maximum allowed size of the sub-block based merge list is 5 in VTM3.

[0112] The sub-CU size used in SbTMVP is fixed to 8×8, and as done for affine merge mode, SbTMVP mode is only applicable to CUs whose width and height are both greater than or equal to 8.

[0113] The encoding logic of the additional SbTMVP merge candidates is the same as that of other merge candidates, that is, for each CU in a P or B slice, an additional RD check is performed to decide whether to use the SbTMVP candidate.

[0114] Joint Inter-Intra Prediction (CIIP).

[0115] In VTM3, when a CU is encoded in merge mode, if the CU contains at least 64 luma samples (i.e., the CU width multiplied by the CU height is equal to or greater than 64), an additional flag is signaled to indicate whether the joint inter / intra prediction (CIIP) mode is applied to the current CU.

[0116] To form the CIIP prediction, the intra prediction mode is first derived from two additional semantic elements. Up to four possible intra prediction modes can be used: directional angle prediction (DC), plane prediction (PLANAR), horizontal prediction (HORIZONAL) or vertical prediction (VERTICAL). Then, the conventional intra-frame inter-frame decoding process is used to derive the inter-frame prediction and intra-frame prediction signals. Finally, a weighted average of the inter-frame and intra-frame prediction signals is performed to obtain the CIIP prediction.

[0117] Intra prediction mode export.

[0118] In CIIP mode, up to 4 intra prediction modes (including DC, PLANAR, HORIZONAL, and VERTICAL modes) can be used to predict the luma component. If the CU shape is very wide (i.e., the width is more than twice the height), the HORIZONAL mode is not allowed. If the CU shape is very narrow (i.e., the height is more than twice the width), the VERTICAL mode is not allowed. In these cases, only 3 intra prediction modes are allowed.

[0119] CIIP mode uses 3 most probable modes (MPMs) for intra prediction. The CIIP MPM candidate list is formed as follows:

[0120] - The left and top neighbors are set to A and B respectively

[0121] The intra prediction modes of block A and block B, denoted intraModeA and intraModeB respectively, are derived as follows:

[0122] ○Let X be A or B

[0123] ○ If 1) block X is not available; or 2) block X is not predicted using CIIP mode or intra mode; 3) block B is outside the current CTU, then set intra mode X to DC

[0124] ○ Otherwise, 1) if the intra prediction mode of block X is DC or PLANAR, set intra mode X to DC or PLANAR; or 2) if the intra prediction mode of block X is a "vertical-like" angular mode (greater than 34), set intra mode X to VERTICAL; or 3) if the intra prediction mode of block X is a "horizontal-like" angular mode (less than or equal to 34), set intra mode X to HORIZONAL

[0125] - If intra-mode A and intra-mode B are the same:

[0126] ○ If intra-mode A is PLANAR or DC, the three MPMs are set to {PLANAR, DC, VERTICAL} in the order {PLANAR, DC, VERTICAL},

[0127] ○ Otherwise, set the three MPMs to {Intra Mode A, PLANAR, DC} in the order {Intra Mode A, PLANAR, DC}

[0128] - Otherwise (Intra-mode A and Intra-mode B are different):

[0129] ○ Set the first two MPMs to {Intra Mode A, Intra Mode B} in the order {Intra Mode A, Intra Mode B}

[0130] ○ Check the uniqueness of PLANAR, DC, and VERTICAL against the first two MPM candidate patterns, in that order; once a unique pattern is found, add it as the third MPM.

[0131] If the CU shape is very wide or very narrow (as defined above), the MPM flag is inferred to be 1 and no signaling is required. Otherwise, the MPM flag is signaled to indicate whether the CIIP intra prediction mode is one of the CIIP MPM candidate modes.

[0132] If the MPM flag is 1, the MPM index is further signaled to indicate which of the MPM candidate modes is used in CIIP intra prediction. Otherwise, if the MPM flag is 0, the intra prediction mode is set to the "default" mode in the MPM candidate list. For example, if the PLANAR mode is not in the MPM candidate list, PLANAR is the default mode and the intra prediction mode is set to PLANAR. Since 4 possible intra prediction modes are allowed in CIIP and the MPM candidate list contains only 3 intra prediction modes, one of the 4 possible modes must be the default mode.

[0133] For chroma components, DM mode is always applied without additional signaling; that is, chroma uses the same prediction mode as luma.

[0134] The intra prediction mode of a CIIP-coded CU will be saved and used in the intra-mode coding of future neighboring CUs.

[0135] Joint inter-frame and intra-frame prediction signals.

[0136] The inter prediction signal P in CIIP mode is derived using the same inter prediction process applied to the normal merge mode inter ; and using the CIIP intra prediction mode after the conventional intra prediction process to derive the intra prediction signal P intra Then, the intra and inter prediction signals are combined using a weighted average, where the weight values ​​depend on the intra prediction mode and where the sample is located in the coding block, as follows:

[0137] - If the intra prediction mode is DC or PLANAR mode, or if the block width or height is less than 4, equal weights are applied to the intra prediction and inter prediction signals.

[0138] -Otherwise, the weights are determined based on the intra prediction mode (in this case, either HORIZONAL mode or VERTICAL mode) and the sample position in the block. Take the HORIZONAL prediction mode as an example (the weights for the VERTICAL mode are derived in a similar way but in an orthogonal direction). Denote W as the width of the block and H as the height of the block. First, the coding block is divided into four equal-area parts, each of size (W / 4)xH. Starting from the part closest to the intra prediction reference sample and ending at the part farthest from the intra prediction reference sample, the weight wt for each of the 4 regions is set to 6, 5, 3, and 2, respectively. The following formula is used to derive the final CIIP prediction signal:

[0139] P CIIP =((8-wt)*P inter+wt*P intra )>>3.

[0140] Triangular partitioning for inter prediction.

[0141] In VTM3, a new triangular partition mode is introduced for inter prediction. Triangular partition mode is only applied to CUs that are 8x8 or larger and encoded in skip or merge mode. For CUs that meet these conditions and the merge flag is turned on, the CU level flag is signaled to indicate whether the triangular partition mode is applied.

[0142] When using this mode, use diagonal partitioning or anti-diagonal partitioning ( Fig.13A and Fig. 13B , as described below), the CU is evenly divided into two triangular partitions. Each triangular partition in the CU is inter-predicted using its own motion; only unidirectional prediction is allowed for each partition, that is, each partition has one motion vector and one reference index. Unidirectional prediction motion constraints are applied to ensure that only two motion compensated predictions are required for each CU, the same as conventional bidirectional prediction.

[0143] If the CU level flag indicates that the current CU is encoded using the triangle partition mode, an index in the range of [0,39] is further signaled. Using the triangle partition index, the direction of the triangle partition (diagonal or anti-diagonal) and the motion of each partition can be obtained by looking up the table. After predicting each triangle partition, a hybrid process with adaptive weights is used to adjust the sample values ​​along the diagonal or anti-diagonal edges. This is a prediction signal for the entire CU, and as in other prediction modes, the transform and quantization process will be applied to the entire CU. Finally, the motion field of the CU predicted using the triangle partition mode is stored in 4x4 units.

[0144] Fig.13A Inter-frame prediction based on triangular partitioning according to the present disclosure is shown.

[0145] Fig. 13B Inter-frame prediction based on triangular partitioning according to the present disclosure is shown.

[0146] Context-Adaptive Binary Arithmetic Coding (CABAC).

[0147] Context-Adaptive Binary Arithmetic Coding (CABAC) is a form of entropy coding used in the H.264 / MPEG-4 AVC and High Efficiency Video Coding (HEVC) standards and VVC. CABAC is based on arithmetic coding with some innovations and changes to adapt it to the needs of video coding standards:

[0148] It encodes binary symbols, which keeps complexity low and allows probabilistic modeling for the more frequent bits in any symbol.

[0149] • Adaptively select the probability model based on the local context, allowing better probabilistic modeling since coding patterns are usually well correlated locally.

[0150] It uses multiplication-free range division through the use of quantized probability ranges and probability states.

[0151] CABAC has multiple probability modes for different contexts. It first converts all non-binary symbols into binary. Then, for each binary bit (or bit), the encoder chooses which probability model to use and then uses information from nearby elements to optimize the probability estimate. Finally, arithmetic coding is applied to compress the data.

[0152] Context modeling provides an estimate of the conditional probability of encoding a symbol. With a suitable context model, it is possible to exploit a given inter-symbol redundancy by switching between different probability models depending on the symbols already encoded in the neighborhood of the current symbol to be encoded.

[0153] The encoding of data symbols involves the following stages.

[0154] Binarization: CABAC uses binary arithmetic coding, which means that only binary decisions (1 or 0) are encoded. Non-binary valued symbols (e.g., transform coefficients or motion vectors) are "binarized" or converted into binary codes before arithmetic coding. The process is similar to that of converting data symbols into variable length codes, but the binary codes are further encoded (by an arithmetic encoder) before transmission.

[0155] • The stages are repeated for each binary digit (or "bit") of the binarized symbol.

[0156] Context model selection: A "context model" is a probability model of one or more bins of a binarized symbol. This model may be selected from a selection of available models depending on statistics of recently encoded data symbols. The context model stores the probability of each bin being "1" or "0".

[0157] Arithmetic coding: The arithmetic coder encodes each binary bit according to the chosen probability model. Note that for each binary bit (corresponding to "0" and "1"), there are only two sub-ranges.

[0158] • Probability update: Update the selected context model based on the actual coded value (e.g., if the bin value is "1", increase the frequency count of "1").

[0159] Fig.14 A computing environment 1410 is shown coupled to a user interface 1460. The computing environment 1410 may be part of a data processing server. The computing environment 1410 includes a processor 1420, a memory 1440, and an input / output interface 1450.

[0160] The processor 1420 generally controls the overall operation of the computing environment 1410, such as operations associated with display, data acquisition, data communication, and image processing. The processor 1420 may include one or more processors to execute instructions to implement all or some of the steps in the method described above. In addition, the processor 1420 may include one or more modules that facilitate interaction between the processor 1420 and other components. The processor may be a central processing unit (CPU), a microprocessor, a single-chip machine, a GPU, etc.

[0161] The memory 1440 is configured to store various types of data to support the operation of the computing environment 1410. Examples of such data include instructions for any application or method operating on the computing environment 1410, MRI data sets, image data, etc. The memory 1440 may be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0162] The input / output interface 1450 provides an interface between the processor 1420 and peripheral interface modules such as a keyboard, a click wheel, buttons, etc. These buttons may include, but are not limited to, a home button, a start scanning button, and a stop scanning button. The input / output interface 1450 may be coupled to an encoder and a decoder.

[0163] In one embodiment, a non-transitory computer-readable storage medium is also provided, which includes a plurality of programs, such as the programs included in the memory 1440, which can be executed by the processor 1420 in the computing environment 1410 for implementing the methods described above. For example, the non-transitory computer-readable storage medium can be a ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

[0164] The non-transitory computer-readable storage medium has stored therein a plurality of programs for execution by a computing device having one or more processors, wherein the plurality of programs, when executed by the one or more processors, causes the computing device to carry out the above-described method for motion prediction.

[0165] In one embodiment, the computing environment 1410 can be implemented using one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), graphics processing units (GPUs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0166] According to the method of the present disclosure.

[0167] As described above, in VTM-3.0, the merge mode is further classified into five categories, including conventional merge, merge mode with MVD (MMVD), sub-block merge (including affine merge and temporal motion vector prediction based on sub-blocks), joint inter-frame intra-frame prediction (CIIP) merge, and triangular partition merge. The semantics of merge mode signaling in the current VVC are illustrated in the following table.

[0168] Table 3 Semantics of the merge-related modes in current VVC

[0169]

[0170]

[0171] In summary, in the current VVC, the semantics (associated flags) signaled to indicate the corresponding merge mode are as follows.

[0172] Table 4 Signaling of the merge-related mode in the current VVC

[0173] MMVD logo Sub-block ID CIIP Logo Triangle logo MMVD 1 - - - sub-block 0 1 - - CIIP 0 0 1 - Triangle 0 0 0 1 conventional 0 0 0 0 .

[0174] It is observed that more than 50% of the merge modes are normal merge modes. However, in VTM-3.0, the codeword for normal merge mode is the longest one among the five different merge modes, which is not an efficient design in terms of semantic parsing. In current VVC, skip mode has a similar semantic design to merge mode except that there is no CIIP mode for skipping. However, the same observation is made in skip mode.

[0175] Semantics for regular merges.

[0176] As mentioned above, among several merge-related modes including (conventional merging, MMVD, sub-block merging, CIIP and triangular merging), the scheme of the conventional merge mode in the current VVC is the most frequently used. In an embodiment of the present disclosure, an explicit identifier for the conventional merge mode is signaled to indicate whether the conventional merge mode is used. As shown in the following table, a conventional identifier (or referred to as a conventional merge identifier) ​​is explicitly signaled to the bitstream, and all signaling of the related identifiers is modified accordingly. The conventional merge identifier is context-encoded using CABAC. In one embodiment, only one context is used to encode the conventional merge identifier. In another embodiment, multiple context models are used to encode the conventional merge identifier, and the selection of the context model is based on encoded information, such as the conventional merge identifier of the neighboring block, or the size of the current CU.

[0177] Table 5 Example of signaling of combined related mode in the proposed solution

[0178] General logo MMVD logo Sub-block ID CIIP Logo conventional 1 - - - MMVD 0 1 - - sub-block 0 0 1 - CIIP 0 0 0 1 Triangle 0 0 0 0 .

[0179] In current VVC, the constraints for enabling merge-related modes are different, and thus the signaling of the identity of each merge-related mode is also different, as summarized below.

[0180] Table 6 Constraints for enabling / signaling combined related modes

[0181] .

[0182] Therefore, the signaling of the regular merge flag should also take into account the different constraints applied to each flag signaling. For example, when the block size is 4x4, 8x4, or 4x8, only the regular merge mode and MMVD are valid. Under these conditions (block size is 4x4, 8x4, or 4x8), only the regular merge flag is signaled; when the regular merge flag is equal to 1, the regular merge mode is used; otherwise, when the regular merge flag is equal to 0, MMVD is used. An example based on the semantics of the current VVC working draft is illustrated below.

[0183] Table 7 Examples of semantics in the proposed scheme

[0184]

[0185]

[0186] In this example, it is noted that the regular merge flag is explicitly signaled in the bitstream. However, the regular merge flag may be signaled in any orientation, and the regular merge flag need not be the first orientation as described above. In yet another scheme, the regular merge flag is signaled, but after the MMVD and sub-block merge flags.

[0187] Integrate the related merge mode into the regular merge mode.

[0188] In an embodiment of the present disclosure, MMVD, CIIP and triangle merge are merged into a regular merge mode. In this scheme, all MMVD candidates, CIIP candidates and triangle merge candidates are regarded as regular merge candidates, and the regular merge index is used to indicate the candidates used. Therefore, it is necessary to expand the size of the regular merge candidate list accordingly. In one example, a regular merge index equal to N (N can be any positive integer and is less than the maximum size of the regular merge candidate list) means that the MMVD mode is selected, and signaling sends / receives further semantics to indicate which MMVD candidate is used. The same scheme is also applied to CIIP and triangle merge modes.

[0189] In another example, CIIP and triangle are merged into the regular merge mode. In this scheme, all CIIP candidates and triangle merge candidates are regarded as regular merge candidates, and the regular merge index is used to indicate the candidates used. Therefore, it is necessary to expand the size of the regular merge candidate list accordingly.

[0190] Constraint alignment.

[0191] As mentioned above, the constraints for enabling different merging related modes are different. In an embodiment of the present disclosure, the constraints for enabling different merging modes and signaling related identifiers are more aligned. In one example, the constraints are modified as described in the following table.

[0192] Table 8 Modified constraints for enabling / signaling combined related modes

[0193]

[0194] In yet another example of the present disclosure, the constraints are modified as described in the following table.

[0195] Table 9 Modified constraints for enabling / signaling combined related modes

[0196]

[0197] In another example of the present disclosure, the constraints are modified as described in the following table. In this scheme, it should be noted that when block width = 128 or block height = 128, the CIIP flag is still signaled, and when block width = 128 or block height = 128, the CIIP flag is constrained to always be zero, because intra-frame prediction does not support these conditions.

[0198] Table 10 Modified constraints for enabling / signaling combined related modes

[0199] Constraints conventional No constraints MMVD No constraints sub-block Block width > 8 and block height > 8 CIIP Block width > 8 and block height > 8 Triangle Block width > 8 and block height > 8 .

[0200] In another example of the present disclosure, the constraints are modified as described in the following table. In this scheme, it should be noted that when block width = 128 or block height = 128, the CIIP flag is still signaled, and when block width = 128 or block height = 128, the CIIP flag is constrained to always be zero, because intra-frame prediction does not support these conditions.

[0201] Table 11 Modified constraints for enabling / signaling combined related modes

[0202] Constraints conventional No constraints MMVD No constraints sub-block Block width > 8 and block height > 8 CIIP (block width x block height)>=64 Triangle (block width x block height)>=64 .

[0203] Switch the order of CIIP mark and triangular merger mark.

[0204] The signaling order of the CIIP flag and the delta merging flag is switched because it is observed that the delta merging mode is used more often.

Claims

1. A video encoding method, comprising: Signaling a normal merge flag for a coding unit (CU), wherein the coding unit is encoded as a normal merge mode and a merge-related mode, wherein the merge-related mode includes at least one of MMVD, sub-block merge, CIIP, or delta merge; When the normal merge flag is signaled as 1, indicating that a normal merge mode or a merge mode with motion vector difference (MMVD) is used by the CU, a single merge list is constructed for the CU, wherein the single merge list includes normal motion vector candidates and MMVD motion vector candidates, the normal motion vector candidates and the MMVD motion vector candidates are selected by a normal merge index to indicate the candidates to be used, wherein the single merge list is constructed for both the normal merge mode and the MMVD; as well as When the normal merge flag is signaled as zero, it indicates that the normal merge mode is not used by the CU, and a mode flag is further signaled to indicate that the associated merge-related mode is used when the constraint condition of the mode flag is met.

2. The method of claim 1, wherein: The mode flag is a flag of a merge mode with motion vector difference (MMVD), and the constraints of the MMVD flag include: Get a code block, where the code block has a width and a height; Specifies whether the width and height of the encoding block are both not equal to 4; The width of the specified coding block is not equal to 8 or the height of the coding block is not equal to 4; The width of the specified coding block is not equal to 4 or the height of the specified coding block is not equal to 8; and Specifying that the conventional merge flag is not set; or The method further comprises: When the MMVD flag is equal to 1, the MMVD merge flag, the MMVD distance index and the MMVD direction index are sent by signaling; or The mode identifier is a sub-block identifier, and the constraint conditions of the sub-block identifier include: Get a code block, where the code block has a width and a height; Specifies whether the maximum number of sub-block-based merge MVP candidates (MaxNumSubblockMergeCand) is greater than zero; Specifies whether the width of the encoding block is greater than or equal to 8; and Specifies whether the height of the encoding block is greater than or equal to 8; or The mode identifier is a joint inter-frame intra-frame prediction (CIIP) identifier, and the constraint conditions of the CIIP identifier include: Get a code block, where the code block has a width and a height; Specifies whether sps_mh_intra_enabled_flag is set; Specifies whether cu_skip_flag is equal to zero; Specifies whether the width of the encoding block multiplied by the height of the encoding block is greater than or equal to 64; Specifies whether the encoding block width is less than 128; and Specifies whether the height of the encoding block is less than 128; or The mode identifier is a triangle identifier, and the constraint conditions of the triangle identifier include: Specify whether the normal merge flag is not set; Specifies whether the merge mode with motion vector differences (MMVD) flag is not set; specifies whether the merge sub-block flag is not set; and Specifies whether the mode flag, Joint Inter-Intra Prediction (CIIP) flag, is not set.

3. The method of claim 1, further comprising: When the constraint condition of the mode identifier is satisfied, the conventional merging identifier is signaled before the mode identifier is signaled.

4. The method of claim 3, further comprising: When a constraint condition of the flag of the merge mode with motion vector difference (MMVD) is satisfied, before the flag of the merge mode with motion vector difference (MMVD) is signaled, the normal merge flag is signaled; or The method further comprises: When the constraint condition of the joint inter-frame intra-frame prediction (CIIP) flag is satisfied, the conventional merge flag is signaled before the joint inter-frame intra-frame prediction (CIIP) flag is signaled.

5. The method of claim 1, further comprising: When the constraint condition of the mode identifier is satisfied, the normal combination identifier is signaled after the mode identifier is signaled.

6. The method of claim 5, further comprising: When the constraint condition of the sub-block merging mode flag is satisfied, the normal merging flag is sent by signaling after the sub-block merging mode flag is sent by signaling.

7. The method of claim 1, further comprising: Signaling a regular merging flag using context-adaptive binary arithmetic coding (CABAC) with one context; or The method further comprises: A conventional merge flag is signaled using context-adaptive binary arithmetic coding (CABAC) with multiple context models, and selection of the context model is based on coding information including a size of a current CU.

8. A computing device comprising: one or more processors; a memory coupled to the one or more processors; as well as A plurality of programs stored in the memory, when the plurality of programs are executed by the one or more processors, causes the computing device to perform the method of any one of claims 1-7.

9. A non-transitory computer-readable storage medium storing a plurality of programs, wherein the plurality of programs, when executed by one or more processing units of a computing device, causes the computing device to perform the method according to any one of claims 1 to 7 to generate a bitstream and store the bitstream in the non-transitory computer-readable storage medium.

10. A computer program product comprising a plurality of program codes for execution by a computing device having one or more processing units, wherein the plurality of program codes, when executed by the one or more processing units, causes the computing device to perform the method according to any one of claims 1 to 7.

11. A method for storing a bit stream, wherein: The bit stream is generated according to the method according to any one of claims 1-7.

Images