Method and apparatus for signaling merge mode in video coding

Abstract

A method for video coding and decoding is provided. The method comprises: deriving a variable for a current block, the variable indicating an enabled state of at least one merge mode in a merge mode set; generating a codeword set for signaling of the merge mode based on the variable, each codeword corresponding to one of the merge modes; determining a merge mode selected from the merge mode set for the current block based on the variable and / or the signaling; and deriving motion information of the current block using the selected merge mode.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 62 / 845,315, filed on May 8, 2019, entitled “The Signaling of Merge Modes for Video Coding,” which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0003] The present application relates generally to video coding and compression, and in particular, but not limited to, methods and apparatus for signaling merge modes in video codecs. Background Art

[0004] Digital video is supported by various electronic devices, such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smart phones, video teleconferencing devices, video streaming devices, etc. These electronic devices transmit, receive, encode, decode and / or store digital video data by implementing video compression / decompression. Digital video devices implement video codec technologies such as those described in standards defined by Versatile Video Codec (VVC), Joint Exploration Test Model (JEM), MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 Part 10, Advanced Video Codec (AVC), ITU-T H.265 / High Efficiency Video Codec (HEVC), and extensions of such standards.

[0005] Video codecs typically utilize prediction methods (e.g., inter-frame prediction, intra-frame prediction) that exploit the redundancy present in video images or sequences. An important goal of video codec technology is to compress video data into a form that uses a lower bit rate while avoiding or minimizing the degradation of video quality. As evolving video services become available, coding techniques with better codec efficiency are needed.

[0006] Video compression typically involves performing spatial (intra) prediction and / or temporal (inter) prediction to reduce or remove redundancy inherent in video data. For block-based video codecs, a video frame is partitioned into one or more slices, each with multiple video blocks, which may also be referred to as codec tree units (CTUs). Each CTU may contain a codec unit (CU) or be recursively partitioned into smaller CUs until a predefined minimum CU size is reached. Each CU (also referred to as a leaf CU) contains one or more transform units (TUs), and each CU also contains one or more prediction units (PUs). Each CU may be encoded in intra, inter, or IBC mode. Video blocks in an intra (I) slice of a video frame are encoded using spatial prediction with respect to reference samples in neighboring blocks within the same video frame. Video blocks in an inter (P or B) slice of a video frame may use spatial prediction with respect to reference samples in neighboring blocks within the same video frame or temporal prediction with respect to reference samples in other previous and / or future reference video frames.

[0007] A prediction block for the current video block to be encoded is generated based on spatial or temporal prediction of previously encoded reference blocks (e.g., neighboring blocks). The process of finding a reference block can be accomplished by a block matching algorithm. The residual data representing the pixel differences between the current block to be encoded and the prediction block is called a residual block or prediction error. The inter-frame encoding and decoding block is encoded based on the motion vector pointing to the reference block in the reference frame that forms the prediction block and the residual block. The process of determining the motion vector is generally called motion estimation. The intra-frame encoding and decoding block is encoded based on the intra-frame prediction mode and the residual block. For further compression, the residual block is transformed from the pixel domain to a transform domain, such as the frequency domain, to produce residual transform coefficients, which can then be quantized. The quantized transform coefficients, initially arranged in a two-dimensional array, can be scanned to produce a one-dimensional vector of transform coefficients, which are then entropy encoded into the video bitstream to achieve even further compression.

[0008] The encoded video bitstream is then stored in a computer-readable storage medium (e.g., flash memory) for access by another electronic device with digital video capabilities or is directly transmitted to the electronic device by wire or wireless. The electronic device then performs video decompression (which is the reverse process of the above-mentioned video compression) by, for example, parsing the encoded video bitstream to obtain syntax elements from the bitstream and reconstructing the digital video data from the encoded video bitstream to its original format based at least in part on the syntax elements obtained from the bitstream, and presents the reconstructed digital video data on a display of the electronic device.

[0009] As digital video quality increases from HD to 4Kx2K or even 8Kx4K, the amount of video data to be encoded / decoded increases exponentially. How to encode / decode video data more efficiently while maintaining the image quality of the decoded video data is an ongoing challenge.

[0010] At the Joint Video Experts Team (JVET) meeting, JVET defined the first draft of the Versatile Video Codec (VVC) and the VVC Test Model 1 (VTM l) coding method. It was decided to include quadtrees with nested multi-type trees using binary and ternary partitioning codec block structures as the initial new codec features for VVC. Since then, the reference software VTM for implementing the coding method and the draft VVC decoding process has been developed during the JVET meetings. Summary of the invention

[0011] In general, this disclosure describes examples of techniques related to signaling of motion merging modes for video codecs.

[0012] According to a first aspect of the present disclosure, a method for video encoding and decoding is provided, comprising: deriving a variable for a current block, the variable indicating an enabled state of at least one merge mode in a merge mode set; generating a codeword set for signaling of the merge mode based on the variable, each codeword corresponding to one of the merge modes; determining a merge mode selected from the merge mode set for the current block based on the variable and / or the signaling; and deriving motion information of the current block using the selected merge mode.

[0013] According to a second aspect of the present disclosure, a device for video encoding and decoding is provided, comprising: one or more processors; and a memory configured to store instructions executable by the one or more processors; wherein the one or more processors are configured to: derive a variable for a current block, the variable indicating an enabled state of at least one merge mode in a merge mode set; generate a codeword set for signaling of the merge mode based on the variable, each codeword corresponding to one of the merge modes; determine a merge mode selected from the merge mode set for the current block based on the variable and / or the signaling; and derive motion information of the current block using the selected merge mode.

[0014] According to a third aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, comprising instructions stored therein, wherein when one or more processors execute the instructions, the instructions cause the one or more processors to perform actions including the following: deriving a variable for a current block, the variable indicating an enabled state of at least one merge mode in a merge mode set; generating a codeword set for signaling of the merge mode based on the variable, each codeword corresponding to one of the merge modes; determining a merge mode selected from the merge mode set for the current block based on the variable and / or the signaling; and deriving motion information of the current block using the selected merge mode. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] A more specific description of examples of the present disclosure will be presented by reference to specific examples illustrated in the accompanying drawings. Given that these drawings depict only some examples and are therefore not to be considered limiting of scope, these examples will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0016] Figure 1 is a block diagram illustrating an exemplary video encoder according to some embodiments of the present disclosure.

[0017] Figure 2 is a block diagram illustrating an exemplary video decoder according to some embodiments of the present disclosure.

[0018] Figure 3 is a schematic diagram illustrating search points of a merge mode with a motion vector difference (MMVD) according to some embodiments of the present disclosure.

[0019] Figure 4 is a schematic diagram illustrating an example of a control point-based affine motion model according to some embodiments of the present disclosure.

[0020] Figure 5 is a diagram illustrating an example of an affine motion vector field (MVF) per sub-block of a block according to some embodiments of the present disclosure.

[0021] Figure 6 is a schematic diagram illustrating the location of inherited affine motion prediction values ​​according to some embodiments of the present disclosure.

[0022] Figure 7 is a schematic diagram illustrating inheritance of control point motion vectors according to some embodiments of the present disclosure.

[0023] Figure 8 is a schematic diagram illustrating the positions of candidate positions for constructed affine merge mode candidates according to some embodiments of the present disclosure.

[0024] Fig.9Ais a schematic diagram illustrating spatial neighboring blocks used by sub-block based temporal motion vector prediction (SbTMVP) according to some embodiments of the present disclosure.

[0025] Fig. 9B is a schematic diagram illustrating the SbTMVP process of deriving a sub-CU motion field according to some embodiments of the present disclosure.

[0026] Fig.10 is a schematic diagram illustrating an example of inter-frame prediction based on triangular partitions according to some embodiments of the present disclosure.

[0027] Fig.11 is a block diagram illustrating an exemplary apparatus for video encoding and decoding according to some embodiments of the present disclosure.

[0028] Fig.12 is a flow chart illustrating an exemplary process for signaling of merge mode for video encoding and decoding according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0029] Reference will now be made in detail to specific embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth to aid in understanding the subject matter presented herein. However, it will be apparent to one of ordinary skill in the art that various alternatives may be used. For example, it will be apparent to one of ordinary skill in the art that the subject matter presented herein may be implemented on various types of electronic devices having digital video capabilities.

[0030] References throughout this specification to "one embodiment," "an embodiment," "an example," "some embodiments," "some examples," or similar language mean that the particular feature, structure, or characteristic being described is included in at least one embodiment or example. Unless explicitly stated otherwise, features, structures, elements, or characteristics described in conjunction with one or some embodiments are also applicable to other embodiments.

[0031] Throughout the disclosure, the terms "first", "second", "third", etc. are used as special terms and are used only to refer to relevant elements, such as devices, parts, components, steps, etc., and do not imply any spatial or temporal order unless otherwise explicitly stated. For example, "first device" and "second device" may refer to two separately formed devices, two parts, components or operating states of the same device, and may be named arbitrarily.

[0032] The terms "module", "sub-module", "circuit", "sub-circuit", "circuitry", "sub-circuitry", "unit", or "sub-unit" may include a memory (shared, dedicated, or group) that stores code or instructions that can be executed by one or more processors. A module may include one or more circuits with or without stored code or instructions. A module or circuit may include one or more components connected directly or indirectly. These components may or may not be physically attached to each other or located adjacent to each other.

[0033] As used herein, the terms "if" or "when" may be understood to mean "upon" or "in response to" depending on the context. These terms, if they appear in a claim, may not indicate that the associated limitation or feature is conditional or optional. For example, a method may include the following steps: i) when condition X exists or if condition X exists, then perform function or action X', and ii) when condition Y exists or if condition Y exists, then perform function or action Y'. The method may be implemented with the ability to perform function or action X' and the ability to perform function or action Y'. Therefore, both functions X' and Y' may be implemented at different times for multiple executions of the method.

[0034] A unit or module may be implemented purely by software, purely by hardware, or by a combination of hardware and software. For example, in a pure software implementation, a unit or module may include functionally related code blocks or software components that are linked together directly or indirectly to perform a specific function.

[0035] Figure 1 A block diagram illustrating an exemplary block-based hybrid video encoder 100 is shown, which can be used in conjunction with many video codec standards that use block-based processing. In the encoder 100, a video frame is segmented into multiple video blocks for processing. For each given video block, a prediction is formed based on an inter-frame prediction method or an intra-frame prediction method. In inter-frame prediction, one or more prediction values ​​are formed through motion estimation and motion compensation based on pixels from a previously reconstructed frame. In intra-frame prediction, prediction values ​​are formed based on reconstructed pixels in the current frame. Through mode decision, the best prediction value can be selected to predict the current block.

[0036] The prediction residual representing the difference between the current video block and its predicted value is sent to the transform circuitry 102. The transform coefficients are then sent from the transform circuitry 102 to the quantization circuitry 104 for entropy reduction. The quantized coefficients are then fed to the entropy encoding and decoding circuitry 106 to generate a compressed video bitstream. Figure 1As shown in , prediction related information 110 (such as video block partition information, motion vectors, reference picture indexes and intra prediction modes) from the inter prediction circuitry and / or the intra prediction circuitry 112 is also fed through the entropy encoding and decoding circuitry 106 and saved as a compressed video bitstream 114.

[0037] In encoder 100, decoder-related circuitry is also required to reconstruct pixels for prediction purposes. First, the prediction residual is reconstructed by inverse quantization circuitry 116 and inverse transform circuitry 118. The reconstructed prediction residual is combined with block prediction value 120 to generate unfiltered reconstructed pixels of the current video block.

[0038] Spatial prediction (or "intra-frame prediction") uses pixels from samples of already coded neighboring blocks (called reference samples) in the same video frame as the current video block to predict the current video block.

[0039] Temporal prediction (also called "inter prediction") uses reconstructed pixels from already coded video pictures to predict the current video block. Temporal prediction reduces temporal redundancy inherent in video signals. The temporal prediction signal for a given codec unit (CU) or codec block is typically signaled by one or more motion vectors (MVs) that indicate the amount and direction of motion between the current CU and its temporal reference picture. Furthermore, if multiple reference pictures are supported, a reference picture index is additionally sent to identify which reference picture in the reference picture store the temporal prediction signal comes from.

[0040] After performing spatial and / or temporal prediction, the intra / inter mode decision circuitry 121 in the encoder 100 selects the best prediction mode, for example based on a rate distortion optimization method. The block prediction value 120 is then subtracted from the current video block; and the resulting prediction residual is decorrelated using the transform circuitry 102 and the quantization circuitry 104. The resulting quantized residual coefficients are inversely quantized by the inverse quantization circuitry 116 and inversely transformed by the inverse transform circuitry 118 to form a reconstructed residual, which is then added back to the prediction block to form a reconstructed signal for the CU. Further loop filtering 115 (such as a deblocking filter, sample adaptive offset (SAO) and / or adaptive loop filter (ALF)) may be applied to the reconstructed CU before it is placed into the reference picture storage of the picture buffer 117 and used to encode and decode future video blocks. To form the output video bitstream 114, the encoding and decoding mode (inter or intra), prediction mode information, motion information, and quantized residual coefficients are all sent to the entropy encoding and decoding unit 106 to be further compressed and packaged to form a bitstream.

[0041] For example, deblocking filters are available in AVC, HEVC, and the current versions of VVC. In HEVC, an additional loop filter called SAO (Sample Adaptive Offset) is defined to further improve the coding efficiency. In the current version of the VVC standard, another loop filter called ALF (Adaptive Loop Filter) is being actively studied and is likely to be included in the final standard.

[0042] These loop filter operations are optional. Performing these operations helps improve codec efficiency and visual quality. They can also be turned off as a decision presented by the encoder 100 to save computational complexity.

[0043] It should be noted that intra prediction is usually based on unfiltered reconstructed pixels, while inter prediction is based on filtered reconstructed pixels if these filter options are turned on by the encoder 100.

[0044] Figure 2 is a block diagram illustrating an exemplary block-based video decoder 200 that can be used in conjunction with many video codec standards. The decoder 200 is similar to the one residing in Figure 1 The decoder 200 is a decoder 201 that first decodes the incoming video bitstream 201 by entropy decoding 202 to derive quantization coefficient levels and prediction related information. The quantization coefficient levels are then processed by inverse quantization 204 and inverse transform 206 to obtain reconstructed prediction residuals. The block prediction value mechanism implemented in the intra / inter mode selector 212 is configured to perform intra prediction 208 or motion compensation 210 based on the decoded prediction information. A set of unfiltered reconstructed pixel points is obtained by adding the reconstructed prediction residuals from the inverse transform 206 and the prediction output generated by the block prediction value mechanism using an adder 214.

[0045] The reconstructed block may also pass through a loop filter 209 before it is stored in a picture buffer 213, which is used as a reference picture store. The reconstructed video in the picture buffer 213 may be sent to drive a display device, and used to predict future video blocks. With the loop filter 209 turned on, filtering operations are performed on these reconstructed pixels to derive a final reconstructed video output 222.

[0046] The video codec / decoding standards mentioned above (such as VVC, JEM, HEVC, MPEG-4 Part 10) are similar in concept. For example, they all use block-based processing. In the Joint Video Experts Group (JVET) meeting, JVET defined the first draft of the Universal Video Codec (VVC) and VVC Test Model 1 (VTM1) coding methods. It was decided to include a quadtree with nested multi-type trees as the initial new codec feature of VVC, which uses a binary and ternary segmentation codec block structure.

[0047] In VVC, the picture partitioning structure divides the input video into blocks called codec tree units (CTUs). A quadtree with a nested multi-type tree structure is used to partition CTUs into codec units (CUs), where a leaf codec unit (CU) defines an area that shares the same prediction mode (e.g., intra or inter). In this disclosure, the term "unit" defines an image area covering all components; the term "block" is used to define an area covering a specific component (e.g., luminance), and when considering a chroma sampling format such as 4:2:0, the term "block" may be different in spatial location.

[0048] Regular merge mode.

[0049] In VVC Test Model 4 (VTM4), the merge candidate list is constructed by including five types of candidates in the following order:

[0050] 1. Spatial motion vector prediction value (MVP) from spatially neighboring CUs

[0051] 2. Time MVP from the same CU

[0052] 3. History-based MVP from FIFO (First-In-First-Out) Table

[0053] 4. Average MVP by pair; and

[0054] 5. Zero MV.

[0055] The size of the merge list is signaled in the slice header, and the maximum allowed size of the merge list is 6 in VTM4. For each CU code in merge mode, the index of the best merge candidate is encoded using truncated unary binarization (TU). The first bin of the merge index is encoded using context, and bypass encoding is used for the other bins. In the following context of this disclosure, this merge mode is also referred to as normal merge mode, because this concept is the same as the merge mode used in HEVC.

[0056] Merge mode (MMVD) with motion vector difference (MVD).

[0057] In addition to the conventional merge mode in which implicitly derived motion information is directly used for prediction sample generation of the current CU, a merge mode with motion vector difference (MMVD) is introduced in VVC. In some examples, the MMVD flag can be signaled immediately after the skip flag and the merge flag are sent to specify whether MMVD is used for the CU.

[0058] Figure 3 is a schematic diagram illustrating search points of a merge mode with a motion vector difference (MMVD) according to some embodiments of the present disclosure.

[0059] In MMVD, after the merge candidates are selected, the merge candidates are further refined by the motion vector difference (MVD) information transmitted by signaling. This information includes the merge candidate identifier, the index used to specify the motion magnitude (i.e., the distance index), and the index used to indicate the motion direction (i.e., the direction index). In MMVD, one of the first two candidates in the merge list is selected as the MV basis (or starting point). The merge candidate identifier is transmitted by signaling to specify which one is used.

[0060] The distance index specifies the motion magnitude information and indicates a predefined offset from the starting point. Figure 3 As shown in , the offset is added to the horizontal component or the vertical component of the starting MV to obtain the MMVD search point. The relationship between the distance index and the predefined offset is specified in Table 1:

[0061] Table 1 Relationship between distance index and predefined offset

[0062] Distance from IDX 0 1 2 3 4 5 6 7 Bias (in luma samples) 1 / 4 1 / 2 1 2 4 8 16 32

[0063] The direction index indicates the direction of the MVD relative to the starting point. The direction index can indicate four directions, as shown in Table 2. The meaning of the MVD sign (e.g., + or -) may be different depending on the information of the starting MV. When the starting MV is a unidirectionally predicted MV or a bidirectionally predicted MV and both MVs point to the same side of the current picture (i.e., the picture order counts (POCs) of both reference pictures are greater than the POC of the current picture, or are less than the POC of the current picture), the sign in Table 2 specifies the sign of the MV offset added to the starting MV. When the starting MV is a bidirectionally predicted MV and the two MVs point to different sides of the current picture (i.e., the POC of one reference picture is greater than the POC of the current picture, and the POC of the other reference picture is less than the POC of the current picture), the sign in Table 2 specifies the sign of the MV offset added to the list 0 MV component of the starting MV, and the sign of the list 1 MV has an opposite value.

[0064] Table 2 Signs of MV offsets specified by direction index

[0065] Directions to IDX 00 01 10 11 X-axis + – N / A N / A y-axis N / A N / A + –

[0066] Affine motion compensated prediction.

[0067] Figure 4 is a schematic diagram illustrating an example of a control point-based affine motion model according to some embodiments of the present disclosure.

[0068] In HEVC, only the translational motion model is applied to motion compensated prediction (MCP). However, in the real world, there are many kinds of motion, such as zooming in / out, rotation, perspective motion, and other irregular motions. In VVC, especially in VTM4, block-based affine transformation motion compensated prediction is applied. Figure 4 As shown in , the affine motion field of a block is described by the motion information of two control point motion vectors (ie, 4 parameters) or three control point motion vectors (6 parameters).

[0069] For the 4-parameter affine motion model 410, the motion vector at a sample position (x, y) in a block is derived as:

[0070] (1).

[0071] For the 6-parameter affine motion model 420, the motion vector at a sample position (x, y) in a block is derived as:

[0072] (2).

[0073] in( mv 0x , mv 0y ) is the motion vector of the upper left control point, ( mv 1x , mv 1y ) is the motion vector of the upper right control point, and ( mv 2x , mv 2y ) is the motion vector of the lower left control point.

[0074] Figure 5 is a schematic diagram illustrating an affine motion vector field (MVF) 501 of each sub-block of a block according to some embodiments of the present disclosure. In order to simplify motion compensation prediction, block-based affine transformation prediction is applied. In order to derive the motion vector of each 4×4 luminance sub-block, the following equations (1) and (2) are used to calculate: Figure 5The motion vector of the center sample of each sub-block shown in is obtained and rounded to 1 / 16 fractional accuracy. A motion compensated interpolation filter is then applied to generate a prediction for each sub-block using the derived motion vector. The sub-block size of the chroma component can also be set to 4×4. The motion vector (MV) of a 4×4 chroma sub-block is calculated as the average of the MVs of the four corresponding 4×4 luminance sub-blocks.

[0075] Similar to translational motion inter prediction, there are also two affine motion inter prediction modes: affine merge mode and affine AMVP (Advanced Motion Vector Prediction) mode.

[0076] Affine merge mode (AF_MERGE mode) can be applied to CUs whose width and height are both greater than or equal to 8. In this mode, the control point motion vector (CPMV) of the current CU is generated based on the motion information of the spatially neighboring CUs. There can be up to five CPMV prediction value (CPMVP) candidates, and the signaling index is used to indicate which candidate to be used for the current CU. The following three types of CPMVP candidates are used to form the affine merge candidate list:

[0077] 1) Inherited affine merging candidates extrapolated from the CPMV of neighboring CUs;

[0078] 2) constructing affine merge candidates derived using the translated MVs of neighboring CUs; ​​and

[0079] 3) Zero MV.

[0080] Figure 6 is a schematic diagram illustrating the location of inherited affine motion prediction values ​​according to some embodiments of the present disclosure. In VTM4, there are at most two inherited affine candidates derived from the affine motion model of the neighboring blocks, one from the left neighboring CU and one from the top neighboring CU. The candidate blocks of the current CU are as follows: Figure 6 As shown in (Error! Reference source not found). For the left prediction value, the scanning order is A0->A1, and for the upper prediction value, the scanning order is B0->B1->B2. Only the first inheritance candidate from each side is selected. No pruning check is performed between two inheritance candidates.

[0081] Figure 7 is a schematic diagram illustrating the inheritance of control point motion vectors according to some embodiments of the present disclosure. When a neighboring affine CU 720 is identified, its control point motion vector is used to derive the CPMVP candidate in the affine merge list of the current CU 710. Figure 7As shown in , when the lower left neighboring block A is encoded and decoded in affine mode, the motion vectors v2, v3 and v4 of the upper left corner, upper right corner and lower left corner of the CU720 containing block A are obtained. When block A is encoded using a 4-parameter affine model, two CPMVs of the current CU710 are calculated based on v2 and v3. In the case where block A is encoded using a 6-parameter affine model, three CPMVs of the current CU710 are calculated based on v2, v3 and v4.

[0082] Figure 8 is a schematic diagram illustrating the positions of candidate items for constructing affine merge mode candidates according to some embodiments of the present disclosure. The constructed affine candidate items are constructed by combining the adjacent translation motion information of each control point. The motion information of the control point is obtained from Figure 8 The specified spatial and temporal neighbors are derived from CPMV. k (k=1,2,3,4) represents the kth control point of the current block 810. For CPMV1, the B2->B3->A2 blocks are checked in order, and the MV of the first available block is used. Similarly, CPMV2 is derived by checking the B1->B0 blocks in order. CPMV3 is derived by checking the A1->A0 blocks in order. The temporal motion vector prediction value T is used as CPMV4 when it is available.

[0083] After obtaining the MVs of the four control points, affine merge candidates are constructed based on the motion information. The following combinations of the control point MVs are used to construct the following in order:

[0084] {CPMV1, CPMV2, CPMV3},

[0085] {CPMV1, CPMV2, CPMV4},

[0086] {CPMV1, CPMV3, CPMV4},

[0087] {CPMV2, CPMV3, CPMV4},

[0088] {CPMV1, CPMV2}, and

[0089] {CPMV1, CPMV3}.

[0090] The combination of 3 CPMVs constructs a 6-parameter affine merge candidate, and the combination of 2 CPMVs constructs a 4-parameter affine merge candidate. To avoid the motion scaling process, the relevant combination of control point MVs is discarded if the reference index of the control points is different.

[0091] After checking the inherited affine merge candidates and the constructed affine merge candidates, if the affine merge candidate list is still not full, a zero MV is inserted at the end of the list.

[0092] Sub-block based temporal motion vector prediction (SbTMVP).

[0093] The reference software VTM supports the sub-block based temporal motion vector prediction (SbTMVP) method. In some examples, the sub-block merge mode or sub-block based merge mode can include sub-block based temporal motion vector prediction (SbTMVP) and affine merge prediction (AMP) candidates. The SbTVMP process is Fig.9A and Fig. 9B Medium picture.

[0094] Fig.9A is a schematic diagram illustrating spatial neighboring blocks used by sub-block based temporal motion vector prediction (SbTMVP) according to some embodiments of the present disclosure.

[0095] Fig. 9B is a schematic diagram illustrating an SbTMVP process according to some embodiments of the present disclosure, which derives a sub-CU motion field by applying motion shifts from spatial neighbors and scaling the motion information of the corresponding co-located sub-CU.

[0096] Similar to the temporal motion vector prediction (TMVP) in HEVC, SbTMVP uses the motion field in the co-located picture to improve the motion vector prediction and merge mode of the CU in the current picture. The same co-located pictures used by TMVP are used for SbTVMP. SbTMVP differs from TMVP in the following two main aspects:

[0097] 1) TMVP predicts motion at CU level, but SbTMVP predicts motion at sub-CU level; and

[0098] 2) Although TMVP obtains the temporal motion vector from the co-located block in the co-located picture (the co-located block is the bottom right or center block relative to the current CU), SbTMVP applies motion shift before obtaining the temporal motion information from the co-located picture, where the motion shift is obtained from the motion vector of one of the spatial neighboring blocks of the current CU.

[0099] SbTMVP predicts the motion vector of a sub-CU within the current CU in two steps. In the first step, check the following order A1, B1, B0 and A0 Fig.9AOnce the first spatial neighboring block with a motion vector that uses the co-located picture as its reference picture is identified, that motion vector is selected as the motion shift to be applied. When no such motion is identified from the spatial neighbors, the motion shift is set to (0, 0).

[0100] In the second step, the motion shift identified in the first step is applied (i.e., added to the coordinates of the current block) to obtain sub-CU level motion information (e.g., motion vectors and reference indices) from the co-located picture, as Fig. 9B As shown in . Fig. 9B The example in assumes that the motion shift is set to the motion of block A1. Then, for each sub-CU, the motion information of its corresponding block in the co-located picture (the minimum motion grid covering the center sample) is used to derive the motion information of the sub-CU. After the motion information of the co-located sub-CU is identified, it is converted into the motion vector and reference index of the current sub-CU in a manner similar to the TMVP process of HEVC, in which temporal motion scaling is applied to align the reference pictures of the temporal motion vectors with those of the current CU.

[0101] In VTM4, a sub-block based merge list containing a combination of both SbTVMP candidates and affine merge candidates is used for signaling of sub-block based merge mode, which may be referred to as sub-block merge mode or SbTVMP mode. The SbTVMP mode is enabled / disabled through a sequence parameter set (SPS) flag. When the SbTMVP mode is enabled, the SbTMVP prediction value is added as the first entry in the list of sub-block based merge candidates, and followed by the affine merge candidates. The size of the sub-block based merge list is signaled in the SPS, and the maximum allowed size of the sub-block based merge list is 5 in VTM4.

[0102] The sub-CU size used in SbTMVP is fixed to 8×8, and similar to the affine merge mode, the SbTMVP mode is only applicable to CUs whose width and height are both greater than or equal to 8.

[0103] The encoding logic of the additional SbTMVP merge candidates is the same as that of other merge candidates, that is, for each CU in a P or B slice, an additional RD check is performed to decide whether to use the SbTMVP candidate.

[0104] Joint inter and intra prediction (CIIP).

[0105] In VTM4, when a CU is encoded and decoded in merge mode and when the CU contains at least 64 luma samples (i.e., the CU width multiplied by the CU height is equal to or greater than 64), an additional flag is signaled to indicate when the joint inter / intra prediction (CIIP) mode is applied to the current CU.

[0106] To form a CIIP prediction, the intra prediction mode is first derived from two additional syntax elements. Up to four possible intra prediction modes can be used: directional angle prediction (DC), plane prediction (PLANAR), horizontal prediction (HORIZONAL) or vertical prediction (VERTICAL). Then, the conventional intra and inter decoding process is used to derive the inter prediction and intra prediction signals. Finally, a weighted average of the inter and intra prediction signals is performed to obtain the CIIP prediction. CIIP prediction can be adopted so that only the plane mode is used for CIIP, thereby simplifying the CIIP mode; for example, for CIIP, the intra prediction mode derivation described in the following paragraphs can be removed. In addition, in some CIIP designs, the weights of intra and inter prediction samples are adaptively selected based on the number of adjacent intra-frame coding blocks. Specifically, the weights (wIntra, wInter) are adaptively set as follows. When both the top and left neighbors are intra-coded, (wIntra, wInter) is set to be equal to (3, 1). Otherwise, when one of the blocks is intra-coded, the weights are the same, i.e., (2, 2). When no block is intra-coded, the weights are set equal to (1, 3). In these CIIP designs, the weight derivation method described in the following paragraphs may not be used.

[0107] In some examples, for intra prediction mode derivation, up to 4 intra prediction modes (including DC, PLANAR, HORIZONAL, and VERTICAL modes) can be used in CIIP mode to predict the luma component. When the CU shape is very wide (i.e., its width is greater than twice its height), the HORIZONAL mode is not allowed. When the CU shape is very narrow (i.e., its height is greater than twice its width), the VERTICAL mode is not allowed. In these cases, only 3 intra prediction modes are allowed.

[0108] CIIP mode uses 3 most probable modes (MPMs) for intra prediction. The CIIP MPM candidate list is formed as follows:

[0109] i. The left and top adjacent blocks are set to A and B respectively;

[0110] ii. The intra prediction modes of blocks A and B, denoted as intraModeA and intraModeB respectively, are derived as follows:

[0111] a. Let X be A or B

[0112] b. When 1) block X is not available; or 2) block X is not predicted using CIIP mode or intra mode; or 3) block B is outside the current CTU, set intraModeX to DC;

[0113] c. Otherwise, 1) when the intra prediction mode of block X is DC or PLANAR, set intraModeX to DC or PLANAR; or 2) when the intra prediction mode of block X is a "vertical-like" angular mode (greater than 34), set intraModeX to VERTICAL; or 3) when the intra prediction mode of block X is a "horizontal-like" angular mode (less than or equal to 34), set intraModeX to HORIZONAL;

[0114] iii. When intraModeA and intraModeB are the same:

[0115] a. When intraModeA is PLANAR or DC, set the three MPMs to {PLANAR, DC, VERTICAL} in the order of {PLANAR, DC, VERTICAL};

[0116] b. Otherwise, set the three MPMs to {intraModeA, PLANAR, DC} in the order of {intraModeA, PLANAR, DC};

[0117] iv. Otherwise (intraModeA and intraModeB are different):

[0118] a. Set the first two MPMs to {intraModeA, intraModeB} in the order {intraModeA, intraModeB}

[0119] b. Check the uniqueness of PLANAR, DC, and VERTICAL against the first two MPM candidate patterns, in that order; once a unique pattern is found, add it as the third MPM.

[0120] When the CU shape is very wide or very narrow (as defined above), the MPM flag is inferred to be 1 and does not need to be signaled. Otherwise, the MPM flag is signaled to indicate when the CIIP intra prediction mode is one of the CIIP MPM candidate modes.

[0121] When the MPM flag is 1, the MPM index is further signaled to indicate which of the MPM candidate modes is used in CIIP intra prediction. Otherwise, when the MPM flag is 0, the intra prediction mode is set to the "default" mode in the MPM candidate list. For example, if the PLANAR mode is not in the MPM candidate list, PLANAR is the default mode and the intra prediction mode is set to PLANAR. Since 4 possible intra prediction modes are allowed in CIIP and the MPM candidate list contains only 3 intra prediction modes, one of the 4 possible modes must be the default mode.

[0122] For chroma components, DM mode is always applied without additional signaling; that is, for a CU, chroma uses the same prediction mode as luma.

[0123] The intra prediction mode of a CIIP-coded CU is saved and used in the intra mode coding of future neighboring CUs.

[0124] In order to combine the inter and intra prediction signals, the inter prediction signal in CIIP mode is derived using the same inter prediction process applied to the regular merge mode. ; and use the CIIP intra prediction mode to derive the intra prediction signal after the conventional intra prediction process Then, the intra and inter prediction signals are combined using a weighted average. In some examples, the weight values ​​depend on the intra prediction mode and where the sample is located in the codec block, as follows:

[0125] 1) When the intra prediction mode is DC or PLANAR mode, or when the block width or height is less than 4, equal weights are applied to the intra prediction and inter prediction signals;

[0126] 2) Otherwise, the weights are determined based on the intra prediction mode (in this case, either HORIZONAL mode or VERTICAL mode) and the sample position in the block. Take the HORIZONAL prediction mode as an example (the weights for VERTICAL mode are derived in a similar way but in an orthogonal direction). Denote W as the width of the block and H as the height of the block. First, the codec block is divided into four equal parts, each of size (W / 4)xH. Starting from the part closest to the intra prediction reference sample and ending at the part farthest from the intra prediction reference sample, the weight wt for each of the 4 regions is set to 6, 5, 3 and 2, respectively. The following equation is used to derive the final CIIP prediction signal:

[0127] (3).

[0128] Triangular partitioning for inter prediction.

[0129] In VTM4, a new triangle partition mode is introduced for inter prediction. Triangle partition mode (TPM) is only applied to CUs that are 8x8 or larger and are encoded or decoded in skip or merge mode. For CUs that meet these conditions and the merge flag is turned on, a CU level flag is signaled to indicate whether the triangle partition mode is applied. Triangle partition mode (TPM) can also be called triangle merge mode.

[0130] Fig.10 is a schematic diagram illustrating inter-frame prediction based on triangular partitions according to some embodiments of the present disclosure.

[0131] When the triangle merge mode is used, the CU is evenly divided into two triangle partitions using diagonal split 1010 or anti-diagonal split 1020, such as Fig.10 As shown in . Each triangular partition in the CU is inter-predicted using its own motion; and only unidirectional prediction is allowed for each partition, that is, each partition has one motion vector and one reference index. The unidirectional prediction motion constraint is applied to ensure that in triangular prediction mode, only two motion compensated predictions are required for the CU, which is the same as conventional bidirectional prediction.

[0132] When the CU level flag indicates that the current CU is encoded and decoded using the triangle partition mode, the flag is signaled to indicate the triangle partition direction (i.e. diagonal or anti-diagonal). Then, for each of the two partitions, an index is signaled to indicate the merged motion vector candidates for each triangle partition. After predicting each triangle partition, a hybrid process with adaptive weights is used to adjust the sample values ​​along the diagonal or anti-diagonal edges. After the prediction process, the transformation and quantization process is applied to the entire CU. It is worth mentioning that the motion field of the CU predicted using the triangle partition mode is stored in 4x4 units.

[0133] Context-Adaptive Binary Arithmetic Codec (CABAC).

[0134] Context-Adaptive Binary Arithmetic Codec (CABAC) is a form of entropy codec used in many video codec standards, such as H.264 / MPEG-4 AVC and High Efficiency Video Codec (HEVC) and VVC. CABAC is based on arithmetic codec with some innovations and changes to adapt it to the needs of video codec standards:

[0135] i. It encodes and decodes binary symbols, which keeps the complexity low and allows probabilistic modeling of the more frequently used bits in any symbol.

[0136] ii. Adaptively select the probability model based on the local context, allowing for better probabilistic modeling since the encoding and decoding patterns are usually well correlated locally.

[0137] iii. It uses multiplication-free range division through the use of quantized probability ranges and probability states.

[0138] CABAC has multiple probability modes for different scenarios. It first converts all non-binary symbols into binary. Then, for each binary bit (or bit), the codec chooses which probability model to use and uses information from nearby elements to optimize the probability estimate. Finally, arithmetic coding and decoding are applied to compress the data.

[0139] Context modeling provides an estimate of the conditional probability of encoding and decoding symbols. With a suitable context model, a given inter-symbol redundancy can be exploited by switching between different probability models depending on the already encoded and decoded symbols in the neighborhood of the current symbol to be encoded.

[0140] The encoding and decoding of data symbols involves the following stages:

[0141] i. Binarization: CABAC uses binary arithmetic coding, which means that only binary decisions (1 or 0) are encoded. Non-binary valued symbols (e.g., transform coefficients or motion vectors) are "binarized" or converted into binary codes before arithmetic coding. The process is similar to that of converting data symbols into variable length codes, but the binary codes are further encoded (by the arithmetic codec) before transmission.

[0142] ii. Repeat the stages for each binary digit (or "bit") of the binarized symbol.

[0143] iii. Context model selection: A "Context model" is a probability model for one or more bins of the binarized symbol. The model may be selected from a selection of available models depending on the statistics of the most recently coded data symbols. The context model stores the probability of each bin being "1" or "0".

[0144] iv. Arithmetic Coding: The arithmetic codec encodes each binary bit according to the chosen probability model. Note that for each binary bit (corresponding to "0" and "1"), there are only two sub-ranges.

[0145] v. Probability update: Update the selected context model based on the actual codec value (e.g., when the binary bit value is "1", increase the frequency count of "1").

[0146] In summary, in VTM-4.0, merge modes are classified into five categories, including normal merge mode, merge mode with MVD (MMVD), sub-block merge mode (including affine merge and sub-block based temporal motion vector prediction), joint inter-frame and intra-frame prediction (CIIP) merge mode, and triangle partition merge mode. The syntax of merge mode signaling in the current VVC is illustrated in Table 3. The symbol ae(v) indicates a context adaptive arithmetic entropy codec syntax element.

[0147] Table 3 Syntax of the merge-related mode in the current VVC

[0148] .

[0149] That is, in the current VVC, the syntax (associated flag) signaled to indicate the corresponding merge mode is shown in Table 4.

[0150] Table 4 Signaling of merge-related modes in current VVC

[0151] MMVD logo Sub-block ID CIIP Logo Triangle logo MMVD 1 - - - sub-block 0 1 - - CIIP 0 0 1 - Triangle 0 0 0 1 conventional 0 0 0 0

[0152] It is observed that more than 50% of the merge modes are regular merge modes. However, in VTM-4.0, the codeword for regular merge mode is the longest one among the five different merge modes (as shown in Table 4), which is not an efficient design in terms of syntax parsing. In current VVC, skip mode has a similar syntax design to merge mode except that there is no CIIP mode for skipping. And, the same problem is observed in skip mode. Several methods are proposed to improve the efficiency of syntax signaling of merge-related modes. These methods can be applied independently or jointly.

[0153] In some examples, it is proposed to signal an explicit flag for the normal merge mode to indicate whether the normal merge mode is used. As shown in the examples of Tables 5 and 6 below, a normal flag (which may be referred to as a normal merge flag) is explicitly signaled into the bitstream, and all signaling of the related flags is modified accordingly. The normal merge flag is contextually encoded and decoded using CABAC. In one embodiment, only one context is used to encode and decode the normal merge flag. In another embodiment, multiple context models are used to encode and decode the normal merge flag, and the selection of the context model is based on the information of the codec, such as the normal merge flag of the neighboring blocks, the size of the current CU, or the skip flag of the current CU.

[0154] Table 5 Example of signaling for combined related modes

[0155] General logo MMVD logo Sub-block ID CIIP Logo conventional 1 - - - MMVD 0 1 - - sub-block 0 0 1 - CIIP 0 0 0 1 Triangle 0 0 0 0

[0156] Table 6 Example of signaling of skip-related mode

[0157] General logo MMVD logo Sub-block ID conventional 1 - - MMVD 0 1 - sub-block 0 0 1 Triangle 0 0 0

[0158] In the example shown in Table 5, if the regular flag is signaled as 1 (i.e., the signaling of the merge mode includes a positive signal of the regular merge mode), it indicates that the regular merge mode is used and the motion information of the current block is derived using the regular merge mode. If the regular flag is 0 (i.e., the signaling of the merge mode does not include a positive signal of the regular merge mode), it indicates that other merge modes are used. Depending on the value of (one or more) other merge mode flags, one of these other merge modes including MMVD, sub-block, CIIP and delta merge modes can be indicated for deriving the motion information of the current block. For example, if the MMVD flag is a positive signal, it indicates that the motion information of the current block is derived using MMVD.

[0159] In the current VVC, the constraints and / or conditions for enabling each different merge-related mode are different, which are summarized in Table 7 below. In addition, a flag is signaled in the sequence parameter set (SPS) to indicate the on / off of each merge mode including MMVD, sub-block (including affine and sub-block TMVP), CIIP and triangle mode. Therefore, a variable indicating the enabled state of the merge mode can be derived.

[0160] Table 7 Constraints for enabling / signaling merge / skip related modes

[0161] .

[0162] Tables 5 and 6 show the binarized codewords, assuming that all merge-related modes are available (i.e., enabled). However, due to the constraints on the enablement of the merge mode described above, in some cases, only a subset of the merge-related modes, rather than all of them, are available. That is, a specific merge mode may not be allowed or enabled as a mode option. In this case, the corresponding codeword binarization can also be adjusted accordingly to improve the encoding and decoding efficiency. For example, in some cases where only the conventional merge mode and the MMVD mode are enabled for a given CU, only one binary bit needs to be signaled to indicate which of the two merge modes is used for the given CU, as shown in Table 8. In this case, there is no need to signal identifiers such as MMVD, sub-block, or CIIP identifiers.

[0163] Table 8 Special cases of merge mode signaling

[0164] General logo MMVD logo Sub-block ID CIIP Logo conventional 1 - - - MMVD 0 - - -

[0165] There are other examples where only a subset of merge-related modes are available due to constraints on merge mode enablement. Whenever only a subset of merge-related modes are available, the corresponding codeword binarization can be adjusted accordingly to save signaling overhead.

[0166] According to the present disclosure, the necessity of signaling each different merge mode identifier can be adjusted accordingly according to the enable / disable state (or enable state) of each merge mode. More specifically, based on the order of signaling (for example, the signaling order can be conventional merge identifier -> MMVD identifier -> sub-block identifier -> CIIP identifier), a given merge mode identifier is only signaled when there is at least one other type of merge mode following the given merge mode in the signaling order and the at least one other type of merge mode is also allowed (i.e., enabled as a mode option) for the current CU. Based on this concept, an exemplary signaling method is provided. Taking the above-mentioned signaling order as an example, if the CIIP and triangular merge modes are not enabled, the sub-block identifier may not be signaled.

[0167] The above signaling order is only an example, and other signaling orders are possible.

[0168] In some examples, when a specific merge mode is allowed (enabled) but the corresponding flag is not signaled as shown above, the value of the corresponding flag can be inferred or derived. When the merge mode flags explicitly signaled for the current CU are all false, the value of the corresponding flag is inferred to be true. Otherwise, the value is inferred to be false. In the example shown in Table 8, the MMVD mode is allowed but the MMVD flag is not signaled, and if the conventional flag signaled is 0, the value of the MMVD flag is inferred to be true (or 1). Otherwise, the value of the MMVD flag is inferred to be false (or 0).

[0169] Table 9 is a syntax table illustrating an example of the syntax of the merge mode signaling according to the present disclosure. In an embodiment based on VTM 5.0, the values ​​of four Boolean variables MMVDAallowed, MergeSubblockAllowed, MergeCIIPAllowed, and MergeTriangleAllowed are first derived according to the following content, and the values ​​of the four Boolean variables respectively specify whether each mode is allowed (e.g., enabled) for the current codec block:

[0170] i. MMVDAllowed = sps_MMVD_enabled_flag;

[0171] ii. MergeSubblockAllowed = (sps_affine_enabled_flag || (sps_sbtmvp_enabled_flag && slice_temporal_mvp_enabled_flag)) && block width>=8 && block height>=8;

[0172] iii. MergeCIIPAllowed = sps_ciip_enabled_flag && The current codec block is not encoded in skip mode && (block width x block height) >=64 && block width !=128 && block height !=128;

[0173] iv. MergeTriangleAllowed = sps_triangle_enabled_flag && (block width x block height) >=64.

[0174] In the above expression, the symbols "||" and "&&" represent logical OR and AND operations, respectively. The definition of the semantics (eg, sps_MMVD_enabled_flag) in the above expression is given as follows.

[0175] sps_mmvd_enabled_flag equal to 1 specifies that merge mode with motion vector differences (MMVD) is enabled. sps_mmvd_enabled_flag equal to 0 specifies that merge mode with motion vector differences is disabled.

[0176] sps_affine_enabled_flag specifies whether affine-based motion compensation can be used for inter prediction. When sps_affine_enabled_flag is equal to 0, the syntax should be constrained so that affine-based motion compensation is not used in the coded video sequence (CVS), and inter_affine_flag and cu_affine_type_flag are not present in the codec unit syntax of the CVS. Otherwise (i.e., sps_affine_enabled_flag is equal to 1), affine-based motion compensation can be used in the CVS.

[0177] sps_sbtmvp_enabled_flag equal to 1 specifies that subblock based temporal motion vector prediction can be used for decoding pictures of all slices in the CVS with slice_type not equal to 1. sps_sbtmvp_enabled_flag equal to 0 specifies that subblock based temporal motion vector prediction is not used in the CVS. When sps_sbtmvp_enabled_flag is not present, it is inferred to be equal to 0.

[0178] slice_temporal_mvp_enabled_flag specifies whether the temporal motion vector predictor can be used for inter prediction. When slice_temporal_mvp_enabled_flag is equal to 0, the syntax elements of the current picture shall be constrained so that the temporal motion vector predictor is not used in the decoding of the current picture. Otherwise (i.e., slice_temporal_mvp_enabled_flag is equal to 1), the temporal motion vector predictor can be used for the decoding of the current picture. When it is not present, the value of slice_temporal_mvp_enabled_flag is inferred to be equal to 0.

[0179] sps_ciip_enabled_flag specifies that ciip_flag may be present in the codec unit syntax of an inter codec unit. sps_ciip_enabled_flag equal to 0 specifies that ciip_flag is not present in the codec unit syntax of an inter codec unit.

[0180] sps_triangle_enabled_flag specifies whether triangle based motion compensation can be used for inter prediction. sps_triangle_enabled_flag equal to 0 specifies that the syntax should be constrained so that triangle based motion compensation is not used in CVS and merge_triangle_split_dir, merge_triangle_idx0 and merge_triangle_idx1 are not present in the codec unit syntax of CVS. sps_triangle_enabled_flag equal to 1 specifies that triangle based motion compensation can be used in CVS.

[0181] It is worth noting that the conditions for each Boolean variable listed above correspond to the current VVC. When the associated constraints for a certain merge mode change, the corresponding equation or expression can be updated accordingly, and the method described in the present disclosure still applies. The Boolean variable is used to determine whether each merge mode identifier needs to be signaled as shown in Table 9 and the related semantics below. In the syntax table, the bold variables represent the syntax transmitted by signaling. The symbol ae(v) indicates a context-adaptive arithmetic entropy coding and decoding syntax element.

[0182] Table 9: Syntax for merging related patterns in an example

[0183] .

[0184] Merge data semantics.

[0185] regular_merge_flag[x0][y0] equal to 1 specifies that regular merge mode is used to generate inter prediction parameters for the current codec unit. Array indices x0, y0 specify the position (x0, y0) of the top-left luma sample of the considered codec block relative to the top-left luma sample of the picture.

[0186] When regular_merge_flag[x0][y0] is not present, it is inferred to be equal to 1.

[0187] mmvd_flag[x0][y0] equal to 1 specifies that merge mode with motion vector difference is used to generate inter prediction parameters for the current codec unit. Array index x0, y0 specifies the position (x0, y0) of the top left luma sample of the considered codec block relative to the top left luma sample of the picture.

[0188] When mmvd_flag[x0][y0] is not present, it is inferred as follows:

[0189] - mmvd_flag[ x0 ][ y0 ] is inferred to be equal to 1 when all of the following are true:

[0190] - sps_mmvd_enabled_flag is equal to 1.

[0191] - general_merge_flag[ x0 ][ y0 ] is equal to 1.

[0192] - cbWidth*cbHeight is less than or equal to 32.

[0193] -regular_merge_flag[ x0 ][ y0 ] is equal to 0.

[0194] - Otherwise, mmvd_flag[ x0 ][ y0 ] is inferred to be equal to 0.

[0195] merge_subblock_flag[x0][y0] specifies whether to infer subblock-based inter prediction parameters for the current codec unit from neighboring blocks. Array index x0, y0 specifies the position (x0, y0) of the top-left luma sample of the considered codec block relative to the top-left luma sample of the picture.

[0196] When merge_subblock_flag[x0][y0] is not present, it is inferred as follows:

[0197] - merge_subblock_flag[ x0 ][ y0 ] is inferred to be equal to 1 when all of the following are true:

[0198] -sps_affine_enabled_flag is equal to 1 or (sps_sbtmvp_enabled_flag is equal to 1 and slice_temporal_mvp_enabled_flag is equal to 1).

[0199] -general_merge_flag[x0][y0] is equal to 1.

[0200] -cbWidth is greater than or equal to 8.

[0201] -cbHeight is greater than or equal to 8.

[0202] -regular_merge_flag[x0][y0] is equal to 0.

[0203] -mmvd_flag[x0][y0] is equal to 0.

[0204] Otherwise, merge_subblock_flag[x0][y0] is inferred to be equal to 0.

[0205] ciip_flag[x0][y0] specifies whether combined inter-picture merging and intra-picture prediction is applied for the current codec unit. Array index x0, y0 specifies the position (x0, y0) of the top-left luma sample of the considered codec block relative to the top-left luma sample of the picture.

[0206] When ciip_flag[x0][y0] is not present, it is inferred as follows:

[0207] - ciip_flag[x0][y0] is inferred to be equal to 1 when all of the following are true:

[0208] -sps_ciip_enabled_flag is equal to 1.

[0209] -general_merge_flag[x0][y0] is equal to 1.

[0210] -cu_skip_flag[x0][y0] is equal to 0.

[0211] -cbWidth*cbHeight is greater than or equal to 64.

[0212] -regular_merge_flag[x0][y0] is equal to 0.

[0213] -mmvd_flag[x0][y0] is equal to 0.

[0214] -merge_subblock_flag[x0][y0] is equal to 0.

[0215] Otherwise, ciip_flag[x0][y0] is inferred to be equal to 0.

[0216] The variable MergeTriangleFlag[x0][y0] specifies whether triangle-based motion compensation is used to generate the prediction samples for the current codec unit. When decoding a B slice, the variable MergeTriangleFlag[x0][y0] is derived as follows:

[0217] -MergeTriangleFlag[x0][y0] is set equal to 1 when all of the following conditions are true:

[0218] -sps_triangle_enabled_flag is equal to 1.

[0219] -slice_type is equal to B.

[0220] -general_merge_flag[x0][y0] is equal to 1.

[0221] -MaxNumTriangleMergeCand is greater than or equal to 2.

[0222] -cbWidth*cbHeight is greater than or equal to 64.

[0223] -regular_merge_flag[x0][y0] is equal to 0.

[0224] -mmvd_flag[x0][y0] is equal to 0.

[0225] -merge_subblock_flag[x0][y0] is equal to 0.

[0226] -ciip_flag[x0][y0] is equal to 0.

[0227] Otherwise, MergeTriangleFlag[x0][y0] is set equal to 0.

[0228] In some examples, different merging modes may be considered in groups. For example, a first group of merging modes may include a conventional merging mode, and a second group of merging modes may include a CIIP merging mode and a triangular merging mode (TPM). Alternatively or additionally, a sub-block based merging mode may be considered as a third group. A signaling order may be provided based on each group of merging modes. According to different embodiments, each group of merging modes may adopt any predefined order.

[0229] In one example, the first group may be listed before the second group in the signaling order, and then the normal merge mode identification may be signaled only if there is at least one merge mode allowed or enabled as a mode option in the second group of merge modes (ie, one or both of CIIP and TPM are allowed).

[0230] In some other examples, instead of signaling a separate mode identifier, a signaling syntax may be used to indicate which merge / skip related mode is used for the current CU. For CABAC encoding and decoding of mode syntax, different codeword binarizations may be used and different CABAC context models may be applied to each binary bit. An exemplary codeword binarization may be a truncated unary codeword, where the maximum codeword index is N (e.g., N=4), as shown in Table 10 below. In this example, all five merge-related modes are allowed (i.e., enabled), where MMVD, sub-block, CIIP, and triangular merge modes are indicated as enabled by Boolean variables MMVDAllowed, MergeSubblockAllowed, MergeCIIPAllowed, and MergeTriangleAllowed. Based on these variables, a set of codewords for signaling of merge modes is generated, as shown in Table 10, where each codeword corresponds to one of the merge modes. The maximum length of the codeword in this example is 4 (i.e., the length of the codeword corresponding to the CIIP and triangular merge modes). Therefore, the merge mode for the current block can be determined based on the codeword transmitted by signaling, rather than using the separate mode identifiers described above (e.g., conventional identifier, MMVD identifier, sub-block identifier, and CIIP identifier). For example, the codeword "01" transmitted by signaling indicates that MMVD is used for the current codec block.

[0231] Table 10 Example of syntax for signaling the merge mode

[0232] Codeword Index Codeword Merge Mode 0 0 Normal merge mode 1 01 MMVD 2 001 sub-block 3 0001 CIIP 4 0000 Triangle

[0233] In some examples, codewords and merge modes may be mapped in different combinations. For the case where all five merge-related modes are allowed, a syntax with a mapping different from those shown in Table 10 may be used. For example, a codeword "0" may be used to indicate that sub-block merge mode is used for the current codec block, and a codeword "001" may be used to indicate that a normal merge mode is used for the current codec block.

[0234] It should be noted that the 0s and 1s in the codeword can all be reversed and the resulting codeword will function the same.

[0235] It should also be noted that the value of N (e.g., in Fig.10 In the example of N=4) and the codeword mapping merging mode can be different.

[0236] In one example, when the CIIP merge mode is not allowed (e.g., not enabled) and all four other modes are allowed, the associated codewords may be those shown in Table 11 below, where N = 3. The maximum length of the codeword in this example is 3 (i.e., the length of the codeword corresponding to the sub-block and delta merge modes).

[0237] Table 11 Example of syntax for signaling merging mode

[0238] Codeword Index Codeword Merge Mode 0 0 Normal merge mode 1 01 MMVD 2 001 sub-block 3 000 Triangle

[0239] Therefore, the number of codewords in the generated codeword set and the maximum length of the codewords vary based on the variables.

[0240] In another example, the variable may indicate that only one merge mode is enabled and other merge modes are not enabled. The only enabled merge mode may be inferred as the merge mode for the current block without explicit signaling of the codeword.

[0241] In yet another example, only two merge modes may be indicated as allowed or enabled. When the signaled codeword indicates that the first of the two merge modes is used or the other merge mode is not used, the first of the two merge modes may be determined as the merge mode for the current block.

[0242] In some other examples, the merge mode set may include all or part of a regular merge mode (Regular), a merge mode with motion vector difference (MMVD), a subblock-based merge mode (Subblock), a joint inter-frame and intra-frame merge mode (CIIP), and a triangle merge mode (Triangle).

[0243] Fig.11 1 is a block diagram illustrating an apparatus for video encoding and decoding according to some embodiments of the present disclosure. The apparatus 1100 may be a terminal such as a mobile phone, a tablet computer, a digital broadcast terminal, a tablet device, or a personal digital assistant.

[0244] like Fig.11As shown in , device 1100 may include one or more of the following components: processing component 1102, memory 1104, power supply component 1106, multimedia component 1108, audio component 1110, input / output (I / O) interface 1112, sensor component 1114 and communication component 1116.

[0245] The processing component 1102 generally controls the overall operation of the device 1100, such as operations related to display, phone calls, data communications, camera operations, and recording operations. The processing component 1102 may include one or more processors 1120 for executing instructions to complete all or part of the steps of the above method. In addition, the processing component 1102 may include one or more modules to facilitate interaction between the processing component 1102 and other components. For example, the processing component 1102 may include a multimedia module for facilitating interaction between the multimedia component 1108 and the processing component 1102.

[0246] The memory 1104 is configured to store different types of data to support the operation of the device 1100. Examples of such data include instructions for any application or method running on the device 1100, contact data, phone book data, messages, pictures, videos, etc. The memory 1104 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, and the memory 1104 can be a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk.

[0247] The power supply component 1106 supplies power to the various components of the device 1100. The power supply component 1106 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 1100.

[0248] The multimedia component 1108 includes a screen that provides an output interface between the device 1100 and the user. In some examples, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen that receives input signals from the user. The touch panel may include one or more touch sensors for sensing touch, sliding, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or sliding action, but may also detect the duration and pressure associated with the touch or sliding operation. In some examples, the multimedia component 1108 may include a front camera and / or a rear camera. When the device 1100 is in an operating mode (such as a shooting mode or a video mode), the front camera and / or the rear camera may receive external multimedia data.

[0249] The audio component 1110 is configured to output and / or input audio signals. For example, the audio component 1110 includes a microphone (MIC). When the device 1100 is in an operating mode (such as a call mode, a recording mode, and a speech recognition mode), the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 1104 or sent via the communication component 1116. In some examples, the audio component 1110 also includes a speaker for outputting an audio signal.

[0250] The I / O interface 1112 provides an interface between the processing unit 1102 and the peripheral interface module. The peripheral interface module may be a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to a home button, a volume button, an on button, and a lock button.

[0251] The sensor component 1114 includes one or more sensors for providing status assessment for the device 1100 in different aspects. For example, the sensor component 1114 can detect the on / off state of the device 1100 and the relative position of the components. For example, the components are the display and keyboard of the device 1100. The sensor component 1114 can also detect the position change of the device 1100 or the components of the device 1100, the presence or absence of the user's contact on the device 1100, the orientation or acceleration / deceleration of the device 1100, and the temperature change of the device 1100. The sensor component 1114 may include a proximity sensor, which is configured to detect the presence of nearby objects without any physical contact. The sensor component 1114 may also include an optical sensor, such as a CMOS or CCD image sensor used in imaging applications. In some examples, the sensor component 1114 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0252] Communication component 1116 is configured to promote wired or wireless communication between device 1100 and other equipment. Device 1100 can access wireless network based on communication standards such as WiFi, 4G or its combination. In an example, communication component 1116 receives broadcast signal or broadcast related information from external broadcast management system via broadcast channel. In an example, communication component 1116 may also include a near field communication (NFC) module for promoting short-range communication. For example, NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0253] In an example, apparatus 1100 may be implemented by one or more of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic components to perform the above method.

[0254] The non-transitory computer-readable storage medium may be, for example, a hard disk drive (HDD), a solid-state drive (SSD), a flash memory, a hybrid drive or a solid-state hybrid drive (SSHD), a read-only memory (ROM), a compact disk read-only memory (CD-ROM), a magnetic tape, a floppy disk, etc.

[0255] Fig.12 is a flow chart illustrating an exemplary process for signaling of merge mode for video encoding and decoding according to some embodiments of the present disclosure.

[0256] In step 1202 , the processor 1120 derives a variable for a current block, the variable indicating an enabled state of at least one merge mode in a merge mode set.

[0257] In step 1204, the processor 1120 generates a set of codewords for signaling the merge modes based on the variables, each codeword corresponding to one of the merge modes.

[0258] In step 1206 , the processor 1120 determines a merge mode selected from the merge mode set for the current block based on the variable and / or the signaling.

[0259] In step 1208 , the processor 1120 derives motion information of the current block using the selected merge mode.

[0260] The processor 1120 may determine the first merge mode for the current block when it is determined that the variable indicates that the first merge mode in the set is enabled and other merge modes in the set are not enabled. The processor 1120 determines the first merge mode may include deriving explicit signaling indicating the selection of the first merge mode using the codeword, or inferring the selection of the first merge mode based on the variable without explicit signaling.

[0261] The processor 1120 may determine the first merge mode for the current block when it is determined that the variable indicates that the first merge mode and at least one second merge mode in the set are enabled and the signaling indicates that the first merge mode is selected.

[0262] The processor 1120 may determine the first merge mode for the current block when it is determined that the variable indicates that the first merge mode and one second merge mode in the set are enabled, other merge modes in the set are not enabled, and the signaling indicates that the second merge mode is not selected.

[0263] In some examples, a device for video encoding and decoding is provided. The device includes: one or more processors 1120; and a memory 1104 configured to store instructions that can be executed by the one or more processors; when the processor executes the instructions, it is configured to perform the following steps: Fig.12 The method shown in .

[0264] In some other examples, a non-transitory computer-readable storage medium 1104 is provided having instructions stored therein. When the instructions are executed by one or more processors 1120, the instructions cause the processors to perform the following steps: Fig.12 The method shown in .

[0265] The description of the present disclosure is presented for purposes of illustration and is not intended to be exhaustive or limiting of the present disclosure. Many modifications, variations, and alternative embodiments will be apparent to one of ordinary skill in the art having the benefit of the teachings presented in the foregoing description and the associated drawings.

[0266] These examples are selected and described in order to explain the principles of the present disclosure and enable other persons skilled in the art to understand the various embodiments of the present disclosure and to best utilize the basic principles and various embodiments with various modifications adapted to the specific use contemplated. Therefore, it should be understood that the scope of the present disclosure is not limited to the specific examples of the disclosed embodiments, and modifications and other embodiments are intended to be included within the scope of the present disclosure.

Claims

1. A video encoding and decoding method, comprising: Exporting a variable for the current block, the variable indicating the allowed states of a plurality of merge modes in the merge mode set; generating a codeword set including a plurality of codewords based on the variable for signaling the merge mode, each codeword in the plurality of codewords corresponding to one of the plurality of merge modes, and the number of codewords in the codeword set and the maximum length of the codewords varying based on the variable; Determining, based on the signaled codeword, a merge mode selected from the merge mode set for the current block according to the codeword set; as well as The motion information of the current block is derived using the selected merge mode.

2. The method according to claim 1, wherein: Upon determining that the variable indicates that a first merge mode and at least one second merge mode in the set are allowed and the signaling indicates that the first merge mode is selected, determining the first merge mode for the current block.

3. The method according to claim 1, wherein when it is determined that the variable indicates that the first merge mode and one second merge mode in the set are allowed, other merge modes in the set are not allowed, and the signaling indicates that the second merge mode is not selected, the first merge mode is determined for the current block.

4. The method according to claim 1, wherein the merge mode set comprises: Regular merge mode (Regular), merge mode with motion vector difference (MMVD), sub-block based merge mode (Subblock), joint inter-frame and intra-frame prediction merge mode (CIIP) and triangle merge mode (Triangle); and the at least one merge mode includes one of the following or a combination thereof: MMVD, Subblock, CIIP and Triangle.

5. A device for video encoding and decoding, comprising: one or more processors; as well as a memory configured to store instructions executable by the one or more processors; Wherein, when executing the instructions, the one or more processors are configured to: Exporting a variable for the current block, the variable indicating the allowed states of a plurality of merge modes in the merge mode set; generating a codeword set including a plurality of codewords based on the variable for signaling the merge mode, each codeword in the plurality of codewords corresponding to one of the plurality of merge modes, and the number of codewords in the codeword set and the maximum length of the codewords varying based on the variable; Determining a merge mode selected from the merge mode set for the current block according to the codeword set based on the signaled codeword; and The motion information of the current block is derived using the selected merge mode.

6. The device according to claim 5, wherein: Upon determining that the variable indicates that a first merge mode and at least one second merge mode in the set are allowed and the signaling indicates that the first merge mode is selected, determining the first merge mode for the current block.

7. The apparatus of claim 5, wherein when it is determined that the variable indicates that the first merge mode and one second merge mode in the set are allowed, other merge modes in the set are not allowed, and the signaling indicates that the second merge mode is not selected, the first merge mode is determined for the current block.

8. The apparatus according to claim 5, wherein the merging mode set comprises: Regular merge mode (Regular), merge mode with motion vector difference (MMVD), sub-block based merge mode (Subblock), joint inter-frame and intra-frame prediction merge mode (CIIP) and triangle merge mode (Triangle); and the at least one merge mode includes one of the following or a combination thereof: MMVD, Subblock, CIIP and Triangle.

9. A non-transitory computer-readable storage medium comprising instructions stored therein, wherein: When one or more processors execute the instructions, the instructions cause the one or more processors to perform actions including: Exporting a variable for the current block, the variable indicating the allowed states of a plurality of merge modes in the merge mode set; generating a codeword set including a plurality of codewords based on the variable for signaling the merge mode, each codeword in the plurality of codewords corresponding to one of the plurality of merge modes, and the number of codewords in the codeword set and the maximum length of the codewords varying based on the variable; Determining, based on the signaled codeword, a merge mode selected from the merge mode set for the current block according to the codeword set; as well as The motion information of the current block is derived using the selected merge mode.

10. The non-transitory computer-readable storage medium of claim 9, wherein upon determining that the variable indicates that a first merge mode and at least one second merge mode in the set are allowed and the signaling indicates that the first merge mode is selected, the first merge mode is determined for the current block.

11. The non-transitory computer-readable storage medium of claim 9, wherein: When it is determined that the variable indicates that the first merge mode and one second merge mode in the set are allowed, other merge modes in the set are not allowed, and the signaling indicates that the second merge mode is not selected, the first merge mode is determined for the current block.

Images