Method and apparatus for video encoding and decoding using triangular partitioning

Abstract

A method and apparatus for filling motion vectors for blocks within a coding unit (CU) during video coding and decoding are provided. The method includes: dividing the CU into two prediction units (PUs), including a first PU and a second PU; respectively obtaining a first unidirectional prediction motion vector MV0 for the first PU and a second unidirectional prediction motion vector MV1 for the second PU; forming a motion vector MV01 based on the characteristics of MV0 and MV1; and filling corresponding blocks within the CU with a motion vector selected from the group consisting of MV0, MV1, and MV01.

Description

Technical Field

[0001] Embodiments of the present application generally relate to video encoding, decoding, and compression, and in particular, but not limited to, methods and apparatuses for motion compensation prediction using a triangular prediction unit (i.e., a special case of a geometric partitioning prediction unit) in video encoding and decoding. Background Art

[0002] Digital video is supported by various electronic devices, such as digital televisions, laptop or desktop computers, tablet computers, digital cameras, digital recording devices, digital media players, video game consoles, smart phones, video teleconferencing devices, video streaming devices, etc. These electronic devices transmit, receive, encode, decode, and / or store digital video data by implementing video compression / decompression. Digital video devices implement video encoding and decoding technologies, such as those described in standards defined by General Video Coding (VVC), Joint Exploration Test Model (JEM), MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264 / MPEG-4 Part 10, Advanced Video Coding (AVC), ITU-T H.265 / High Efficiency Video Coding (HEVC), and extensions of such standards.

[0003] Video encoding and decoding generally utilize prediction methods (e.g., inter-frame prediction, intra-frame prediction), which utilize the redundancy present in video images or sequences. An important goal of video encoding and decoding technologies is to compress video data into a form that uses a lower bit rate while avoiding or minimizing the degradation of video quality. As evolving video services become available, there is a need for encoding technologies with better encoding and decoding efficiency. The block partitioning schemes in each standard are also evolving.

[0004] Video compression generally includes performing spatial (intra-frame) prediction and / or temporal (inter-frame) prediction to reduce or remove the redundancy inherent in video data. For block-based video encoding and decoding, a video frame is divided into one or more stripes, each stripe having a plurality of video blocks, which may also be referred to as coding tree units (CTUs). Each CTU may contain one coding unit (CU) or be recursively divided into smaller CUs until a predefined minimum CU size is reached. Each CU (also referred to as a leaf CU) contains one or more transform units (TUs), and each CU also contains one or more prediction units (PUs). Each CU can be encoded and decoded in intra-frame, inter-frame, or IBC mode. Video blocks in the intra-coded (I) stripes of a video frame are encoded using spatial prediction with respect to reference samples in adjacent blocks within the same video frame. Video blocks in the inter-coded (P or B) stripes of a video frame can use spatial prediction with respect to reference samples in adjacent blocks within the same video frame or temporal prediction with respect to reference samples in other previous and / or future reference video frames.

[0005] A prediction block of a current video block to be coded / decoded is generated based on spatial domain or temporal domain prediction of previously coded reference blocks (e.g., neighboring blocks). The process of finding the reference blocks can be accomplished by a block matching algorithm. Residual data representing the pixel point differences between the current block to be coded / decoded and the prediction block is referred to as a residual block or prediction error. Inter-frame coded blocks are coded based on motion vectors pointing to reference blocks in a reference frame forming the prediction block and the residual block. The process of determining the motion vectors is generally referred to as motion estimation. Intra-frame coded blocks are coded based on intra-frame prediction modes and the residual block. For further compression, the residual block is transformed from the pixel domain to a transform domain, such as the frequency domain, to generate residual transform coefficients, and then the residual transform coefficients can be quantized. The quantized transform coefficients initially arranged in a two-dimensional array can be scanned to generate a one-dimensional vector of the transform coefficients, and then entropy coded into the video bitstream to achieve even further compression.

[0006] The coded video bitstream is then stored in a computer-readable storage medium (e.g., flash memory) for access by another electronic device having digital video capabilities or is directly transmitted to the electronic device wired or wirelessly. The electronic device then performs video decompression (which is a process opposite to the above video compression) by, for example, parsing the coded video bitstream to obtain semantic elements from the bitstream and reconstructing the digital video data from the coded video bitstream into its original format at least partially based on the semantic elements obtained from the bitstream, and presenting the reconstructed digital video data on a display of the electronic device.

[0007] As digital video quality goes from high definition to 4Kx2K and even 8Kx4K, the amount of video data to be coded / decoded grows exponentially. How to more efficiently code / decode video data while maintaining the image quality of the decoded video data is an ongoing challenge. [[ID=**10**]]

[0008] At the Joint Video Exploration Team (JVET) meetings, the JVET defined a first draft of Versatile Video Coding (VVC) and the coding method of the VVC Test Model 1 (VTM 1). It was decided to include a quadtree with a nested multi-type tree using binary and ternary partitioned coding block structures as an initial new coding / decoding feature of VVC. Since then, a reference software VTM has been developed during JVET meetings for implementing the coding method and the draft VVC decoding process. SUMMARY OF THE INVENTION

[0009] Generally, the present disclosure describes examples of techniques related to motion compensation prediction using geometric shape prediction units in video coding / decoding.

[0010] According to a first aspect of the present disclosure, a method for filling a motion vector for a block within a coding unit (CU) during video encoding and decoding is provided, comprising: dividing the CU into two prediction units (PUs), including a first PU and a second PU; obtaining a first unidirectional prediction motion vector MV0 for the first PU and a second unidirectional prediction motion vector MV1 for the second PU; forming a motion vector MV0 based on characteristics of MV0 and MV1. 01 ; and using MV0, MV1 and MV 01 The motion vectors selected from the group fill the corresponding blocks in the CU.

[0011] According to a second aspect of the present disclosure, a video encoding and decoding device with geometric partitioning is provided, comprising: one or more processors; and a memory for storing instructions executable by the one or more processors; wherein, when executing these instructions, the one or more processors are used to perform the aforementioned method for filling motion vectors for blocks within a coding unit (CU) during video encoding and decoding.

[0012] According to a third aspect of the present disclosure, a non-transitory computer-readable storage medium for video encoding and decoding is provided, which has a geometric partition storing computer-executable instructions. When executed by one or more computer processors, these instructions cause the one or more computer processors to perform the aforementioned method for filling motion vectors for blocks within a coding unit (CU) during video encoding and decoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] A more particular description of examples of the present disclosure will be presented by reference to specific examples shown in the accompanying drawings. Given that these drawings depict only some examples and are therefore not to be considered limiting of scope, these examples will be described and explained with additional specificity and detail through use of the accompanying drawings.

[0014] Figure 1 is a block diagram illustrating an exemplary video encoder according to some embodiments of the present disclosure.

[0015] Figure 2 is a block diagram illustrating an exemplary video decoder according to some embodiments of the present disclosure.

[0016] Figure 3 is a schematic diagram illustrating a quadtree plus binary tree (QTBT) structure according to some embodiments of the present disclosure.

[0017] Figure 4 is a diagram illustrating an example of a picture divided into CTUs according to some embodiments of the present disclosure.

[0018] Figure 5is a schematic diagram showing a multi-type tree partitioning pattern according to some embodiments of the present disclosure.

[0019] Figure 6 is a schematic diagram showing the positions of adjacent blocks according to some embodiments of the present disclosure.

[0020] Figure 7 is a schematic diagram showing the motion vector scaling of temporal merge candidates according to some embodiments of the present disclosure.

[0021] Figure 8 is a schematic diagram showing the candidate positions of temporal merge candidates according to some embodiments of the present disclosure.

[0022] Figure 9 is a schematic diagram showing the partitioning of a CU into triangular prediction units according to some embodiments of the present disclosure.

[0023] Figure 10 is a schematic diagram showing an example of unidirectional prediction motion vector (MV) selection in a triangular partitioning pattern according to some embodiments of the present disclosure.

[0024] Figure 11 is a schematic diagram showing an example of motion vector (MV) filling in a triangular prediction mode according to some embodiments of the present disclosure.

[0025] Figure 12A and 12B is a schematic diagram showing two examples of simplified motion vector filling in a triangular prediction mode according to some embodiments of the present disclosure.

[0026] Figure 12C and 12D is a schematic diagram showing another example of simplified motion vector filling in a triangular prediction mode according to some embodiments of the present disclosure.

[0027] Figure 13A is a schematic diagram showing another example of simplified motion vector filling in a triangular prediction mode according to some embodiments of the present disclosure.

[0028] Figure 13B is a schematic diagram showing another example of simplified motion vector filling in a triangular prediction mode according to some embodiments of the present disclosure.

[0029] Figure 14 is a block diagram showing an exemplary apparatus for video coding and decoding according to some embodiments of the present disclosure.

[0030] Figure 15is a flowchart showing an exemplary process of video coding and decoding for motion compensation prediction using a geometric prediction unit according to some embodiments of the present disclosure. Detailed Embodiments

[0031] Reference will now be made in detail to specific embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous non-limiting specific details are set forth in order to assist in understanding the subject matter presented herein. However, it will be apparent to one of ordinary skill in the art that various alternatives may be used without departing from the scope of the claims, and the subject matter may be practiced without these specific details. For example, it will be apparent to one of ordinary skill in the art that the subject matter presented herein may be implemented on a variety of types of electronic devices having digital video capabilities.

[0032] References throughout this specification to "one embodiment", "an embodiment", "an example", "some embodiments", "some examples" or similar language mean that a particular feature, structure, or characteristic described is included in at least one embodiment or example. Features, structures, elements, or characteristics described in connection with one or some embodiments are also applicable to other embodiments, unless otherwise stated.

[0033] Throughout this disclosure, the terms "first", "second", "third", etc. are used as terms only for referring to related elements, such as devices, components, compositions, steps, etc., and do not imply any spatial or temporal order, unless otherwise expressly stated. For example, "a first device" and "a second device" may refer to two separately formed devices, or two parts, components, or working states of the same device, and may be named arbitrarily.

[0034] The terms "if" or "when" as used herein may be understood in context to mean "in" or "in response to". These terms, if they appear in a claim, may not indicate that the related limitation or feature is conditional or optional.

[0035] The terms "module", "sub-module", "circuit", "sub-circuit", "circuitry", "sub-circuitry", "unit" or "sub-unit" may include a memory (shared, dedicated, or group) storing code or instructions executable by one or more processors. A module may include one or more circuits with or without stored code or instructions. The module or circuit may include one or more components connected directly or indirectly. These components may or may not be physically attached to each other or adjacent to each other.

[0036] A unit or module can be implemented purely by software, purely by hardware, or by a combination of hardware and software. For example, in a pure software implementation, a unit or module can include functionally related code blocks or software components that are directly or indirectly linked together to perform a specific function.

[0037] Figure 1 A block diagram showing an exemplary block-based hybrid video encoder 100 is presented. The hybrid video encoder 100 can be used in conjunction with many video coding and decoding standards that utilize block-based processing. In the encoder 100, a video frame is segmented into multiple video blocks for processing. For each given video block, a prediction is formed based on either an inter-frame prediction method or an intra-frame prediction method. In inter-frame prediction, one or more predicted values are formed through motion estimation and motion compensation based on pixel points from a previously reconstructed frame. In intra-frame prediction, a predicted value is formed based on the reconstructed pixel points in the current frame. Through mode decision, the best predicted value can be selected to predict the current block.

[0038] The prediction residual representing the difference between the current video block and its predicted value is sent to the transform circuitry 102. The transform coefficients are then sent from the transform circuitry 102 to the quantization circuitry 104 for entropy reduction. The quantized coefficients are then fed to the entropy coding / decoding circuitry 106 to generate a compressed video bitstream. As Figure 1 shown, prediction-related information 110 (such as video block partitioning information, motion vectors, reference picture indices, and intra-frame prediction modes) from the inter-frame prediction circuitry and / or the intra-frame prediction circuitry 112 is also fed through the entropy coding / decoding circuitry 106 and saved as the compressed video bitstream 114.

[0039] In the encoder 100, circuitry related to the decoder is also required to reconstruct pixel points for prediction purposes. First, the prediction residual is reconstructed through the inverse quantization circuitry 116 and the inverse transform circuitry 118. The reconstructed prediction residual is combined with the block predicted value 120 to generate the unfiltered reconstructed pixel points of the current video block. Spatial prediction (or "intra-frame prediction") uses pixel points from samples (referred to as reference samples) of already coded and decoded adjacent blocks in the same video frame as the current video block to predict the current video block.

[0040] Temporal prediction (also known as "inter - frame prediction") uses the reconstructed pixel values from already decoded video pictures to predict the current video block. Temporal prediction reduces the temporal redundancy inherent in the video signal. The temporal prediction signal for a given coding unit (CU) or coding block is typically signaled by one or more motion vectors (MVs), which indicate the amount and direction of motion between the current CU and its temporal reference picture. Further, if multiple reference pictures are supported, an additional reference picture index is sent to identify which reference picture in the reference picture buffer the temporal prediction signal is from.

[0041] After performing spatial and / or temporal prediction, the intra - / inter - frame mode decision circuitry 121 in the encoder 100 selects the best prediction mode, e.g., based on rate - distortion optimization methods. Then the block prediction value 120 is subtracted from the current video block; and the resulting prediction residual is decorrelated using the transform circuitry 102 and quantization circuitry 104. The resulting quantized residual coefficients are inverse - quantized by the inverse quantization circuitry 116 and inverse - transformed by the inverse transform circuitry 118 to form the reconstructed residual, which is then added back to the prediction block to form the reconstructed signal for the CU. Further loop filtering 115 (such as de - block filter, sample - adaptive offset (SAO), and / or adaptive loop filter (ALF)) can be applied to the reconstructed CU before the reconstructed CU is placed in the reference picture buffer 117 of the picture buffer and used for coding future video blocks. To form the output video bitstream 114, the coding mode (inter - frame or intra - frame), prediction mode information, motion information, and quantized residual coefficients are all sent to the entropy coding unit 106 to be further compressed and packetized to form the bitstream.

[0042] For example, the de - block filter can be used in the current versions of AVC, HEVC, and VVC. In HEVC, an additional loop filter called SAO (sample - adaptive offset) is defined to further improve coding efficiency. In the current version of the VVC standard, another loop filter called ALF (adaptive loop filter) is being actively studied and it is very likely to be included in the final standard.

[0043] These loop filter operations are optional. Performing these operations helps to improve coding efficiency and visual quality. They can also be turned off as a decision presented by the encoder 100 to save computational complexity.

[0044] It should be noted that intra - frame prediction is typically based on unfiltered reconstructed pixels, while inter - frame prediction is based on filtered reconstructed pixels if these filter options are turned on by the encoder 100.

[0045] Figure 2FIG. 0 is a block diagram illustrating an exemplary block-based video decoder 200, which can be used in conjunction with many video coding and decoding standards. The decoder 200 is similar to the reconstruction-related part of the encoder 100 residing in Figure 1 In the decoder 200, first, the incoming video bitstream 201 is decoded by entropy decoding 202 to derive quantization coefficient levels and prediction-related information. Then, the quantization coefficient levels are processed by inverse quantization 204 and inverse transform 206 to obtain the reconstructed prediction residuals. The block prediction value mechanism implemented in the intra / inter mode selector 212 is configured to perform intra prediction 208 or motion compensation 210 based on the decoded prediction information. A set of unfiltered reconstructed pixels is obtained by adding the reconstructed prediction residuals from the inverse transform 206 and the prediction output generated by the block prediction value mechanism using an adder 214.

[0046] The reconstructed blocks may also be passed through a loop filter 209 before being stored in the picture buffer 213, which serves as a reference picture store. The reconstructed video in the picture buffer 213 can be sent to drive a display device and for predicting future video blocks. When the loop filter 209 is turned on, a filtering operation is performed on these reconstructed pixel points to derive the final reconstructed video output 222.

[0047] The aforementioned video coding / decoding standards, such as VVC, JEM, HEVC, MPEG-4, Part 10, are conceptually similar. For example, they all use block-based processing. The block partitioning schemes in some of the standards will be described in detail below.

[0048] High Efficiency Video Codec (HEVC)

[0049] HEVC is based on a hybrid block-based motion compensation transform coding architecture. The basic unit for compression is called a coding tree unit (CTU). For the 4:2:0 chroma format, the maximum CTU size is defined as a block of up to 64 by 64 luma pixels and two 32 by 32 chroma pixels. Each CTU may contain one coding unit (CU) or be recursively divided into four smaller CUs until a predefined minimum CU size is reached. Each CU (also called a leaf CU) contains one or more prediction units (PUs) and a transform unit (TU) tree.

[0050] Generally, except for monochromatic content, a CTU may include one luminance coding tree block (CTB) and two corresponding chrominance CTBs; a CU may include one luminance coding block (CB) and two corresponding chrominance CBs; a PU may include one luminance prediction block (PB) and two corresponding chrominance PBs; and a TU may include one luminance transform block (TB) and two corresponding chrominance TBs. However, exceptions may occur because for both luminance and chrominance, the minimum TB size is 4×4 (i.e., 2×2 chrominance TBs are not supported for the 4:2:0 color format) and each intra-chrominance CB always has only one intra-chrominance PB regardless of the number of intra-luminance PBs in the corresponding intra-luminance CB.

[0051] For an intra-CU, the luminance CB may be predicted by one or four luminance PBs, and each of the two chrominance CBs is always predicted by one chrominance PB, where each luminance PB has one intra-luminance prediction mode, and the two chrominance PBs share one intra-chrominance prediction mode. In addition, for an intra-CU, the TB size cannot be larger than the PB size. In each PB, intra prediction is applied to predict the samples of each TB inside the PB from the neighboring reconstructed samples of the TB. For each PB, in addition to 33 directional intra prediction modes, the DC mode and the planar mode are also supported to predict flat regions and gradually changing regions respectively.

[0052] For each inter-PU, one of three prediction modes including inter, skip, and merge can be selected. Generally, a motion vector competition (MVC) scheme is introduced to select a motion candidate from a given candidate set including spatial motion candidates and temporal motion candidates. Multiple references for motion estimation enable finding the best reference in 2 possible reconstructed reference picture lists (i.e., list 0 and list 1). For the inter mode (referred to as the AMVP mode, where AMVP stands for advanced motion vector prediction), an inter prediction indicator (list 0, list 1, or bi-prediction), a reference index, a motion candidate index, a motion vector difference (MVD), and a prediction residual are sent. For the skip mode and the merge mode, only the merge index is sent, and the current PU inherits the inter prediction indicator, the reference index, and the motion vector from the neighboring PU referenced by the encoded merge index. In the case of a skipped encoded CU, the residual signal is also omitted.

[0053] Joint Exploration Test Model (JEM)

[0054] The Joint Exploration Test Model (JEM) is built on top of the HEVC Test Model. The basic encoding and decoding processes of HEVC remain unchanged in JEM; however, the design elements of the most important modules (including modules for block structure, intra and inter prediction, residual transform, loop filter, and entropy coding and decoding) are slightly modified, and additional coding tools are added. The following new coding features are included in JEM.

[0055] In HEVC, the CTU is divided into CUs by using a quadtree structure represented as an encoding tree to adapt to various local characteristics. The decision of whether to use inter-picture (temporal) prediction or intra-picture (spatial) prediction to encode a picture region is made at the CU level. Each CU can be further divided into one, two, or four PUs according to the PU partition type. Inside a PU, the same prediction process is applied, and relevant information is sent to the decoder based on the PU. After obtaining the residual block by applying the prediction process based on the PU partition type, the CU can be divided into transform units (TUs) according to another quadtree structure similar to the encoding tree of the CU. One of the key features of the HEVC structure is that it has multiple partition concepts including CUs, PUs, and TUs.

[0056] Figure 3 is a schematic diagram showing a quadtree plus binary tree (QTBT) structure according to some embodiments of the present disclosure.

[0057] The QTBT structure removes the concept of multiple partition types, that is, it removes the distinction between the CU, PU, and TU concepts, and supports greater flexibility in the CU partition shape. In this QTBT block structure, the CU can have a square or rectangular shape. As Figure 3 shown, the coding tree unit (CTU) is first divided by a quadtree structure. The quadtree leaf nodes can be further divided by a binary tree structure. There are two partition types in the binary tree partition: symmetric horizontal partition and symmetric vertical partition. The binary tree leaf nodes are called coding units (CUs), and this segmentation is used for prediction and transformation processing without further partitioning. This means that the CUs, PUs, and TUs have the same block size in the QTBT coding block structure. In JEM, a CU sometimes consists of coding blocks (CBs) of different color components. For example, in the case of P and B slices in the 4:2:0 chroma format, a CU contains one luminance CB and two chroma CBs, and sometimes consists of a CB of a single component. For example, in the case of I slices, a CU contains only one luminance CB or only two chroma CBs.

[0058] The following parameters are defined for this QTBT partitioning scheme.

[0059] - CTUsize: The size of the root node of the quadtree, which is the same as the concept in HEVC;

[0060] - MinQTSize: The minimum allowable size of the quadtree leaf nodes;

[0061] - MaxBTSize: The maximum allowable size of the binary tree root nodes;

[0062] - MaxBTDepth: The maximum allowable depth of the binary tree;

[0063] -MinBTSize: The minimum allowable size of a binary tree leaf node.

[0064] In an example of the QTBT partitioning structure, the CTU size is set to 128×128 luma samples with two corresponding 64×64 chroma sample blocks (with 4:2:0 chroma format), MinQTSize is set to 16×16, MaxBTSize is set to 64×64, MinBTSize (width and height) is set to 4×4, and MaxBTDepth is set to 4. First, quadtree partitioning is applied to the CTU to generate quadtree leaf nodes. The sizes of these quadtree leaf nodes can range from 16×16 (i.e., MinQTSize) to 128×128 (i.e., the CTU size). If a quadtree leaf node is 128×128, it will no longer be further partitioned by the binary tree because its size exceeds MaxBTSize (i.e., 64×64). Otherwise, the quadtree leaf node can be further partitioned by the binary tree. Therefore, the quadtree leaf node is also the root node of the binary tree, and the binary tree depth is 0. When the binary tree depth reaches MaxBTDepth (i.e., 4), further partitioning is no longer considered. When a binary tree node has a width equal to MinBTSize (i.e., 4), further horizontal partitioning is no longer considered. Similarly, when a binary tree node has a height equal to MinBTSize, further vertical partitioning is no longer considered. The leaf nodes of the binary tree are further subjected to prediction and transformation processing without further partitioning. In JEM, the maximum CTU size is 256×256 luma samples.

[0065] An example of block partitioning using this QTBT scheme and the corresponding tree representation are shown in Figure 3 as follows. Solid lines represent quadtree partitioning, and dashed lines represent binary tree partitioning. As Figure 3As shown, first, the coding tree unit (CTU) 400 is segmented by a quadtree structure, and three of the four quadtree leaf nodes 402, 404, 406, 408 are further divided by a quadtree structure or a binary tree structure. For example, the quadtree leaf node 406 is further divided by quadtree partitioning; the quadtree leaf node 404 is further divided into two leaf nodes 404a and 404b by binary tree partitioning; and the quadtree leaf node 402 is also further divided by binary tree partitioning. In each partitioning (i.e., non-leaf) node of the binary tree, a flag is marked to indicate which partitioning type (i.e., horizontal or vertical) is used, where 0 indicates horizontal partitioning and 1 indicates vertical partitioning. For example, for the quadtree leaf node 404, it is marked as 0 to indicate horizontal partitioning, and for the quadtree leaf node 402, it is marked as 1 to indicate vertical partitioning. For quadtree partitioning, there is no need to indicate the partitioning type because quadtree partitioning always divides the block horizontally and vertically to produce 4 sub-blocks of equal size.

[0066] In addition, the QTBT scheme supports the ability for luminance and chrominance to have separate QTBT structures. Currently, for P and B slices, the luminance and chrominance CTBs in a CTU share the same QTBT structure. However, for I slices, the luminance CTB is segmented into CUs by the QTBT structure, and the chrominance CTB is segmented into chrominance CUs by another QTBT structure. This means that the CUs in I slices are composed of coding blocks of the luminance component or coding blocks of the two chrominance components, and the CUs in P or B slices are composed of coding blocks of all three color components.

[0067] Versatile Video Codec (VVC)

[0068] At the Joint Video Experts Team (JVET) meeting, JVET defined the first draft of Versatile Video Coding (VVC) and the coding method of VVC Test Model 1 (VTM1). The decisions included using a quadtree with a nested multi-type tree of binary and ternary partitioning coding block structures as the initial new coding feature of VVC.

[0069] In VVC, the picture partitioning structure divides the input video into blocks called coding tree units (CTUs). The CTUs are divided into coding units (CUs) using a quadtree with a nested multi-type tree structure, where the leaf coding units (CUs) define regions that share the same prediction mode (e.g., intra or inter). Here, the term "unit" defines a region of the image covering all components; the term "block" is used to define a region covering a specific component (e.g., luminance), and can be different in spatial position when considering chrominance sampling formats such as 4:2:0.

[0070] Divide the image into CTUs

[0071] Figure 4 It is a schematic diagram showing an example of a picture divided into CTUs according to some embodiments of the present disclosure.

[0072] In VVC, a picture is divided into a sequence of CTUs, and the CTU concept is the same as the CTU concept in HEVC. For a picture with three sample arrays, a CTU consists of an N×N block of luma samples and two corresponding chroma sample blocks. Figure 4 An example of a picture 400 divided into CTU 402 is shown.

[0073] The maximum allowed size of the luma block in a CTU is specified as 128×128 (although the maximum size of the luma transform block is 64×64).

[0074] Use tree structure to divide CTU

[0075] Figure 5 It is a schematic diagram showing a multi-type tree partitioning pattern according to some embodiments of the present disclosure.

[0076] In HEVC, a CTU is divided into CUs using a quadtree structure represented as an encoding tree to adapt to various local characteristics. The decision of whether to use inter-picture (temporal) prediction or intra-picture (spatial) prediction to encode a picture region is made at the leaf CU level. Each leaf CU can be further divided into one, two, or four PUs according to the PU partitioning type. Inside a PU, the same prediction process is applied, and relevant information is sent to the decoder based on the PU. After obtaining the residual block by applying the prediction process based on the PU partitioning type, the leaf CU can be divided into transform units (TUs) according to another quadtree structure similar to the encoding tree of the CU. One of the key features of the HEVC structure is that it has multiple partitioning concepts, including CUs, PUs, and TUs.

[0077] In VVC, a quadtree with a nested multi-type tree using a binary and ternary partitioning structure is used to replace the concept of multiple partitioning unit types, that is, it removes the distinction between the CU, PU, and TU concepts (except for CUs with sizes too large for the maximum transform length which require this distinction), and supports greater flexibility for the CU split shape. In the encoding tree structure, a CU can have a square or rectangular shape. First, the coding tree unit (CTU) is divided by a quadtree (i.e., a four-way tree) structure. Then, the quadtree leaf nodes can be further divided by a multi-type tree structure. As Figure 5As shown, there are four partition types in the multi-type tree structure: vertical binary partition 502 (SPLIT_BT_VER), horizontal binary partition 504 (SPLIT_BT_HOR), vertical ternary partition 506 (SPLIT_TT_VER), and horizontal ternary partition 508 (SPLIT_TT_HOR). The multi-type tree leaf nodes are called coding units (CUs), and unless the CU is too large for the maximum transform length, this partition is used for prediction and transform processing without any further splitting. This means that in most cases, the CUs, PUs, and TUs have the same block size in a quadtree with a nested multi-type tree coding block structure. Exceptions occur when the maximum supported transform length is less than the width or height of the color components of the CU. In VTM1, a CU consists of coding blocks (CBs) of different color components. For example, a CU contains one luma CB and two chroma CBs (unless the video is monochrome, i.e., has only one color component).

[0078] Divide the CU into multiple prediction units

[0079] In VVC, for each CU segmented based on the structure described above, the prediction of the block content can be performed on the entire CU block or in the sub-block manner explained in the following paragraphs. The operation unit of this prediction is called the prediction unit (or PU).

[0080] In the case of intra prediction (or intra-frame prediction), the size of the PU is usually equal to the size of the CU. In other words, the prediction is performed on the entire CU block. For inter prediction (or prediction between frames), the size of the PU can be equal to or less than the size of the CU. In other words, there are cases where a CU can be divided into multiple PUs for prediction.

[0081] Some examples where the PU size is less than the CU size include the affine prediction mode, the advanced temporal motion vector prediction (ATMVP) mode, and the triangular prediction mode, etc.

[0082] In the affine prediction mode, a CU can be split into multiple 4×4 PUs for prediction. Motion vectors can be derived for each 4×4 PU and the 4×4 PUs can be motion compensated accordingly. In the ATMVP mode, a CU can be split into one or more 8×8 PUs for prediction. Motion vectors are derived for each 8×8 PU and the 8×8 PUs can be motion compensated accordingly. In the triangular prediction mode, a CU can be divided into two triangular prediction units. Motion vectors are derived for each PU and motion compensation is performed accordingly. Inter prediction supports the triangular prediction mode. More details of the triangular prediction mode are described below.

[0083] Normal merge mode motion vector candidate list

[0084] According to the current VVC, in the normal merge mode of predicting the entire CU without partitioning it into more than one PU, a process different from the triangular prediction mode is used to construct the motion vector candidate list or the merge candidate list.

[0085] First, based on the motion vectors from adjacent blocks, spatial motion vector candidates are selected. Figure 6 A schematic diagram showing the positions of spatial merge candidates according to some embodiments of the present disclosure is as Figure 6 shown. In the derivation of spatial merge candidates for the current block 602, up to four merge candidates are selected from the candidates located at the positions as Figure 6 shown. These candidates are selected in a certain order. An exemplary derivation order is A1 → B1 → B0 → A0 → (B2). Position B2 is considered only when any of the PUs at positions A1, B1, B0, A0 is unavailable or intra-coded. It should be noted that other different orders can also be used. For example, in a later stage of VVC, the order is changed to B1 → A1 → B0 → A0 (B2).

[0086] Next, temporal merge candidates are derived. In the derivation of temporal merge candidates, the scaled motion vectors are derived based on the co-located PUs belonging to the picture with the smallest picture order count (POC) difference from the current picture within a given reference picture list. The reference picture list to be used for deriving co-located PUs is explicitly indicated in the slice header. Figure 7 A schematic diagram showing the motion vector scaling for temporal merge candidates according to some embodiments of the present disclosure is as Figure 7 shown. To obtain the scaled motion vector of the temporal merge candidate, the motion vector of the co-located PU col_PU is used to scale the scaled motion vector of the temporal merge candidate using the POC distances tb and td, where tb is defined as the POC difference between the reference picture curr_ref of the current picture and the current picture curr_pic, and td is defined as the POC difference between the reference picture col_ref of the co-located picture and the co-located picture col_pic. The reference picture index of the temporal merge candidate is set to zero. The actual implementation of this scaling process is described in the HEVC specification draft. For B slices, two motion vectors are obtained and combined, one for reference picture list 0 and the other for reference picture list 1, to form a bi-predictive merge candidate.

[0087] Figure 8 A schematic diagram showing the candidate positions for temporal merge candidates according to some embodiments of the present disclosure.

[0088] As Figure 8As shown, the position of the co-located PU is selected between the two candidate positions C3 and H. If the PU at position H is not available, or is intra-coded, or is outside the current CTU, position C3 is used to derive the temporal merge candidate. Otherwise, position H is used to derive the temporal merge candidate.

[0089] After inserting both spatial and temporal motion vectors into the merge candidate list as described above, history-based merge candidates are added. These so-called history-based merge candidates include motion vectors from previously coded CUs, which are maintained in a separate motion vector list and managed based on certain rules.

[0090] After inserting the history-based candidates, if the merge candidate list is not full, pairwise average motion vector candidates are further added to it. As the name suggests, this type of candidate is constructed by averaging the candidates already in the current list. More specifically, based on a certain order or rule, two candidates are taken from the merge candidate list at a time and their average motion vector is appended to the current list.

[0091] After inserting the pairwise average motion vectors, if the merge candidate list is still not full, zero motion vectors will be added to fill the list.

[0092] Triangle prediction mode (or triangle partitioning mode)

[0093] Figure 9 FIG. 2 is a schematic diagram illustrating dividing a CU into triangular prediction units according to some embodiments of the present disclosure.

[0094] The concept of triangular prediction mode is to introduce triangular partitioning for motion compensation prediction. Triangular prediction mode can also be called triangular prediction unit mode or triangular partition mode. Figure 9 As shown, CU 902 or 904 is divided into two triangular prediction unit partitions 0 and 1 in a diagonal direction or an anti-diagonal direction (i.e., divided from the upper left corner to the lower right corner as shown in CU 902 or from the upper right corner to the lower left corner as shown in CU 904). Each triangular prediction unit in the CU uses its own unidirectional prediction motion vector and reference frame index derived directly and / or indirectly from the conventional merge mode motion vector candidate list for inter-frame prediction. After the two triangular prediction units are predicted, the diagonal edges separating the two triangular prediction units are weighted. Then, the transform and quantization process is applied to the entire CU 902 or 904. It should be noted that this mode is only applied to the skip and merge modes in the current VVC. Although in Figure 9 The CU is shown as a square block, but the triangular prediction mode can also be applied to CUs with non-square (ie rectangular) shapes.

[0095] Unidirectional prediction motion vector derivation

[0096] Figure 10 is a schematic diagram showing unidirectional prediction motion vector selection for a triangular partitioning pattern according to some embodiments of the present disclosure.

[0097] In some examples, as described in the previous section "Conventional Merge Mode Motion Vector Candidate List", the unidirectional prediction motion vectors for each triangular partition are directly derived from the merge candidate list formed for the conventional merge mode. Given a merge candidate index, the candidate can be located from the merge candidate list. Then for this candidate, the list X motion vectors whose X is equal to the parity value (p) of the merge candidate index value are used as the unidirectional prediction motion vectors for the triangular partitioning pattern. These motion vectors are marked with "x" in Figure 10 In the case where the corresponding table X (or table p) motion vector does not exist, the table (1 - X) (or table (1 - p)) motion vector of the same candidate is used as the unidirectional prediction motion vector for the triangular partition pattern.

[0098] Boundary Weighted Process

[0099] A prediction value is derived for each triangular PU based on the motion vector of each triangular PU. It should be noted that the derived prediction value covers a larger area than the actual triangular PU, so there is an overlapping area of two prediction values along the shared diagonal edge of these two triangular PUs (for example, see partitions 0 and 1 of CU 902 or 904). A weighting process is applied to the diagonal edge area between these two prediction values to derive the final prediction for this CU. Currently, the weighting factors for luma and chroma samples are {7 / 8, 6 / 8, 5 / 8, 4 / 8, 3 / 8, 2 / 8, 1 / 8} and {6 / 8, 4 / 8, 2 / 8}, respectively.

[0100] Triangular prediction mode semantics and signaling

[0101] Here, when a CU is encoded or decoded in skip mode or merge mode, a triangular prediction flag is used to identify the triangular prediction mode. For a given CU, if the binary value of the triangular prediction flag is 1, it means that the corresponding CU is encoded or decoded using the triangular prediction mode. Otherwise, a prediction mode different from the triangular prediction mode is used to encode or decode this CU.

[0102] First, before identifying the triangular prediction flag, identify the triangular prediction tool enable / disable flag in the sequence parameter set (or SPS). As long as the triangular prediction tool enable / disable flag is true, identify the triangular prediction flag at the CU level. Second, enable the triangular prediction tool only in B slices. Therefore, only in B slices is the triangular prediction flag identified at the CU level. Third, identify the triangular prediction mode only for CUs with a size equal to or greater than a certain threshold (such as 64). If the size of the CU is less than the threshold, the triangular prediction flag is not identified. Fourth, the triangular prediction mode is allowed for a CU only when the CU is not decoded or encoded in the regular merge mode, or the merge mode with motion vector difference (MMVD), or the sub-block merge mode, or the combined inter and intra prediction (CIIP) mode. For CUs that meet these conditions, apply the triangular partitioning mode.

[0103] If the triangular prediction mode is used, also identify the triangular partition orientation flag to indicate whether the partition is oriented from the upper left corner to the lower right corner (such as CU 902) or from the upper right corner to the lower left corner (such as CU 904).

[0104] When the triangular prediction flag is identified, use a context-adaptive binary arithmetic coding (CABAC) entropy encoder with certain contexts to identify the triangular prediction flag. These contexts are formed based on the triangular prediction flag values of the top block and / or the left block of the current CU.

[0105] To encode or decode (i.e., encoder or decoder) the triangular prediction flag of the current block (or current CU), derive the triangular prediction flags from the top block and / or the left block (or CU) and add their values. This results in three possible contexts corresponding to the following cases:

[0106] 1) The triangular prediction flags of both the left block and the top block are 0;

[0107] 2) The triangular prediction flags of both the left block and the top block are 1; and

[0108] 3) Other cases.

[0109] Maintain separate probabilities for each of the three contexts. Once the context value for the current block is determined, use the CABAC probability model corresponding to that context value to encode or decode the triangular prediction flag of the current block.

[0110] When the triangular prediction mode is used for a CU, two merge index values are identified to indicate the merge index values for the first and second uni-prediction merge candidates for triangular prediction respectively. These two merge index values are used to locate two merge candidates from the above-mentioned uni-prediction motion vector candidate list for the first and second partitions respectively. For triangular prediction, it is required that these two merge index values be different so that the two prediction values of the two triangular partitions can be different from each other. Therefore, the first merge index value is directly identified. To identify the second merge index value, if it is less than the first merge index value, the second merge index value is directly identified. Otherwise, the second merge index value is subtracted by 1 before being identified by the decoder. On the decoder side, the first merge index is directly decoded and used. To decode the second merge index value, first, a value represented as "idx" is decoded from the CABAC engine. If idx is less than the first merge index value, the second merge index value is equal to the value of idx. Otherwise, the second merge index value is equal to (idx + 1).

[0111] Block motion vector filling in triangular prediction mode

[0112] According to the VVC draft standard, if a CU is encoded and decoded in the triangular prediction mode, different motion vectors are used to fill (i.e., store in) the motion vector buffer of the 4×4 sub-blocks within the CU according to the sub-block position. Performing such motion vector filling is for motion vector prediction when encoding and decoding other CUs that may be adjacent to the current CU in space or time. More specifically, the uni-prediction motion vector represented as MV0 of the first triangular partition is used to fill the sub-blocks within the first triangular partition (i.e., partition 0); the uni-prediction motion vector represented as MV1 of the second triangular partition is used to fill the sub-blocks within the second triangular partition (i.e., partition 1); and the motion vector formed from MV0 and MV1 is used to fill the 4×4 sub-blocks located on the edge of the diagonal partition. According to the characteristics of MV0 and MV1, the formed motion vector is represented by MV 01 and can be uni-predicted or bi-predicted. If MV0 and MV1 come from different reference lists, the two uni-prediction motion vectors are directly combined to form a bi-prediction motion vector. If they refer to the same reference list, the reference picture of MV1 is checked to see if it exists in other reference lists. If so, MV1 is converted to refer to the same reference picture in another reference list, and then it is combined with MV0 to form the bi-prediction motion vector MV01. If the reference picture of MV1 does not exist in another reference list, the reference picture of MV0 is checked to see if it exists in another reference list. If so, MV0 is converted to refer to the same reference picture in another reference list, and then it is combined with MV1 to form the bi-prediction motion vector MV 01If there is no reference picture of MV0 in another reference list, MV0 is directly used as the formed motion vector. In this case, the formed motion vector is a unidirectional prediction motion vector.

[0113] As Figure 11 shown, a CU with a size of 32×32 is encoded and decoded in the triangular prediction mode. In this case, the 4×4 blocks within partition 0 are filled with the unidirectional prediction motion vectors of partition 0; the 4×4 blocks within partition 1 are filled with the unidirectional prediction motion vectors of partition 1; and the 4×4 blocks located on this diagonal edge line (represented by the solid dividing line squares) are filled with the aforementioned formed motion vector MV 01 . It should be noted that, in the above process, the motion vector used to fill the 4×4 block may be the same as or different from the motion vector for the inter prediction for forming this 4×4 block. Although the sub-blocks of the present disclosure have a size of 4×4 pixels, the sub-block size can be adapted to 2×2, 8×8 or other sizes, and the disclosed method can be adapted accordingly.

[0114] Unidirectional prediction motion vector derivation with limited motion vector pruning

[0115] In some examples, the unidirectional prediction motion vectors for each triangular partition are directly derived from the merge candidate list formed for the regular merge mode, as described in the previous section "Motion Vector Candidate List for Regular Merge Mode". However, as Figure 10 shown, the number of selectable motion vectors can be limited for triangular partitions. For example, when there are motion vectors marked with "x" in the figure, the corresponding unidirectional prediction motion vectors from other reference lists of the same merge candidate (i.e., the motion vectors not marked with "x" in the figure) will have no chance to be used for triangular prediction. At the same time, it often occurs that some motion vectors marked with "x" may be the same as each other, which may further limit the diversity of motion vectors and sacrifice the coding and decoding efficiency.

[0116] Another problem with triangular prediction is related to its current block motion vector filling method, as described in the previous section "Block Motion Vector Filling in Triangular Prediction Mode", because the corresponding operations when filling the motion vector are not so simple. More implementation-friendly methods may be better.

[0117] According to some examples of the present disclosure, given two merge index values in a given triangular prediction mode, these two unidirectional prediction motion vectors can be located based on the process described in the previous section "Derivation of Unidirectional Prediction Motion Vector". In addition, a motion vector pruning operation can also be performed. If these two unidirectional prediction motion vectors derived for partition 0 and partition 1 respectively are the same, the unidirectional prediction motion vectors corresponding to these two unidirectional prediction motion vectors from another reference list (if any) can be alternatively used.

[0118] The previously disclosed examples can be implemented in different ways. Suppose the two uni-directional predicted motion vectors located according to the process described in the "Uni-directional Prediction Motion Vector Derivation" section are MV0 and MV1 for triangular partition 0 and partition 1 respectively. In one example, if MV1 is the same as MV0, then the corresponding motion vector that shares the same merge index as MV1 but from another reference list (if it exists) is alternatively used for partition 1. If it does not exist, then MV1 is still used. In another example, if MV1 is the same as MV0, then the corresponding motion vector that shares the same merge index as MV1 but from another reference list (if it exists) is alternatively used. If it does not exist or is the same as MV0, then MV1 is still used. In this case, the corresponding motion vector that shares the same merge index as MV0 but from another reference list (if it exists) is used for partition 0. If it does not exist, then MV0 is still used for partition 0.

[0119] In the above description, the check and processing order of partition numbers 0 and 1 together with their respective MV0 and MV1 are relative. Therefore, their check and processing order can be interchanged in the specification, and the resulting method still falls within the same spirit of the present disclosure. For example, instead of performing the pruning operation on MV1 relative to MV0 as described in the above example, the pruning operation on MV0 relative to MV1 can also be performed first.

[0120] Based on these disclosed examples, different methods can be used to determine whether two uni-directional predicted motion vectors are the same. In one example, when the codec device determines that the two vectors have the same X and Y components and the same POC (i.e., Picture Order Count) of their reference pictures, the two uni-directional predicted motion vectors are considered to be the same. The X and Y components of the motion vector respectively represent the relative horizontal and vertical offset values from the current block to its corresponding reference block. In another example, when the codec device determines that the two vectors have the same X and Y components, the same reference list, and the same reference picture index, the two uni-directional predicted motion vectors are considered to be the same. In yet another example, when the codec device determines that the two vectors have the same X and Y components, the two uni-directional predicted motion vectors are considered to be the same regardless of their reference list or reference picture index. Here, the codec device can be an electronic device having a chip for encoding or decoding video data.

[0121] Through the disclosed method as described above, more motion vectors can be selected and used for triangular prediction without additional signaling overhead. In this way, the codec efficiency is improved while the complexity of the associated motion vector pruning operation is limited.

[0122] Simplified block motion vector filling

[0123] According to some examples of the present disclosure, the block motion vector filling operation does not have to follow the process described in the section "Block Motion Vector Filling in Triangular Prediction Mode", but instead some simplified schemes can be used. In the following description of the present disclosure, the motion vectors for triangular partitions 0 and 1 are denoted as MV0 and MV1 respectively; the motion vector formed based on the process described in the section "Block Motion Vector Filling in Triangular Prediction Mode" is denoted as MV 01 which represents the motion vector formed by MV0 and MV1. As described above, MV 01 can be a bi - directional prediction or uni - directional prediction motion vector.

[0124] In an example of the present disclosure, the formed motion vector MV 01 is used to fill each 4×4 block in the current CU, rather than filling these 4×4 blocks with different motion vectors.

[0125] In another example of the present disclosure, the uni - directional prediction motion vector associated with the triangular partition located at the bottom of the CU is used to fill each 4×4 block in the current CU, rather than filling these 4×4 blocks with different motion vectors. Figure 9 An example is shown in

[0126] where partition 1 is the triangular partition located at the bottom, and its motion vector MV1 is used to fill each 4×4 block in CU 902 or 904. 01 In yet another example of the present disclosure, the uni - directional prediction motion vector associated with the triangular partition located at the bottom of the CU is used to fill each 4×4 block in the current CU except for the two 4×4 blocks at both ends located on the diagonal partition boundary. For these two 4×4 blocks at both ends of the diagonal partition boundary, the formed motion vector MV Figure 12A and Figure 12B are shown, where only the two 4×4 blocks with solid boundary lines are filled with the formed motion vector MV 01 More specifically, as shown in Figure 12A when the current CU is partitioned from the upper left corner to the lower right corner, the upper left 4×4 block and the lower right 4×4 block are filled with the formed motion vector MV 01 When the current CU is partitioned from the upper right corner to the lower left corner, as shown in Figure 12B the upper right 4×4 block and the lower left 4×4 block are filled with the formed motion vector MV 01 In another example of the present disclosure, the formed motion vector MV

[0127] is used to fill with the formed motion vector MV 01Fill each block in the current CU except for two 4×4 corner blocks located at two corners of partition 1 and partition 2 respectively. These two corner blocks are not located on the diagonal partition boundary. An example is shown in Figure 12C and Figure 12D , where these two corner blocks are represented by solid boundary lines. According to this embodiment of the present disclosure, the two corner blocks are filled with corresponding unidirectional prediction motion vectors of the triangular partition of these two corner blocks. More specifically, as shown in Figure 12C , when the current CU is divided from the upper left corner to the lower right corner, the upper right 4×4 block and the lower left 4×4 block are filled with MV0 and MV1. When the current CU is divided from the upper right corner to the lower left corner, as shown in Figure 12D , the upper left 4×4 block and the lower right 4×4 block are filled with MV0 and MV1 respectively. Although the example in Figure 11-1 3 uses sub-blocks with a size of 4×4 pixels, this method can be applied to different sub-block sizes, such as 2×2, 8×8 or other sizes.

[0128] In another example of the present disclosure, the block motion vector filling operation still follows the process described in the "Block Motion Vector Filling in Triangular Prediction Mode" section, except for the 4×4 blocks located on the diagonal boundary line (the boundary line in Figure 11 ). For the two 4×4 blocks located at two corners of the diagonal partition boundary, the formed motion vector MV01 is used to fill them. For the other 4×4 blocks located on the diagonal boundary line, the unidirectional prediction motion vector associated with the triangular partition located at the bottom of the CU is used to fill them.

[0129] In another example of the present disclosure, the block motion vector filling operation still follows the process described in the section "Block Motion Vector Filling in Triangular Prediction Mode", except for the 4×4 blocks located on the diagonal boundary line (marked with solid boundary lines in Figure 11 ). For the two 4×4 blocks located at two corners of the diagonal partition boundary, the formed motion vector MV 01 is used to fill them. For the other 4×4 blocks located on the diagonal boundary line, the unidirectional prediction motion vector associated with the triangular partition located at the upper part of the CU is used to fill them.

[0130] In another example of the present disclosure, the current CU is divided into four quarter-sized regions. The 4×4 blocks in each region are filled with the same motion vector, while the 4×4 blocks in different regions can be filled with different motion vectors. More specifically, the blocks in the quarter-sized region located on the diagonal boundary are filled with MV 01 , and the blocks in the quarter-sized region within each triangular partition are filled with the unidirectional prediction motion vector of this partition. An example is shown in Figure 13A . In this figure, the blocks are filled with MV 01Fill the 4×4 blocks in two quarter-sized regions (marked with solid lines) that contain the diagonal partition boundaries, and fill the 4×4 blocks in the other two quarter-sized regions with MV0 or MV1 according to the triangular partition where MV0 and MV1 are located. In Figure 13A the case of, fill the 4×4 blocks in the upper-right quarter-sized region with MV0, and fill the 4×4 blocks in the lower-left quarter-sized region with MV1. In Figure 13B the case of, fill the 4×4 blocks in the upper-left quarter-sized region with MV0, and fill the 4×4 blocks in the lower-right quarter-sized region with MV1.

[0131] In another example of the present disclosure, in the case where the current CU has a width equal to 4 pixels or a height equal to 4 pixels, if each block in the current CU is encoded and decoded using the triangular prediction mode, then fill each block in the current CU with MV 01 . It should be noted that this example can be used together with each of the examples described above.

[0132] In another example of the present disclosure, in the case where the current CU has a size smaller than a specific threshold, if each block in the current CU is encoded and decoded using the triangular prediction mode, then fill each block in the current CU with MV 01 . Otherwise, one of the above methods can be used. The threshold can be defined according to the actual area size of the CU. The threshold can also be defined according to the CU width and / or CU height, or according to some other metric (e.g., the ratio of width to height). In one example, for a CU with an area size smaller than 256 pixels, if each 4×4 block in the current CU is encoded and decoded using the triangular prediction mode, then fill each 4×4 block in the current CU with MV 01 ; and, for a CU with an area size equal to or greater than 256 pixels, fill the blocks of the CU using the methods described in conjunction with Figure 13A and Figure 13B . In another example, for a CU with a width and / or height smaller than 16 pixels, if each block in the current CU is encoded and decoded using the triangular prediction mode, then fill each block in the current CU with MV 01 ; and, for a CU with a width and height both equal to or greater than 16 pixels, fill the blocks of the CU using the methods described in conjunction with Figure 13A and Figure 13B .

[0133] It should be noted that the MV in the present disclosure 01 is used to refer to the potential bi-predicted motion vector constructed from these two uni-predicted motion vectors in the triangular prediction mode. However, constructing MV 01The method is not limited to what is disclosed in the section "Block Motion Vector Filling in Triangular Prediction Mode". It is obvious to those skilled in the art that other methods for deriving the MV 01 are also covered under the same spirit of this disclosure.

[0134] In the above process, although the first merge list containing 5 merge candidates is used for illustration in all examples of this disclosure, in practice, the size of the first merge list can be defined differently, such as 6 or 4, or some other value. All the methods described in this disclosure are equally applicable to the case where the size of the first merge list is not 5.

[0135] Although the method of forming a unidirectional prediction merge list in this disclosure is described with respect to the triangular prediction mode, these methods are applicable to other prediction modes of a similar kind. For example, in a more general geometric partition prediction mode, where a CU is divided into two PUs along a line that is not completely diagonal, the two PUs can have geometric shapes such as triangles, wedges, or trapezoids. In these cases, the prediction for each PU can be formed in a manner similar to the triangular prediction mode, and the methods described here are equally applicable.

[0136] Figure 14 is a block diagram illustrating a device for video coding and decoding according to some embodiments of this disclosure. The device 1400 can be a terminal, such as a mobile phone, a tablet computer, a digital broadcast terminal, a tablet device, or a personal digital assistant, or a chipset thereof.

[0137] As Figure 14 shown, the device 1400 can include one or more of the following components: a processing component 1402, a memory 1404, a power supply component 1406, a multimedia component 1408, an audio component 1410, an input / output (I / O) interface 1412, a sensor component 1414, and a communication component 1416.

[0138] The processing component 1402 generally controls the overall operation of the device 1400, such as operations related to display, phone call, data communication, camera operation, and recording operation. The processing component 1402 can include one or more processors 1420 for executing instructions to complete all or part of the steps of the above method. In addition, the processing component 1402 can include one or more modules to facilitate the interaction between the processing component 1402 and other components. For example, the processing component 1402 can include a multimedia module to facilitate the interaction between the multimedia component 1408 and the processing component 1402.

[0139] The memory 1404 is configured to store different types of data to support the operation of the device 1400. Examples of such data include instructions for any application or method running on the device 1400, contact data, phone book data, messages, pictures, videos, etc. The memory 1404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, and the memory 1404 can be a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk.

[0140] The power component 1406 supplies power to different components of the device 1400. The power component 1406 can include a power supply management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 1400.

[0141] The multimedia component 1408 includes a screen that provides an output interface between the device 1400 and the user. In some examples, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen for receiving input signals from the user. The touch panel can include one or more touch sensors for sensing touches, swipes, and gestures on the touch panel. The touch sensors can not only sense the boundaries of touch or swipe actions but also detect the duration and pressure associated with the touch or swipe operation. In some examples, the multimedia component 1408 can include a front camera and / or a rear camera. When the device 1400 is in an operating mode (such as a shooting mode or a video mode), the front camera and / or the rear camera can receive external multimedia data.

[0142] The audio component 1410 is configured to output and / or input audio signals. For example, the audio component 1410 includes a microphone (MIC). When the device 1400 is in an operating mode (such as a call mode, a recording mode, and a voice recognition mode), the microphone is configured to receive external audio signals. The received audio signals can be further stored in the memory 1404 or transmitted via the communication component 1416. In some examples, the audio component 1410 also includes a speaker for outputting audio signals.

[0143] The I / O interface 1412 provides an interface between the processing component 1402 and the peripheral interface module. The above-mentioned peripheral interface module can be a keyboard, a click wheel, buttons, etc. These buttons can include, but are not limited to, a home button, a volume button, a power-on button, and a lock button.

[0144] The sensor component 1414 includes one or more sensors for providing a status assessment of the device 1400 in different aspects. For example, the sensor component 1414 can detect the on / off state of the device 1400 and the relative positions of components. For example, the components are the display and the keyboard of the device 1400. The sensor component 1414 can also detect changes in the position of the device 1400 or components of the device 1400, the presence or absence of user contact on the device 1400, the orientation or acceleration / deceleration of the device 1400, and changes in the temperature of the device 1400. The sensor component 1414 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor component 1414 can also include an optical sensor, such as a CMOS or CCD image sensor used in imaging applications. In some examples, the sensor component 1414 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0145] The communication component 1416 is configured to facilitate wired or wireless communication between the device 1400 and other devices. The device 1400 can access a wireless network based on communication standards such as WiFi, 4G, or a combination thereof. In an example, the communication component 1416 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an example, the communication component 1416 can also include a near field communication (NFC) module for facilitating short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0146] In an example, the device 1400 can be implemented by one or more of an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor, or other electronic components to perform the above method.

[0147] The non-transitory computer-readable storage medium can be, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, a hybrid drive, or a solid state hybrid drive (SSHD), a read-only memory (ROM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, etc.

[0148] As an overview of the present disclosure, Figure 15FIG. 0 is a flowchart illustrating an exemplary process of video coding and decoding for motion compensation prediction using a geometric prediction unit in accordance with some embodiments of the present disclosure. In this process, a video codec (e.g., video encoder 100 or video decoder 200) is responsible for filling motion vectors for blocks within a coding unit (CU) during video coding and decoding. For illustrative purposes, the following description is provided from the perspective of video decoder 200.

[0149] Video decoder 200 first divides (1510) the CU into two PUs including a first prediction unit PU and a second PU. As Figure 9 shown, CU 902 or 904 is divided into two triangular prediction units (partition 0 and partition 1) in triangular prediction mode. Then, video decoder 200 respectively obtains (1520) a first unidirectional prediction motion vector MV0 for the first PU and a second unidirectional prediction motion vector MV1 for the second PU. Next, video decoder 200 forms (1530) a motion vector MV based on the characteristics of MV0 and MV1 as described in the section "Block Motion Vector Filling in Triangular Prediction Mode" above. 01 Finally, video decoder 200 fills (1540) the corresponding blocks within the CU with a motion vector selected from the group consisting of MV0, MV1, and MV 01 .

[0150] In some embodiments, video decoder 200 fills (1540-1) each block within the CU with the same one of the motion vectors MV0, MV1, and MV01. In other words, different blocks within the CU have the same motion vector.

[0151] In some embodiments, video decoder 200 fills (1540-3) each block at the corresponding end of the boundary line between the first PU and the second PU within the CU with MV 01 (e.g., see Figure 12A and 12B ). Next, video decoder 200 fills (1540-5) the remaining blocks within the CU with the same one of the motion vectors MV0 and MV1.

[0152] In some embodiments, video decoder 200 fills (1540-7) the corner blocks within the first PU with MV0 and fills (1540-7) the corner blocks within the second PU with MV1 (e.g., see Figure 12C and 12D ). Next, video decoder 200 fills (1540-9) the remaining blocks within the CU with MV 01 .

[0153] In some embodiments, the video decoder 200 completely fills (1540-11) the blocks within the first PU with MV0 respectively, and completely fills (1540-11) the blocks within the second PU with MV1 (e.g., see Figure 11 ). Next, the video decoder 200 fills (1540-13) each block at the corresponding ends of the boundary line within the CU that is between the first PU and the second PU with MV 01 , and then fills (1540-15) the remaining blocks on the boundary line between the first PU and the second PU with the same one of MV0 and MV1.

[0154] In some embodiments, the video decoder 200 divides (1540-17) the CU into four quarter regions, where the four quarter regions include a first quarter region entirely within the first PU, a second quarter region entirely within the second PU, and two quarter regions shared by the first PU and the second PU (e.g., see Figure 13A and 13B ). Next, the video decoder 200 fills (1540-19) the blocks within the first quarter region with MV0 respectively, fills the blocks within the second quarter region with MV1, and then fills (1540-21) the blocks within the two shared quarter regions with MV 01 .

[0155] In some embodiments, before filling the motion vectors of the blocks within the CU, the video decoder 200 compares the size of the CU with a predefined threshold. If the size of the CU is smaller than the predefined threshold, the video decoder 200 fills all the blocks within the CU with MV 01 . Otherwise, the video decoder 200 fills the blocks within the CU according to different schemes as described above.

[0156] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted across a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. The computer-readable medium may include a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, the computer-readable medium generally may correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code, and / or data structures to implement the embodiments described in this application. A computer program product may include a computer-readable medium.

[0157] The terms used in the description of the embodiments herein are for the purpose of describing particular embodiments only and are not intended to limit the scope of the claims. The singular forms “a,” “an,” and “the” as used in the description of the embodiments and the appended claims are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term “and / or” as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items. It will be further understood that when the terms “comprises” and / or “comprising” are used in this specification, they specify the presence of the stated features, elements, and / or components, but do not preclude the presence or addition of one or more other features, elements, components, and / or groups thereof.

[0158] It should also be understood that although terms such as first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the embodiments, a first electrode may be referred to as a second electrode, and similarly, a second electrode may be referred to as a first electrode. The first electrode and the second electrode are both electrodes, but not the same electrode.

[0159] The description of the present application has been presented for purposes of illustration and description, and is not intended to be exhaustive or to limit the disclosure in the form disclosed. Many modifications, variations, and alternative embodiments will be apparent to those of ordinary skill in the art in light of the foregoing description and the teachings presented in the associated drawings.

[0160] The embodiments are chosen and described in order to best explain the principles of the present disclosure, its practical application, and to enable others skilled in the art to understand the various implementations of the present disclosure and to best utilize the basic principles and various implementations with various modifications, as are suitable for the particular use contemplated. It is therefore to be understood that the scope of the present disclosure is not limited to the specific examples of the disclosed embodiments, and that modifications and other embodiments are intended to be included within the scope of the present disclosure.

Claims

1. A method for filling motion vectors for blocks within a coding unit (CU) during video coding and decoding, comprising: Dividing the CU into two prediction units (PUs) including a first PU and a second PU; Obtaining a first unidirectional prediction motion vector MV0 for the first PU and a second unidirectional prediction motion vector MV1 for the second PU, respectively; Form a motion vector MV based on the features of MV0 and MV1 01 ; and Fill the corresponding blocks in the CU with motion vectors selected from the group consisting of MV0, MV1, and MV 01 ; Among them, filling the corresponding block in the CU with a motion vector selected from the group consisting of MV0, MV1, and MV 01 comprises: Comparing the size of the CU with a predefined threshold; Based on determining that the size of the CU is less than the predefined threshold, fill all blocks within the CU with MV 01 and; Based on determining that the size of the CU is not less than the predefined threshold, filling the blocks within the CU according to at least one of the following schemes: Using MV 01 Fill each block at the respective ends of the boundary line within the CU that is between the first PU and the second PU; fill the remaining blocks in the CU with the same one of the motion vectors MV0 and MV1; or, Fill the corner blocks in the first PU with MV0 respectively, fill the corner blocks in the second PU with MV1; and fill the remaining blocks in the CU with MV 01 ; or, Fully fill the blocks within the first PU with MV0 and the blocks within the second PU with MV1; use MV 01 to fill each block at the corresponding end of the boundary line between the first PU and the second PU within the CU; and use the same motion vector among MV0 and MV1 to fill the remaining blocks on the boundary line between the first PU and the second PU.

2. The method according to claim 1, wherein the CU is divided into two PUs in a triangular prediction mode.

3. The method according to claim 1, wherein the size of the CU is one selected from the group consisting of the width of the CU, the height of the CU, and the number of pixels of the CU.

4. The method according to claim 1, wherein MV 01 is a uni - directional predicted motion vector or a bi - directional predicted motion vector.

5. An electronic device, comprising: One or more processors; A memory coupled to the one or more processors; Wherein, the one or more processors are configured to: Divide a coding unit (CU) into two prediction units (PUs) including a first PU and a second PU; Obtain a first unidirectional prediction motion vector MV0 for the first PU and a second unidirectional prediction motion vector MV1 for the second PU, respectively; Form a motion vector MV based on the features of MV0 and MV1 01 ; and Fill the corresponding blocks in the CU with motion vectors selected from the group consisting of MV0, MV1, and MV 01 ; Wherein, the one or more processors are configured to fill corresponding blocks in the CU with motion vectors selected from the group consisting of MV0, MV1, and MV 01 comprising: Compare the size of the CU with a predefined threshold; Based on determining that the size of the CU is less than the predefined threshold, fill all the blocks within the CU with MVs 01 ; and Based on determining that the size of the CU is not less than the predefined threshold, fill the blocks within the CU according to at least one of the following schemes: Using MV 01 Fill each block at the respective ends of the boundary line within the CU that is between the first PU and the second PU; fill the remaining blocks in the CU with the same one of the motion vectors MV0 and MV1; or, Fill the corner blocks in the first PU with MV0 respectively, fill the corner blocks in the second PU with MV1; and fill the remaining blocks in the CU with MV 01 ; or, Completely fill the blocks within the first PU with MV0, and completely fill the blocks within the second PU with MV1; use MV 01 to fill each block at the corresponding ends of the boundary line between the first PU and the second PU within the CU; and use the same one of the motion vectors MV0 and MV1 to fill the remaining blocks on the boundary line between the first PU and the second PU.

6. The electronic device according to claim 5, wherein the CU is divided into two PUs in a triangular prediction mode.

7. The electronic device according to claim 5, wherein the size of the CU is one selected from the group consisting of the width of the CU, the height of the CU, and the number of pixels of the CU.

8. The electronic device according to claim 5, wherein MV 01 is a uni - directional prediction motion vector or a bi - directional prediction motion vector.

9. A non-transitory computer-readable storage medium storing a plurality of programs executed by an electronic device having one or more processors, wherein the plurality of programs, when executed by the one or more processors, cause the electronic device to execute the method according to any one of claims 1-4.

10. A computer program product includes a plurality of programs, which are executed by an electronic device having one or more processors, wherein, When the plurality of programs are executed by the one or more processors, cause the electronic device to execute the method according to any one of claims 1-4.

Images