Method or apparatus for video encoding or decoding

Abstract

The present disclosure relates to a method and apparatus for video encoding or decoding. A method includes determining to enable both multi-shape partitioning and multiple hypotheses for an image block; and encoding the image block based on a combination of a multi-shape partition and a multi-hypothesis prediction mode.

Description

[0001] This application is a divisional application of the invention patent application with application number 201980078737.6, application date November 19, 2019, and invention name “Triangle and multiple hypothesis combinations for video encoding and decoding”. Technical Field

[0002] At least one of the present embodiments generally relates to a method or apparatus for video encoding or decoding, and more specifically to a method or apparatus for efficiently providing combined video compression and / or decompression with multi-shape and multi-hypothesis prediction, etc. Background Art

[0003] To achieve high compression efficiency, image and video coding schemes typically employ predictions including motion vector predictions, and transforms to exploit spatial and temporal redundancies in video content. Typically, intra-frame or inter-frame predictions are used to exploit intra-frame or inter-frame correlations, and then the difference between the original image and the predicted image (usually denoted as the prediction error or prediction residual) is transformed, quantized, and entropy coded. To reconstruct the video, the compressed data is decoded by the inverse process corresponding to entropy coding, quantization, transform, and prediction.

[0004] Recent additions to video compression technology include versions of various industry standards, reference software and / or documents, such as the Joint Development Model (JEM) and the follow-up VTM (Versatile Video Coding (VVC) Test Model) being developed by the Joint Video Development Team (JVET) group to further improve upon the existing HEVC (High Efficiency Video Coding) standard. Summary of the invention

[0005] The deficiencies and disadvantages of the prior art are addressed by one or more aspects described in this application.

[0006] According to one embodiment, a method for video encoding is provided, the method comprising: determining to enable both multi-shape partitioning and multiple hypotheses for an image block; and encoding the image block based on a combination of the multi-shape partitioning and the multiple hypotheses prediction mode.

[0007] According to another embodiment, a device for video encoding is provided, the device comprising: one or more processors, wherein the one or more processors are configured to implement: determining to enable both multi-shape partitioning and multiple hypotheses for an image block; and encoding the image block based on a combination of multi-shape partitioning and multiple hypothesis prediction modes.

[0008] According to another embodiment, a method for video decoding is provided, the method comprising: determining to enable both multi-shape partitioning and multiple hypotheses for an image block; and decoding the image block based on a combination of multi-shape partitioning and multiple hypothesis prediction modes.

[0009] According to another embodiment, a device for video decoding is provided, the device comprising: one or more processors, wherein the one or more processors are configured to implement: determining to enable both multi-shape partitioning and multiple hypotheses for an image block; and decoding the image block based on a combination of multi-shape partitioning and multiple hypothesis prediction modes. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A block diagram of an embodiment of a video encoder is shown.

[0011] Figure 2 A block diagram of an embodiment of a video decoder is shown.

[0012] Figure 3 A block diagram of a system is shown in which aspects of the present embodiments may be implemented.

[0013] Figure 4 The partitioning of the CU into two triangular prediction units is shown.

[0014] Figure 5 The weighting process on the diagonal edge between two triangular prediction units is shown.

[0015] Figure 6 The derivation process of unidirectional prediction for triangle partitioning is shown.

[0016] Figure 7 Sub-block motion vector storage for a triangle partition (PU) is shown.

[0017] Figure 8 An additional partition (PU) is shown.

[0018] Fig. 9 Additional non-triangle partitions (PUs) are shown.

[0019] Fig.10 Multiple hypothesis prediction is shown for the case of inter-mode and intra-mode combination.

[0020] Fig.11 Intra prediction regions and their corresponding weights in multiple hypothesis mode are shown.

[0021] Fig.12 A 2x8 sub-region of an 8x8 CU with vertical prediction for multiple hypothesis mode is shown.

[0022] Fig.13 It is shown how horizontal prediction is used for the second partition in the upper right to lower left triangular partition.

[0023] Fig.14 A process or apparatus for multiple hypothesis prediction is shown.

[0024] Fig.15 A process or apparatus for multiple hypothesis prediction with only one weighting step is shown.

[0025] Fig.16 A triangular prediction with one weight per sub-block is shown.

[0026] Fig.17 The storage of weights used by the sub-blocks is shown.

[0027] Fig.18 An 8x8 CU predicted by multiple hypothesis and vertical intra prediction is shown.

[0028] Fig.19 A process or apparatus is shown for multiple hypothesis prediction with unidirectional inter prediction restriction.

[0029] Fig. 20 A process or apparatus is shown for multiple hypothesis prediction with unidirectional inter prediction restriction and intra-inter combining followed by clipping.

[0030] Fig.21 The derivation of unidirectional or intra predictors for multiple shape partitions is shown.

[0031] Fig. 22 Intra direction prediction inferred from the predicted shape is shown.

[0032] Fig.23 The derivation of predictors for multiple shape partitions is shown, where the intra direction is added at the end of the list.

[0033] Fig.24 A separate predictor derivation process for each of List 0 and List 1 is shown.

[0034] Fig.25 The spatial and temporal locations checked for each PU are shown.

[0035] Fig.26 An example of separate candidate derivation for 2 PUs in a TL2BR triangle partition (part 1 / 2) is shown.

[0036] Fig. 27 An example of separate candidate derivation for 2 PUs in a TL2BR triangle partition (part 2 / 2) is shown.

[0037] Fig.28 New locations checked for candidate list derivation are shown.

[0038] Fig.29 The storage of prediction information for multiple shapes and multiple hypothesis combinations is shown.

[0039] Fig.30 The storage of prediction information for multiple shapes and multiple hypothesis combinations is shown. DETAILED DESCRIPTION

[0040] The present disclosure is in the field of video compression and decompression, and at least one embodiment more particularly relates to block prediction performed using two different inter predictions or a combination of inter prediction and intra prediction.

[0041] For clarity of description, the following description will describe various aspects with reference to embodiments involving video compression techniques such as, for example, HEVC, JEM, and / or H.266. However, the various aspects described are applicable to other video processing techniques and standards.

[0042] This application describes various aspects including tools, features, embodiments, models, methods, etc. Many of these aspects are specifically described, and at least to illustrate individual characteristics, are often described in a manner that may sound limited. However, this is for the sake of clarity of description and does not limit the application or scope of those aspects. In fact, all different aspects can be combined and interchanged to provide further aspects. In addition, these aspects can also be combined and interchanged with aspects described in previous applications.

[0043] The aspects described and contemplated in this application may be implemented in many different forms. Figure 1 , 2 and 3 provide some embodiments, but other embodiments are contemplated, and Figure 1 , 2 The discussion of 3 does not limit the breadth of the embodiments. At least one of these aspects generally relates to video encoding and decoding, and at least another aspect generally relates to sending a generated or encoded bitstream. These and other aspects can be implemented as a method, an apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0044] In this application, the terms "reconstructed" and "decoded" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image", "picture" and "frame" are used interchangeably. Usually, but not necessarily, the term "reconstructed" is used on the encoder side, and "decoded" is used on the decoder side.

[0045] The terms HDR (high dynamic range) and SDR (standard dynamic range) are used in this disclosure. These terms generally convey to one of ordinary skill in the art a specific value of dynamic range. However, additional embodiments are also intended in which references to HDR are understood to mean "higher dynamic range" and references to SDR are understood to mean "lower dynamic range". Such additional embodiments are not bound by any specific value of dynamic range that might be generally associated with the terms "high dynamic range" and "standard dynamic range".

[0046] Various methods are described herein, and each of the methods includes one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for the correct operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0047] The various methods and other aspects described in this application can be used to modify Figure 1 and 2 Modules of the video encoder 100 and decoder 200 shown, for example, intra prediction, entropy coding and / or decoding modules (160, 360, 145, 330). In addition, the present aspects are not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations, whether pre-existing or developed in the future, and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this application can be used alone or in combination.

[0048] Various values ​​are used in this application, for example, with respect to block size. The specific values ​​are for example purposes, and the described aspects are not limited to these specific values.

[0049] Figure 1 An encoder 100 is shown. Variations of the encoder 100 are contemplated, but for clarity, the encoder 100 is described below without describing all contemplated variations. Before being encoded, the video sequence may undergo pre-encoding processing (101), such as color transformation of input color pictures (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or remapping of input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0050] In an encoder 100, a picture is encoded by encoder elements as described below. The picture to be encoded is partitioned (102) and processed in units such as CUs. Each unit is encoded using, for example, intra or inter mode. When the unit is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder determines (105) which mode, intra mode or inter mode, is used to encode the unit and indicates the intra / inter decision by, for example, a prediction mode flag. The prediction residual is calculated, for example, by subtracting (110) the predicted block from the original image block.

[0051] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy encoded (145) to output a bitstream. The encoder may skip the transform and apply quantization directly to the untransformed residual signal. The encoder may bypass both the transform and quantization, i.e., encode the residual directly without applying the transform or quantization process.

[0052] The encoder decodes the encoded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. An in-loop filter (165) is applied to the reconstructed picture to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0053] Figure 2 2 shows a block diagram of a video decoder 200. In the decoder 200, the bitstream is decoded by the decoder elements as described below. The video decoder 200 generally performs the same operations as described below. Figure 1 The encoding channel is opposite to the decoding channel. The encoder 100 also typically performs video decoding as part of encoding the video data. Specifically, the input to the decoder includes a video bitstream, which can be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors and other encoding information. The picture partition information indicates how the picture is partitioned. Therefore, the decoder can divide (235) the picture according to the decoded picture partition information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (255) to reconstruct the image block. The prediction block can be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0054] The decoded picture may further undergo post-decoding processing (285), such as an inverse color transform (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping, which performs the inverse of the remapping process performed in the pre-encoding process (101). The post-decoding process may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0055] Figure 3 A block diagram of an example of a system in which various aspects and embodiments are implemented is shown. System 1000 may be embodied as a device including various components described below, and is configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, networked home appliances, and servers. The elements of system 1000 may be embodied in a single integrated circuit (IC), multiple ICs, and / or discrete components, either individually or in combination. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed over multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0056] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing, for example, various aspects described in this document. The processor 1010 may include embedded memory, input and output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include a non-volatile memory and / or a volatile memory, including but not limited to an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), a programmable read-only memory (PROM), a random access memory (RAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a magnetic disk drive, and / or an optical disk drive. As a non-limiting example, the storage device 1040 may include an internal storage device, an attached storage device (including a removable and non-removable storage device), and / or a network accessible storage device.

[0057] The system 1000 includes an encoder / decoder module 1030, which is configured to, for example, process data to provide encoded video or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a (multiple) module that can be included in a device to perform encoding and / or decoding functions. As is well known, a device may include one or both of the encoding and decoding modules. In addition, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000, or may be incorporated into the processor 1010 as a combination of hardware and software known to those skilled in the art.

[0058] Program code to be loaded onto the processor 1010 or the encoder / decoder 1030 to perform various aspects described in this document may be stored in the storage device 1040 and then loaded onto the memory 1020 to be executed by the processor 1010. According to various embodiments, during the execution of the processes described in this document, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items. Such stored items may include, but are not limited to, input video, decoded video or a portion of decoded video, bitstreams, matrices, variables, and intermediate or final results of processing equations, formulas, operations, and operational logic.

[0059] In some embodiments, memory internal to the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for the processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or a storage device 1040, such as a dynamic volatile memory and / or a non-volatile flash memory. In several embodiments, the external non-volatile flash memory is used to store, for example, an operating system for a television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations such as MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, i.e., a new standard developed by the JVET Joint Video Experts Team).

[0060] As indicated at block 1130, input to the elements of system 1000 may be provided through various input devices. Such input devices include, but are not limited to: (i) a radio frequency (RF) section that receives an RF signal sent over the air by, for example, a broadcaster; (ii) a component (COMP) input terminal (or a collection of COMP input terminals); (iii) a universal serial bus (USB) input terminal; and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Fig.10 Other examples not shown include composite video.

[0061] In various embodiments, the input device of block 1130 has associated corresponding input processing elements, as known in the art. For example, the RF portion may be associated with elements suitable for: (i) selecting a desired frequency (also referred to as selecting a signal, or band-limiting a signal to a frequency band); (ii) down-converting the selected signal; (iii) again band-limiting a narrower frequency band to select a signal frequency band that may be referred to as a channel in some embodiments; (iv) down-demodulating the down-converted and band-limited signal; (v) performing error correction; and (vi) performing demultiplexing to select a desired data packet stream. The RF portion of various embodiments includes one or more elements for performing these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF portion may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or a baseband. In a set-top box embodiment, the RF part and its associated input processing element receive the RF signal sent on a wired (e.g., cable) medium, and perform frequency selection by filtering, down-conversion and filtering to the desired frequency band again. Each embodiment rearranges the order of the above-mentioned (and other) elements, removes some elements in these elements, and / or adds other elements that perform similar or different functions. Adding elements can include inserting elements between existing elements, such as, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF part includes an antenna. In addition, USB and / or HDMI terminals can include corresponding interface processors to connect system 1000 to other electronic devices by USB and / or HDMI. It should be understood that the various aspects of input processing (e.g., Reed-Solomon error correction) can be implemented as needed, for example, in a separate input processing IC or in processor 1010. Similarly, the various aspects of USB or HDMI interface processing can be implemented as needed in a separate interface IC or in processor 1010. The demodulated, error corrected and demultiplexed stream is provided to various processing elements including, for example, a processor 1010 and an encoder / decoder 1030 operating in conjunction with memory and storage elements to process the data stream as necessary for presentation on an output device.

[0062] The various elements of system 1000 may be provided within an integrated housing. Within the integrated housing, the various elements may be interconnected and data may be transmitted between the various elements using a suitable connection arrangement (eg, an internal bus known in the art, including an Inter-IC (I2C) bus, wiring, and printed circuit boards).

[0063] The system 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data over the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0064] In various embodiments, a wireless network such as a Wi-Fi network (e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers)) is used to send or otherwise provide data streams to the system 1000. The Wi-Fi signals of these embodiments are received on a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 of these embodiments is typically connected to an access point or router that provides access to an external network including the Internet to allow streaming applications and other over-the-top communications. Other embodiments provide streaming data to the system 1000 using a set-top box that passes data through an HDMI connection of an input block 1130. Other embodiments use an RF connection of an input block 1130 to provide streaming data to the system 1000. As indicated above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use a wireless network other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0065] The system 1000 can provide output signals to various output devices including a display 1100, a speaker 1110, and other peripheral devices 1120. The display 1100 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1100 can be used for a television, a tablet computer, a laptop computer, a mobile phone (mobile phone), or other devices. The display 1100 can also be integrated with other components (e.g., integrated in a smart phone) or be separate (e.g., an external monitor of a laptop computer). In various examples of embodiments, other peripheral devices 1120 include independent digital video disks (or digital versatile disks) (DVR, for both terms), disk players, stereo systems, and / or lighting systems. Various embodiments use one or more peripheral devices 1120 that provide functions based on the output of the system 1000. For example, a disk player performs the function of playing the output of the system 1000.

[0066] In various embodiments, control signals are communicated between the system 1000 and the display 1100, speaker 1110, or other peripheral device 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices may be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, output devices may be connected to the system 1000 using a communication channel 1060 via a communication interface 1050. The display 1100 and speaker 1110 may be integrated into a single unit with other components of the system 1000 in an electronic device such as, for example, a television. In various embodiments, the display interface 1070 includes a display driver, such as, for example, a timing controller (T Con) chip.

[0067] For example, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 may alternatively be separate from one or more other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signals may be provided via dedicated output connections including, for example, an HDMI port, a USB port, or a COMP output.

[0068] The embodiment may be implemented by computer software implemented by the processor 1010 or by hardware or by a combination of hardware and software. As a non-limiting example, the embodiment may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as, as a non-limiting example, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, the processor 1010 may be of any type suitable for the technical environment and may include one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0069] Figure 1 Encoder 100, Figure 2 The decoder 200 and Figure 3 The system 1000 is suitable for implementing at least one of the embodiments described below.

[0070] When using triangular prediction in merge mode, the coding unit (CU) is split into two triangular prediction units (PUs) along the diagonal or anti-diagonal direction, such as Figure 4 As shown. Each triangle prediction unit in the CU performs inter prediction using its own motion vector and reference frame index, which are derived from the merge candidate list.

[0071] An adaptive weighting process is applied to the prediction samples of the diagonal or anti-diagonal edges between two triangle prediction units to derive the final prediction sample value of the entire CU, such as Figure 5 shown.

[0072] The triangle prediction unit mode is only applied to CUs in skip or merge mode. When the triangle prediction unit mode is applied to a CU, an index (merge_idx) indicating the direction in which the CU is split into two triangle prediction units and the motion vectors of the two triangle prediction units are signaled.

[0073] For two prediction units, a common list of 5 unidirectional predictors is derived. Compared to the regular merge process, the same spatial and temporal positions are checked, but only unidirectional vectors are used. Redundant motion vectors are not added to the list, and if there are not enough candidates, a zero motion vector is added to the end of the list. Figure 6 The process of deriving one-way prediction quantities is depicted.

[0074] For a given prediction unit, the number of motion vector predictors is 5. 20 combinations are tested for each diagonal (5*4=20, the two PUs cannot use the same motion vector predictor). The index range is 0 to 39, and a lookup table consisting of a list of triplets is used to derive the partition direction and motion vector for each PU from the index. The first element of a given triplet gives the diagonal direction, and the second and third elements give the predictor index for PU0 and PU1, respectively. Table 1 shows the index syntax.

[0075]

[0076] surface : Triangle partitioning (PU) and corresponding merge index syntax

[0077] Motion vectors may be stored for each 4x4 sub-block. When a triangular PU is used for a CU, the motion vector for a given partition (PU) is stored for each of the sub-blocks that are not located on a diagonal or anti-diagonal edge. For sub-blocks located on a diagonal or anti-diagonal edge, if the two motion vectors in the two partitions (PUs) refer to different lists (Mv0 refers to LIST_0 and Mv1 refers to LIST_1, or conversely, Mv1 refers to LIST_0 and Mv0 refers to LIST_1), they are stored, if the two motion vectors refer to the same list, only one vector is stored, e.g. Figure 7 shown.

[0078] like Figure 8 and Fig. 9 As depicted, additional shapes (such as non-triangles) can also be introduced to weight the two unidirectional predictions.

[0079] In a new mode called Intra-Inter Multiple Hypothesis (MH) mode, inter prediction performed in merge mode (signaling a merge index to derive motion information for motion compensated prediction) is combined with an intra prediction mode or with another inter mode (e.g., uni-prediction AMVP, skip, or merge). The final prediction is a weighted average of the merge index prediction and the prediction generated by the intra prediction mode, where different weights are applied depending on the intra direction and the distance between the current sample and the intra reference sample. The intra prediction mode is signaled (it can be a subset of the full set of prediction modes (e.g., 4)).

[0080] like Fig.11 As shown in , the current block is divided into 4 equal-sized regions. As the region moves away from the intra reference sample, the weight gradually decreases, thereby gradually reducing the influence of the intra image on the predicted inter frame. Each weight set (denoted as (w_intrai, w_interi), where i ranges from 1 to 4 and (w_intra1, w_inter1) = (6, 2), (w_intra2, w_inter2) = (5, 3), (w_intra3, w_inter3) = (3, 5) and (w_intra4, w_inter4) = (2, 6)) is applied to the corresponding region, as shown in Fig.10 As depicted in the example for intra vertical prediction in . When DC or planar mode is selected, or when the CU width is less than or equal to 4, equal weight is applied to all samples. When intra prediction is performed in a multiple hypothesis CU, the chroma components use direct mode (same intra direction as luma).

[0081] Fig.11 The shapes used for 4 intra predictions in multiple hypotheses are shown.

[0082] In the following, the terms triangle partition and various shape partition are used but are interchangeable depending on the context and application.

[0083] However, for triangle merging, the weighting of the two predictions is done pixel by pixel, and pixel-by-pixel operation is more difficult to optimize than block or sub-block operation.

[0084] If horizontal or vertical prediction is selected, the Multiple Hypothesis (MH) uses 4 sub-regions corresponding to 4 different weights for weighting. MH mode is available for blocks with an area greater than or equal to 64 samples, which means that 4x16 and 16x4 blocks are allowed. In this case, MH can use 16x1 and 1x16 regions for horizontal or vertical prediction. For 8x8 blocks, 2x8 sub-regions can be used, such as Fig.12However, the region used for prediction is 4x4. In other words, depending on the respective sizes of these regions, there may be inconsistencies between the regions where prediction information is stored and the regions that are actually used to perform these predictions.

[0085] Furthermore, for a triangular PU, except for a sub-block covering a diagonal or anti-diagonal edge, only one motion vector is stored even if the sub-block is close to the edge, resulting in information loss.

[0086] For multiple hypotheses, the motion vector and intra direction are stored, but only the inter information is used as the predictor for inter neighboring blocks, while the intra information is used as the predictor for intra neighboring blocks.

[0087] These two aspects make the design of the two tools (triangular partitioning and multiple hypothesis forecasting model) inconsistent.

[0088] Additionally, in the combination of triangle partitioning (PU) and multi-hypothesis prediction, the orientation of the triangle may be incompatible with intra prediction and the way the weighting process is done. Fig.13 In , the top right to bottom left (TR2BL) triangle partition is selected. Because the left side of PU 1 is far away from the reconstructed samples for intra prediction at the left row outside the current CU, the horizontal intra prediction efficiency for the second PU (PU1) is low.

[0089] Multiple hypothesis is a combination of inter prediction and intra prediction. Bidirectional prediction can be used in the middle. The multiple hypothesis pipeline consists of many steps: two-step motion compensation, bidirectional optical flow (BIO), generalized bidirectional prediction (GBi) weighting, clipping of inter and intra prediction, weighting of inter and intra prediction, such as Fig.14 shown.

[0090] In one embodiment, only one weighting step is performed to add the two unidirectional inter predictions and the intra prediction, such as Fig.15As shown. For inter prediction, the motion compensation step is performed with higher precision (usually 14 bits), and intra prediction can also be performed with higher precision, so the weighting of all predictions can be done with higher precision. In VTM-3.0, for planar or DC intra prediction, the weight used to combine all multi-hypothesis predictions is 1 / 4 for unidirectional inter prediction and 1 / 2 for intra prediction. In this embodiment, since all predictions are added during the same weighting step, the weight can be changed to be close to 1 / 3 (due to integer operations, it is preferred to use division by shifting, so the number of bits of precision is divided by a power of 2, for example), for example, the weight of 2 inter unidirectional predictions is 3 / 8, and the weight of intra prediction is 2 / 8 (3 / 8 + 3 / 8 + 2 / 8 = 1). Or if 5 bits of precision are used instead of 3 bits of precision, the weight of inter unidirectional prediction is 11 / 32, and the weight of intra prediction is 10 / 32.

[0091] At least one embodiment is directed to processing predictions for blocks when encoding or decoding a video, where predictions are performed using two different inter predictions or a combination of inter prediction and intra prediction, in contexts where these predictions may potentially be completely unrelated. Therefore, a design of a unified multi-shape PU partition mode (more specifically a triangle partition mode) and a multiple hypothesis mode is proposed, and different exemplary aspects are proposed as follows:

[0092] Apply weighting to the triangle partitions (more generally to multi-shape PU partitions) of each sub-block;

[0093] Perform storage of weights for each sub-block;

[0094] Use weights that are powers of 2;

[0095] Adjust the number of regions in multiple hypotheses according to the CU size;

[0096] Force a CU that combines multiple hypothesis prediction and multi-shape PU partitioning to use only unidirectional prediction, so when multiple hypothesis prediction and multi-shape PU partitioning are enabled for a CU, disable BIO and GBi;

[0097] Add intra prediction mode as an additional candidate in the candidate list of multi-shape PU partition mode;

[0098] When the reference samples used for intra prediction are far away from the samples of the PU to be predicted, some intra prediction modes are disabled for a specific PU of the multi-shape PU partition;

[0099] When multiple hypothesis prediction is combined with multi-shape PU partitioning, storage of a motion vector or intra mode is performed for each subblock.

[0100] One or more aspects of the present embodiment (such as, for example, bidirectional prediction for inter-coded blocks) may be implemented, for example, in Figure 3 In the system 1000 and in accordance with Figure 1 The elements 145, 170, 175 of the encoder 100 and in accordance with Figure 2 The components 230 and 275 of the video decoder 200 are implemented.

[0101] Weighted processing by sub-block

[0102] In one embodiment, the same weight is applied to all samples within a given sub-block. For example, for a triangle partition, a sub-block covering an edge can use the same weight for the two predictions P0 and P1. For a sub-block close to an edge but not covering it, the weight is larger for the prediction corresponding to the triangle (or prediction unit) to which the sub-block belongs, such as Fig.16 As shown. For example, the sub-blocks marked as 6+2 on the diagonal indicate that the weighting for P0 is 6 / 8, while the weighting for P1 is 2 / 8. The diagonal weights the two predictions equally. This principle applies similarly to rectangular partitions.

[0103] In one embodiment, the weights are specified as powers of 2, which results in a simplification of the calculations by replacing multiplications with shift operations. In the conventional example of using 2 multiplications per sample and then dividing by 8, using weights equal to a power of 2 will result in 2 shifts and 1 division (applied via the shift operation), and is therefore much more efficient.

[0104] Storage by weight of sub-blocks

[0105] In at least one embodiment, weighting by sub-block allows consistency between weighting process and information storage. Since generalized bi-prediction mode (GBi) uses weights already stored for each CU, there is no cost for storage of weights used by weighting process of triangles or multiple hypotheses. Fig.17 An example of storage of motion vectors and associated weights for each sub-block is shown. As in the case of GBi, the weights can be used to predict or infer the weights of the next CU that is processed later in the decoding process.

[0106] Adjust the number of sub-regions of a CU encoded in multiple hypothesis mode

[0107] In at least one embodiment, the number of subregions of a CU encoded in multiple hypothesis mode is adjusted. If the size of the block in the prediction direction is smaller than a given size S (e.g., S=16), fewer subregions are used so as to have a minimum number of samples S in a given subregion. min (For example, S min=4). In one embodiment, the number of sub-regions is set to one sub-region for a 4xN CU and to two sub-regions for an 8xN CU. In one embodiment, an 8×8 CU coded with multiple hypotheses and using vertical intra prediction is used. The weighted process for weighting intra and inter prediction uses 2 sub-regions of size 4x8, such as Fig.18 shown.

[0108] Combination of multiple hypotheses and multiple shape prediction units

[0109] Multiple Hypothesis Unidirectional Inter Prediction Constraints

[0110] In the following embodiments, the multiple hypothesis mode is replaced by a multiple shape prediction mode where one of the predictors is an intra mode.

[0111] In one embodiment, the following restrictions are specified for combining multiple hypothesis prediction mode and multiple shape prediction mode. When multiple hypothesis prediction mode and multiple shape prediction mode are enabled for the current CU, inter prediction is restricted to unidirectional. By designing BIO and GBi weights, this restriction is disabled because these two processes are only used for bidirectional prediction. Therefore, syntax elements associated with these two modes will not be decoded. The multiple hypothesis pipeline with this unidirectional restriction is Fig.19 Depicted as Fig.14 or 15 for comparison. This allows the process to be more efficient by streamlining the pipeline.

[0112] To further simplify the multiple hypothesis pipeline, the prediction can be clipped only once after the intra prediction is combined with the inter prediction, e.g. Fig. 20 This allows for higher precision (typically 14 bits instead of 10) in the weighting process between inter- and intra-frame prediction.

[0113] The following table shows an example of the syntax and semantics for applying this restriction. First, the flags merge_triangle_flag and mh_merge_flag are signaled in the bitstream. When both flags are equal to 1, bidirectional prediction cannot be applied. Therefore, the syntax element gbi_idx (which signals the weighted index) is not signaled, and the parameter of the activation BIO (bioEnableFlag) is forced to 0. If at least one of the flags is equal to 0 and the bidirectional prediction mode is activated, gbi_idx is signaled and bioEnableFlag may be set to 1.

[0114]

[0115] Intra prediction as an additional candidate for multiple shapes

[0116] In one embodiment, to coordinate multiple shape prediction and multiple hypothesis prediction, intra mode is added to the list of predictors for multiple shape prediction. For triangle partition mode, a list of 5 unidirectional motion vectors can be derived, and then 2 motion vectors are selected from the 5 predictors, one motion vector for each partition (PU) to predict the current CU.

[0117] Intra mode derivation from neighboring blocks

[0118] The first way to add intra directions in the predictor list is to propagate intra directions from neighboring blocks. Instead of discarding neighboring intra-coded blocks, intra directions from neighboring blocks are added to the candidate list derivation for multiple shape prediction modes. Compared to the derivation process of prediction predictors for triangular / multiple shape partitions, intra modes from neighboring blocks are added to the final predictor list, as shown in Figure 2. Fig.21 shown.

[0119] Implicit intra prediction direction from partition shape

[0120] A second way to add intra directions to the prediction list is to implicitly add one or more intra directions to the prediction list based on the PU shape. For example, in one embodiment, the following items (such as Fig. 22 shown):

[0121] Add horizontal prediction for vertical PU shapes,

[0122] Vertical prediction of horizontal PU shape,

[0123] TR2BL (upper right to lower left) PU shape prediction in the upper left direction,

[0124] predictions in the lower left and upper right directions for PU TL2BR (upper left to lower right) shapes,

[0125] You can add DC and Planar for all partitions.

[0126] like Fig.23 As shown, the derivation process of the prediction quantity for triangle / multiple shape partitions is modified.

[0127] In a first variant, some intra directional prediction may be added for the first prediction (PU), but not for the second prediction (PU), since the reference samples used for prediction may be far away from the second prediction.

[0128] In another variation, intra DC, Planar mode or other intra directions may be added based on the predicted shape.

[0129] In another variation, intra prediction may be selected for PU0 and PU1, in which case the multiple hypotheses are intra-intra and the weighting depends on the selected intra direction.

[0130] Separate predictor candidates for list0 and list1 of multiple shapes

[0131] In one embodiment, two lists of prediction quantities are derived for a CU using multiple shape prediction units, such as Fig.24 As shown. The first list of predictors contains only motion vectors that reference LIST_0 or intra-frame directions, and the second list of predictors contains only motion vectors that reference LIST_1 or intra-frame directions. If the first prediction unit uses a candidate from List 0, the second prediction unit uses a predictor from List 1, and if the first prediction unit uses a candidate from List 1, the second prediction unit uses a predictor from List 0. If 4 candidates are available in each list, the total number of combinations is 2*4*4= 32. The first PU can use 4 candidates from List_0 and 4 candidates from List_1, and the second PU can use only 4 candidates from another list different from the first PU.

[0132] HMVP (history-based motion vector prediction) keeps a buffer of the 6 last motion vectors used by the previously encoded CU. This list of predictors can be used in an additional step to derive unidirectional motion vectors for multiple shape partitions. In the same way, the unidirectional ATMVP candidate derived from the first spatial candidate can be added to the list of unidirectional predictors.

[0133] Merge with Motion Vector Difference (MMVD) adds a small motion vector difference (MVD) to the first or second merge candidate. Using MMVD for multiple shape partitions can improve coding efficiency. The only constraint of MMVD is that it is only applied to inter predictors.

[0134] A problem with embodiments where the intra predictor is in the merge list is that during the parsing phase, it is not known whether the parsed merge index indicates the use of the intra predictor. One solution is to add the intra predictor after multiple inter candidates (e.g., 2 inter candidates) available for use by the MMVD.

[0135] Separate prediction unit candidates for each prediction unit

[0136] In one embodiment, two lists of prediction quantities are derived for a CU using multiple shape prediction units, one list for each prediction unit. The spatial locations checked are not the same for each partition. The candidate list derivation is also different for all shapes, e.g. Fig.25 PU0 or PU1 uses the spatial position upper left, upper, upper right, left, lower left and the temporal position. For example, in Fig.25 , for the TL2BR shape (top left), the top, top-right, and temporal positions are checked to derive the motion vector predictor list for PU0, and the top-left, left, and bottom-left positions are checked to derive the motion vector predictor list for PU1.

[0137] Fig.26 and Fig. 27 A separate candidate derivation for each of the TL2BR triangle partitions is shown. The spatial and temporal locations examined are different for the two partitions, but the process is the same for both partitions.

[0138] In this embodiment, planar or affine unidirectional candidates can be added to the predictor derivation process by using two or more spatial positions (the same spatial positions as those checked for motion vectors). When affine uses an inherited motion model, the same process is applied to select candidates. When affine uses a constructed motion model, the same process is applied to each control point motion vector (CPMV) candidate used to construct the motion model. For planes or regressed motion vector fields (RMVFs), the same process is applied, and only sub-blocks close to the partition can be used to derive the motion model. This mode is not available if the partition has no adjacent sub-blocks. For example, Planar is not available for PU1 in a TR2BL partition.

[0139] In one variant, the spatial position is close to the border between 2 partitions, which position can be used for both lists, such as the top left position of a TL2BR partition. In case of AMVP, only one mvd is sent for a shared spatial position between two PUs. In one variant, the spatial position checked is different for the two PUs, but the same for all shapes. In another variant, other positions besides the spatial position are checked in HEVC merge, and depend on the partition, such as Fig.28 An example of horizontal partitioning is shown in Figure 1.

[0140] Intra- and inter-frame information propagation for multiple hypotheses and various shape combinations

[0141] When a CU is encoded with multiple hypotheses, both inter-motion information and intra-direction are stored and can be used for neighboring block prediction. An intra-coded block can use the intra-direction used in a neighboring block that has been encoded in multiple-hypothesis mode (and vice versa). In a CU using multiple hypotheses, motion vectors and intra-directions for prediction are stored for all sub-blocks within the CU, regardless of the intra-direction, such as Fig.29 Depicted.

[0142] In at least one embodiment, inter-frame and intra-frame information are stored for each sub-block, depending on how the sub-block is predicted, according to the storage by weight of the sub-block. Typically, for some sub-blocks, the intra-frame (inter-frame) information is not stored because the intra-frame (inter-frame) information has not been used in the prediction process of the sub-block, such as Fig.30 This embodiment advantageously reduces storage memory requirements.

[0143] Various embodiments relate to decoding. As used in this application, "decoding" may encompass, for example, all or part of a process performed on a received coded sequence in order to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by a decoder of various embodiments described in this application, such as Figure 4 and 8 The process presented in .

[0144] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or broadly to a broader decoding process will be apparent based on the context of the particular description and will be well understood by those skilled in the art.

[0145] Various embodiments relate to encoding. In a manner similar to the discussion above regarding "decoding", "encoding" as used in this application may encompass, for example, all or part of the processes performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more of the processes typically performed by an encoder, such as partitioning, differential encoding, transforms, quantization, and entropy encoding. In various embodiments, such processes also or instead include processes performed by an encoder of the various embodiments described in this application, such as Figure 4 and 8 process.

[0146] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or broadly to a broader encoding process will be apparent based on the context of the particular description and will be well understood by those skilled in the art.

[0147] Note that the grammatical elements used herein are descriptive terms. Thus, they do not exclude the use of other grammatical element names.

[0148] When the figures are presented as flow charts, it should be understood that they also provide block diagrams of the corresponding apparatus. Similarly, when the figures are presented as block diagrams, it should be understood that they also provide flow charts of the corresponding methods / processes.

[0149] The embodiments and aspects described herein may be implemented, for example, in a method or process, device, software program, data stream or signal. Even if only discussed in a single form of embodiment (e.g., discussed only as a method), the embodiments of the features discussed may also be implemented in other forms (e.g., devices or programs). The device may be implemented, for example, with appropriate hardware, software and firmware. The method may be implemented, for example, in a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit or a programmable logic device. The processor also includes a communication device, such as, for example, a computer, a mobile phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate information communication between end users.

[0150] References to "one embodiment" or "an embodiment" or "an implementation" or "an implementation" and other variations thereof indicate that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0151] Additionally, the present application may refer to “determining” various information. Determining information may include, for example, estimating information, calculating information, predicting information, or retrieving information from a memory.

[0152] Furthermore, the present application may refer to "accessing" various information. Accessing information may include, for example, one or more of: receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0153] Additionally, the present application may involve "receiving" various information. Receiving, like "accessing," is intended as a broad term. Receiving information may include, for example, one or more of: accessing the information or retrieving the information (e.g., from a memory). Furthermore, during operations such as storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is often involved in one way or another.

[0154] It should be understood that the use of any of the following " / ", "and / or", and "at least one of" is intended to cover selecting only the first listed option (A), or only the second listed option (B), or both options (A and B), such as in the case of "A / B", "where A and / or B", and "at least one of A and B". As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C", such phrases are intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A and B and C). This may be extended to as many items as listed as would be apparent to one of ordinary skill in this and related arts.

[0155] In addition, as used herein, the word "signal" especially refers to indicating something to the corresponding decoder. For example, in some embodiments, the encoder signals a specific mode. In this way, in an embodiment, the same parameters are used on both the encoder side and the decoder side. Therefore, for example, the encoder can send (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. On the contrary, if the decoder already has specific parameters as well as other parameters, signaling can be used without sending (implicit signaling) to only allow the decoder to know and select specific parameters. By avoiding sending any actual function, bit saving is achieved in various embodiments. It should be understood that signaling can be implemented in a variety of ways. For example, in various embodiments, one or more grammatical elements, flags, etc. are used to signal information to the corresponding decoder. Although the verb form of the word "signal" is mentioned above, the word "signal" can also be used as a noun in this article.

[0156] It will be apparent to one of ordinary skill in the art that embodiments may generate various signals that are formatted to carry information that may be stored or transmitted, for example. The information may include, for example, instructions for executing a method or data generated by one of the described embodiments. For example, a signal may be formatted to carry a bit stream of the described embodiment. Such a signal may be formatted as, for example, an electromagnetic wave (e.g., using a radio frequency portion of a spectrum) or a baseband signal. Formatting may include, for example, encoding a data stream and modulating a carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. As is well known, the signal may be sent over a variety of different wired or wireless links. The signal may be stored on a processor-readable medium.

[0157] We have described a number of embodiments. Features of these embodiments may be provided individually or in any combination. In addition, embodiments may include, individually or in any combination, one or more of the following features, devices, or aspects across various claim categories and types:

[0158] A syntax for handling prediction of a block when encoding or decoding a video, the prediction being performed using two different inter-frame predictions or a combination of inter-frame prediction and intra-frame prediction, wherein the syntax includes elements related to the prediction of the block. The syntax includes weighting between the two predictions at a sub-block level. The syntax includes storing motion vectors with corresponding weighted values ​​at a sub-block level.

[0159] A video encoding method comprises performing block prediction using two different inter predictions or a combination of inter prediction and intra prediction, and wherein weighting between the two predictions is performed at a sub-block level. A video encoding method comprises performing block prediction using two different inter predictions or a combination of inter prediction and intra prediction, and wherein weighting between the two predictions is performed at a sub-block level and storage of motion vectors with corresponding weighted values.

[0160] A video decoding method comprises performing block prediction using two different inter predictions or a combination of inter prediction and intra prediction, and wherein weighting between the two predictions is performed at a sub-block level. A video encoding method comprises performing block prediction using two different inter predictions or a combination of inter prediction and intra prediction, and wherein weighting between the two predictions is performed at a sub-block level and storage of motion vectors with corresponding weighted values.

[0161] A video encoder, a transcoder or another electronic device performs video encoding, including performing block prediction using two different inter-frame predictions or a combination of inter-frame prediction and intra-frame prediction, and wherein weighting between the two predictions is performed at a sub-block level. A video encoding method includes performing block prediction using two different inter-frame predictions or a combination of inter-frame prediction and intra-frame prediction, and wherein weighting between the two predictions is performed at a sub-block level and storage of motion vectors with corresponding weighted values.

[0162] A TV, set-top box, mobile phone, tablet computer or another electronic device performs video decoding, including performing block prediction using two different inter-frame predictions or a combination of inter-frame prediction and intra-frame prediction, and wherein weighting between the two predictions is performed at a sub-block level. A video encoding method includes performing block prediction using two different inter-frame predictions or a combination of inter-frame prediction and intra-frame prediction, and wherein weighting between the two predictions is performed at a sub-block level and storage of motion vectors with corresponding weighted values.

Claims

1. A method comprising: Determining to enable both multi-shape partitioning and multiple hypotheses for the image patch; as well as The image block is encoded based on a combination of multi-shape partitions and multiple hypothesis prediction modes.

2. The method according to claim 1, wherein: The multi-shape partition is a weighted triangular partition of each sub-block of the image block.

3. The method according to claim 2, wherein: The weight of the weighting is a power of 2, and the motion vector and the weight are stored for each sub-block of the image block.

4. The method according to claim 1, wherein: The number of sub-regions in the multiple hypotheses depends on the size of the image block.

5. The method according to claim 1, wherein: For the combination of multi-shape partitions and multiple hypothesis prediction modes, inter prediction is restricted to be unidirectional.

6. The method according to claim 1, wherein: The intra prediction mode is added as an additional candidate in the candidate list of the multi-shape partition prediction mode.

7. The method according to claim 1, wherein: Separate prediction quantity candidates are derived for List 0 and List 1 of multi-shape partition prediction modes.

8. The method of claim 1, further comprising deriving a separate predictor candidate for each prediction block of the image block for a multi-shape partition prediction mode.

9. The method according to claim 1, wherein: The image block is partitioned into at least two prediction units according to the multi-shape partitioning, and a weight used in the prediction is larger for a prediction corresponding to a prediction unit to which a sub-block of the image block belongs.

10. The method according to claim 1, wherein: Inter-frame and intra-frame information is stored for each sub-block of the image block depending on how the sub-block was predicted.

11. An apparatus comprising one or more processors, wherein: The one or more processors are configured to implement: Determining to enable both multi-shape partitioning and multiple hypotheses for the image block; and The image block is encoded based on a combination of multi-shape partitions and multiple hypothesis prediction modes.

12. The device according to claim 11, wherein: The multi-shape partition is a weighted triangular partition of each sub-block of the image block.

13. The device according to claim 12, wherein: The weight of the weighting is a power of 2, and the motion vector and the weight are stored for each sub-block of the image block.

14. The device according to claim 11, wherein: The number of sub-regions in the multiple hypotheses depends on the size of the image block.

15. The device according to claim 11, wherein For the combination of multi-shape partitions and multiple hypothesis prediction modes, inter prediction is restricted to be unidirectional.

16. The device according to claim 11, wherein The intra prediction mode is added as an additional candidate in the candidate list of the multi-shape partition prediction mode.

17. The device according to claim 11, wherein: Separate prediction quantity candidates are derived for List 0 and List 1 of multi-shape partition prediction modes.

18. The apparatus of claim 11, further comprising deriving a separate predictor candidate for each prediction block of the image block for a multi-shape partition prediction mode.

19. The device according to claim 11, wherein: The image block is partitioned into at least two prediction units according to the multi-shape partitioning, and a weight used in the prediction is larger for a prediction corresponding to a prediction unit to which a sub-block of the image block belongs.

20. The device according to claim 11, wherein Inter-frame and intra-frame information is stored for each sub-block of the image block depending on how the sub-block was predicted.

21. A method comprising: Determining to enable both multi-shape partitioning and multiple hypotheses for the image patch; as well as The image block is decoded based on a combination of multi-shape partitions and multiple hypothesis prediction modes.

22. The method according to claim 21, wherein: The multi-shape partition is a weighted triangular partition of each sub-block of the image block.

23. The method according to claim 22, wherein: The weight of the weighting is a power of 2, and the motion vector and the weight are stored for each sub-block of the image block.

24. The method according to claim 21, wherein: The number of sub-regions in the multiple hypotheses depends on the size of the image block.

25. The method according to claim 21, wherein: For the combination of multi-shape partitions and multiple hypothesis prediction modes, inter prediction is restricted to be unidirectional.

26. The method according to claim 21, wherein: The intra prediction mode is added as an additional candidate in the candidate list of the multi-shape partition prediction mode.

27. The method of claim 21, wherein: Separate prediction quantity candidates are derived for List 0 and List 1 of multi-shape partition prediction modes.

28. The method of claim 21, further comprising deriving a separate predictor candidate for each prediction block of the image block for a multi-shape partition prediction mode.

29. The method according to claim 21, wherein: The image block is partitioned into at least two prediction units according to the multi-shape partitioning, and a weight used in the prediction is larger for a prediction corresponding to a prediction unit to which a sub-block of the image block belongs.

30. The method of claim 21, wherein: Inter-frame and intra-frame information is stored for each sub-block of the image block depending on how the sub-block was predicted.

31. An apparatus comprising one or more processors, wherein: The one or more processors are configured to implement: Determining to enable both multi-shape partitioning and multiple hypotheses for the image block; and The image block is decoded based on a combination of multi-shape partitions and multiple hypothesis prediction modes.

32. The device according to claim 31, wherein The multi-shape partition is a weighted triangular partition of each sub-block of the image block.

33. The device according to claim 32, wherein: The weight of the weighting is a power of 2, and the motion vector and the weight are stored for each sub-block of the image block.

34. The device according to claim 31, wherein The number of sub-regions in the multiple hypotheses depends on the size of the image block.

35. The device according to claim 31, wherein For the combination of multi-shape partitions and multiple hypothesis prediction modes, inter prediction is restricted to be unidirectional.

36. The device according to claim 31, wherein The intra prediction mode is added as an additional candidate in the candidate list of the multi-shape partition prediction mode.

37. The device according to claim 31, wherein Separate prediction quantity candidates are derived for List 0 and List 1 of multi-shape partition prediction modes.

38. The apparatus of claim 31 , further comprising deriving a separate predictor candidate for each prediction block of the image block for a multi-shape partition prediction mode.

39. The device according to claim 31, wherein The image block is partitioned into at least two prediction units according to the multi-shape partitioning, and a weight used in the prediction is larger for a prediction corresponding to a prediction unit to which a sub-block of the image block belongs.

40. The device according to claim 31, wherein Inter-frame and intra-frame information is stored for each sub-block of the image block depending on how the sub-block was predicted.

Images