Video encoder, video decoder, and corresponding method

Abstract

To provide a method that is used for a video coding and decoding for performing a coding unit division and partitioning of an image or a video signal.SOLUTION: A coding method contains steps of: determining whether or not a size of a current block is larger than a minimum permission quadtree leaf node size; applying a binary tree division to the current block on the basis of the maximum permission binary tree root node size in the case where the size of the current block is not larger than the minimum permission quadtree leaf node size; generating a prediction block of the current block after the application of the binary tree division to the current block; acquiring a residual block of the current block on the basis of the current block and the prediction block; acquiring a conversion factor of the current block by applying a conversion to a sample value of the residual block; and coding a bit stream.SELECTED DRAWING: None

Description

[Technical field]

[0001] FIELD Embodiments of the present application relate generally to the field of video coding, and more specifically to coding unit splitting and partitioning. [Background technology]

[0002] Video coding (video encoding and decoding) is used in a wide range of digital video applications, such as broadcast digital TV, video transmission over the Internet and mobile networks, real-time conversation applications like video chat, video conferencing, DVDs and Blu-ray Discs, video content capture and editing systems, and camcorders for security applications.

[0003] With the development of the block-based hybrid video coding approach in the H.261 standard in 1990, new video coding techniques and tools have been developed, laying the foundation for new video coding standards. Further video coding standards include MPEG-1 Video, MPEG-2 Video, ITU-T H.262 / MPEG-2, ITU-T H.263, ITU-T H.264 / MPEG-4 Part 10, Advanced Video Coding (AVC), ITU-T H.265 / High Efficiency Video Coding (HEVC), ITU-T H.266 / Generic Video Coding (VVC) and extensions, e.g. scalability and / or three-dimensional (3D) extensions of such standards. As video production and usage becomes more and more prevalent, video traffic is the greatest burden on communication networks and data storage, and therefore one of the objectives of many video coding standards has been to achieve bit rate reductions compared to their predecessors without sacrificing image quality. The latest High Efficiency Video Coding (HEVC) is capable of compressing video at roughly twice the rate of AVC without sacrificing image quality, but new techniques to compress video even further compared to HEVC are still highly desirable.

[0004] The amount of video data required to depict even a relatively short video can be substantial, which can create difficulties when the data is streamed or otherwise communicated over communication networks that have limited bandwidth capacity. Thus, video data is typically compressed before being communicated over today's telecommunications networks. The size of the video can also be an issue when the video is stored on a storage device, since memory resources can be limited. Video compression devices often use software and / or hardware at the source to code the video data before transmission or storage, thereby reducing the amount of data required to represent a digital video image. The compressed data is then received at the destination by a video decompression device, which decodes the video data. With limited network resources and an ever-increasing demand for higher video quality, improved compression and decompression techniques that improve compression ratios with little sacrifice in image quality are desirable. Summary of the Invention

[0005] Embodiments of the present application (or disclosure) provide devices and methods for encoding and decoding according to the independent claims. These and other objects are achieved by the subject matter of the independent claims. Further implementation forms are evident from the dependent claims, the description and the drawings.

[0006] According to a first aspect, the invention relates to a video decoding method, the method being executed by a decoding device, comprising the steps of determining whether a size of a current block is greater than a minimum allowed quadtree leaf node size, and applying a multitype tree split to the current block if the size of the current block is not greater than the minimum allowed quadtree leaf node size, wherein the minimum allowed quadtree leaf node size is not greater than a maximum allowed binary tree root node size or the minimum allowed quadtree leaf node size is not greater than a maximum allowed ternary tree root node size.

[0007] The current block can be obtained by dividing an image or a coding tree unit (CTU).

[0008] The method may include two cases: 1) treeType is equal to SINGLE_TREE or DUAL_TREE_LUMA; 2) treeType is equal to DUAL_TREE_CHROMA. In case 1), the current block is a luma block, and in case 2), the current block is a chroma block.

[0009] The maximum allowable binary tree root node size may be the maximum luma size in luma samples of the luma coding root block that can be split using binary tree splitting.

[0010] The maximum allowable ternary tree root node size may be the maximum luma size in luma samples of a luma coding root block that can be divided using ternary tree division.

[0011] The minimum allowable quadtree leaf node size may be the minimum luma size in luma samples of a luma leaf block resulting from the quadtree partitioning.

[0012] This approach facilitates efficient partitioning or signaling of partitioning parameters for image / video blocks.

[0013] Furthermore, in a possible implementation form of the method according to the first aspect, the method further comprises a step of determining whether the current block of the picture is a border block. The step of applying a multi-type tree partition to the current block if the size of the current block is not larger than the minimum allowed quad-tree leaf node size comprises a step of applying a bisection to the current block if the current block is a border block and the size of the current block is not larger than the minimum allowed quad-tree leaf node size. It should be noted that in this case, the minimum allowed quad-tree leaf node size is not larger than the maximum allowed binary tree root node size. Thus, the above-mentioned step of applying a multi-type tree partition to the current block if the size of the current block is not larger than the minimum allowed quad-tree leaf node size and if the size of the current block is not larger than the maximum allowed binary tree root node size comprises a step of applying a bisection to the current block if the current block is a border block and the size of the current block is not larger than the minimum allowed quad-tree leaf node size.

[0014] The method may further comprise the step of obtaining a reconstructed block of a block obtained directly or indirectly from applying the bisection to the current block.

[0015] Providing a bisection can be particularly beneficial for blocks at image / video frame boundaries, e.g., blocks that are cut by the boundary. Thus, in some implementations, it may be beneficial to apply this approach to boundary blocks but not to the remaining blocks. However, the disclosure is not limited in this respect, and as discussed above, this approach of applying a bisection can also be applied and efficiently signaled for non-border blocks.

[0016] In a possible implementation form of the method according to the first aspect or the above-mentioned embodiment, the minimum allowable quadtree leaf node size is not greater than the maximum allowable binary tree root node size, and the minimum allowable quadtree leaf node size is not greater than the maximum allowable ternary tree root node size.

[0017] In a possible implementation of the method according to the first aspect or the above embodiments, applying a multi-type tree partition to the current block may include applying a three-way partition to the current block, or applying a two-way partition to the current block, although the disclosure is not so limited and in general the multi-type tree partition may also include further or other different types of partitions.

[0018] In a possible implementation of the method according to the first aspect or the above-mentioned embodiment, the method may further include a step of determining a maximum allowable binary tree root node size based on a minimum allowable quad-tree leaf node size. This facilitates efficient signaling / storage of the parameters. For example, the maximum allowable binary tree root node size may be considered to be equal to the minimum allowable quad-tree leaf node size. For another example, a lower bound of the maximum allowable binary tree root node size may be considered to be equal to the minimum allowable quad-tree leaf node size, and the minimum allowable quad-tree leaf node size may be used to determine the validity of the maximum allowable binary tree root node size. However, the present disclosure is not limited thereto, and other relationships may be assumed to derive the maximum allowable binary tree root node size.

[0019] According to an example embodiment, additionally or alternatively to the first aspect or the above-described embodiment, the method may further include dividing the image into blocks, the blocks including the current block. Applying the bisection to the current block includes applying the bisection to a boundary block having a maximum boundary multitype partition depth, the maximum boundary multitype partition depth being at least the sum of the maximum multitype tree depth and the maximum multitype tree depth offset, the maximum multitype tree depth being greater than 0. Further, in some implementations, when applying the bisection to the boundary block, the maximum multitype tree depth is greater than 0.

[0020] In a possible implementation of the method according to the first aspect or the above-mentioned embodiment, the method may further comprise a step of dividing the image into blocks (including the current block). The step of applying the multitype tree partitioning to the current block comprises a step of applying the multitype tree partitioning to the current block of a block having a final maximum multitype tree depth, the final maximum multitype tree depth being at least the sum of the maximum multitype tree depth and the maximum multitype tree depth offset, the maximum multitype tree depth being equal to or greater than the Lo2 value of the minimum allowed quadtree leaf node size minus the Lo2 value of the minimum allowed transformation block size, or the maximum multitype tree depth being equal to or greater than the Lo2 value of the minimum allowed quadtree leaf node size minus the Lo2 value of the minimum allowed coding block size. This encourages further partitioning even for larger partition depths.

[0021] The current block may be a non-border block. The maximum multitype tree depth offset may be 0. The current block may alternatively or additionally be a border block and the multitype tree split is a bipartite. The multitype tree split may be or include a tertiary split.

[0022] According to a second aspect, the invention relates to a method for encoding, the method being executed by an encoding device, comprising the steps of determining whether a size of a current block is greater than a minimum allowed quadtree leaf node size, and applying a multitype tree split to the current block if the size of the current block is not greater than the minimum allowed quadtree leaf node size, wherein the minimum allowed quadtree leaf node size is not greater than a maximum allowed binary tree root node size or the minimum allowed quadtree leaf node size is not greater than a maximum allowed ternary tree root node size.

[0023] The encoding method can apply any rules and constraints described above for the decoding method. Since the encoder and decoder sides must share a bitstream, in particular, the encoding side generates a bitstream after coding the partitions resulting from the above partitioning, while the decoding side analyzes the bitstream and reconstructs the decoded partitions accordingly. The same applies to the embodiments related to the encoding device (encoder) and the decoding device (decoder) described below.

[0024] According to a third aspect, the invention relates to a decoding device including a circuit configured to determine whether a size of a current block is larger than a minimum allowed quadtree leaf node size and to apply a multitype tree split to the current block if the size of the current block is not larger than the minimum allowed quadtree leaf node size, where the minimum allowed quadtree leaf node size is not larger than the maximum allowed binary tree root node size or the minimum allowed quadtree leaf node size is not larger than the maximum allowed ternary tree root node size. It is noted that determining whether a size of a current block is larger than the minimum allowed quadtree leaf node size may be performed based on a signaling in the bitstream at the decoding side.

[0025] According to a fourth aspect, the invention relates to an encoding device including a circuit configured to determine whether a size of a current block is greater than a minimum allowed quadtree leaf node size and to apply a multitype tree split to the current block if the size of the current block is not greater than the minimum allowed quadtree leaf node size, wherein the minimum allowed quadtree leaf node size is not greater than a maximum allowed binary tree root node size or wherein the minimum allowed quadtree leaf node size is not greater than a maximum allowed ternary tree root node size.

[0026] The method according to the first aspect of the invention can be performed by an apparatus or device according to the third aspect of the invention. Further features and implementation forms of the method according to the third aspect of the invention correspond to the features and implementation forms of the apparatus according to the first aspect of the invention.

[0027] The method according to the second aspect of the invention can be performed by an apparatus or device according to the fourth aspect of the invention. Further features and implementation forms of the method according to the fourth aspect of the invention correspond to the features and implementation forms of the apparatus according to the second aspect of the invention.

[0028] According to a fifth aspect, the invention relates to an apparatus for decoding a video stream comprising a processor and a memory, the memory storing instructions for causing the processor to carry out the method according to the first aspect.

[0029] According to a sixth aspect, the invention relates to an apparatus for encoding a video stream comprising a processor and a memory, the memory storing instructions for causing the processor to carry out the method according to the second aspect.

[0030] According to a seventh aspect, a computer readable storage medium is proposed having stored thereon instructions which, when executed, cause one or more processors configured for coding video data to perform a method according to the first or second aspect or any possible embodiment of the first or second aspect.

[0031] According to an eighth aspect, the present invention relates to a computer program comprising a program code for performing the method according to the first or second aspect or any possible embodiment of the first or second aspect, when the program is executed on a computer.

[0032] According to a ninth aspect, there is provided a non-transitory computer readable storage medium storing programming for execution by a processing circuit, the programming which, when executed by the processing circuit, configures the processing circuit to perform any of the methods set forth above.

[0033] For purposes of clarity, any one embodiment disclosed in the present application can be combined with any one or more other embodiments to produce new embodiments within the scope of the present disclosure.

[0034] The details of one or more embodiments are set forth in the accompanying drawings and description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims. [Brief description of the drawings]

[0035] Hereinafter, embodiments of the invention will be described in more detail with reference to the accompanying drawings and figures. [Figure 1A] FIG. 1 is a block diagram illustrating an example of a video coding system configured to implement embodiments of the present invention. [Figure 1B] FIG. 2 is a block diagram illustrating another example of a video coding system configured to implement embodiments of the present invention. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a video encoder configured to implement embodiments of the present invention. [Diagram 3] FIG. 2 is a block diagram illustrating an exemplary structure of a video decoder configured to implement an embodiment of the present invention. [Figure 4] FIG. 1 is a block diagram showing an example of an encoding device or a decoding device. [Diagram 5] FIG. 11 is a block diagram showing another example of an encoding device or a decoding device. [Figure 6] FIG. 1 is an illustrative diagram of an example of block partitioning using a quad-tree binary tree (QTBT) structure. [Figure 7] FIG. 7 is an illustrative diagram of an example of a tree structure corresponding to block partitioning using the QTBT structure of FIG. [Figure 8] FIG. 13 is an explanatory diagram of an example of a partition type of a horizontal ternary tree. [Figure 9] FIG. 13 is an explanatory diagram of an example of a vertical ternary tree partition type. [Figure 10] AF shows various CU division modes in VVC. [Figure 11A] This shows a forced QT partition for the lower border CTU (128x128) of HD (1920x1080). [Figure 11B] 1 illustrates a forced BT partition for a lower border CTU (128×128) of HD (1920×1080) according to an embodiment of the present disclosure. [Figure 12] 1 illustrates an exemplary boundary definition. [Figure 13A] 1 illustrates an example of a corner case forced QTBT partition according to an embodiment of the present disclosure. [Figure 13B] 1 illustrates an example of mandatory QTBT partitions for corner-located blocks according to an embodiment of the present disclosure. [Figure 14] 1 illustrates an embodiment of a boundary definition. [Figure 15] FIG. 2 is a block diagram illustrating an example of a video encoder configured to implement an embodiment of the present invention. [Figure 16] FIG. 2 is a block diagram illustrating an example structure of a video decoder configured to implement an embodiment of the present invention. [Figure 17] FIG. 31 is a block diagram showing an example of the structure of a content supply system 3100 that realizes a content distribution service. [Figure 18] FIG. 2 is a block diagram illustrating the structure of an example of a terminal device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] In the following description, reference is made to the accompanying drawings which form a part of this disclosure and which show, by way of example, specific aspects of embodiments of the invention or in which embodiments of the invention may be used. It is understood that embodiments of the invention may be used in other ways and may include structural or logical changes that are not shown in the accompanying drawings. Therefore, the following detailed description is not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.

[0037] For example, it is understood that disclosure related to a described method may also apply to a corresponding device or system configured to perform the method, and vice versa. For example, if one or more particular method steps are described, the corresponding device may include one or more units, such as functional units, for performing the described one or more method steps (e.g., one unit performing one or more steps, or multiple units each performing one or more of the multiple steps), even if such one or more units are not explicitly described or illustrated in the drawings. On the other hand, if a particular apparatus is described based on one or more units, such as functional units, the corresponding method may include one step for performing the function of the one or more units (e.g., one step for performing the function of the one or more units, or multiple steps each performing one or more functions of the multiple units), even if such one or more steps are not explicitly described or illustrated in the drawings. Furthermore, it is understood that the features of various exemplary embodiments and / or aspects described herein may be combined with each other, unless otherwise specified.

[0038] Video coding typically refers to the processing of a video or a series of pictures forming a video sequence. In the field of video coding, the terms "frame" or "image" may be used synonymously instead of the term "picture". Video coding as used in this application (or in this disclosure) refers to video encoding or video decoding. Video encoding is performed at the source side and typically involves processing an original video picture (e.g., by compression) to reduce the amount of data required to represent the video picture (for more efficient storage and / or transmission). Video decoding is performed at the destination side and typically involves the reverse processing compared to the encoder to reconstruct the video picture. Any embodiment to the "coding" of a video picture (or, as explained below, a picture in general) shall be understood to relate to the "encoding" or "decoding" with respect to a video sequence. The combination of the encoding and decoding parts is also referred to as CODEC (Coding and Decoding).

[0039] In the case of lossless video coding, it is possible to reconstruct the original video picture, i.e. the reconstructed video picture has the same quality as the original video picture (assuming that no transmission or other data losses occur during storage and transmission). In the case of non-lossless video coding, further compression is performed, e.g. by quantization, to reduce the amount of data representing the video picture, and the video picture cannot be perfectly reconstructed at the decoder side, i.e. the quality of the reconstructed video picture is lower or worse than the quality of the original video picture.

[0040] Several video coding standards since H.261 belong to the group of "non-lossless hybrid video codecs" (i.e. they combine spatial and temporal prediction in the sample domain with 2D transform coding to apply quantization in the transform domain). Each picture of a video sequence is typically partitioned into a set of non-overlapping blocks and coding is typically performed at the block level. In other words, at the encoder side, a video is typically processed or encoded at the block (video block) level, for example by generating a prediction block using spatial (intra-picture) and temporal (inter-picture) prediction, subtracting the prediction block from a current block (current processed / to-be processed block) to obtain a residual block, transforming the residual block, and quantizing the residual block in the transform domain to reduce the amount of data to be transmitted (compression); at the decoder side, the inverse processing compared to the encoder is partially applied to the coded or compressed block to reconstruct the current block for representation. Furthermore, the encoder repeats the decoder's processing loop, so that they will both generate the same predictions (eg, intra and inter predictions) and / or reconstructions for processing, i.e., coding, future blocks.

[0041] As used herein, the term "block" may be a portion of a picture or a frame. For ease of explanation, embodiments of the present invention are described with reference to the reference software of Generic Video Coding (VVC) or High Efficiency Video Coding (HEVC) developed by the ITU-T Video Coding Experts Group (VCEG) and the Joint Research Team on Video Coding (JCT-VC) of the ISO / IEC Moving Picture Experts Group (MPEG). Those skilled in the art will appreciate that embodiments of the present invention are not limited to HEVC or VVC. This may refer to coding units (CU), prediction units (PU) or transform units (TU). In HEVC, a coding tree unit (CTU) is divided into multiple CUs by using a quadtree structure, denoted as a coding tree. The decision of whether to code a picture using inter-picture (temporal) or intra-picture (spatial) prediction is made at the CU level. Each CU may be further divided into one, two or four PUs according to the PU division type. Within one PU, the same prediction process is applied and related information is transmitted to the decoder based on the PU. After obtaining the residual block by applying the prediction process based on the PU partition type, the CU can be divided into transform units (TUs) according to another quadtree structure similar to the coding tree for CUs. In the latest development of video compression technology, quadtree and binary tree (QTBT) partitioning frames are used to divide the coding blocks. In the QTBT block structure, the CUs can be square or rectangular. For example, the coding tree units (CTUs) are first partitioned by the quadtree structure. The quadtree leaf nodes are further partitioned by the binary tree structure. The binary tree leaf nodes are referred to as coding units (CUs), and their segmentation is used for prediction and transform processing without any further partitioning.This means that the CU, PU, ​​and TU have the same block size in the QTBT coding block structure. At the same time, multiple partitions, such as ternary tree (TT) partitions, are also proposed to be used with the QTBT block structure. The term "device" may also be "equipment", "decoder", or "encoder".

[0042] In the following, embodiments of the encoder 20, the decoder 30 and the coding system 10 will be described with reference to FIGS. 1-3.

[0043] 1A is a conceptual or schematic block diagram illustrating an example coding system 10, e.g., a video coding system 10 capable of using techniques of the present disclosure. An encoder 20 (e.g., video encoder 20) and a decoder 30 (e.g., video decoder 30) of video coding system 10 represent example devices that may be configured to perform techniques according to various examples described herein. As shown in FIG. 1A, coding system 10 includes a source device 12 configured to provide encoded data 13, e.g., encoded pictures 13, to a destination device 14, e.g., which decodes encoded data 13.

[0044] The source device 12 includes an encoder 20 and may additionally, i.e. optionally, include a picture source 16 , a pre-processing unit 18 , such as a picture pre-processing unit 18 , and a communication interface or unit 22 .

[0045] The video source 16 may include or be any kind of picture capture device, e.g. capturing real-world pictures, and / or e.g. a picture or comment generation device (in case of screen content coding, any text on the screen is also considered as part of the picture or image to be coded), e.g. a computer graphics processor generating computer animated pictures, or any kind of device obtaining and / or providing real-world or computer animated pictures (e.g. screen content or Virtual Reality (VR) pictures), and / or any combination thereof (e.g. Augmented Reality (AR) pictures). The picture source could be any kind of memory or storage storing any of the above mentioned pictures.

[0046] A (digital) picture is, or can be thought of as, a two-dimensional array or matrix of samples with intensity values. The samples in the array may be called pixels (short for picture element) or pels. The amount of samples in the horizontal and vertical directions (or axes) of the array or picture defines the size and / or resolution of the picture. For color representation, typically three color components are used, i.e. a picture may be represented with or may contain three sample arrays. In an RBG format or color space, a picture contains corresponding red, green, and blue sample arrays. However, in video coding, each pixel is typically represented in a luminance / chrominance format or color space, e.g. YCbCr, which contains a luminance component denoted Y (sometimes L is used instead) and two chrominance components denoted Cb and Cr. The luminance (or abbreviated luma) component Y represents the brightness or gray level intensity (as in a gray scale picture, for example), and the two chrominance (or abbreviated chroma) components Cb and Cr represent the chrominance or color information components. Thus, a picture in YCbCr format contains a luminance sample array of luminance sample values ​​(Y) and two chrominance sample arrays of chrominance values ​​(Cb and Cr). A picture in RGB format can be converted or transformed to YCbCr format and vice versa, a process also known as color conversion or transformation. If a picture is monochrome, it may contain only a luminance sample array.

[0047] The picture source 16 (e.g. video source 16) may be, for example, a camera capturing a picture, a memory, such as, for example, a picture memory, containing or storing a pre-captured or generated picture, and / or any kind of (internal or external) interface for acquiring or receiving a picture. The camera may be, for example, a local or integrated camera integrated in the source device, and the memory may be, for example, a local or integrated memory integrated in the source device. The interface may be, for example, an external interface for receiving a picture from an external video source, such as, for example, an external picture capturing device, such as, for example, a camera, an external memory, or an external picture generating device, such as, for example, an external computer graphics processor, computer or server. The interface can be any kind of interface, such as, for example, a wired or wireless interface, an optical interface, following any proprietary or standardized interface protocol. The interface for acquiring the picture data 17 may be the same interface as the communication interface 22 or may be part of it.

[0048] As distinct from pre-processing unit 18 and the processing performed by pre-processing unit 18 , the pictures and picture data 17 (eg, video data 16 ) may also be referred to as unprocessed pictures or unprocessed picture data 17 .

[0049] The pre-processing unit 18 is configured to receive the (unprocessed) picture data 17 and perform pre-processing on the picture data 17 to obtain a pre-processed picture 19 or pre-processed picture data 19. The pre-processing performed by the pre-processing unit 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It is possible to understand that the pre-processing unit 18 may be an optional component.

[0050] An encoder 20 (e.g. a video encoder 20) is configured to receive the pre-processed picture data 19 and to provide encoded picture data 21 (further details will be explained, e.g. on the basis of FIG. 2 or FIG. 4 below).

[0051] The communications interface 22 of the source device 12 can be configured to receive the encoded picture data 21 and to transmit the encoded picture data 21 (or any further processed version of it) via the communications channel 13 to another device, such as the destination device 14 or any other device, for storage or direct reconstruction.

[0052] The communications interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and transmit it to another device, such as the destination device 14 or any other device, for storage or direct reconstruction, or to process the encoded picture data 21 before storing the encoded data 13 and / or transmitting the encoded data 13 to another device, such as the destination device 14 or any other device, for decoding or storage, respectively.

[0053] The destination device 14 includes a decoder 30 (eg, a video decoder 30 ) and may additionally, i.e. optionally, include a communications interface or unit 28 , a post-processing unit 32 , and a display device 34 .

[0054] The communications interface 28 of the destination device 14 is configured to receive the encoded picture data 21 (or any further processed version thereof), e.g. directly from the source device 12 or from any other source, e.g. from a source device, e.g. from a coded picture data storage device, and to provide the encoded picture data 21 to a decoder 30.

[0055] The communications interface 28 of the destination device 14 is configured to receive the encoded picture data 21 or the encoded data 13, e.g. directly from the source device 12 or from any other source, e.g. from a storage device, e.g. from a coded picture data storage device.

[0056] The communication interface 22 and the communication interface 28 may be configured to transmit or receive the encoded picture data 21 or the encoded data 13 via a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or via any kind of network, such as a wired or wireless network or any combination thereof, or any kind of private and public network, or any combination thereof.

[0057] The communications interface 22 may, for example, be configured to package the encoded picture data 21 into a suitable format, such as packets, and / or process the encoded picture data using any type of transmission coding or processing for transmission over a communications link or network.

[0058] The communications interface 28 , which forms a counterpart of the communications interface 22 , may for example be configured to unpackage the encoded data 13 to obtain the encoded picture data 21 .

[0059] The communications interface 28, which forms a counterpart of the communications interface 22, for example receives the transmitted data and processes the transmitted data using any kind of corresponding transmission decoding or processing and / or unpackaging to obtain the encoded picture data 21.

[0060] Both communication interface 22 and communication interface 28 may be configured as a unidirectional communication interface, as indicated by the arrow in FIG. 1A for encoded picture data 13 from source device 12 to destination device 14, or as a bidirectional communication interface, and may be configured, e.g., to send and receive messages, e.g., to set up connections, to verify and exchange communication links and / or any other information related to data transmission, such as encoded picture data transmission.

[0061] The decoder 30 is arranged to receive the encoded picture data 21 and to provide decoded picture data 31 or decoded pictures 31 (further details will be explained below, e.g. on the basis of FIG. 3 or FIG. 5).

[0062] The post-processing processor 32 of the destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), e.g., a decoded picture 31, to obtain post-processed picture data 33, e.g., a post-processed picture 33. The post-processing performed by the post-processing unit 32 may include, e.g., color format conversion (e.g., from YCbCr to RGB), color correction, cropping, resampling, or any other processing to prepare, e.g., the decoded picture data 31 for display by, e.g., a display device 34.

[0063] A display device 34 of destination device 14 is configured to receive the post-processed picture data 33 and display the picture, for example to a user or viewer. The display device 34 may be or include any type of display that presents the reconstructed picture, for example an integrated or external display or monitor. The display may include, for example, a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0064] 1A depicts source device 12 and destination device 14 as separate devices, an embodiment of the devices may also include both or both functionality, source device 12 or corresponding functionality and destination device 14 or corresponding functionality. In such an embodiment, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0065] As will be clear to a person skilled in the art based on the specification, the presence and (exact) division of functions or functions of various units in the source device 12 and / or destination device 14 shown in FIG. 1A may vary depending on the actual device and application.

[0066] The encoder 20 (e.g., video encoder 20) and the decoder 30 (e.g., video decoder 30), each may be implemented as any of a variety of suitable circuits, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, hardware, or any combination thereof. Where the techniques are implemented in part in software, the device may store instructions of the software in a suitable non-transitory computer readable storage medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Any of the above (including hardware, software, a combination of hardware and software, etc.) may be considered to be one or more processors. The video encoder 20 and the video decoder 30 may each be included in one or more encoders or decoders, and either may be integrated as part of a combined encoder / decoder (CODEC) within a respective device.

[0067] The encoder 20 may be implemented via a processing circuit 46 to embody various modules as described with respect to the encoder 20 of FIG. 2 and / or any other encoder system or subsystem described herein. The decoder 30 may be implemented via a processing circuit 46 to embody various modules as described with respect to the decoder 30 of FIG. 3 and / or any other decoder system or subsystem described herein. The processing circuit may be configured to perform various operations as described below. If the techniques are implemented in part in software, as shown in FIG. 5, a device may store the software instructions in a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Either the video encoder 20 or the video decoder 30 may be integrated as part of a combined encoder / decoder (CODEC) in a single device, as shown in FIG. 1B, for example.

[0068] Source device 12 may be referred to as a video encoding device or video encoding apparatus, and destination device 14 may be referred to as a video decoding device or video decoding apparatus. Source device 12 and destination device 14 may be examples of video coding devices or video coding apparatus.

[0069] The source device 12 and destination device 14 may include any of a wide range of devices, including any type of handheld or stationary device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or content delivery server), a broadcast receiving device, a broadcast transmitting device, etc., and may or may not use any type of operating system.

[0070] In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

[0071] In some cases, the video coding system 10 shown in FIG. 1A is merely an example, and the techniques of the present application may be applied to video coding settings (e.g., video encoding or video decoding) that do not need to include any data communication between the encoding device and the decoding device. In other examples, data may be retrieved from a local memory, streamed over a network, etc. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other, but simply encode data to memory and / or retrieve data from memory and decode it.

[0072] For ease of explanation, embodiments of the present invention are described herein with reference to reference software of, for example, High Efficiency Video Coding (HEVC) or Generic Video Coding (VVC), a next-generation video coding standard developed by the ITU-T Video Coding Experts Group (VCEG) and the ISO / IEC Moving Picture Experts Group (MPEG) Joint Task Force on Video Coding (JCT-VC). Those skilled in the art will appreciate that embodiments of the present invention are not limited to HEVC or VVC. With respect to each of the above examples described with respect to video encoder 20, it should be understood that video decoder 30 can be configured to perform the reverse process. With respect to signaling syntax elements, video decoder 30 can be configured to receive and parse such syntax elements and decode associated video data accordingly. In some examples, video encoder 20 can entropy encode one or more semantics into an encoded video bitstream. In such examples, video decoder 30 can parse such syntax elements and decode associated video data accordingly.

[0073] 1B is an example diagram of another example video coding system 40 including the encoder 20 of FIG. 2 according to an example embodiment and / or the decoder 30 of FIG. 3 according to an example embodiment. The system 40 can implement techniques according to various examples described herein. In the illustrated implementation, the video coding system 40 can include an imager 41, a video encoder 100, a video decoder 30 (and / or a video coder implemented by logic circuitry 47 of a processing unit 46), an antenna 42, one or more processors 43, one or more memory stores 44, and / or a display device 45.

[0074] As shown, imager 41, antenna 42, processing unit 46, logic circuitry 47, video encoder 20, video decoder 30, processor 43, memory store 44, and / or display device 45 may be in communication with one another. As noted above, although illustrated with both video encoder 20 and video decoder 30, video coding system 40 may include only video encoder 20 or only video decoder 30 in various examples.

[0075] As shown, in some examples, video coding system 40 may include antenna 42. Antenna 42 may be configured to transmit or receive, for example, an encoded bitstream of video data. Additionally, in some examples, video coding system 40 may include display device 45. Display device 45 may be configured to present the video data. As shown, in some examples, logic circuitry 47 may be implemented by processing unit 46. Processing unit 46 may include application specific integrated circuit (ASIC) logic, graphics processor, general purpose processor, etc. Additionally, video coding system 40 may also include optional processor 43, which may also include application specific integrated circuit (ASIC) logic, graphics processor, general purpose processor, etc. In some examples, logic circuitry 47 may be implemented by hardware, video coding dedicated hardware, etc., and processor 43 may implement general purpose software, an operating system, etc. Further, memory store 44 may be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM)), or non-volatile memory (e.g., flash memory). In a non-limiting example, memory store 44 may be implemented by a cache memory. In some examples, logic circuitry 47 may access memory store 44 (e.g., for implementation of an image buffer). In other examples, logic circuitry 47 and / or processing unit 46 may include a memory store (e.g., a cache, etc.) for implementation of an image buffer, etc.

[0076] In some examples, video encoder 100 implemented with logic circuitry may include an image buffer (e.g., by processing unit 46 or memory store 44) and a graphics processing unit (e.g., by processing unit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video encoder 100 implemented with logic circuitry 47 to embody various modules such as those described with respect to FIG. 2 and / or any other encoder system or subsystems described herein. The logic circuitry may be configured to perform various operations as described herein.

[0077] Video decoder 30 may be implemented in a manner similar to that implemented by logic circuitry 47 to implement various modules as described with respect to decoder 30 of FIG. 3 and / or any other decoder system or subsystem described herein. In some examples, video decoder 30 may be implemented by logic circuitry and may include an image buffer (e.g., by processing unit 420 or memory storage store 44) and a graphics processing unit (e.g., by processing unit 46). The graphics processing unit may be communicatively coupled to the image buffer. The graphics processing unit may include video decoder 30 as implemented by logic circuitry 47 to implement various modules as described with respect to decoder 30 of FIG. 3 and / or any other decoder system or subsystem described herein.

[0078] In some examples, antenna 42 of video coding system 40 can be configured to receive an encoded bitstream of video data. As mentioned above, the encoded bitstream may include data related to encoding video frames as described herein, such as data related to coding partitions, indicators, index values, mode selection data, etc. (e.g., transform coefficients or quantized transform coefficients, optional indicators (as discussed), and / or data defining coding partitions). Video coding system 40 may also include video decoder 30 coupled to antenna 42 and configured to decode the encoded bitstream. Display device 45 is configured to present the video frames.

[0079] FIG. 2 is a schematic / conceptual block diagram of an example video encoder 20 configured to implement the techniques of the present disclosure. In the example of FIG. 2, the video encoder 20 includes a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a buffer 216, a loop filter unit 220, a decoded picture buffer (DPB) 230, a prediction processing unit 260, and an entropy coding unit 270. The prediction processing unit 260 may include an inter prediction unit 244, an intra prediction unit 254, and a mode selection unit 262. The inter prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). The video encoder 20 illustrated in FIG. 2 may also be referred to as a hybrid video encoder or a video encoder with a hybrid video codec.

[0080] For example, the residual calculation unit 204, the transform processing unit 206, the quantization unit 208, the prediction processing unit 260, and the entropy coding unit 270 form a forward signal path of the encoder 20, and for example, the inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the buffer 216, the loop filter 220, the decoded picture buffer (DPB) 230, and the prediction processing unit 260 form a backward signal path of the encoder, which corresponds to the signal path of the decoder (see decoder 30 in FIG. 3).

[0081] The inverse quantization unit 210, the inverse transform processing unit 212, the reconstruction unit 214, the loop filter 220, the decoded picture buffer (DPB) 230, the inter prediction unit 244, and the intra prediction unit 254 are also referred to as forming a "built-in decoder" of the video encoder 20. The encoder 20 is configured to receive, e.g., a picture of a sequence of pictures forming a video or a video sequence, e.g., via an input 202, a picture 201, or a block 203 of the picture 201. The picture block 203 is also referred to as a current picture block or a picture block to be coded, and the picture 201 is referred to as a current picture or a picture to be coded (particularly in video coding to distinguish the current picture from other pictures, e.g., from previously coded and / or decoded pictures of the same video sequence, i.e., a video sequence that also includes the current picture).

[0082] A (digital) picture is, or can be thought of as, a two-dimensional array or matrix of samples with intensity values. The samples in the array may also be called pixels (short for picture element) or pels. The amount of samples in the horizontal and vertical directions (or axes) of the array or picture defines the size and / or resolution of the picture. For color representation, typically three color components are used, i.e. a picture may be represented with or may contain three sample arrays. In an RBG format or color space, a picture contains corresponding red, green, and blue sample arrays. However, in video coding, each sample is typically represented in a luminance / chrominance format or color space, e.g. YCbCr, which contains a luminance component denoted Y (sometimes L is used instead) and two chrominance components denoted Cb and Cr. The luminance (or abbreviated luma) component Y represents the brightness or gray level intensity (as in a gray scale picture, for example), and the two chrominance (or abbreviated chroma) components Cb and Cr represent the chrominance or color information components. Thus, a picture in YCbCr format contains a luminance sample array of luminance sample values ​​(Y) and two chrominance sample arrays of chrominance values ​​(Cb and Cr). A picture in RGB format can be converted or transformed into YCbCr format and vice versa, a process also known as color transformation or conversion. If the picture is monochrome, the picture may contain only a luminance sample array. Thus, a picture may be, for example, an array of luma samples in monochrome format, or an array of luma samples and two corresponding arrays of chroma samples in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0083] Partitioning An embodiment of the encoder 20 may include a partitioning unit (not shown in FIG. 2) configured to partition a picture 201 into multiple (typically non-overlapping) picture blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC) or coding tree blocks (CTBs) or coding tree units (CTUs) (H.265 / HEVC and VVC). The partitioning unit may be configured to use the same block size for all pictures of a video sequence and a corresponding grid that defines the block size, or to vary the block size between pictures, subsets, or groups of pictures and partition each picture into corresponding blocks.

[0084] In further embodiments, the video encoder may be configured to directly receive blocks 203 of picture 201, such as one, some, or all of the blocks forming picture 201. Picture blocks 203 may also be referred to as current picture blocks or picture blocks to be coded. In one example, prediction processing unit 260 of video encoder 20 may be configured to perform any combination of the partitioning techniques described above.

[0085] Similar to the picture 201, the block 203 is or can be considered as a two-dimensional array or matrix of samples with intensity values ​​(sample values), again with smaller dimensions than the picture 201. In other words, the block 203 may contain, for example, one sample array (e.g., a luma array in the case of a monochrome picture 201) or three sample arrays (e.g., a luma and two chroma arrays in the case of a color picture 201), or any other quantity and / or type of arrays depending on the color format applied. The number of samples in the horizontal and vertical directions (or axes) of the block 203 defines the size of the block 203. Thus, the block may be, for example, an MxN (M columns and N rows) array of samples, or an MxN array of transform coefficients.

[0086] As shown in FIG. 2, the encoder 20 is arranged to encode a block of a picture 201 on a block-by-block basis, eg encoding and prediction is performed on a block-by-block basis 203 .

[0087] The embodiment of video encoder 20 shown in FIG. 2 may be further configured to partition and / or encode a picture through the use of slices (also referred to as video slices), where a picture may be partitioned into or encoded using one or more slices (typically non-overlapping), each of which may include one or more blocks (e.g., CTUs) or one or more groups of blocks (e.g., tiles (H.265 / HEVC and VVC) or bricks (VVC)).

[0088] The embodiment of video encoder 20 shown in FIG. 2 may be further configured to partition and / or encode a picture by using slice / tile groups (also referred to as video tile groups) and / or tiles (also referred to as video tiles), where a picture may be partitioned into or encoded using one or more slice / tile groups (typically non-overlapping), each slice / tile group may include, for example, one or more blocks (e.g., CTUs) or one or more tiles, where each tile may be, for example, rectangular in shape and may include one or more blocks (e.g., CTUs), e.g., complete or fractional blocks.

[0089] Residual calculation The residual calculation unit 204 is configured to calculate a residual block 205 based on the picture block 203 and the prediction block 265 (further details about the prediction block 265 will be described later), e.g. by subtracting sample values ​​of the prediction block 265 sample-by-sample (pixel-by-pixel) from sample values ​​of the picture block 203 to obtain the residual block 205 in the sample domain.

[0090] conversion The transform processing unit 206 is configured to apply a transform, such as a discrete cosine transform (DCT) or a discrete sine transform (DST), on the sample values ​​of the residual block 205 to obtain transform coefficients 207 in a transform domain. The transform coefficients 207 are also called transform residual coefficients and represent the residual block 205 in the transform domain.

[0091] The transform processing unit 206 may be configured to apply an integer approximation of a DCT / DST, such as the transform specified for HEVC / H.265. Compared to an orthogonal DCT transform, such an integer approximation is typically scaled by a factor. To ensure the norm of the residual block processed by the forward and inverse transforms, an additional scaling factor is applied as part of the transform process. The scaling factor is typically selected based on certain constraints, such as a scaling factor that is a power of two due to shift operations, the bit depth of the transform coefficients, a trade-off between accuracy and implementation cost, etc. A particular scaling factor may be specified for the inverse transform, e.g., by the inverse transform processing unit 212 in the decoder 30 (and the corresponding inverse transform, e.g., by the inverse transform processing unit 212 in the encoder 20), and a corresponding scaling factor for the forward transform, e.g., by the transform processing unit 206 in the encoder 20, may be specified accordingly.

[0092] An embodiment of the video encoder 20 (individual transform processing units 206) may be configured to output transform parameters, e.g., a type of transform or multiple transforms, either directly or encoded or compressed by an entropy coding unit 270, so that the video decoder 30 can receive and use the transform parameters for decoding.

[0093] Quantization The quantization unit 208 is configured to quantize the transform coefficients 207, for example by applying scalar quantization or vector quantization, to obtain quantized transform coefficients 209. The quantized transform coefficients 209 may also be referred to as quantized residual coefficients 209. The quantization process may reduce a bit depth associated with some or all of the transform coefficients 207. For example, an n-bit transform coefficient may be rounded to an m-bit transform coefficient during quantization, where n is greater than m. The degree of quantization may be modified by adjusting a quantization parameter (QP). For example, for scalar quantization, different scaling may be applied to achieve finer or coarser quantization. A smaller quantization step corresponds to a finer quantization, and a larger quantization step corresponds to a coarser quantization. The applicable quantization step size may be indicated by the quantization parameter (QP). For example, the quantization parameter may be an index to a predefined set of applicable quantization step sizes. For example, a smaller quantization parameter may correspond to a finer quantization (smaller quantization step size) and a larger quantization parameter may correspond to a coarser quantization (larger quantization step size), and vice versa. Quantization may include division by the quantization step size and corresponding or inverse quantization, such as by the inverse quantization unit 210, and may include multiplication by the quantization step size. Some standards, such as HEVC, may be configured to use the quantization parameter to determine the quantization step size. In general, the quantization step size may be calculated based on the quantization parameter using a fixed-point approximation of an equation that includes a division. An additional scaling factor may be introduced for quantization and inverse quantization to restore norms of the residual block that may be modified due to the scaling used in the fixed-point approximation of the equation for the quantization step size and the quantization parameter. In one example implementation, the scaling of the inverse transform and inverse quantization may be combined.Alternatively, customized quantization tables may be used and signaled from the encoder to the decoder, e.g., in the bitstream. Quantization is a non-lossless operation: increasing the quantization step size increases the loss.

[0094] An embodiment of the video encoder 20 (individual transform processing units 206) may be configured to output a quantization parameter (QP), e.g., directly or encoded by an entropy encoding unit 270, so that the video decoder 30 can receive and apply the quantization parameter for decoding.

[0095] The inverse quantization unit 210 is configured to apply the inverse quantization of the quantization unit 208 to the quantized coefficients to obtain inverse quantized coefficients 211, e.g., by applying the inverse of the quantization scheme applied by the quantization unit 208 based on or using the same quantization step as the quantization unit 208. The inverse quantized coefficients 211, which may also be referred to as inverse quantized residual coefficients 211, correspond to the transform coefficients 207, but due to losses due to quantization, typically do not match the transform coefficients.

[0096] The inverse transform processing unit 212 is configured to apply an inverse transform of the transform applied by the transform processing unit 206, such as an inverse discrete cosine transform (DCT) or an inverse discrete sine transform (DST), to obtain an inverse transform block 213 in the sample domain. The inverse transform block 213 may also be referred to as an inverse transform inverse quantization block 213 or an inverse transform residual block 213.

[0097] The reconstruction unit 214 (e.g., adder 214) is configured to add the inverse transform block 213 (i.e., the reconstructed residual block 213) to the prediction block 265 to obtain a reconstructed block 215 in the sample domain, e.g. by adding sample values ​​of the reconstructed residual block 213 and sample values ​​of the prediction block 265.

[0098] Optionally, a buffer unit 216 (or "buffer" 216 for short), e.g. a line buffer 216, is configured to buffer or store the reconstructed block 215 and the individual sample values, e.g. for intra prediction. In further embodiments, the encoder may be configured to use the unfiltered reconstructed block and / or the corresponding sample values ​​stored in the buffer unit 216 for any kind of estimation and / or prediction, e.g. intra prediction.

[0099] Embodiments of the encoder 20 may be configured, for example, such that the buffer unit 216 is used not only for storing the reconstructed block 215 for intra prediction 254, but also for the loop filter unit 220 (not shown in FIG. 2), and / or such that the buffer unit 216 and the decoded picture buffer unit 230 form one buffer. Other embodiments may be configured to use blocks or samples from the filtered block 221 and / or the decoded picture buffer 230 (both not shown in FIG. 2) as input or basis for the intra prediction 254.

[0100] The loop filter unit 220 (or "loop filter" 220 for short) is configured to filter the reconstructed block 215, e.g., to smooth pixel transitions or otherwise improve video quality, to obtain a filtered block 221. The loop filter unit 220 is intended to represent one or more loop filters, such as a deblocking filter, a sample adaptive offset (SAO) filter, or other filters, e.g., a bilateral filter, an adaptive loop filter (ALF), a sharpening or smoothing filter, or a collaborative filter. Although the loop filter unit 220 is illustrated in FIG. 2 as an in-loop filter, in other configurations, the loop filter unit 220 may be implemented as a post-loop filter. The filtered block 221 may also be referred to as a filtered reconstructed block 221. The decoded picture buffer 230 may store the reconstructed coding block after the loop filter unit 220 performs a filtering operation on the reconstructed coding block.

[0101] The loop filter unit 220 (or "loop filter" 220 for short) is configured to filter the reconstruction block 215 to obtain a filtered block 221, and in general to filter the reconstructed samples to obtain filtered sample values. The loop filter unit is configured to, for example, smooth pixel transitions or otherwise improve video quality. The loop filter unit 220 may include one or more loop filters, such as a deblocking filter, a sample adaptive offset (SAO) filter, or one or more other filters, for example an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. In one example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of filtering processes may be deblocking filter, SAO, and ALF. In another example, a process called luma mapping with chroma scaling (LMCS) (i.e., an adaptive in-loop reshaper) is added. This process is performed before deblocking. In another example, the deblocking filter process may also be applied to interior sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra-sub-partition (ISP) edges. Although the loop filter unit 220 is illustrated in FIG. 2 as an in-loop filter, in other configurations, the loop filter unit 220 may be implemented as a post-loop filter. The filtered block 221 may also be referred to as a filtered reconstruction block 221.

[0102] An embodiment of the video encoder 20 (individual loop filter unit 220) may be configured to output loop filter parameters (such as SAO filter parameters or ALF filter parameters or LMCS parameters), e.g., directly or encoded by the entropy decoding unit 270, so that e.g. the decoder 30 can receive and apply the same loop filter parameters or individual loop filters for decoding.

[0103] An embodiment of the encoder 20 (individual loop filter units 220) may be configured to output output filter parameters (such as sample adaptive offset information), e.g., directly or after being entropy coded by the entropy coding unit 270 or any other entropy coding unit, so that the decoder 30 can receive and apply the same loop filter parameters for decoding.

[0104] The decoded picture buffer (DPB) 230 may be a reference picture memory that stores reference picture data for use in encoding the video data by the video encoder 20. The DPB 230 may be formed by any of a variety of memory devices, such as dynamic random access memory (DRAM), including synchronous dynamic random access memory (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The DPB 230 and the buffer 216 may be provided by the same memory device or separate memory devices. In some examples, the decoded picture buffer (DPB) 230 is configured to store the filtered block 221. The decoded picture buffer 230 may further be configured to store other previously filtered blocks, e.g., previously reconstructed filtered block 221, of the same current picture or of a different picture, e.g., a previously reconstructed picture, providing a complete previously reconstructed, i.e., decoded picture (and corresponding reference blocks and corresponding samples) and / or a partially reconstructed current picture (and corresponding reference blocks and corresponding samples), e.g., for inter prediction. In some examples, if the reconstructed block 215 is reconstructed but without in-loop filtering, the decoded picture buffer (DPB) 230 may be configured to store one or more unfiltered reconstructed blocks 215, or, in general, unfiltered reconstructed samples, e.g., if the reconstructed block 215 has not been filtered by the loop filter unit 220, or any other further processed version of the reconstructed block or sample.

[0105] The prediction processing unit 260, also referred to as block prediction processing unit 260, is configured to receive or obtain the block 203 (the current block 203 of the current picture 201) and reconstructed picture data, e.g. reference samples of the same (current) picture from the buffer 216 and / or reference picture data 231 from one or more previously decoded pictures from the decoded picture buffer 230, and to process the data for prediction, i.e. to provide a prediction block 265, which may be an inter-prediction block 245 or an intra-prediction block 255.

[0106] The mode selection unit 262 may be configured to select a prediction mode (e.g., intra or inter prediction mode) to be used as the prediction block 265 and / or the corresponding prediction block 245 or 255 for calculation of the residual block 205 and for reconstruction of the reconstruction block 215.

[0107] An embodiment of the mode selection unit 262 may be configured to select a prediction mode (e.g., from prediction modes supported by the prediction processing unit 260) that provides the best match or in other words the smallest residual (smallest residual means better compression for transmission or storage), or the smallest signaling overhead (smallest signaling overhead means better compression for transmission or storage), or a consideration or balance of both. The mode selection unit 262 may also be configured to determine the prediction mode based on rate-distortion optimization (RDO), i.e., to select a prediction mode that provides the smallest rate-distortion optimization, or a prediction mode whose associated rate-distortion at least satisfies a prediction mode selection criterion.

[0108] The prediction process (eg, prediction processing unit 260) and mode selection (eg, by mode selection unit 262) performed by the exemplary encoder 20 are described in detail below.

[0109] Additionally or alternatively to the above-mentioned embodiment, in another embodiment according to Fig. 15, the mode selection unit 260 includes a partitioning unit 262, an inter prediction unit 244 and an intra prediction unit 254 and is configured to receive or obtain original picture data, e.g. an original block 203 (current block 203 of current picture 17) and reconstructed picture data, e.g. filtered and / or unfiltered samples or blocks of the same (current) picture and / or from one or more previously decoded pictures, e.g. from the decoded picture buffer 230 or another buffer (e.g. a line buffer, not shown). The reconstructed picture data is used as reference picture data for prediction, e.g. inter prediction or intra prediction, to obtain a prediction block 265 or a predictor 265.

[0110] The mode selection unit 260 may be configured to determine or select a prediction mode (including no partitioning) and partitioning for the prediction mode (e.g., intra or inter prediction mode) of the current block and generate a corresponding prediction block 265 used for calculation of the residual block 205 and reconstruction of the reconstruction block 215.

[0111] An embodiment of the mode selection unit 260 can be configured to select a partitioning and prediction mode (e.g., from those supported by or available to the mode selection unit 260) that provides the best match, or in other words, the smallest residual (smallest residual means better compression for transmission or storage), or the smallest signaling overhead (smallest signaling overhead means better compression for transmission or storage), or that considers or balances both. The mode selection unit 260 may be configured to determine the partitioning and prediction mode based on rate-distortion optimization (RDO), i.e., to select the prediction mode that provides the smallest rate-distortion. Terms such as "best," "lowest," "optimal" in this context do not necessarily refer to an overall "best," "lowest," "optimal," etc., but may also refer to a termination or selection criterion, such as a value being above or below a threshold, or the achievement of other constraints that may result in a "suboptimal selection," but that reduce complexity and processing time.

[0112] In other words, the partitioning unit 262 may be configured to partition pictures from a video sequence into a sequence of coding tree units (CTUs), the CTUs 203 being further partitioned into smaller block partitions or sub-blocks (which again form blocks), e.g. using quadtree partitioning (QT), binary tree partitioning (BT) or ternary tree partitioning (TT) or any combination thereof iteratively, and prediction may be performed for each of the block partitions or sub-blocks, e.g. the mode selection includes selection of a tree structure of the partitioned blocks 203, and a prediction mode is applied to each of the block partitions or sub-blocks.

[0113] The partitioning (eg, by partitioning unit 260) and prediction processes (eg, by inter prediction unit 244 and intra prediction unit 254) performed by exemplary video encoder 20 are described in more detail below.

[0114] Partitioning The partitioning unit 262 may be configured to partition a picture from a video sequence into a sequence of coding tree units (CTUs), which may partition (or split) the coding tree units (CTUs) 203 into smaller partitions, e.g., smaller blocks of square or rectangular size. For a picture with three sample arrays, a CTU consists of an N×N block of luma samples and two corresponding blocks of chroma samples. The maximum allowed size of a luma block of a CTU is specified to be 128×128 in the evolving Generic Video Coding (VVC) standard, but may be specified in the future to be a value other than 128×128, e.g., 256×256. The CTUs of a picture may be clustered / grouped as slices / tile groups, tiles, or bricks. A tile covers a rectangular area of ​​the picture, and a tile may be divided into one or more bricks. A brick consists of multiple CTU rows within a tile. A tile that is not partitioned into multiple bricks can be referred to as a brick. However, a brick is a true subset of a tile and is not referred to as a tile. There are two tile group modes supported by VVC: raster scan slice / tile group mode and rectangular slice mode. In raster scan tile group mode, a slice / tile group contains a sequence of tiles in the tile raster scan of a picture. In rectangular slice mode, a slice contains a number of bricks of a picture that collectively form a rectangular region of the picture. The bricks within a rectangular slice are in the order of the brick raster scan of the slice. These smaller blocks (sometimes called sub-blocks) can be further partitioned into even smaller partitions.This is also called tree partitioning or hierarchical tree partitioning, where for example a root block at root tree level 0 (hierarchical level 0, depth 0) can be partitioned recursively, for example into two or more blocks at the next lower tree level, for example at a node at tree level 1 (hierarchical level 1, depth 1), and these blocks can be partitioned again into two or more blocks at the next lower level, for example at tree level 2 (hierarchical level 2, depth 2), etc., until the partitioning is terminated, for example due to a termination criterion being met, for example due to reaching a maximum tree depth or a minimum block size. The blocks that are not further partitioned are also called leaf blocks or leaf nodes of the tree. A tree that uses a partition into two partitions is called a binary tree (BT), a tree that uses a partition into three partitions is called a ternary tree (TT), and a tree that uses a partition into four partitions is called a quad tree (QT).

[0115] For example, a coding tree unit (CTU) may be or include a CTB of luma samples, two corresponding CTBs of chroma samples for a picture with three sample arrays, or a CTB of samples of a picture or monochrome picture coded using three separate color planes and syntax structures used to code the samples. Correspondingly, a coding tree block (CTB) may be an NxN block of samples for some value N, such that the division of the components into CTBs is a partitioning. A coding unit (CU) may be or include a coding block of luma samples, two corresponding coding blocks of chroma samples for a picture with three sample arrays, or a coding block of samples of a picture or monochrome picture coded using three separate color planes and syntax structures used to code the samples. Correspondingly, a coding block (CB) may be an MxN block of samples for some value of M and N, such that the division of the CTBs into coding blocks is a partitioning.

[0116] In an embodiment, for example according to HEVC, coding tree units (CTUs) can be divided into CUs by using a quadtree structure, denoted as coding tree. The decision of whether to code a picture area using inter-picture (temporal) or intra-picture (spatial) prediction is made at the leaf CU level. Each leaf CU can be further divided into one, two or four PUs depending on the PU partition type. Within one PU, the same prediction process is applied and related information is transmitted to the decoder on a PU basis. After obtaining the residual blocks by applying the prediction process based on the PU partition type, the leaf CUs can be partitioned into transform units (TUs) according to another quadtree structure similar to the coding tree of the CU.

[0117] In an embodiment, for example, according to the latest video coding standard currently being developed, called Universal Video Coding (VVC), a combined quad-tree nested multi-type tree using, for example, a binary tree and a ternary tree segmentation structure is used to partition the coding tree unit. In the coding tree structure in the coding tree unit, the CU can have either a square or a rectangular shape. For example, the coding tree unit (CTU) is first partitioned by a quad-tree. Then, the quad-tree leaf node can be further partitioned by a multi-type tree structure. The multi-type tree structure has four partition types: vertical bisection (SPLIT_BT_VER), horizontal bisection (SPLIT_BT_HOR), vertical trisection (SPLIT_TT_VER), and horizontal trisection (SPLIT_TT_HOR). The multitype tree leaf nodes are called coding units (CUs) and this segmentation is used for prediction and transformation processes without any further partitioning, unless the CU is too large for the maximum transform length. This means that in most cases, CUs, PUs, and TUs have the same block size in a quadtree with nested multitype tree coding block structures. An exception occurs when the maximum supported transform length is smaller than the width or height of the color components of the CU. VVC develops a unique signaling mechanism for partitioning information in a quadtree with nested multitype tree coding tree structures. In the signaling mechanism, the coding tree unit (CTU) is treated as the root of the quadtree and is first partitioned by the quadtree structure. Each quadtree leaf node (if it is large enough to allow it) is then further partitioned by the multitype tree structure.In a multi-type tree structure, a first flag (mtt_split_cu_flag) is signaled to indicate whether the node is further partitioned, and if the node is further partitioned, a second flag (mtt_split_cu_vertical_flag) is signaled to indicate the split direction, and then a third flag (mtt_split_cu_binary_flag) is signaled to indicate whether the split is bipartite or tripartite. Based on the values ​​of mtt_split_cu_vertical_flag and mtt_split_cu_binary_flag, the multi-type tree split mode (MttSplitMode) of the CU can be derived by the decoder based on a predefined rule or table. It should be noted that in certain designs, such as the 64x64 luma block and 32x32 chroma pipeline design in the VVC hardware decoder, TT split is prohibited if either the width or height of the luma coding block is greater than 64, as shown in Figure 6. TT partitioning is also prohibited if the width or height of a chroma coding block exceeds 32. The pipeline design divides a picture into Virtual Pipeline Data Units (VPDUs), which are defined as non-overlapping units within a picture. In hardware decoders, consecutive VPDUs are processed simultaneously by multiple pipeline stages. It is important to keep the VPDU size small, since it is roughly proportional to the buffer size of most pipeline stages. In most hardware decoders, the VPDU size can be set to the maximum transform block (TB) size. However, in VVC, ternary tree (TT) and binary tree (BT) partitions can lead to an increase in VPDU size. Furthermore, it should be noted that if any part of a tree node block exceeds the bottom or right picture boundary, the tree node block is forced to be split until all samples of all coded CUs are located within the picture boundary.

[0118] As an example, an Intra Sub-Partition (ISP) tool can split a luma intra prediction block vertically or horizontally into two or four sub-partitions, depending on the block size.

[0119] In one example, mode selection unit 260 of video encoder 20 may be configured to perform any combination of the partitioning techniques described herein. As mentioned above, encoder 20 may be configured to determine or select a best or optimal prediction mode from a set of (predetermined) prediction modes. The set of prediction modes may include, for example, intra-prediction modes and / or inter-prediction modes.

[0120] The set of intra prediction modes may include 35 different intra prediction modes, e.g., non-directional modes such as DC (or average) mode and planar mode, or directional modes, e.g., as specified in H.265, or may include 67 different intra prediction modes, e.g., non-directional modes such as DC (or average) mode and planar mode, or directional modes, e.g., as specified in VVC. As an example, some conventional angular intra prediction modes are adaptively replaced with wide-angle intra prediction modes for non-square blocks, e.g., as specified in VVC. As another example, to avoid division operations for DC prediction, only the long side is used to calculate the average for non-square blocks. Also, the result of intra prediction in planar mode may be further modified by a position-dependent intra prediction combining (PDPC) method. The intra prediction unit 254 is configured to generate an intra prediction block 265 using reconstructed samples of neighboring blocks of the same current picture according to an intra prediction mode of the set of intra prediction modes.

[0121] The intra prediction unit 254 (or, in general, the mode selection unit 260) is further configured to output intra prediction parameters (or, in general, information indicating the selected intra prediction mode for the block) to the entropy coding unit 270 in the form of a syntax element 266 for inclusion in the encoded picture data 21, so that, for example, the video decoder 30 can receive and use the prediction parameters for decoding.

[0122] The set of (possible) intra prediction modes depends on the available reference pictures (i.e., previous pictures that have been at least partially decoded, e.g., stored in DBP 230) and other inter prediction parameters, such as whether the entire reference picture or only a part of it, e.g., a search window area around the area of ​​the current block of the reference picture, is used to search for the best matching reference block, and / or whether pixel interpolation, such as half / semi-pel, quarter-pel, and / or 1 / 16-pel interpolation, is applied.

[0123] In addition to the prediction modes listed above, skip mode, direct mode, and / or other inter prediction modes may be applied.

[0124] For example, the extended merge prediction, the merge candidate list of such a mode is constructed by including five types of candidates: spatial MVP from spatially neighboring CUs, temporal MVP from co-located CUs, history-based MVP from FIFO table, pair-wise average MVP, and zero MV, in order. Also, to improve the accuracy of MV in merge mode, bilateral matching based decoder-side motion vector refinement (DMVR) may be applied. Merge mode with MVD (MMVD) is derived from merge mode with motion vector differential. MMVD flag is signaled to specify whether MMVD mode is used for a CU immediately after sending skip flag and merge flag. Then, CU-level adaptive motion vector decomposition (AMVR) scheme may be applied. AMVR allows MVD of a CU to be decoded with different accuracy. Depending on the prediction mode of the current CU, the MVD of the current CU can be adaptively selected. When the CU is coded in merge mode, combined inter / intra prediction (CIIP) mode can be applied to the current CU. To obtain the CIIP prediction, a weighted average of the inter and intra prediction signals is performed. Affine motion compensation prediction, the affine motion field of a block is described by motion information of 2-control points (4 parameters) or 3-control point motion vectors (6 parameters). Subblock-based temporal motion vector prediction (SbTMVP) is similar to the temporal motion vector prediction (TMVP) in HEVC, but predicts the motion vectors of sub-CUs in the current CU. Bidirectional optical flow (BDOF), previously called BIO, is a simpler version that requires much less computational effort, especially in terms of the number of multiplications and the size of the multipliers. In modes such as the triangular partition mode, the CU is divided equally into two triangular partitions, using either diagonal or anti-diagonal partitioning. In addition, the bidirectional prediction mode is extended beyond a simple average to allow a weighted average of the two prediction signals.

[0125] In addition to the above prediction modes, skip mode and / or direct mode may also be applied.

[0126] The prediction processing unit 260 may further be configured to partition the block 203 into smaller block partitions or sub-blocks, e.g. by recursively using quadtree partitions (QT), bipartite partitions (BT), tertile partitions (TT) or combinations thereof, and to perform prediction for each of the block partitions or sub-blocks, e.g., the mode selection including selecting a tree structure of the partitioned block 203 and a prediction mode to be applied to each of the block partitions or sub-blocks.

[0127] The inter prediction unit 244 may include a motion estimation (ME) unit (not shown in FIG. 2) and a motion compensation (MC) unit (not shown in FIG. 2). The motion estimation unit is configured to receive or obtain a picture block 203 (current picture block 203 of current picture 201) and a decoded picture 231, or at least one or more previously reconstructed blocks, e.g. reconstructed blocks of one or more other / different previously decoded pictures 231, for motion estimation. For example, a video sequence may include the current picture and the previously decoded picture 231, or in other words, the current picture and the previously decoded picture 231 may be part of pictures forming a video sequence or may form a sequence of pictures.

[0128] The encoder 20 may be configured, for example, to select a reference block from multiple reference blocks of the same or different pictures among multiple other pictures and provide the reference picture (or reference picture index, ...) and / or the offset (spatial offset) between the position (x,y coordinates) of the reference block and the position of the current block as inter prediction parameters to a motion estimation unit (not shown in Fig. 2). This offset is also called a motion vector (MV).

[0129] The motion compensation unit is configured to obtain, e.g., receive, inter prediction parameters and perform inter prediction based on or using the inter prediction parameters to obtain an inter prediction block 265. The motion compensation performed by the motion compensation unit may include fetching or generating a prediction block based on a motion / block vector determined by motion estimation, possibly performing interpolation up to sub-pixel accuracy. Interpolation filtering may generate additional pixel samples from known pixel samples, thus potentially increasing the number of candidate prediction blocks that may be used to code the picture block. Upon receiving a motion vector for the PU of the current picture block, the motion compensation unit may locate the prediction block to which the motion vector points in one of the reference picture lists.

[0130] The intra prediction unit 254 is configured to obtain, e.g. receive, the picture block 203 (current picture block) and one or more previously reconstructed blocks, e.g. reconstructed neighboring blocks, of the same picture for intra estimation. The encoder 20 may be configured to select, e.g., an intra prediction mode from a plurality of (predetermined) intra prediction modes.

[0131] An embodiment of the encoder 20 may be configured to select the intra prediction mode based on an optimization criterion, such as minimum residual (e.g., the intra prediction mode provides the predicted block 255 that is most similar to the current picture block 203) or minimum rate distortion.

[0132] The intra prediction unit 254 is further configured to determine intra prediction parameters, e.g., a selected intra prediction mode, based on the intra prediction block 255. In any case, after selecting an intra prediction mode for the block, the intra prediction unit 254 is also configured to provide the intra prediction parameters, i.e., information indicative of the selected intra prediction mode for the block, to the entropy coding unit 270. In one example, the intra prediction unit 254 may be configured to perform any combination of the intra prediction techniques described below.

[0133] The entropy encoding unit 270 is configured to apply an entropy encoding algorithm or scheme (e.g., a variable length coding (VLC) scheme, a context-adaptive VLC scheme (CALVC), an arithmetic coding scheme, a context-adaptive binary arithmetic coding (CABAC), a syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy encoding method or technique) to the quantized residual coefficients 209, the inter prediction parameters, the intra prediction parameters, and / or the loop filter parameters to obtain coded picture data 21, e.g., in the form of a coded bitstream 21, which may be output by an output 272, either individually or together (or not all). The coded bitstream 21 may be transmitted to the video decoder 30 or may be stored for later transmission or retrieval by the video decoder 30. The entropy encoding unit 270 may further be configured to entropy encode other syntax elements for the current video slice being coded.

[0134] Other structural variations of the video encoder 20 may be used to encode the video stream. For example, a non-transform-based encoder 20 may directly quantize the residual signal for a particular block or frame without the transform processing unit 206. In another implementation, the encoder 20 may combine the quantization unit 208 and the inverse quantization unit 210 into a single unit.

[0135] 3 illustrates an exemplary video decoder 30 configured to implement implementations of the present application. The video decoder 30 is configured to receive encoded picture data (e.g., encoded bitstream) 21, e.g., encoded by an encoder 100, to obtain a decoded picture 131. During the decoding process, the video decoder 30 receives from the video encoder 100 the encoded video bitstream representing picture blocks of video data, e.g., encoded video slices and associated syntax elements.

[0136] 3, decoder 30 includes an entropy decoding unit 304, an inverse quantization unit 310, an inverse transform processing unit 312, a reconstruction unit 314 (e.g., summer 314), a buffer 316, a loop filter 320, a decoded picture buffer 330, and a prediction processing unit 360. Prediction processing unit 360 may include an inter prediction unit 344, an intra prediction unit 354, and a mode selection unit 362. Video decoder 30 may, in some examples, perform a decoding path that is generally the reverse of the encoding path described with respect to video encoder 100 in accordance with FIG.

[0137] As described with respect to encoder 20, inverse quantization unit 210, inverse transform processing unit 212, reconstruction unit 214, loop filter 220, decoded picture buffer (DPB) 230, inter prediction unit 344, and intra prediction unit 354 are also referred to as forming a "built-in decoder" of video encoder 20. Accordingly, inverse quantization unit 310 may be functionally identical to inverse quantization unit 110, inverse transform processing unit 312 may be functionally identical to inverse transform processing unit 212, reconstruction unit 314 may be functionally identical to reconstruction unit 214, loop filter 320 may be functionally identical to loop filter 220, and decoded picture buffer 330 may be functionally identical to decoded picture buffer 230. Accordingly, the descriptions provided for each unit and function of video encoder 20 apply correspondingly to each unit and function of video decoder 30.

[0138] The entropy decoding unit 304 is configured to perform entropy decoding on the encoded picture data 21 to obtain, e.g., quantization coefficients 309 and / or decoded coding parameters (not shown in FIG. 3), e.g., (decoded) inter-prediction parameters, intra-prediction parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 is further configured to forward the inter-prediction parameters, intra-prediction parameters, and / or other syntax elements to the prediction processing unit 360. The video decoder 30 may receive the syntax elements at a video slice level and / or a video block level.

[0139] The entropy decoding unit 304 is configured to analyze the bitstream 21 (or the coded picture data 21 in general) and to perform, e.g., entropy decoding on the coded picture data 21 to obtain, e.g., quantization coefficients 309 and / or decoded coding parameters (not shown in FIG. 3 ), e.g., any or all of inter prediction parameters (e.g., reference picture indexes and motion vectors), intra prediction parameters (e.g., intra prediction modes or indices), transform parameters, quantization parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 may be configured to apply a decoding algorithm or scheme corresponding to an encoding scheme as described with respect to the entropy coding unit 270 of the encoder 20. The entropy decoding unit 304 may be further configured to provide the inter prediction parameters, intra prediction parameters and / or other syntax elements to the mode application unit 360 and to provide other parameters to other units of the decoder 30. Video decoder 30 may receive syntax elements at the video slice level and / or the video block level. In addition to or as an alternative to slices and their respective syntax elements, tile groups and / or tiles and their respective syntax elements may be received and / or used.

[0140] The inverse quantization unit 310 may be functionally identical to the inverse quantization unit 110, the inverse transform processing unit 312 may be functionally identical to the inverse transform processing unit 112, the reconstruction unit 314 may be functionally identical, the buffer 316 may be functionally identical to the buffer 116, the loop filter 320 may be functionally identical to the loop filter 120, and the decoded picture buffer 330 may be functionally identical to the decoded picture buffer 130.

[0141] An embodiment of decoder 30 may include a partitioning unit (not shown in FIG. 3). In one example, prediction processing unit 360 of video decoder 30 may be configured to perform any combination of the partitioning techniques described above.

[0142] Prediction processing unit 360 includes an inter prediction unit 344, which may be functionally similar to inter prediction unit 144, and an intra prediction unit 354, which may be functionally similar to intra prediction unit 154. Prediction processing unit 360 is typically configured to perform block prediction and / or obtain a prediction block 365 from the encoded data 21, and to receive or obtain (explicitly or implicitly) prediction-related parameters and / or information regarding a selected prediction mode, for example from entropy decoding unit 304.

[0143] When a video slice is coded as an intra-coded (I) slice, intra prediction unit 354 of prediction processing unit 360 is configured to generate a prediction block 365 for a picture block of the current video slice based on data from a previously decoded block of the current frame or picture and a signaled intra prediction mode. When a video frame is coded as an inter-coded (i.e., B or P) slice, inter prediction unit 344 (e.g., a motion compensation unit) of prediction processing unit 360 is configured to generate a prediction block 365 for a video block of the current video slice based on a motion vector and another syntax element received from entropy decoding unit 304. For inter prediction, the prediction block may be generated from one reference picture in one reference picture list. Video decoder 30 may construct the reference frame lists, List0 and List1, by using a default construction technique based on the reference pictures stored in DPB 330.

[0144] Prediction processing unit 360 is configured to determine prediction information for video blocks of the current video slice by parsing the motion vectors and other syntax elements, and to use the prediction information to generate a prediction block for the current video block to be decoded. For example, prediction processing unit 360 uses some of the received syntax elements to determine a prediction mode (e.g., intra or inter prediction) to be used to code video blocks of the video slice, an inter-prediction slice type (e.g., a B slice, a P slice, or a GPB slice), configuration information for one or more of the reference picture lists for the slice, a motion vector for each inter-coded video block of the slice, an inter-prediction status for each inter-coded video block of the slice, and other information for decoding video blocks in the current video slice.

[0145] The inverse quantization unit 310 is configured to inverse quantize, i.e., dequantize, the quantized transform coefficients provided in the bitstream and decoded by the entropy decoding unit 304. The inverse quantization process may include determining the degree of quantization, and similarly, the degree of inverse quantization to be applied, using a quantization parameter calculated by the video encoder 100 for each video block in a video slice.

[0146] The inverse quantization unit 310 may be configured to receive a quantization parameter (QP) (or in general information regarding inverse quantization) and quantized coefficients from the coded picture data 21 (e.g. by analysis and / or decoding, e.g. by the entropy decoding unit 304) and to apply inverse quantization on the decoded quantized coefficients based on the quantization parameter to obtain unquantized coefficients 311, also referred to as transform coefficients 311.

[0147] Inverse transform processing unit 312 is configured to apply an inverse transform, eg, an inverse DCT, an inverse integer transform, or a conceptually similar inverse transform process, to the transform coefficients to produce residual blocks in the pixel domain.

[0148] The inverse transform processing unit 312 may be configured to receive the unquantized coefficients 311, also referred to as transform coefficients 311, and to apply a transform to the unquantized coefficients 311 to obtain a reconstructed residual block 213 in the sample domain. The reconstructed residual block 213 may also be referred to as a transform block 313. The transform may be an inverse transform, e.g., an inverse DCT, an inverse DST, an inverse integer transform, or a conceptually similar inverse transform process. The inverse transform processing unit 312 may further be configured to receive transform parameters or corresponding information from the coded picture data 21 (e.g., by analysis and / or decoding, e.g., by the entropy decoding unit 304) to determine the transform to be applied to the unquantized coefficients 311.

[0149] The reconstruction unit 314 (e.g., adder 314) is configured to add the inverse transform block 313 (i.e., the reconstructed residual block 313) to the prediction block 365 to obtain a reconstructed block 315 in the sample domain, e.g. by adding sample values ​​of the reconstructed residual block 313 and sample values ​​of the prediction block 365.

[0150] A loop filter unit 320 (either in the coding loop or after the coding loop) is configured to filter the reconstruction block 315 to obtain a filtered block 321, for example to smooth pixel transitions or otherwise improve video quality. The loop filter unit 320 may include one or more loop filters, such as a deblocking filter, a sample adaptive offset (SAO) filter, or one or more other filters, for example an adaptive loop filter (ALF), a noise suppression filter (NSF), or any combination thereof. In one example, the loop filter unit 220 may include a deblocking filter, an SAO filter, and an ALF filter. The order of the filtering process may be deblocking filter, SAO, and ALF. In another example, a process called Luma Mapping with Chroma Scaling (LMCS) (i.e., adaptive in-loop reshaper) is added. This process is performed before deblocking. In another example, the deblocking filter process may also be applied to interior sub-block edges, such as affine sub-block edges, ATMVP sub-block edges, sub-block transform (SBT) edges, and intra-sub-partition (ISP) edges. Although loop filter unit 320 is illustrated as an in-loop filter in FIG. 3, in other configurations, loop filter unit 320 may be implemented as a post-loop filter.

[0151] The decoded video blocks 321 in a given frame or picture are then stored in a decoded picture buffer 330, which stores reference pictures used for subsequent motion compensation.

[0152] The decoded video blocks 321 of the picture are then stored in a decoded picture buffer 330, which stores the decoded picture 331 as a reference picture for subsequent motion compensation for other pictures and / or output, respectively.

[0153] The decoder 30 is arranged to output the decoded pictures 331 for presentation or display to a user, for example via an output 332.

[0154] Other variations of the video encoder 30 may be used to decode the compressed bitstream. For example, the decoder 30 may generate an output video stream without the loop filtering unit 320. For example, the non-transform-based decoder 30 may inverse quantize the residual signal directly for a particular block or frame without the inverse transform processing unit 312. In another implementation, the video decoder 30 may combine the inverse quantization unit 310 and the inverse transform processing unit 312 into a single unit.

[0155] Additionally or alternatively to the above-mentioned embodiment, in another embodiment according to Fig. 16, the inter prediction unit 344 may be identical to the inter prediction unit 244 (in particular the motion compensation unit) and the intra prediction unit 354 may be functionally identical to the inter prediction unit 254 and performs the division or partitioning decision and prediction based on the partitioning and / or prediction parameters or respective information received from the coded picture data 21 (e.g. by analysis and / or decoding, by the entropy decoding unit 304). The mode application unit 360 may be configured to perform prediction (intra prediction or inter prediction) for each block based on the reconstructed picture, block or respective samples (filtered or unfiltered) to obtain a prediction block 365.

[0156] When a video slice is coded as an intra-coded (I) slice, intra prediction unit 354 of mode application unit 360 is configured to generate a prediction block 365 for a picture block of the current video slice based on data from a previously decoded block of the current picture and a signaled intra prediction mode. When a video picture is coded as an inter-coded (i.e., B or P) slice, inter prediction unit 344 (e.g., a motion compensation unit) of mode application unit 360 is configured to generate a prediction block 365 for a video block of the current video slice based on a motion vector and another syntax element received from entropy decoding unit 304. For inter prediction, the prediction block may be generated from a reference picture in a reference picture list. Video decoder 30 may construct the reference frame lists, List0 and List1, using a default construction technique based on the reference pictures stored in DPB 330. The same or similar may apply to or with embodiments that use tile groups (e.g., video tile groups) and / or tiles (e.g., video tiles) in addition to or instead of slices (e.g., video slices), e.g., video may be encoded using I, P or B tile groups and / or tiles.

[0157] Mode application unit 360 is configured to determine prediction information for the video blocks of the current video slice by parsing the motion vectors or related information and other syntax elements, and to generate a prediction block for the current video block to be decoded using the prediction information. For example, mode application unit 360 uses some of the received syntax elements to determine a prediction mode (e.g., intra or inter prediction) used to code the video blocks of the video slice, an inter-prediction slice type (e.g., a B slice, a P slice, or a GPB slice), configuration information for one or more of the reference picture lists of the slice, a motion vector for each inter-coded video block of the slice, an inter-prediction status for each inter-coded video block of the slice, and other information for decoding video blocks in the current video slice. The same or similar may apply to or with embodiments that use tile groups (e.g., video tile groups) and / or tiles (e.g., video tiles) in addition to or instead of slices (e.g., video slices), e.g., video may be coded using I, P or B tile groups and / or tiles.

[0158] The embodiment of video decoder 30 shown in FIG. 3 may be configured to partition and / or decode a picture by using slices (also referred to as video slices), where a picture may be partitioned or decoded into one or more slices (typically non-overlapping), and each slice may include one or more blocks (e.g., CTUs) or one or more groups of blocks (e.g., tiles (H.265 / HEVC and VVC) or bricks (VC)).

[0159] The embodiment of video decoder 30 shown in FIG. 3 may be configured to partition and / or decode a picture by using slice / tile groups (also referred to as video tile groups) and / or tiles (also referred to as video tiles), where a picture may be partitioned or decoded into one or more slice / tile groups (typically non-overlapping), where each slice / tile group may include, for example, one or more blocks (e.g., CTUs) or one or more tiles, where each tile may be, for example, rectangular in shape and may include one or more blocks (e.g., CTUs), e.g., complete or fractional blocks.

[0160] Other variations of the video decoder 30 may be used to decode the encoded picture data 21. For example, the decoder 30 may generate an output video stream without a loop filtering unit 320. For example, a non-transform-based decoder 30 may inverse quantize the residual signal directly for a particular block or frame, without the inverse transform processing unit 312. In another implementation, the video decoder 30 may combine the inverse quantization unit 310 and the inverse transform processing unit 312 into a single unit.

[0161] It should be understood that in the encoder 20 and the decoder 30, the processing result of the current step may be further processed and output to the next step, for example, after the interpolation filtering, the motion vector derivation, or the loop filtering, further processing such as clipping or shifting may be performed on the processing result of the interpolation filtering, the motion vector derivation, or the loop filtering.

[0162] 4 is a schematic diagram of a video encoding device 400 according to an embodiment of the present disclosure. The video coding device 400 is suitable for implementing the disclosed embodiments as described herein. In an embodiment, the video encoding device 400 may be a decoder, such as the video decoder 30 of FIG. 1A, or an encoder, such as the video encoder 20 of FIG. 1A. In an embodiment, the video encoding device 400 may be one or more components of the video decoder 30 of FIG. 1A or the video encoder 20 of FIG. 1A, as described above.

[0163] The video coding device 400 includes an ingress port 410 and a receiver unit (Rx) 420 for receiving data, a processor, logic unit, or central processing unit 430 for processing data, a transmitter unit (Tx) 440 and an egress port 450 for transmitting data, and a memory 460 for storing data. The video coding device 400 may also include optical-to-electrical (OE) and electrical-to-optical (EO) components coupled to the ingress port 410, the receiver unit 420, the transmitter unit 440, and the egress port 450 for entering and exiting optical or electrical signals.

[0164] The processor 430 is implemented by hardware and software. The processor 430 may be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 430 communicates with the ingress port 410, the receiver unit 420, the transmitter unit 440, the egress port 450, and the memory 460. The processor 430 includes a coding module 470. The coding module 470 implements the disclosed embodiments described above. For example, the coding module 470 implements, processes, prepares, or provides various coding operations. Thus, the inclusion of the coding module 470 provides a significant improvement to the functionality of the video coding device 400 and affects the transformation of the video coding device 400 into different states. Alternatively, the coding module 470 is implemented as instructions stored in the memory 460 and executed by the processor 430.

[0165] Memory 460 may include one or more disks, tape drives, and solid state drives, may be used as overflow data storage devices, may store programs when such programs are selected for execution, and may store instructions and data read during the execution of the programs. Memory 460 may be volatile and / or non-volatile, and may be read only memory (ROM), random access memory (RAM), ternary content addressable memory (TCAM), and / or static random access memory (SRAM).

[0166] 5 is a simplified block diagram of an apparatus 500 that can be used as either or both of the source device 310 and the destination device 320 of FIG. 1, according to an example embodiment. The apparatus 500 can implement the techniques of the present application described above. The apparatus 500 can be in the form of a computing system including multiple computing devices, or in the form of a single computing device, such as a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, etc.

[0167] The processor 502 in the device 500 may be a central processing unit. Alternatively, the processor 502 may be any other type of device or devices capable of manipulating or processing information, now existing or later developed. Although the disclosed implementations may be implemented with a single processor, such as the processor 502, as shown, multiple processors may be used to achieve advantages in speed and efficiency. The memory 504 in the device 500 may be a read-only memory (ROM) device or a random access memory (RAM) device in implementations. Any other suitable type of storage device may be used as the memory 504. The memory 504 may include code and data 506 that are accessed by the processor 502 using a bus 512. The memory 504 may further include an operating system 508 and application programs 510, which include at least one program that enables the processor 502 to execute the methods described herein. For example, application programs 510 may include applications 1 through N, further including a video coding application that performs the methods described herein. The device 500 may also include additional memory in the form of a secondary storage 514, which may be, for example, a memory card used with a mobile computing device. Because video communication sessions may include a significant amount of information, they may be stored, in whole or in part, in the secondary storage 514 and loaded into the memory 504 as needed for processing. The device 500 may also include one or more output devices, such as a display 518. The display 518 may, in one example, be a touch-sensitive display that combines a display with touch-sensitive elements operable to sense touch input. The display 518 may be coupled to the processor 502 via a bus 512.

[0168] The device 500 may also include one or more output devices, such as a display 518. The display 518 may, in one example, be a touch-sensitive display that combines a display with touch-sensitive elements operable to sense touch input. The display 518 may be coupled to the processor 502 via the bus 512. Other output devices that enable a user to program or otherwise use the device 500 may be provided in addition to or as an alternative to the display 518. If the output device is or includes a display, the display may be implemented in a variety of ways, including with a liquid crystal display (LCD), a cathode ray tube (CRT) display, a plasma display, or a light emitting diode (LED) display, such as an organic LED (OLED) display.

[0169] The device 500 may also include or communicate with an image sensing device 520, such as a camera or any other existing or later developed image sensing device 520 capable of sensing images, such as an image of a user operating the device 500. The image sensing device 520 may be positioned such that it is pointed towards a user operating the device 500. In one example, the position and optical axis of the image sensing device 520 may be set such that the field of view includes an area immediately adjacent to the display 518 from which the display 518 is viewable.

[0170] Apparatus 500 may also include or be in communication with an acoustic sensing device 522, such as a microphone or any other present or later developed acoustic sensing device capable of sensing sounds in the vicinity of apparatus 500. Acoustic sensing device 522 may be positioned such that it is pointed towards a user operating apparatus 500 and can be configured to receive sounds, such as speech or other utterances, made by the user while the user is operating apparatus 500.

[0171] Although FIG. 5 depicts the processor 502 and memory 504 of device 500 as integrated into a single unit, other configurations may be used. The operations of processor 502 may be distributed across multiple machines (each having one or more processors) that may be coupled directly or through a local area or other network. Memory 504 may be distributed across multiple machines, such as a network-based memory or memory within multiple machines that perform the operations of device 500. Although depicted here as a single bus, bus 512 of device 500 may be comprised of multiple buses. Additionally, secondary storage 514 may be directly coupled to other components of device 500 or may be accessed over a network, and may include a single integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, device 500 may be implemented in a wide variety of configurations.

[0172] Next Generation Video Coding (NGVC) removes the distinction between the concepts of CU, PU and TU and supports more flexibility for CU partition shapes: the size of a CU corresponds to the size of the coding node and may be square or non-square (e.g. rectangular) in shape.

[0173] J. An et al., “Block partitioning structure for next generation video coding”, International Telecommunication Union, COM16-C966, September 2015 (hereinafter, “VCEG proposal COM16-C966”), In JEM, a quad-tree binary tree (QTBT) partitioning technique was proposed for future video coding standards beyond HEVC. Simulations show that the proposed QTBT structure is more efficient than the quad-tree structure used in HEVC. In HEVC, inter prediction for small blocks is restricted and inter prediction for 4x4 blocks is not supported to reduce memory access for motion compensation. In JEM's QTBT, these constraints are removed.

[0174] In QTBT, a CU can have either a square or rectangular shape. As shown in Figure 6, a coding tree unit (CTU) is first partitioned by a quadtree structure. The quadtree leaf nodes can be further partitioned by a binary tree structure. In binary tree partitioning, there are two partition types: symmetric horizontal partitioning and symmetric vertical partitioning. In each case, the node is partitioned by splitting horizontally or vertically from the center of the node. The binary tree leaf node is called a coding unit (CU), and its segmentation is used for prediction and transform processes without any further partitioning. This means that CU, PU, ​​and TU have the same block size in the QTBT coding block structure. A CU is often composed of coding blocks (CBs) of different color components, e.g. in the case of P and B slices in 4:2:0 chroma format, a CU contains one luma CB and two chroma CBs, but often also consists of a single component CB, e.g. in the case of an I slice, a CU contains only one luma CB or only two chroma CBs.

[0175] The following parameters are defined for the QTBT partitioning scheme: - CTU size: The size of the root node of the quadtree, the same concept as in HEVC. - MinQTSize: The minimum allowed quadtree leaf node size. - MaxBTSize: The maximum allowed binary tree root node size. - MaxBTDepth: Maximum allowed binary tree depth - MinBTSize: The minimum allowable binary tree leaf node size.

[0176] In one example of a QTBT partitioning structure, if a quadtree node has a size less than or equal to MinQTSize, no further quadtrees are considered. Since the size (MinQTSize) does not exceed MaxBTSize, it will not be further split by the binary tree. Otherwise, the leaf quadtree node can be further partitioned by the binary tree. Thus, the quadtree leaf node is also the root node of the binary tree, which has a depth of 0 (zero). If the depth of the binary tree reaches MaxBTDepth (i.e. 4), no further splits are considered. If a binary tree node has a width equal to MinBTSize (i.e. 4), no further horizontal splits are considered. Similarly, if a binary tree node has a height equal to MinBTSize, no further vertical splits are considered. The leaf node of the binary tree is further processed by the prediction and transformation process without any further partitioning. In JEM, the maximum CTU size is 256x256 luma samples. The leaf nodes of the binary tree (CU) may be further processed (e.g., by performing a prediction process and a transformation process) without any further partitioning.

[0177] FIG. 6 illustrates an example of a block 30 (e.g., CTB) partitioned using the QTBT partitioning technique. As shown in FIG. 6, using the QTBT partitioning technique, each block is partitioned symmetrically through the center of each block. FIG. 7 illustrates a tree structure corresponding to the block partitioning of FIG. 6. The solid lines in FIG. 7 indicate quadtree partitioning, and the dotted lines indicate binary tree partitioning. In one example, at each partition (i.e., non-leaf) node of the binary tree, a syntax element (e.g., a flag) is signaled to indicate the type of partitioning (e.g., horizontal or vertical) to be performed, with 0 indicating a horizontal partition and 1 indicating a vertical partition. In the case of quadtree partitioning, there is no need to specify the partition type, since the quadtree partition always partitions a block horizontally and vertically into four sub-blocks of equal size.

[0178] As shown in FIG. 7, at node 50, block 30 (corresponding to the root 50) is split into four blocks 31, 32, 33, and 34 shown in FIG. 6 using QT partitioning. Block 34 is not further split and is therefore a leaf node. At node 52, block 31 is further split into two blocks using BT partitioning. As shown in FIG. 7, node 52 is marked with a 1 indicating a vertical split. Thus, the split at node 52 results in block 37 and a block containing both blocks 35 and 36. Blocks 35 and 36 are generated by a further vertical split at node 54. At node 56, block 32 is further split into two blocks 38 and 39 using BT partitioning.

[0179] At node 58, block 33 is split into four equal-sized blocks using QT partitioning. Blocks 43 and 44 are generated from this QT partitioning and are not further split. At node 60, the top left block is first split using vertical binary tree partitioning, resulting in block 40 and a right vertical block. The right vertical block is then split into blocks 41 and 42 using horizontal binary tree partitioning. The bottom right block created from the quadtree partitioning at node 58 is split at node 62 into blocks 45 and 46 using horizontal binary tree partitioning. As shown in FIG. 7, node 62 is marked with a 0 to indicate a horizontal split.

[0180] In addition to QTBT, a block partitioning structure named Multi-Type Tree (MTT) is proposed to replace BT in the QTBT-based CU structure, which means that a CTU may first be partitioned by QT partitioning to obtain blocks of CTUs, and then the blocks may be secondarily partitioned by MTT partitioning.

[0181] The MTT partitioning structure is still a recursive tree structure. In MTT, multiple different partition structures (e.g., two or more) are used. For example, with the MTT technique, two or more different partition structures may be used for each and every non-leaf node of the tree structure at each depth of the tree structure. The depth of a node in the tree structure may indicate the length of the path (e.g., the number of splits) from the node to the root of the tree structure.

[0182] In MTT, there are two partition types: BT partitioning and ternary tree (TT) partitioning. The partition type can be selected from BT partitioning and TT partitioning. The TT partition structure differs from that of the QT and BT structures in that the TT partition structure does not split blocks from the center. The center regions of the blocks remain together in the same subblock. Unlike QT, which generates four blocks, or binary tree, which generates two blocks, a division by the TT partition structure generates three blocks. Exemplary partition types by the TT partition structure include asymmetric partition types (both horizontal and vertical) in addition to symmetric partition types (both horizontal and vertical). Furthermore, the symmetric partition types by the TT partition structure may be non-uniform / non-uniform or uniform / uniform. The asymmetric partition type by the TT partition structure is non-uniform / non-uniform. In one example, the TT partition structure may include at least one of the following partition types: horizontal uniform / uniform symmetric ternary tree, vertical uniform / uniform symmetric ternary tree, horizontal uneven / non-uniform symmetric ternary tree, vertical uneven / non-uniform symmetric ternary tree, horizontal uneven / non-uniform asymmetric ternary tree, or vertical uneven / non-uniform asymmetric ternary tree.

[0183] In general, a non-uniform / non-uniform symmetric ternary tree partition type is a partition type that is symmetric with respect to the centerline of the blocks, in which at least one of the three resulting blocks is not the same size as the other two. One preferred example is when the side blocks are 1 / 4 the size of the block and the central block is 1 / 2 the size of the block. A uniform / uniform symmetric ternary tree partition type is a partition type that is symmetric with respect to the centerline of the blocks, in which the resulting blocks are all the same size. Such partitions are possible when the height or width of the blocks is a multiple of three, depending on the vertical or horizontal division. A non-uniform / non-uniform asymmetric ternary tree partition type is a partition type that is not symmetric with respect to the centerline of the blocks, in which at least one of the resulting blocks is not the same size as the other two.

[0184] FIG. 8 is a conceptual diagram illustrating an optional exemplary horizontal ternary tree partition type. FIG. 9 is a conceptual diagram illustrating an optional exemplary vertical ternary tree partition type. In both FIG. 8 and FIG. 9, h represents the height of the block in luma or chroma samples, and w represents the width of the block in luma or chroma samples. Note that the center lines of each of the blocks do not represent the boundaries of the blocks (i.e., ternary tree partitions do not divide blocks through the center lines). Rather, the center lines are used to represent whether a particular partition type is symmetric or asymmetric with respect to the center line of the original block. Also, the center lines are along the direction of the division.

[0185] As shown in Figure 8, the block 71 is partitioned with a horizontal uniform / uniform symmetric partition type. The horizontal uniform / uniform symmetric partition type creates an upper and lower half that are symmetric about the centerline of the block 71. The horizontal uniform / uniform symmetric partition type creates three sub-blocks of equal size, each with a height of h / 3 and a width of w. The horizontal uniform / uniform symmetric partition type is possible if the height of the block 71 is evenly divisible by 3.

[0186] The block 73 is partitioned with a horizontally uneven / non-uniform symmetric partition type. The horizontally uneven / non-uniform symmetric partition type creates upper and lower halves that are symmetric about the centerline of the block 73. The horizontally uneven / non-uniform symmetric partition type creates two blocks of equal size (e.g., upper and lower blocks with a height of h / 4) and a central block of a different size (e.g., a central block with a height of h / 2). In one example, according to the horizontally uneven / non-uniform symmetric partition type, the area of ​​the central block is equal to the sum of the areas of the upper and lower blocks. In some examples, the horizontally uneven / non-uniform symmetric partition type may be preferred for blocks with heights that are powers of two (e.g., 2, 4, 8, 16, 32, etc.).

[0187] The block 75 is partitioned with a horizontally uneven / non-uniform asymmetric partition type. The horizontally uneven / non-uniform asymmetric partition type does not produce upper and lower halves that are symmetrical about the centerline of the block 75 (i.e., the upper and lower halves are asymmetric). In the example of FIG. 8, the horizontally uneven / non-uniform asymmetric partition type produces an upper block with a height of h / 4, a central block with a height of 3h / 8, and a lower block with a height of 3h / 8. Of course, other asymmetric arrangements may be used.

[0188] As shown in Figure 9, the block 81 is partitioned with a vertical uniform / uniform symmetric partition type. The vertical uniform / uniform symmetric partition type creates left and right halves that are symmetric about the centerline of the block 81. The vertical uniform / uniform symmetric partition type creates three sub-blocks of equal size, each with a width of w / 3 and a height of h. The vertical uniform / uniform symmetric partition type is possible if the width of the block 81 is evenly divisible by 3.

[0189] The block 83 is partitioned with a vertical non-uniform / non-uniform symmetric partition type. The vertical non-uniform / non-uniform symmetric partition type creates left and right halves that are symmetric about the centerline of the block 83. The vertical non-uniform / non-uniform symmetric partition type creates left and right halves that are symmetric about the centerline of 83. The vertical non-uniform / non-uniform symmetric partition type creates two blocks of equal size (e.g., left and right blocks with a width of w / 4) and a central block of different size (e.g., central block with a width of w / 2). In one example, according to the vertical non-uniform / non-uniform symmetric partition type, the area of ​​the central block is equal to the combined area of ​​the left and right blocks. In some examples, the vertical non-uniform / non-uniform symmetric partition type may be preferred for blocks with widths that are powers of two (e.g., 2, 4, 8, 16, 32, etc.).

[0190] Block 85 is partitioned with a vertical non-uniform / non-uniform asymmetric partition type that does not produce left and right halves that are symmetrical about the centerline of block 85 (i.e., the left and right halves are asymmetric). In the example of FIG. 9, the vertical non-uniform / non-uniform asymmetric partition type produces a left block with a width of w / 4, a middle block with a width of 3w / 8, and a right block with a width of 3w / 8. Of course, other asymmetric arrangements may be used.

[0191] In addition to (or instead of) the QTBT parameters specified above, the following parameters are specified for the MTT partitioning scheme: - MaxBTSize: The maximum allowed binary tree root node size. - MinBtSize: The minimum allowed binary tree root node size. - MaxMttDepth: Maximum multitype tree depth - MaxMttDepth offset: Maximum multitype tree depth offset - MaxTtSize: Maximum allowed ternary tree root node size - MinTtSize: The minimum allowed ternary tree root node size. - MinCbSize: The minimum allowed coding block size.

[0192] The embodiments of the present disclosure may be implemented by a video encoder or a video decoder, such as the video encoder 20 of Figure 2 or the video decoder 30 of Figure 3, according to embodiments of the present application. One or more structural elements of the video encoder 20 or the video decoder 30, including a partition unit, may be configured to perform the techniques of the disclosed embodiments.

[0193] In the disclosed embodiments: In JVET-K1001-v4, log2_ctu_size_minus2, log2_min_qt_size_intra_slices_minus2 and log2_min_qt_size_inter_slices_minus2 are signaled (as syntax elements) in the SPS.

[0194] The parameter log2_ctu_size_minus2 plus 2 specifies the luma coding tree block size for each CTU. In particular: CtbLog2SizeY = log2_ctu_size_minus2 + 2 (7-5) CtbSizeY = 1 << CtbLog2SizeY (7-6)

[0195] In other words, CtbLog2SizeY specifies the log2 value of the CTU size CtbSizeY, which corresponds to the coding tree block (CTB) size for luma (Y).

[0196] Further configuration is provided below: MinCbLog2SizeY = 2 (7-7) MinCbSizeY = 1 << MinCbLog2SizeY (7-8) MinTbSizeY = 4 (7-9) MaxTbSizeY = 64 (7-10)

[0197] The parameter log2_min_qt_size_intra_slices_minus2 plus 2 specifies the minimum luma size of leaf blocks resulting from the quadtree decomposition of a CTU in slices with slice_type equal to 2(I), i.e., intra slices. The value of log2_min_qt_size_intra_slices_minus2 shall be in the range from 0 to CtbLog2SizeY - 2, inclusive. MinQtLog2SizeIntraY = log2_min_qt_size_intra_slices_minus2 + 2 (7-22)

[0198] The parameter log2_min_qt_size_inter_slices_minus2 plus 2 specifies the minimum luma size of leaf blocks resulting from the quadtree decomposition of a CTU in slices with slice_type equal to 0 (B) or 1 (P), i.e., inter slices. The value of log2_min_qt_size_inter_slices_minus2 shall be in the range from 0 to CtbLog2SizeY - 2, inclusive. MinQtLog2SizeInterY = log2_min_qt_size_inter_slices_minus2 + 2 (7-23)

[0199] MinQtSizeY is specified as (7-30), which means the minimum allowed quadtree division size in luma samples. If the coding block size is smaller than or equal to MinQtSizeY, no quadtree division is allowed. Further settings are provided as follows: MinQtLog2SizeY = ( slice_type = = I ) ? MinQtLog2SizeIntraY : MinQtLog2SizeInterY (7-25) MaxBtLog2SizeY = CtbLog2SizeY - log2_diff_ctu_max_bt_size (7-26) MinBtLog2SizeY = MinCbLog2SizeY (7-27) MaxTtLog2SizeY = ( slice_type = = I ) ? 5 : 6 (7-28) MinTtLog2SizeY = MinCbLog2SizeY (7-29) MinQtSizeY = 1 << MinQtLog2SizeY (7-30) MaxBtSizeY = 1 << MaxBtLog2SizeY (7-31) MinBtSizeY = 1 << MinBtLog2SizeY (7-32) MaxTtSizeY = 1 << MaxTtLog2SizeY (7-33) MinTtSizeY = 1 << MinTtLog2SizeY (7-34) MaxMttDepth = ( slice_type = = I ) ? max_mtt_hierarchy_depth_intra_slices : max_mtt_hierarchy_depth_inter_slices (7-35)

[0200] The parameters max_mtt_hierarchy_depth_intra_slices and max_mtt_hierarchy_depth_inter_slices indicate the maximum hierarchical depth of MTT type partitioning for intra and inter slices, respectively.

[0201] Based on the semantics of log2_min_qt_size_intra_slices_minus2 and log2_min_qt_size_inter_slices_minus2, the ranges of log2_min_qt_size_intra_slices_minus2 and log2_min_qt_size_inter_slices_minus2 are from 0 to CtbLog2SizeY - 2. Here, CtbLog2SizeY is defined by the semantics of log2_ctu_size_minus2, which means the log2 value of the luma coding tree block size of each CTU, and CtbLog2SizeY in VTM2.0 is equal to 7.

[0202] Based on (7-22) and (7-23), the ranges of MinQtLog2SizeIntraY and MinQtLog2SizeInterY are from 2 to CtbLog2SizeY.

[0203] Based on (7-25), the range of MinQtLog2SizeY is from 2 to CtbLog2SizeY.

[0204] Based on (7-30), the range of MinQtSizeY is from (1<<2) to (1<<CtbLog2SizeY) in JVET-K1001-v4, and the range in VTM2.0 is from (1<<2) to (1<<7), which is equal to 4 to 128.

[0205] In JVET-K1001-v4, log2_diff_ctu_max_bt_size is conditionally signaled in the slice header.

[0206] The parameter log2_diff_ctu_max_bt_size specifies the difference between the maximum luma size (width or height) of a coding block that can be split using bisection and the luma CTB size. The value of log2_diff_ctu_max_bt_size shall be in the range from 0 to CtbLog2SizeY - MinCbLog2SizeY, inclusive.

[0207] If log2_diff_ctu_max_bt_size is not present, the value of log2_diff_ctu_max_bt_size is inferred to be equal to 2.

[0208] MinCbLog2SizeY is specified in (7-7), which denotes the minimum allowable coding block size.

[0209] Based on the semantics of log2_diff_ctu_max_bt_size, the range of log2_diff_ctu_max_bt_size is 0 to CtbLog2SizeY - MinCbLog2SizeY.

[0210] Based on (7-26), the range of MaxBtLog2SizeY is from CtbLog2SizeY to MinCbLog2SizeY.

[0211] Based on (7-31), the range of MaxBtSizeY is (1<< CtbLog2SizeY) to (1<< MinCbLog2SizeY).

[0212] Based on (7-7), the range of MaxBtSizeY is from (1<< CtbLog2SizeY ) to (1<< 2) in JVET-K1001-v4, and CtbLog2SizeY is equal to 7 in VTM2.0, so the range of MaxBtSizeY in VTM2.0 is from 128 to 4.

[0213] Therefore, MinQtSizeY ranges from 4 to (1 << CtbLog2SizeY), which is from 4 to 128 in VTM2.0, and MaxBtSizeY ranges from (1 << CtbLog2SizeY) to 4, which is from 128 to 4 in VTM2.0.

[0214] Therefore, MinQtSizeY may be larger than MaxBtSizeY.

[0215] Furthermore, based on the current boundary processing in VVC2.0, only QT and BT partitioning are allowed for blocks located at the boundary (TT is not allowed, and non - splitting is not allowed).

[0216] If the current coding block is on the boundary and the current coding block size cbSizeY satisfies the condition: MinQtSizeY > cbSizeY > MaxBtSizeY then neither QT splitting nor BT splitting is possible for the current coding block. Therefore, there is no available partitioning mode for the current block.

[0217] Embodiment 1 The solutions (embodiments of the present invention) to the above problems including boundary - case issues are described in more detail below.

[0218] According to an embodiment, to solve the above problem, the lower limit of MaxBtSizeY should be limited to MinQtSizeY, ensuring that MaxBtSizeY is not smaller than MinQtSizeY. In particular, the lower limit of MaxBtSizeY may be equal to MinQtSizeY, so the range of MaxBtSizeY should be from (1<< CtbLog2SizeY) to (1<< MinQtLog2SizeY), so the range of MaxBtLog2SizeY should be from CtbLog2SizeY to MinQtLog2SizeY, so the range of log2_diff_ctu_max_bt_size should be from 0 to CtbLog2SizeY - MinQtLog2SizeY. Therefore, the information of MinQtSizeY may be used to determine the validity of MaxBtSizeY. In other words, MaxBtSizeY can be determined based on the information of MinQtSizeY.

[0219] The corresponding change in the draft text (of the video standard) in the semantics of log2_diff_ctu_max_bt_size is as follows: log2_diff_ctu_max_bt_size specifies the difference between the maximum luma size (width or height) of a coding block that can be split using bisection and the luma CTB size. The value of log2_diff_ctu_max_bt_size shall be in the range from 0 to CtbLog2SizeY - MinQtLog2SizeY, inclusive.

[0220] The corresponding method of coding realized by a coding device (decoder or encoder) can be as follows: determining whether the current block of the picture is a border block; Determine if the size of the current block is greater than the minimum allowable quadtree leaf node size; Bisection is applied to the current block if it is a border block and the size of the current block is not larger than the minimum allowed quadtree leaf node size; the minimum allowed quadtree leaf node size (MinQtSizeY) is not larger than the maximum allowed binary tree root node size (MaxBtSizeY). In this case, applying the bisection to the current block may include applying a forced bisection to the current block, where the coding corresponds to image, video, or motion picture coding.

[0221] Being a border block means that an image / frame boundary cuts the block, or in other words, the block is an image / frame boundary. In the above embodiment, bisection is applied to the current block if it is a border block (condition 1) and its size is not larger than the minimum allowed quadtree leaf node size (condition 2). Note that in some embodiments, a trisection or other division may be used instead of a bisection. Furthermore, in some embodiments, a bisection may be applied under condition 2, regardless of condition 1. In other words, condition 1 does not need to be evaluated. If the size of the current block is in fact larger than the minimum allowed quadtree leaf node size (i.e., condition 2 is not met), a quadtree division may be applied.

[0222] Note that there are embodiments in which bisection is used only for boundary blocks (Condition 1). For non-border blocks, quadtree partitioning may be the only partitioning used. Applying bisection (or trisection) to image / frame boundaries offers the advantage of potentially more efficient partitioning, such as horizontal bisection / trisection at horizontal boundaries and vertical bisection / trisection at vertical boundaries.

[0223] Another corresponding method of coding implemented by a coding device (decoder or encoder) may be as follows: determining whether the size of the boundary block is greater than the minimum allowed quadtree leaf node size, and if the size of the boundary block is not greater than the minimum allowed quadtree leaf node size and the minimum allowed quadtree leaf node size is not greater than the maximum allowed binary tree root node size (e.g. as specified by the standard), bisection is applied to the boundary block.

[0224] Optionally, border blocks may not include corner blocks - in other words, corner blocks that are cut off by both vertical and horizontal image / frame boundaries are not considered border blocks for the purposes of condition 1 above.

[0225] EMBODIMENT 2 Other embodiments of the disclosure (which may be combined with the above-mentioned embodiments) are described below.

[0226] In JVET-K1001-v4, max_mtt_hierarchy_depth_inter_slices and max_mtt_hierarchy_depth_intra_slices are signaled in the SPS, in other words, max_mtt_hierarchy_depth_inter_slices and max_mtt_hierarchy_depth_intra_slices are syntax elements, meaning that their values ​​are included in the bitstream that also contains the coded image or video.

[0227] In particular, max_mtt_hierarchy_depth_inter_slices specifies the maximum hierarchical depth for coding units resulting from a multitype tree split of quadtree leaves in slices with slice_type equal to 0 (B) or 1 (P). The value of max_mtt_hierarchy_depth_inter_slices shall be in the range from 0 to CtbLog2SizeY - MinTbLog2SizeY, inclusive.

[0228] max_mtt_hierarchy_depth_intra_slices specifies the maximum hierarchical depth for coding units resulting from a multitype tree split of quadtree leaves in slices with slice_type equal to 2(I). The value of max_mtt_hierarchy_depth_intra_slices shall be in the range from 0 to CtbLog2SizeY - MinTbLog2SizeY, inclusive.

[0229] MinTbSizeY is specified by (7-9), which is fixed to 4, therefore MinTbLog2SizeY = log2 MinTbSizeY, which is fixed to 2.

[0230] MaxMttDepth is specified, which is the maximum allowed depth of a multitype tree partition. If the current multitype tree partition depth is greater than or equal to MaxMttDepth, then a multitype tree partition is not allowed (applied).

[0231] Based on the semantics of max_mtt_hierarchy_depth_inter_slices and max_mtt_hierarchy_depth_intra_slices, the range of max_mtt_hierarchy_depth_inter_slices and max_mtt_hierarchy_depth_intra_slices is 0 to CtbLog2SizeY - MinTbLog2SizeY.

[0232] Based on (7-35), the range of MaxMttDepth is 0 to CtbLog2SizeY - MinTbLog2SizeY. In VTM2.0, CtbLog2SizeY equals 7, so the range of MaxMttDepth is 0 to 5.

[0233] Thus, MaxMttDepth has a range of 0 to CtbLog2SizeY - MinTbLog2SizeY, which is 0 to 5 in VTM2.0.

[0234] Based on the current boundary handling in VVC2.0, only QT and BT partitioning is allowed for blocks that lie on boundaries (TT is not allowed, no partitioning is allowed).

[0235] If the first problem above is solved (MaxBtSizeY >= MinQtSizeY), then the following additional conditions are true: cbSizeY <= MinQtSizeY MaxMttDepth =0

[0236] There is no BT (or, in general, any MTT that includes TT) partition at a sufficient level for boundary processing.

[0237] For example, MinQtSizeY is equal to 16, MinTbSizeY is equal to 4, and MaxMttDepth is 0.

[0238] If a border block has cbSizeY = 16, and the parent partition is QT, and this block is still located on the border, then no further partitions can be performed since the MttDepth of the current block reaches MaxMttDepth.

[0239] Solution to this problem for boundary cases (embodiment of the present invention): To solve the above problem, the lower bound of MaxMttDepth should be bounded to 1 (in other words, it cannot take the value of zero) to ensure that after QT partitioning there is a sufficient level of multitype tree partitioning for the boundary cases. Alternatively, the lower bound of MaxMttDepth should be bounded to (MinQtLog2SizeY- MinTbLog2SizeY) to ensure that after QT partitioning there is a sufficient level of multitype tree partitioning for both boundary and non-boundary cases.

[0240] The corresponding changes in the draft text to the semantics of max_mtt_hierarchy_depth_inter_slices and max_mtt_hierarchy_depth_intra_slices are as follows: max_mtt_hierarchy_depth_inter_slices specifies the maximum hierarchical depth for coding units resulting from a multitype tree split of quadtree leaves in slices with slice_type equal to 0 (B) or 1 (P). The value of max_mtt_hierarchy_depth_inter_slices shall be in the range from 1 to CtbLog2SizeY - MinTbLog2SizeY, inclusive. max_mtt_hierarchy_depth_intra_slices specifies the maximum hierarchical depth for coding units resulting from a multitype tree split of quadtree leaves in slices with slice_type equal to 2(I). The value of max_mtt_hierarchy_depth_intra_slices shall be in the range from 1 to CtbLog2SizeY - MinTbLog2SizeY, inclusive. or, max_mtt_hierarchy_depth_inter_slices specifies the maximum hierarchical depth for coding units resulting from a multitype tree split of quadtree leaves in slices with slice_type equal to 0 (B) or 1 (P). The value of max_mtt_hierarchy_depth_inter_slices shall be in the range MinQtLog2SizeY- MinTbLog2SizeY to CtbLog2SizeY - MinTbLog2SizeY, inclusive. max_mtt_hierarchy_depth_intra_slices specifies the maximum hierarchical depth for coding units resulting from a multitype tree split of quadtree leaves in slices with slice_type equal to 2(I). The value of max_mtt_hierarchy_depth_intra_slices shall be in the range MinQtLog2SizeY- MinTbLog2SizeY to CtbLog2SizeY - MinTbLog2SizeY, inclusive.

[0241] The corresponding method of coding realized by a coding device (decoder or encoder) can be as follows: Divide an image into blocks, the blocks including a border block. Apply bisection to the border block having a maximum border multitype partition depth, the maximum border multitype partition depth being at least the sum of a maximum multitype tree depth and a maximum multitype tree depth offset, the maximum multitype tree depth being greater than 0. This embodiment may be combined with embodiment 1 or may be applied without relying on embodiment 1.

[0242] Optionally, if bisection is applied to border blocks, the maximum multitype tree depth is greater than 0.

[0243] Optionally, the border blocks may not include the corner blocks.

[0244] EMBODIMENT 3 Another embodiment of the disclosure:

[0245] In JVET-K1001-v4, when MinQtSizeY>MaxBtSizeY and MinQtSizeY>MaxTtSizeY

[0246] If cbSize = MinQtsizeY, there are no possible partition modes available, so the partition cannot reach MinCbSizeY (MinTbSizeY and MinCbsizeY are fixed and equal to 4).

[0247] Solution to this problem for non-bounded or bounded cases: To solve the above problem, the lower bound of MaxBtSizeY should be bounded by MinQtSizeY to ensure that MaxBtSizeY is not smaller than MinQtSizeY, or the lower bound of MaxTtSizeY should be bounded by MinQtSizeY to ensure that MaxTtSizeY is not smaller than MinQtSizeY.

[0248] The corresponding changes in the draft text are semantically log2_diff_ctu_max_bt_size specifies the difference between the maximum luma size (width or height) of a coding block that can be split using bisection and the luma CTB size. The value of log2_diff_ctu_max_bt_size shall be in the range from 0 to CtbLog2SizeY - MinQtLog2SizeY, inclusive. and / or log2_min_qt_size_intra_slices_minus2 plus 2 specifies the minimum luma size of leaf blocks resulting from the quadtree decomposition of a CTU in slices with slice_type equal to 2(I). The value of log2_min_qt_size_intra_slices_minus2 shall be in the range from 0 to MaxTtLog2SizeY - 2, inclusive. log2_min_qt_size_inter_slices_minus2 plus 2 specifies the minimum luma size of leaf blocks resulting from the quadtree decomposition of CTUs in slices with slice_type equal to 0 (B) or 1 (P). The value of log2_min_qt_size_inter_slices_minus2 shall be in the range from 0 to MaxTtLog2SizeY - 2, inclusive.

[0249] The corresponding method of coding realized by a coding device (decoder or encoder) can be as follows: Determine whether the size of the current block is greater than the minimum allowed quadtree leaf node size; apply multitype tree splitting to the current block if the size of the current block is not larger than the minimum allowed quadtree leaf node size; In this case, the minimum allowable quadtree leaf node size is not greater than the maximum allowable binary tree root node size, or the minimum allowable quadtree leaf node size is not greater than the maximum allowable ternary tree root node size.

[0250] Optionally, the minimum allowable quadtree leaf node size is no greater than the maximum allowable binary tree root node size, and the minimum allowable quadtree leaf node size is no greater than the maximum allowable ternary tree root node size.

[0251] Optionally, applying a multi-type tree partition to the current block includes applying a three-way partition to the current block or applying a two-way partition to the current block.

[0252] Optionally, the border blocks may not include the corner blocks.

[0253] EMBODIMENT 4 In another embodiment of the disclosure:

[0254] If MaxBtSizeY >= MinQtSizeY, MinQtSizeY > MinTbLog2SizeY and MaxMttDepth < (MinQtLog2SizeY - MinTbLog2SizeY), If cbSize = MinQtsizeY, then a partition cannot reach MinCbSizeY because there are not enough levels of allowed multitype tree partitions.

[0255] Solution of this problem for non-bounded case or bounded case: To solve the above problem, the lower bound of MaxMttDepth should be limited to (MinQtLog2SizeY- MinTbLog2SizeY) to ensure that after QT partitioning there is a sufficient level of multitype tree partitioning for both bounded and non-bounded cases.

[0256] The corresponding changes in the draft text are in the semantics of max_mtt_hierarchy_depth_inter_slices and max_mtt_hierarchy_depth_intra_slices: max_mtt_hierarchy_depth_inter_slices specifies the maximum hierarchical depth for coding units resulting from a multitype tree split of quadtree leaves in slices with slice_type equal to 0 (B) or 1 (P). The value of max_mtt_hierarchy_depth_inter_slices shall be in the range MinQtLog2SizeY- MinTbLog2SizeY to CtbLog2SizeY - MinTbLog2SizeY, inclusive. max_mtt_hierarchy_depth_intra_slices specifies the maximum hierarchical depth for coding units resulting from a multitype tree split of quadtree leaves in slices with slice_type equal to 2(I). The value of max_mtt_hierarchy_depth_intra_slices shall be in the range MinQtLog2SizeY- MinTbLog2SizeY to CtbLog2SizeY - MinTbLog2SizeY, inclusive.

[0257] The corresponding method of coding realized by a coding device (decoder or encoder) can be as follows: Divide the image into blocks; Apply multitype tree splitting to the block of blocks with the final maximum multitype tree depth, which is at least the sum of the maximum multitype tree depth and the maximum multitype tree depth offset. The maximum multitype tree depth is greater than or equal to the Log2 value of the minimum allowed quadtree leaf node size minus the Log2 value of the minimum allowed transformation block size, or the maximum multitype tree depth is greater than or equal to the Log2 value of the minimum allowed quadtree leaf node size minus the Log2 value of the minimum allowed coding block size.

[0258] Optionally, the block is a non-border block.

[0259] Optionally, the maximum multitype tree depth offset is 0.

[0260] Optionally, the block is a border block and the multitype tree split is a bipartite.

[0261] Optionally, a multitype tree partition is (or includes) a three-way partition.

[0262] Optionally, the border blocks may not include the corner blocks.

[0263] The embodiments 1 to 4 can be applied on the encoder side to split the images / frames into coding units and to code the coding units. The embodiments 1 to 4 can be applied on the decoder side to provide partitions of the images / frames, i.e. coding units, and to decode the coding units accordingly (e.g. to correctly parse the coding units from the stream and decode them).

[0264] According to some embodiments, there is provided a decoder including one or more processors and a non-transitory computer readable storage medium coupled to the processors and storing programming for execution by the processors, the programming, when executed by the processors, configuring the decoder to perform any of the methods described above in relation to embodiments 1 to 4.

[0265] Further provided is an encoder including one or more processors and a non-transitory computer-readable storage medium coupled to the processors and storing programming for execution by the processors, the programming, when executed by the processors, configuring the encoder to perform any of the methods described above in relation to embodiments 1 to 4.

[0266] Further embodiments related to boundary partitioning In VVC, a multi-type (binary / ternary / quad-tree) segmentation structure (BT / TT / QT or binary / ternary / quad-tree) shall or can replace the concept of multiple partition unit types, i.e., it removes the distinction between the concepts of CU, PU, ​​and TU, except as required for CUs with sizes too large for the maximum transform length, and supports more flexibility for CU partition shapes. [J]

[0267] 10A-F show, by way of example, partition modes currently used in VTM. FIG. 10A shows an unsplit block (no partition), FIG. 10B shows quad or quadtree (QT) partitioning, FIG. 10C shows horizontal bisection or bisection (BT) partitioning, FIG. 10D shows vertical bisection or bisection (BT) partitioning, FIG. 10E shows horizontal ternary or ternary tree (TT) partitioning, and FIG. 10F shows vertical ternary or ternary tree (TT) partitioning of blocks such as CUs or CTUs. An embodiment may be configured to realize the partition modes as shown in FIG. 10A to 10F.

[0268] In an embodiment, the following parameters may be defined and specified by the Sequence Parameter Set (SPS) syntax element for the BT / TT / QT coding tree scheme: - CTU size: The size of the root node of the quadtree. - MinQTSize: The minimum allowed quadtree leaf node size. - MaxBTTSize: Maximum allowed binary and ternary tree root node size. - MaxBTTDepth: The maximum allowed binary and ternary tree depth, and - MinBTTSize: The minimum allowed binary and ternary tree leaf node size.

[0269] In other embodiments, the minimum allowed quadtree leaf node size MinQTSize parameter may be included in another header or set, such as the slice header (SH) or the picture parameter set (PPS).

[0270] In the HEVC standard, a coding tree unit (CTU) or coding unit (CU) that is located on a slice / picture boundary will be forced to be split using a quad tree (QT) until the bottom right sample of the leaf node is located within the slice / picture boundary. The forced QT partition or partitioning does not need to be signaled in the bitstream because both the encoder and the decoder, e.g., both the video encoder 20 and the video decoder 30, know when to apply the forced QT. The purpose of the forced partition is to allow for boundary CTU / CU by the video encoder 20 / video decoder 30.

[0271] International Patent Publication No. WO2016 / 090568 discloses the QTBT (quad-tree plus binary tree) structure, and in VTM1.0, the forced partitioning process of boundary CTU / CU is inherited from HEVC. This means that CTU / CU located on frame boundary will be forced to be partitioned by quad-tree (QT) structure without considering rate-distortion (RD) optimization until the entire current CU is within the picture boundary. These forced partitions are not signaled in the bitstream.

[0272] 11A shows an example of a forced partition of a lower bound CTU (128×128) of a high definition (HD) (1920×1080 pixels) partitioned by a forced QT. In FIG. 11, the HD picture has or is 1920×1080 pixels, and the CTU has or is 128×128 pixels.

[0273] In the San Diego meeting (04.2018) [SubCE2 (Picture Boundary Processing) of CE1 (Partitioning) in JVET-J1021, 15 tests were proposed for picture boundary processing using BT, TT or ABT (Asymmetric BT). For example, in JVET-K0280 and JVET-K0376, the boundary is specified as shown in Figure 12. Figure 12 shows the picture boundary by dot-hash line and the boundary case area in straight line, namely bottom boundary case, corner boundary case, and right boundary case. The bottom boundary can be partitioned by horizontal forced BT or forced QT, the right boundary can be partitioned by vertical forced BT or forced QT, and the corner case can be partitioned only by forced QT, where the decision of which forced BT or forced QT partitioning to use is based on the rate-distortion optimization criterion and is signaled in the bitstream. Forced partitioning means that a block must be partitioned; for example, forced partitioning is applied to boundary blocks that may not be coded using "non-split" as shown in FIG. 10A.

[0274] When forced QT partitioning is used with forced boundary partitioning, the MinQTSize partitioning constraint is ignored. For example, in Figure 13A, if MinQTSize is signaled in the SPS as 32, a QT partitioning down to a block size of 8x8 would be required to match the boundaries with the forced QT method. This ignores the MinQTSize constraint of 32.

[0275] According to embodiments of the present disclosure, when forced QT is used for picture boundary partitioning, the forced QT partitioning follows, e.g., does not ignore, the partitioning constraints signaled, e.g., in the SPS. If further forced partitioning is required, only forced BT is used, which may be referred to in combination as forced QTBT. In embodiments of the present disclosure, e.g., the partition constraint MinQTSize is taken into account for forced QT partitioning at picture boundaries, and no additional signaling for forced BT partitioning is required. Also, the embodiments allow for harmonization of partitioning for normal (non-border) blocks and boundary blocks. For example, in conventional solutions, two "MinQTSize" parameters are needed, one for normal block partitioning and another for boundary block partitioning. The embodiments only need one common "MinQTSize" parameter for both normal block and boundary block partitioning, which can be flexibly set between the encoder and the decoder, e.g., by signaling one "MinQTSize" parameter. Furthermore, the embodiment requires fewer partitions than, for example, the mandatory QT.

[0276] Solutions for the lower and right bound cases In the case of bottom and right boundaries, if the block size is larger than MinQTSize, the partition mode for picture boundary partitioning can be selected between forced BT partitioning and forced QT partitioning, for example based on RDO (rate distortion optimization). Otherwise (i.e., if the block size is less than or equal to MinQTSize), only forced BT partitioning is used for picture boundary partitioning, more specifically, horizontal forced BT is used on the bottom boundary for each border block located at the bottom boundary of the picture, and vertical forced BT is used on the right boundary for each border block located at the right boundary of the picture.

[0277] Enforced BT partitioning may include recursively partitioning the current block by horizontal enforced boundary partitioning until a sub-partition of the current block is located at the bottom boundary of the picture, and recursively partitioning the sub-partitions by vertical enforced boundary partitioning until the leaf node is fully located at the right boundary of the picture. Alternatively, enforced BT partitioning may include recursively partitioning the current block by vertical enforced boundary partitioning until a sub-partition of the current block is located at the bottom boundary, and recursively partitioning the sub-partitions by horizontal enforced boundary partitioning until the leaf node is fully located at the right boundary. MinQTSize may also be applied to control the partitioning of non-border blocks.

[0278] For example, in the case shown in FIG. 11A, if MinQTSize is 32 or is constrained as 32 and a rectangular (non-square) block size of 8 samples high or wide is required to match a picture boundary, then forced BT partitioning will be used to partition the 32×32 block in which the boundary is located. The BT partition can be further partitioned using the same type of forced BT partitioning, e.g., in the case where forced vertical BT partitioning is applied, further only forced vertical BT partitioning is applied, and in the case where forced horizontal BT partitioning is applied, further only forced horizontal BT partitioning is applied. The forced BT partitioning continues until the leaf node is completely within the picture.

[0279] FIG. 11B illustrates an exemplary partitioning of a lower-boundary CTU having a size of 128×128 samples according to an embodiment of the present invention. The lower-boundary CTU, which forms the root block or root node of the partitioning tree, is partitioned into smaller partitions, e.g., smaller blocks of square or rectangular size. These smaller partitions or blocks can be further partitioned into smaller partitions or blocks. In FIG. 11B, the CTU is first quadtree partitioned into four square blocks 710, 720, 730, and 740, each having a size of 64×64 samples. Of these blocks, blocks 710 and 720 are again lower-boundary blocks, while blocks 730 and 740 are outside the picture (respectively located outside the picture) and are not processed.

[0280] Block 710 is further partitioned using a quadtree into four square blocks 750, 760, 770, and 780, each with a size of 32x32 samples. Blocks 750 and 760 are located inside the picture, while blocks 770 and 780 again form bottom border blocks. Since the size of block 770 is not larger than MinQTSize, e.g. 32, a recursive horizontal forced bisection partitioning is applied to block 770 until the leaf node is completely within the picture or is completely within the picture, e.g. until leaf node block 772, a rectangular non-square block with 32x16 samples, is within the picture (after one horizontal bisection) or until leaf node block 774, a rectangular non-square block located on the bottom border of the picture and with 32x8 samples, is within the picture (after two horizontal bisections). The same applies to block 780.

[0281] The embodiments of the present disclosure allow for a harmonization of partitioning for normal blocks that are entirely located within a picture with partitioning for border blocks. A border block is a block that is neither entirely within a picture nor entirely outside a picture. In other words, a border block is a block that includes a part that is located within a picture and a part that is located outside a picture. Furthermore, the embodiments of the present disclosure allow for reduced signaling, since forced BT partitioning below MinQTSize does not need to be signaled.

[0282] Solutions for corner cases In the corner case, some approaches only allow forced QT division, which also ignores the MinQTSize constraint. The embodiments of the present disclosure provide two solutions for the corner case. The corner case occurs when the currently processed block is at a corner of the picture. This is when the current block is intersected by or adjacent to two picture boundaries (vertical and horizontal).

[0283] Solution 1: The corner cases are considered as bottom-boundary cases or right-boundary cases. FIG. 14 shows an embodiment of the boundary definition. FIG. 14 shows the picture boundaries as dot-hash lines and the boundary case areas as straight lines. As shown, the corner case is defined as the bottom-boundary case. Thus, the solution is the same as that described for the bottom-boundary case and the right-boundary case above. In other words, first, horizontal partitioning is applied (as described for the bottom-boundary case) until the block or partition is completely in the picture (vertically), and then vertical partitioning is applied (as described for the right-boundary case) until the leaf node is completely in the picture (horizontally). The boundary case may be a border block.

[0284] Solution 2: The definition of the boundary cases remains as is: if the forced QT is constrained by MinQTSize (currently the block size is less than or equal to MinQTSize), then we use the horizontal forced BT to match the bottom boundary, and if the bottom boundary is matched, we use the vertical forced BT to match the right boundary.

[0285] For example, in FIG. 13A showing an embodiment of forced QTBT for blocks located at the corners of a picture, if MinQTSize is 32 or so limited for the corner case forced QT partition, then after the 32x32 block partition, additional BT partitions will be used until the forced partition is finished.

[0286] 13B shows further details of an exemplary partitioning of a boundary CTU at or within a corner of a picture according to an embodiment of the present invention, the CTU having a size of 128×128 samples. The CTU is first quadtree partitioned into four square blocks, each having a size of 64×64 samples. Of these blocks, only the top-left block 910 is a boundary block, the other three are located outside (completely outside) the picture and are not further processed. Block 910 is further partitioned using a quadtree partitioning into four square blocks 920, 930, 940, and 950, each having a size of 32×32 samples. Block 920 is located inside the picture, while blocks 930, 940, and 950 again form boundary blocks. Since the sizes of these blocks 930, 940, and 950 are not larger than MinQTSize, which is 32, a forced bisection is applied to blocks 930, 940, and 950.

[0287] Block 930 is located at the right boundary and is partitioned using recursive vertical forced bisections until a leaf node is within the picture, e.g., until block 932 is located at the right boundary of the picture (here, after two vertical bisections).

[0288] Block 940 is located at the bottom boundary and is partitioned using recursive horizontal forced bisections until a leaf node is within the picture, e.g., until block 942 is located at the right boundary of the picture (here, after two horizontal bisections).

[0289] Block 950 is located at a corner boundary and is partitioned using first recursive horizontal forced bisection until a sub-partition or block, here block 952, is located at the bottom boundary of the picture (here after two horizontal bisections), and then recursively partition the sub-partitions by vertical forced boundary partitioning until a leaf node or block, e.g. block 954, is located at the right boundary of the picture (here after two vertical bisections) or until the leaf node is located within the picture, respectively.

[0290] The above approach may be applied to both decoding and encoding. For decoding, MinQTSize may be received by the SPS. For encoding, MinQTSize may be transmitted by the SPS. Embodiments may use boundary definitions as shown in Figure 12 or Figure 14, or other boundary definitions.

[0291] Further embodiments of the present disclosure are provided below. It should be noted that the numbers used in the following sections do not necessarily follow the numbers used in the previous sections. Embodiment 1: A partitioning method comprising: determining whether a current block of a picture is a border block; if the current block is a border block, determining whether the size of the current block is greater than the minimum allowed quadtree leaf node size; applying a forced binary tree split to the current block if the size of the current block is not greater than the minimum allowed quadtree leaf node size.

[0292] Embodiment 2: In the partitioning method of embodiment 1, the forced binary tree partitioning is a recursive horizontal forced bisection when the current block is located at the bottom boundary of the picture, or a recursive vertical forced boundary partitioning when the current block is located at the right boundary of the picture.

[0293] Embodiment 3: In the partitioning method of embodiment 1 or 2, the forced bipartitioning includes recursively partitioning the current block by horizontal forced boundary partitioning until a sub-partition of the current block is directly aligned with the bottom boundary of the picture, and recursively partitioning the sub-partitions by vertical forced boundary partitioning until a leaf node is directly and completely aligned with the right boundary of the picture, or vice versa.

[0294] Embodiment 4: In the partitioning method of any one of embodiments 1-3, the minimum allowable quadtree leaf node size is a minimum allowable quadtree leaf node size that is also applied to control partitioning of non-border blocks.

[0295] Embodiment 5: A decoding method for decoding a block by partitioning the block according to the partitioning method of any one of embodiments 1-4.

[0296] Embodiment 6: In the decoding method of embodiment 5, the minimum allowed quadtree leaf node size is received by the SPS.

[0297] Embodiment 7: A method for encoding a block by partitioning the block according to the partitioning method of any one of embodiments 1-4.

[0298] Embodiment 8: In the encoding method of embodiment 7, the minimum allowed quadtree leaf node size is transmitted by the SPS.

[0299] Embodiment 9: A decoding device comprising a logic circuit configured to perform the method of any one of embodiments 5 or 6.

[0300] Embodiment 10: A coding device comprising a logic circuit configured to perform the method of any one of embodiments 7 or 8.

[0301] Embodiment 11: A non-transitory storage medium storing instructions that, when executed by a processor, cause the processor to perform any one of the methods according to embodiments 1-8.

[0302] The apparatus includes a memory element and a processor element coupled to the memory element and configured to determine whether a current block of a picture is a border block, and if the current block is a border block, determine whether a size of the current block is greater than a minimum allowed quad-tree (QT) leaf node size (MinQTSize), and apply mandatory binary tree (BT) partitioning to the current block if the size of the current block is not greater than MinQTSize.

[0303] In summary, embodiments of the present application (or disclosure) provide an apparatus and method for encoding and decoding.

[0304] A first aspect relates to a partitioning method comprising the steps of determining whether a current block of a picture is a border block and whether a size of the current block is greater than a minimum allowed quad-tree leaf node size, and applying a mandatory binary tree (BT) partitioning to the current block if the current block is a border block and the size of the current block is not greater than the minimum allowed quad-tree leaf node size (MinQTSize).

[0305] In a first implementation form of the method according to such a first aspect, the forced binary tree partitioning is a recursive horizontal forced bisection if the current block is located at the bottom boundary of the picture, or a recursive vertical forced boundary partitioning if the current block is located at the right boundary of the picture.

[0306] In such a second implementation form of the method according to the first aspect or any preceding implementation form of the first aspect, the forced binary tree splitting continues until a leaf node block is within the picture.

[0307] In a third implementation form of a method according to such first aspect or any preceding implementation form of the first aspect, the forced bipartitioning includes a step of recursively partitioning the current block by horizontal forced boundary partitioning until a sub-partition of the current block is located at the bottom boundary of the picture, and a step of recursively partitioning the sub-partition by vertical forced boundary partitioning until a leaf node is located entirely at the right boundary of the picture.

[0308] In a fourth implementation form of a method according to such first aspect or any preceding implementation form of the first aspect, the forced BT partitioning includes a step of recursively partitioning the current block by vertical forced boundary partitioning until a sub-partition of the current block is located at the bottom boundary, and a step of recursively partitioning the sub-partition by horizontal forced boundary partitioning until the leaf node is located entirely at the right boundary.

[0309] In a fifth implementation form of the method according to such first aspect or any preceding implementation form of the first aspect, the method further includes applying a minimum allowable quadtree leaf node size to control partitioning of non-border blocks.

[0310] In a sixth implementation form of the method according to such first aspect or any preceding implementation form of the first aspect, a border block is a block that is neither entirely within the picture nor entirely outside the picture.

[0311] A second aspect relates to a decoding method for decoding blocks by partitioning the blocks according to such first aspect or any previous implementation form of the first aspect.

[0312] In a first implementation form of the method according to this second aspect, the method further comprises receiving, via a sequence parameter set (SPS), a minimum allowed quadtree leaf node size.

[0313] A third aspect relates to an encoding method for encoding a block by partitioning the block according to such first aspect or any preceding implementation form of the first aspect.

[0314] In a first implementation form of the method according to such third aspect, the method further comprises the step of transmitting a minimum allowed quadtree leaf node size via a sequence parameter set (SPS).

[0315] A fourth aspect relates to a decoding device including logic circuitry configured to decode blocks by partitioning the blocks according to the partitioning method of such first aspect or any preceding implementation form of the first aspect.

[0316] In a first implementation form of such a decoding device according to the fourth aspect, the logic circuitry is further configured to receive a minimum allowed quadtree leaf node size via a sequence parameter set (SPS).

[0317] A fifth aspect relates to an encoding device including logic circuitry configured to encode a block by partitioning the block according to the partitioning method of such first aspect or any preceding implementation form of the first aspect.

[0318] In a first implementation form of such a decoding device according to the fifth aspect, the logic circuitry is further configured to transmit, via a sequence parameter set (SPS), a minimum allowed quadtree leaf node size.

[0319] A sixth aspect relates to a non-transitory storage medium for storing instructions which, when executed by a processor, cause the processor to perform any such first, second or third aspect or any preceding implementation form of the first, second or third aspect.

[0320] A seventh aspect relates to a method including determining that a current block of a picture is a border block and that a size of the current block is less than or equal to a minimum allowed quad-tree (QT) leaf node size (MinQTSize), and in response to the determination, applying forced binary tree (BT) partitioning to the current block.

[0321] In such a first implementation form of the method according to the seventh aspect, the current block is located at the bottom boundary of the picture, and the forced BT partitioning is a recursive horizontal forced BT partitioning.

[0322] In a second implementation form of the method according to such seventh aspect or any preceding implementation form of the seventh aspect, the current block is located at the right boundary of the picture and the forced BT partitioning is a recursive vertical forced BT partitioning.

[0323] In a third implementation form of a method according to such seventh aspect or any preceding implementation form of the seventh aspect, the forced BT partitioning includes a step of recursively partitioning the current block by horizontal forced boundary partitioning until a sub-partition of the current block is located at the bottom boundary, and a step of recursively partitioning the sub-partition by vertical forced boundary partitioning until a leaf node is located entirely at the right boundary.

[0324] In a fourth implementation form of a method according to such seventh aspect or any preceding implementation form of the seventh aspect, the forced BT partitioning includes a step of recursively partitioning the current block by vertical forced boundary partitioning until a sub-partition of the current block is located at a bottom boundary, and a step of recursively partitioning the sub-partition by horizontal forced boundary partitioning until a leaf node is located entirely at a right boundary.

[0325] In a fifth implementation form of the method according to such seventh aspect or any preceding implementation form of the seventh aspect, the method further comprises applying MinQTSize to control partitioning of non-border blocks.

[0326] In a sixth implementation form of the method according to such seventh aspect or any preceding implementation form of the seventh aspect, the method further includes receiving MinQTSize via a sequence parameter set (SPS).

[0327] In a seventh implementation form of the method according to such seventh aspect or any preceding implementation form of the seventh aspect, the method further comprises transmitting the MinQTSize via a sequence parameter set (SPS).

[0328] An eighth aspect relates to an apparatus that includes a memory; and a processor, coupled to the memory, configured to determine whether a current block of a picture is a border block, and if the current block is a border block, determine whether a size of the current block is greater than a minimum allowed quad-tree (QT) leaf node size (MinQTSize), and apply mandatory binary tree (BT) partitioning to the current block if the size of the current block is not greater than MinQTSize.

[0329] In a first implementation form of such an apparatus according to aspect 8, the forced BT partitioning is a recursive horizontal forced BT partitioning if the current block is located at the bottom boundary of the picture, or a recursive vertical forced BT partitioning if the current block is located at the right boundary of the picture.

[0330] In a second implementation form of the apparatus according to such eighth aspect or any preceding implementation form of the eighth aspect, the forced BT partitioning includes a step of recursively partitioning the current block by horizontal forced boundary partitioning until a sub-partition of the current block is located at the bottom boundary, and a step of recursively partitioning the sub-partition by vertical forced boundary partitioning until the leaf node is located entirely at the right boundary.

[0331] In a third implementation form of an apparatus according to such eighth aspect or any preceding implementation form of the eighth aspect, the forced BT partitioning includes a step of recursively partitioning the current block by vertical forced boundary partitioning until a sub-partition of the current block is located at the bottom boundary, and a step of recursively partitioning the sub-partition by horizontal forced boundary partitioning until the leaf node is located entirely at the right boundary.

[0332] In a fourth implementation form of the apparatus according to such eighth aspect or any previous implementation form of the eighth aspect, the processor is further configured to apply MinQTSize to control partitioning of non-border blocks.

[0333] In a fifth implementation form of the apparatus according to such eighth aspect or any previous implementation form of the eighth aspect, the apparatus further includes a receiver coupled to the processor and configured to receive MinQTSize via a sequence parameter set (SPS).

[0334] In a sixth implementation form of the apparatus according to such eighth aspect or any previous implementation form of the eighth aspect, the apparatus further includes a transmitter coupled to the processor and configured to transmit the MinQTSize via a sequence parameter set (SPS).

[0335] A ninth aspect relates to a computer program product including computer-executable instructions stored on a non-transitory medium that, when executed by a processor, cause an apparatus to determine whether a current block of a picture is a border block, and if the current block is a border block, determine whether a size of the current block is greater than a minimum allowed quad-tree (QT) leaf node size (MinQTSize), and apply mandatory binary tree (BT) partitioning to the current block if the size of the current block is not greater than MinQTSize.

[0336] In a first implementation form of such an apparatus according to aspect 8, the forced BT partitioning is a recursive horizontal forced BT partitioning if the current block is located at the bottom boundary of the picture, or a recursive vertical forced BT partitioning if the current block is located at the right boundary of the picture.

[0337] In a second implementation form of an apparatus according to such ninth aspect or any preceding implementation form of the ninth aspect, the forced BT partitioning includes a step of recursively partitioning the current block by horizontal forced boundary partitioning until a sub-partition of the current block is located at the bottom boundary, and a step of recursively partitioning the sub-partition by vertical forced boundary partitioning until the leaf node is located entirely at the right boundary.

[0338] In a third implementation form of an apparatus according to such ninth aspect or any preceding implementation form of the ninth aspect, the forced BT partitioning includes a step of recursively partitioning the current block by vertical forced boundary partitioning until a sub-partition of the current block is located at the bottom boundary, and a step of recursively partitioning the sub-partition by horizontal forced boundary partitioning until the leaf node is located entirely at the right boundary.

[0339] In a fourth implementation form of the device according to such ninth aspect or any previous implementation form of the ninth aspect, the instructions further cause the device to apply MinQTSize to control partitioning of non-border blocks.

[0340] In a fifth implementation form of an apparatus according to such ninth aspect or any prior implementation form of the ninth aspect, the instructions further cause the apparatus to receive MinQTSize via a sequence parameter set (SPS).

[0341] In a sixth implementation form of the device according to such ninth aspect or any preceding implementation form of the ninth aspect, the instructions further cause the device to transmit the MinQTSize via a sequence parameter set (SPS).

[0342] According to some embodiments, there is provided a decoder including one or more processors and a non-transitory computer readable storage medium coupled to the processors and storing programming for execution by the processors, the programming, when executed by the processors, configuring the decoder to perform any of the methods described above in relation to embodiments 1 to 4.

[0343] Further provided is an encoder including one or more processors and a non-transitory computer-readable storage medium coupled to the processors and storing programming for execution by the processors, the programming, when executed by the processors, configuring the encoder to perform any of the methods described above in relation to embodiments 1 to 4.

[0344] The following is a description of the application of an encoding method similar to the decoding method as shown in the above embodiment, and a system using same.

[0345] 17 is a block diagram showing a content supply system 3100 for implementing a content distribution service. The content supply system 3100 includes a capture device 3102 and a terminal device 3106, and optionally includes a display 3126. The capture device 3102 communicates with the terminal device 3106 via a communication link 3104. The communication link may include the communication channel 13 described above. The communication link 3104 includes, but is not limited to, WIFI, Ethernet, cable, wireless (3G / 4G / 5G), USB, or any type of combination thereof, etc.

[0346] The capture device 3102 may generate data and encode the data by the encoding method as shown in the above embodiment. Alternatively, the capture device 3102 may distribute the data to a streaming server (not shown), which encodes the data and transmits the encoded data to the terminal device 3106. The capture device 3102 may include, but is not limited to, a camera, a smart phone or pad, a computer or laptop, a video conferencing system, a PDA, an in-vehicle device, or any combination thereof. For example, the capture device 3102 may include a source device 12 as described above. If the data includes video, a video encoder 20 included in the capture device 3102 may actually perform a video encoding process. If the data includes audio (i.e., voice), an audio encoder included in the capture device 3102 may actually perform an audio encoding process. In some practical scenarios, the capture device 3102 distributes the encoded video and audio data by multiplexing them together. In other practical scenarios, such as a video conferencing system, the encoded audio data and the encoded video data are not multiplexed. The capture device 3102 distributes the encoded audio data and the encoded video data separately to the terminal device 3106 .

[0347] In the content supply system 3100, the terminal device 310 receives and plays the encoded data. The terminal device 3106 can be a device having data receiving and restoring capabilities, such as a smartphone or pad 3108, a computer or laptop 3110, a network video recorder (NVR) / digital video recorder (DVR) 3112, a TV 3114, a set-top box (STB) 3116, a video conferencing system 3118, a video surveillance system 3120, a portable digital assistant (PDA) 3122, an in-vehicle device 3124, or any combination thereof, or any other device capable of decoding the encoded data as described above. For example, the terminal device 3106 may include a destination device 14 as described above. If the encoded data includes video, the video decoder 30 included in the terminal device is prioritized to perform the video decoding. If the encoded data includes audio, the audio decoder included in the terminal device is prioritized to perform the audio decoding process.

[0348] In case of a terminal device with a display, e.g. a smartphone or pad 3108, a computer or laptop 3110, a network video recorder (NVR) / digital video recorder (DVR) 3112, a TV 3114, a personal digital assistant (PDA) 3122, or an in-vehicle device 3124, the terminal device can provide the decoded data to its display. For terminal devices without a display, such as e.g. a STB 3116, a video conferencing system 3118, or a video surveillance system 3120, an external display 3126 is attached thereto to receive and display the decoded data.

[0349] If each device in the system performs encoding or decoding, it can use a picture encoding device or a picture decoding device as shown in the above embodiments.

[0350] 18 is a diagram showing an example of the configuration of the terminal device 3106. After the terminal device 3106 receives a stream from the capture device 3102, the protocol processing unit 3202 analyzes the transmission protocol of the stream. The protocol includes, but is not limited to, Real Time Streaming Protocol (RTSP), Hypertext Transfer Protocol (HTTP), HTTP Live Streaming Protocol (HLS), MPEG-DASH, Real Time Transport Protocol (RTP), Real Time Messaging Protocol (RTMP), or any kind of combination thereof.

[0351] After the protocol processing unit 3202 processes the stream, a stream file is generated. The file is output to the demultiplexing unit 3204. The demultiplexing unit 3204 can separate the multiplexed data into encoded audio data and encoded video data. As mentioned above, for some practical scenarios, for example in a video conferencing system, the encoded audio data and the encoded video data are not multiplexed. In this situation, the encoded data is sent to the video decoder 3206 and the audio decoder 3208 without passing through the demultiplexing unit 3204.

[0352] The demultiplexing process generates a video elementary stream (ES), an audio ES, and optionally subtitles. The video decoder 3206, which includes the video decoder 30 as described in the above embodiment, decodes the video ES by the decoding method as shown in the above embodiment to generate video frames, and sends this data to the synchronization unit 3212. The audio decoder 3208 decodes the audio ES to generate audio frames, and sends this data to the synchronization unit 3212. Alternatively, the video frames may be stored in a buffer (not shown in FIG. 18) before providing them to the synchronization unit 3212. Similarly, the audio frames may be stored in a buffer (not shown in FIG. 18) before providing them to the synchronization unit 3212.

[0353] The synchronization unit 3212 synchronizes the video and audio frames and provides the video / audio to the video / audio display 3214. For example, the synchronization unit 3212 synchronizes the presentation of video and audio information, which may be coded in a syntax using timestamps for the presentation of the coded audio and visual data and for the delivery of the data stream itself.

[0354] If subtitles are included in the stream, the subtitle decoder 3210 decodes the subtitles, synchronizes them with the video and audio frames, and provides the video / audio / subtitles to the video / audio / subtitle display 3216.

[0355] The present invention is not limited to the above-mentioned system, and either the picture encoding device or the picture decoding device in the above-mentioned embodiments can be embedded in other systems, for example a vehicle system.

[0356] Although embodiments of the present invention have been described primarily in the context of video coding, it should be noted that embodiments of coding system 10, encoder 20 and decoder 30 (and corresponding system 10), as well as other embodiments described herein, may also be configured for processing or coding of still pictures, i.e., processing or coding of individual pictures independent of any preceding or successive pictures, as in video coding. In general, when picture processing coding is limited to a single picture 17, only inter prediction units 244 (encoder) and 344 (decoder) may not be available. All other functions (also called tools or techniques) of the video encoder 20 and the video decoder 30 can be used similarly for still image processing, e.g. residual calculation 204 / 304, transform 206, quantization 208, inverse quantization 210 / 310, (inverse) transform 212 / 312, partitioning 262 / 362, intra prediction 254 / 354, and / or loop filtering 220, 320, and entropy coding 270 and entropy decoding 304.

[0357] For example, embodiments of the encoder 20 and the decoder 30, and the functionality described herein with respect to the encoder 20 and the decoder 30, can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality can be stored on a computer-readable medium or transmitted over a communication medium as one or more instructions or codes and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium, which corresponds to a tangible medium, such as a data storage medium, or a communication medium, which includes any medium that facilitates the transfer of a computer program from one place to another, for example according to a communication protocol. Thus, a computer-readable medium may generally correspond to (1) a tangible computer-readable storage medium that is non-transitory, or (2) a communication medium, such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0358] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored or transmitted as one or more instructions or code in a computer-readable medium and executed by a hardware-based processing unit. A computer-readable medium may include a computer-readable storage medium, which corresponds to a tangible medium, such as a data storage medium, or a communication medium, which includes any medium that facilitates the transfer of a computer program from one place to another, for example according to a communication protocol. As such, a computer-readable medium may generally correspond to (1) a tangible computer-readable storage medium that is non-transitory, or (2) a communication medium, such as a signal or carrier wave. A data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0359] For example, and without limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included within the definition of media. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but rather are directed to non-transitory tangible storage media. Disk and disk, as used herein, include compact discs (CDs), laser discs, optical disks, digital versatile disks (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically and disks reproduce data optically with a laser. Combinations of the above should also be included within the scope of computer readable media.

[0360] The instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Thus, the term "processor" as used herein may refer to any of the foregoing structures, or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or may be incorporated into a combined codec. Additionally, the techniques may be implemented entirely in one or more circuit or logic elements.

[0361] The techniques of the present disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chip set). Various components, modules, or units are described in the present disclosure to highlight functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require implementation by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit or may be provided by a collection of interoperable hardware units including one or more processors as described above, together with appropriate software and / or firmware.

[0362] The following logical or mathematical operators are defined as follows: The mathematical operators used in this application are similar to those used in the C programming language. However, the results of integer division and arithmetic shift operations are more precisely defined, and additional operations such as exponentiation and real division are specified. The numbering and counting conventions generally start from 0, e.g., "first" is equivalent to 0th, "second" is equivalent to 1st, etc.

[0363] Arithmetic Operators The following arithmetic operators are defined as follows: [Table 1]

[0364] Logical Operators The following logical operators are defined as follows: x&&y The Boolean logic "and" of x and y x||y The Boolean "or" of x and y ! Boolean logic “not” x?y:zIf x is TRUE or not equal to 0, then evaluate the value of y, else evaluate the value of z.

[0365] Relational Operators The following relational operators are defined as follows: > Greater than >= Greater than or equal to < Less than <= less than or equal == equal to != not equal When a relational operator is applied to a syntax element or variable that has been assigned the value "na" (not applicable), the value "na" is treated as a separate value of that syntax element or variable. The value "na" is not considered equal to any other value.

[0366] Bitwise Operators The following bitwise operators are defined as follows: & Bit-wise "and". When operating on an integer argument, it operates on the two's complement representation of the integer value. When operating on a binary argument that contains fewer bits than the other argument, the shorter argument is extended by adding more significant bits equal to zero. | Bitwise “or”. When acting on integer arguments, it acts on the two's complement representation of the integer values. When acting on binary arguments that contain fewer bits than the other arguments, the shorter argument is extended by adding additional high-order bits equal to 0. ^ Bitwise “exclusive or”. When acting on integer arguments, it acts on the two's complement representation of the integer values. When acting on binary arguments that contain fewer bits than the other arguments, the shorter argument is extended by adding additional high-order bits equal to 0. x>>y The two's complement integer representation of x arithmetically right-shifted by the binary number y. This function is defined only for non-negative integer values of y. The bit shifted into the most significant bit (MSB) as a result of the right shift is equal to the MSB of x before the shift operation. x<<y The two's complement integer representation of x arithmetically left-shifted by the binary number y. This function is defined only for non-negative integer values of y. The bit shifted into the least significant bit (LSB) as a result of the left shift has a value equal to 0.

[0367] Assignment operators The following assignment operators are defined as follows: = Assignment operator ++ Increment. That is, x++ is equal to x=x+1. When used as an array index, the value of the variable before the increment operation is evaluated. -- Decrement. That is, x-- is equal to x=x-1. When used as an array index, the value of the variable before the decrement operation is evaluated. += Increment by the specified amount. That is, x+=3 is equal to x=x+3. x+=(-3) is equal to x=x+(-3). -= Decrement by the specified amount. That is, x-=3 is equal to x=x-3. x-=(-3) is equal to x=x-(-3).

[0368] Range notation The following notations are used to specify a range of values: x=y..zx takes integer values ​​starting from y up to and including z, where x, y and z are integers and z is greater than y.

[0369] Mathematical Functions The following mathematical functions are defined: [Table 2]

[0370] Operation priority order When precedence is not explicitly specified in an expression by using parentheses, the following rules apply: - An operation with higher precedence is evaluated before any operation with lower precedence. - Operations of equal precedence are evaluated from left to right.

[0371] The following table shows the precedence of operations from highest to lowest, with a higher position in the table indicating a higher precedence.

[0372] With respect to operators that are also used in the C programming language, the precedence used in this specification is the same as that used in the C programming language. Table: Operation precedence from highest (top of table) to lowest (bottom of table) [Table 3]

[0373] Text description of logical operators In the text, the following forms: if( condition 0 ) statement 0 else if( condition 1 ) statement 1 ... else / * informative remark on remaining condition * / statement n A logical statement, mathematically described as: can be written in the following way: …as follows / ...the following applies: - If condition 0, statement 0 - Otherwise, if condition 1, statement 1 - ... - Otherwise (informative remark on remaining condition), statement n

[0374] Each "If ... Otherwise, if ... Otherwise, ..." statement in the text is introduced with "... as follows" or "... the following applies" immediately following the "If...". The last condition in an "If ... Otherwise, if ... Otherwise, ..." is always "Otherwise, ...". Alternating "If ... Otherwise, if ... Otherwise, ..." statements can be identified by matching the "... as follows" or "... the following applies" with the final "Otherwise, ...".

[0375] In the text, the following forms: if( condition 0a && condition 0b ) statement 0 else if( condition 1a || condition 1b ) statement 1 ... else statement n A logical statement, mathematically described as: can be written in the following way: ... as follows / ... the following applies: - If all of the following conditions are true, statement 0: -condition 0a -condition 0b - Otherwise, if one or more of the following conditions are true, statement 1: -condition 1a -condition 1b - ... - Otherwise, statement n

[0376] In the text, the following forms: if( condition 0 ) statement 0 if( condition 1 ) statement 1 A logical statement, mathematically described as: can be written in the following way: If condition is 0, then statement 0 If condition 1, then statement 1

[0377] In summary, the present disclosure relates to methods and devices used for encoding and decoding of image or video signals. These include determining whether the size of the current block is greater than the minimum allowed quadtree leaf node size. If the size of the current block is not greater than the minimum allowed quadtree leaf node size, a multitype tree split is applied to the current block. The minimum allowed quadtree leaf node size is not greater than the maximum allowed binary tree root node size, or the minimum allowed quadtree leaf node size is not greater than the maximum allowed ternary tree root node size.

Claims

1. A method of encoding, comprising the steps of: determining whether the size of the current block is greater than the minimum allowable quadtree leaf node size; applying a ternary tree split to the current block based on a maximum allowable ternary tree root node size, under the condition that the size of the current block is not greater than the minimum allowable quadtree leaf node size, wherein the maximum allowable ternary tree root node size is determined based on the minimum allowable quadtree leaf node size, and the minimum allowable quadtree leaf node size is not greater than the maximum allowable ternary tree root node size; generating a prediction block of the current block after applying the ternary tree partitioning to the current block; obtaining a residual block of the current block based on the current block and the predicted block; obtaining transform coefficients of the current block by applying a transform to sample values ​​of the residual block; encoding the transform coefficients of the current block into a bitstream; The method includes:

2. The method of claim 1 , wherein the current block is a non-border block.

3. 10. The method of claim 1 , Dividing the image into blocks, the blocks including the current block; The step of applying the ternary tree partitioning to the current block comprises: applying a ternary tree partition to the current block of the block having a final maximum multitype tree depth, the final maximum multitype tree depth being at least the sum of a maximum multitype tree depth and a maximum multitype tree depth offset; A method comprising:

4. The method of claim 3 , wherein the maximum multitype tree depth offset is zero.

5. The method of claim 3 , wherein the maximum multitype tree depth is greater than zero.

6. 10. The method of claim 1, further comprising: encoding a first syntax element and a second syntax element into the bitstream, the first syntax element being used to derive the minimum allowable quad-tree leaf node size and the second syntax element being used to derive the maximum allowable ternary tree root node size; The method includes:

7. 7. The method of claim 6, wherein the first syntax element and the second syntax element are signaled in a Sequence Parameter Set (SPS).

8. 10. The method of claim 1, further comprising: transmitting the bitstream; The method includes:

9. 1. A method of decoding, comprising the steps of: receiving a bitstream containing encoded data of an image; determining whether a size of a current block of the image is greater than a minimum allowable quadtree leaf node size; applying a ternary tree split to the current block based on a maximum allowable ternary tree root node size, under the condition that the size of the current block is not greater than the minimum allowable quadtree leaf node size, wherein the maximum allowable ternary tree root node size is determined based on the minimum allowable quadtree leaf node size, and the minimum allowable quadtree leaf node size is not greater than the maximum allowable ternary tree root node size; obtaining transform coefficients of the current block from the bitstream; obtaining a residual block of the current block based on the transform coefficients of the current block; reconstructing the current block based on the residual block of the current block and the prediction block of the current block; The method includes:

10. The method of claim 9 , wherein the current block is a non-border block.

11. 10. The method of claim 9, wherein the bitstream further comprises a first syntax element, the method further comprising: Parsing the first syntax element from the bitstream; deriving the minimum allowable quadtree leaf node size based on the first syntax element; The method includes:

12. 10. The method of claim 9, wherein the bitstream includes a second syntax element, the method further comprising: Parsing the second syntax element from the bitstream; deriving the maximum allowable ternary tree root node size based on the second syntax element; The method includes:

13. A coding device comprising a processing circuit for carrying out the method according to any one of claims 1 to 12.

14. On the coding device: one or more processors; and a non-transitory computer readable storage medium coupled to the one or more processors and storing programming for execution by the one or more processors, the programming, when executed by the one or more processors, configuring the coding device to perform the method of any one of claims 1-12; Including coding device.

15. A computer program comprising a program code for performing the method according to any one of claims 1 to 12 when the computer program is executed on a computer or processor.

16. A non-transitory computer readable storage medium storing program code which, when executed by a computing device, causes said computing device to perform the method of any one of claims 1-12.

17. 1. A device for storing a video or image bitstream, comprising a communications interface and a storage medium, the communications interface configured to receive and / or transmit a bitstream, the storage medium configured to store the bitstream, the bitstream including encoded data of one or more images, a first syntax element for deriving a minimum allowed quadtree leaf node size, and a second syntax element for deriving a maximum allowed ternary tree root node size based on the minimum allowed quadtree leaf node size; the minimum allowed quadtree leaf node size is not greater than the maximum allowed ternary tree root node size, and the bitstream further includes parameters of a prediction process and a transformation process.

Images