A method and apparatus for video decoding performed by a decoder, and a method for video encoding performed by an encoder.

Abstract

Methods and apparatus, including processing circuitry, for decoding coded information of a coding block (CB) in a picture from a coded video bitstream are provided.SOLUTION: The coded information indicates a width W and a height H of the CB. The processing circuitry partitions the CB into sub-processing units (SPUs) having a width that is the smallest one of W and K and a height that is the smallest one of H and K. At least one of the width W and the height H is larger than the processing data unit size K. The processing circuitry determines a partitioning structure for dividing the SPU based on the width and the height of the SPU and a maximum transform unit (TU) size M. At least one of a width and a height of the SPU is greater than M. The processing circuitry divides each of the SPUs into M * M TUs based on the partitioning structure.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present disclosure describes embodiments generally related to video coding.

Background Art

[0002] The description of the background given herein is for the purpose of generally presenting the background of the present disclosure. The research of the presently named inventors, insofar as that research is within the scope described in this background section and aspects of the description that might otherwise qualify as prior art at the time of filing, is not admitted as prior art to the present disclosure, either expressly or implicitly.

[0003] Video coding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a sequence of pictures, each picture having, for example, a spatial dimension of 1920×1080 luminance samples and associated chrominance samples. The sequence of pictures can have, for example, a fixed or variable picture rate of 60 pictures per second, i.e., 60 Hz (commonly known as the frame rate). Uncompressed video has significant bitrate requirements. For example, 1080p60 4:2:0 video (1920×1080 luminance sample resolution at a frame rate of 60 Hz) with 8 bits per sample requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires more than 600 Gbytes of storage space.

[0004] One purpose of video coding and decoding can be to reduce the redundancy of the input video signal through compression. Compression can help reduce the bandwidth or storage space requirements by more than two orders of magnitude in some cases. Both lossless and lossy compression, as well as combinations thereof, can be used. Lossless compression refers to a technique in which an exact copy of the original signal can be reconstructed from the compressed original signal. When lossy compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal useful for its intended purpose. In the case of video, lossy compression is widely used. The amount of distortion that can be tolerated depends on the application; for example, users of certain consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio can reflect that higher tolerable / acceptable distortion can result in a higher compression ratio.

[0005] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transformation, quantization, and entropic coding.

[0006] Video codec techniques may include a technique known as intracoding. In intracoding, sample values ​​are represented without referencing samples or other data from a previously reconstructed reference picture. In some video codecs, a picture is subdivided spatially into blocks of samples. A picture can be an intrapicture if all blocks of samples are coded in intra mode. Intrapictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and can therefore be used as the first picture in a coded video bitstream and video session, or as a still image. Samples in intrablocks can undergo transformations, and the transformation coefficients can be quantized before entropic coding. Intraprediction can be a technique that minimizes sample values ​​in the pre-transformation region. In some cases, the smaller the post-transformation DC value and the smaller the AC coefficient, the fewer bits are needed at a given quantization step size to represent a block after entropic coding.

[0007] For example, traditional intra-coding, as known from MPEG-2 generation coding techniques, does not use intra-prediction. However, newer video compression techniques include, for example, techniques that attempt to use surrounding sample data and / or metadata obtained during the encoding / decoding of spatially adjacent and preceding blocks in the decoding order. Such techniques will hereafter be referred to as “intra-prediction” techniques. Note that in at least some cases, intra-prediction uses reference data only from the current picture being reconstructed, and not from the reference picture.

[0008] A wide variety of intra-prediction forms can exist. If more than one of these techniques can be used in a given video coding technique, the technique in use may be coded in intra-prediction mode. In certain cases, a mode may have submodes and / or parameters, which may be coded independently or included in a mode codeword. Since which codeword should be used for a given mode / submode / parameter combination can affect the coding efficiency gain through intra-prediction, an entropy coding technique may be used to convert the codeword into a bitstream.

[0009] Intra-prediction in specific modes was introduced by H.264, refined in H.265, and further refined with newer coding techniques such as Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). Predictor blocks can be formed using neighboring sample values ​​belonging to already available samples. The sample values ​​of neighboring samples are copied into the predictor block according to the direction. A reference to the direction in use may be coded in the bitstream or may be predicted itself.

[0010] Referring to Figure 1A, the lower right shows a subset of nine predictor directions known from the 33 possible predictor directions of H.265 (corresponding to 33 of the 35 intra-modes, or angular modes). The point where the arrows converge (101) corresponds to the sample being predicted. The arrows correspond to the direction in which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples located to the upper right at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples located to the lower left of sample (101) at an angle of 22.5 degrees from the horizontal.

[0011] Still referring to Figure 1A, the upper left shows a square block (104) of 4x4 samples (indicated by a thick dashed line). The square block (104) contains 16 samples, each sample being denoted by "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in block (104) in both the Y and X dimensions. Since the block is 4x4 samples in size, S44 is in the lower right. Furthermore, a reference sample is shown that follows a similar numbering scheme. The reference sample is denoted by "R", its Y position (e.g., row index), and its X position (column index) relative to block (104). In both H.264 and H.265, the predicted sample is adjacent to the block being reconstructed, and therefore negative values ​​do not need to be used.

[0012] Intra-picture prediction can work by copying reference sample values ​​from adjacent samples as needed, depending on the prediction direction conveyed by the signal. For example, suppose the coded video bitstream includes signaling for this block that indicates a prediction direction corresponding to arrow (102), i.e., that the sample is predicted from one or more prediction samples located to the upper right at an angle of 45 degrees from the horizontal. In that case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0013] In certain cases, the values ​​of multiple reference samples may be combined, for example, through interpolation, in order to calculate the reference sample, especially when the directions are not equally divisible at 45 degrees.

[0014] The number of possible directions has increased with advancements in video coding technology. H.264 (2003) allowed for the representation of nine different directions. This increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions as of this disclosure. Experiments have been conducted to identify the most likely directions, and certain techniques in entropic coding are used to represent those likely directions with a small number of bits, accepting a slight penalty for less likely directions. Furthermore, the direction itself can sometimes be predicted from adjacent directions used in adjacent, already decoded blocks.

[0015] Figure 1B shows a schematic diagram (180) representing the 65 intra-prediction directions by JEM to illustrate the increasing number of prediction directions over time.

[0016] The mapping of intra-predicted direction bits within a coded video bitstream representing direction can vary depending on the video coding technique, ranging from simple direct mapping of predicted direction to intra-predictive modes, codewords, complex adaptive schemes including most probable modes, and similar techniques. In all cases, however, there may be certain directions that are statistically less likely to occur in video content than certain other directions. Since the goal of video compression is to reduce redundancy, those less likely directions will be represented by more bits than more likely directions in a well-functioning video coding technique.

[0017] Motion compensation can be an irreversible compression technique and may relate to a technique used to predict a newly reconstructed picture or picture portion after blocks of sample data from a previously reconstructed picture or portion thereof (reference picture) have been spatially shifted in the direction indicated by a motion vector (MV). In some cases, the reference picture can be the same as the picture currently being reconstructed. The MV can have two dimensions, X and Y, or three dimensions, the third of which indicates the reference picture in use (the latter indirectly being the time dimension).

[0018] In some video compression techniques, the motion vector (MV) applicable to a particular area of ​​sample data can be predicted from other MVs, for example, those relating to other areas of sample data spatially adjacent to the area being reconstructed and preceding that MV in the decoding order. This significantly reduces the amount of data required to code the MV, thereby eliminating redundancy and increasing compression. For example, when coding an input video signal obtained from a camera (known as natural video), MV prediction can be effective because there is a statistical probability that an area larger than the area to which a single MV is applicable will move in a similar direction, and therefore, in some cases, it can be predicted using similar motion vectors derived from the MVs of adjacent areas. As a result, the MV obtained for a given area is similar to or the same as the MV predicted from the surrounding MVs, and can be represented with fewer bits than would be used if the MV were coded directly after entropy coding. In some cases, MV prediction can be an example of lossless compression of the signal (i.e., MV) derived from the original signal (i.e., sample stream). In other cases, MV prediction itself may be lossy, for example, due to rounding errors when calculating the predictor from some surrounding MVs.

[0019] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H265, “High Efficiency Video Coding”, December 2016). Among the many MV prediction mechanisms proposed by H.265, this specification will describe the technique hereafter referred to as “spatial merging”.

[0020] Referring to Figure 2, the current block (201) has samples that the encoder determined during the motion search process to be predictable from a previous block of the same size, but spatially shifted. Instead of directly coding its MV, the MV can be derived from metadata associated with one or more reference pictures, for example, from the most recent reference picture (in decoding order), using the MV associated with one of five surrounding samples represented as A0, A1 and B0, B1, B2 (202 through 206, respectively). In H.265, the MV prediction can use predictors from the same reference pictures used by adjacent blocks. [Overview of the project]

[0021] The disclosure provides methods and apparatus for video encoding / decoding. In some examples, the apparatus for video decoding includes a processing circuit. The processing circuit is configured to decode coded information of coding blocks (CBs) in a picture from a coded video bitstream. The coded information indicates the width of W samples and the height of H samples of the CB. The processing circuit can divide the CB into sub-processing units (SPUs) having a width which is the minimum of W and K, and a height which is the minimum of H and K. At least one of the width W and height H of the CB is greater than the processing data unit size K. The processing circuit can determine a partitioning structure for further dividing the SPU based on the width and height of the SPU and the maximum conversion unit (TU) size of M samples. At least one of the width and height of the SPU is greater than M. Based on the determined partitioning structure, the processing circuit can divide each SPU into M × M TUs.

[0022] In this embodiment, the width and height of the SPU are greater than M. The processing circuit can determine that the partitioning structure is a quadtree partitioning structure. The processing circuit can divide the SPU into the following TUs based on the quadtree partitioning structure.

[0023] In this embodiment, the width of the SPU is greater than M, and the height of the SPU is equal to M. The processing circuit can determine that the partitioning structure is a vertical binary tree partitioning structure. The processing circuit can partition the SPU into TUs based on the vertical binary tree partitioning structure.

[0024] In this embodiment, the height of the SPU is greater than M, and the width of the SPU is equal to M. The processing circuit can determine that the partitioning structure is a horizontal binary tree partitioning structure. The processing circuit can partition the SPU into TUs based on the horizontal binary tree partitioning structure.

[0025] In an embodiment, the processing circuit can recursively divide one of the SPUs into TUs based on a partitioning structure.

[0026] In an embodiment, the processing circuit can process the SPUs according to a first scan order and process the TUs in each of the SPUs according to a second scan order. In an example, at least one of the first scan order and the second scan order is one of (i) a raster scan order, (ii) a vertical scan order, (iii) a zigzag order, and (iv) a diagonal scan order. In an example, the first scan order and the second scan order are the raster scan order. In an example, W is 128, H is 64, K is 64, and M is 32. The first scan order is left or right, and the second scan order is the raster scan order.

[0027] In an embodiment, the processing data unit size K indicates the size of a virtual pipeline data unit (VPDU). In a picture, the first SPU among the SPUs is included in the first VPDU, and the second SPU among the SPUs is included in the second VPDU. After processing the first VPDU in the first stage of a multi-stage pipeline, the processing circuit can process the first VPDU in the second stage of the multi-stage pipeline and the second VPDU in the first stage simultaneously.

[0028] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

Brief Description of the Drawings

[0029] [Figure 1A] It is a schematic diagram of an exemplary subset of intra prediction modes. [Figure 1B] It is an explanatory diagram of an exemplary intra prediction direction. [Figure 2] It is a schematic diagram of a current block and its surrounding spatial merge candidates in an example. [Figure 3]This is a schematic diagram of a simplified block diagram of a communication system (300) according to an embodiment. [Figure 4] This is a schematic diagram of a simplified block diagram of a communication system (400) according to an embodiment. [Figure 5] This is a schematic diagram of a schematic block diagram of a decoder according to the embodiment. [Figure 6] This is a schematic diagram of a simplified block diagram of an encoder according to an embodiment. [Figure 7] A block diagram of an encoder according to another embodiment is shown. [Figure 8] A block diagram of a decoder according to another embodiment is shown. [Figure 9A] This shows the CTU partitioned by a quadtree-plus-binary tree (QTBT) structure (910). [Figure 9B] The QTBT structure (920) is shown. [Figure 9C] This shows a horizontal center-side ternary tree. [Figure 9D] This shows a vertical center-side ternary tree. [Figure 10A] The transformation core matrix for the 4-point DCT-2 transformation is shown. [Figure 10B] The transformation core matrix for the 8-point DCT-2 transformation is shown. [Figure 10C] The transformation core matrix for the 16-point DCT-2 transformation is shown. [Figure 10D] The transformation core matrix for the 32-point DCT-2 transformation is shown. [Figure 11A] The 64x64 transformation core matrix for the 64-point DCT-2 transformation is shown. [Figure 11B] This is a continuation from Figure 11A. [Figure 11C] This is a continuation from Figure 11B. [Figure 11D] This is a continuation from Figure 11C. [Figure 11E] This is a continuation from Figure 11D. [Figure 12] This shows the elementary transformation functions for the selected discrete sine transform (DST) / discrete cosine transform (DCT) of Adaptive Multiple Transforms (AMT). [Figure 13]Table (1300) shows the mapping relationship between the mts_idx value and each horizontal or vertical transformation. [Figure 14A] The transformation core matrix for the 4-point DST-7 transformation is shown. [Figure 14B] The transformation core matrix for the 8-point DST-7 transformation is shown. [Figure 14C] The transformation core matrix for the 16-point DST-7 transformation is shown. [Figure 14D] The transformation core matrix for the 32-point DST-7 transformation is shown. [Figure 15A] The transformation core matrix for the 4-point DCT-8 transformation is shown. [Figure 15B] The transformation core matrix for the 8-point DCT-8 transformation is shown. [Figure 15C] The transformation core matrix for the 16-point DCT-8 transformation is shown. [Figure 15D] The transformation core matrix for the 32-point DCT-8 transformation is shown. [Figure 16] This indicates the number of subpartitions, which depends on the block size. [Figure 17] An example of an intra-subpartition (ISP) is shown. [Figure 18] Here is an example of an ISP. [Figure 19A] An example of syntax elements (1900) for ISP coding mode is shown. [Figure 19B] This is a continuation from Figure 19A. [Figure 20A] An example of a subblock transformation (SBT) is shown. [Figure 20B] An example of a subblock transformation (SBT) is shown. [Figure 20C] An example of a subblock transformation (SBT) is shown. [Figure 20D] An example of a subblock transformation (SBT) is shown. [Figure 21A] Here is an example of the specification text for a video coding standard when SBT is used. [Figure 21B] Here is an example of the specification text for a video coding standard when SBT is used. [Figure 21C]Here is an example of the specification text for a video coding standard when SBT is used. [Figure 21D] Here is an example of the specification text for a video coding standard when SBT is used. [Figure 21E] Here is an example of the specification text for a video coding standard when SBT is used. [Figure 21F] Here is an example of the specification text for a video coding standard when SBT is used. [Figure 21G] Here is an example of the specification text for a video coding standard when SBT is used. [Figure 21H] Here is an example of the specification text for a video coding standard when SBT is used. [Figure 21I] This is a continuation from Figure 21H. [Figure 22] This shows various YUV formats used in several embodiments. [Figure 23] Examples of unacceptable ternary (TT) and binary (BT) partitioning are shown. [Figure 24] An example of transformation tree syntax is shown. [Figure 25] A coding block (2510) with a size of 128 × 64 samples is shown. [Figure 26A] A coding block (2610A) with a size of 128 × 32 samples is shown. [Figure 26B] A coding block (2610B) with a size of 128 × 32 samples is shown. [Figure 27] A flowchart illustrating the process (2700) according to the disclosed embodiment is shown. [Figure 28] This is a schematic diagram of a computer system according to an embodiment. [Modes for carrying out the invention]

[0030] [I. Video Coding Encoders and Decoders] Figure 3 shows a schematic block diagram of a communication system (300) according to an embodiment of the present disclosure. The communication system (300) includes a plurality of terminal devices that can communicate with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In Figure 3, the first pair of terminal devices (310) and (320) perform one-way transmission of data. For example, terminal device (310) may encode video data (e.g., a stream of video data captured by terminal device (310)) for transmission to other terminal devices (320) via the network (350). The encoded video data can be transmitted in the form of one or more encoded video bitstreams. Terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to recover a video picture, and display the video picture according to the recovered video data. One-way data transmission can be common in applications such as media serving.

[0031] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of coded video data that may appear, for example, during a video conference. For bidirectional transmission of data, in this example, each terminal device of terminal devices (330) and (340) may code video data (e.g., a stream of video pictures captured by that terminal device) for transmission to the other terminal device of terminal devices (330) and (340) over the network (350). Each terminal device of terminal devices (330) and (340) may also receive coded video data transmitted by the other terminal device of terminal devices (330) and (340), decode the coded video data to recover video pictures, and display the video pictures on an accessible display device according to the recovered video data.

[0032] In the example in Figure 3, terminal devices (310), (320), (330), and (340) may be represented as a server, a personal computer, and a smartphone, respectively, but the principles of this disclosure are not limited thereto. Embodiments of this disclosure find applications in laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (350) corresponds to any number of networks that transmit coded video data between terminal devices (310), (320), (330), and (340), including, for example, wireline and / or wireless communication networks. Communication network (350) may exchange data over circuit-switched and / or packet-switched channels. Typical networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of network (350) may be irrelevant to the operation of this disclosure unless otherwise described below.

[0033] Figure 4 illustrates the arrangement of a video encoder and video decoder in a streaming environment as an example of the disclosed subject matter. The disclosed subject matter can similarly be applied to other video-enabled applications, including, for example, video conferencing, digital television, and the storage of compressed video on digital media such as CDs, DVDs, and memory sticks.

[0034] The streaming system may include a capture subsystem (413) which may include a video source (401), such as a digital camera, that generates a stream (402) of uncompressed video pictures. In the example, the stream (402) includes a sample captured by the digital camera. The stream (402) is represented by a thick line to highlight its higher data volume compared to encoded video data (404) (or encoded video bitstream) and may be processed by an electronic device (420) which includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement the aspects of the subject disclosed, as will be described in more detail below. The encoded video data (404) (or encoded video bitstream (404)) is represented by a thin line to highlight its lower data volume compared to the stream (402) and may be stored in a streaming server (405) for future use. One or more streaming client subsystems, such as client subsystems (406) and (408) in Figure 4, can access a streaming server (405) to read copies (407) and (409) of the encoded video data (404). The client subsystem (406) may include, for example, a video decoder (410) in an electronic device (430). The video decoder (410) decodes an incoming copy (407) of the encoded video data and generates an output stream (411) of a video picture that can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., video bitstreams) may be encoded according to a specific video coding / compression standard.An example of such a standard is the ITU-T recommended H.265. In this example, the video coding standard under development is commonly known as Versatile Video Coding (VVC). The disclosed subject matter may be used in connection with VVC.

[0035] It is known that electronic devices (420) and (430) may include other components (not shown). For example, electronic device (420) may include a video decoder (not shown), and electronic device (430) may similarly include a video encoder (not shown).

[0036] Figure 5 shows a block diagram of a video decoder (510) according to an embodiment of the present disclosure. The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., a receiving circuit). The video decoder (510) may be used in place of the video decoder (410) in the example of Figure 4.

[0037] The receiver (531) may receive one or more coded video sequences to be decoded by the video decoder (510), or, in the same or other embodiments, one coded video sequence at a time, where the decoding of each coded video sequence is independent of other coded video sequences. The coded video sequences may be received from a channel (501), which may be a hardware / software link to a storage device storing encoded video data. The receiver (531) may receive the encoded video data together with other data, such as coded audio data and / or auxiliary data streams, which may be transferred to their respective usage entities (not shown). The receiver (531) may isolate the coded video sequences from the other data. To counteract network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter "parser (520)"). In certain applications, the buffer memory (515) is part of the video decoder (510). Otherwise, it can be located outside the video decoder (510) (not shown). Further in other cases, there may be a buffer memory outside the video decoder (510) (not shown), for example, to counteract network jitter, and in addition, another buffer memory (515) within the video decoder (510), for example, to manipulate playback timing. When the receiver (531) is receiving data from a sufficiently bandwidth and controllable storage / transfer device or from an isosynchronous network, the buffer memory (515) may not be necessary or may be small. For use in best-effort packet networks such as the Internet, the buffer memory (515) may be necessary, and can be relatively large and, advantageously, adaptively sized, and can be at least partially implemented in an operating system or similar element (not shown) outside the video decoder (510).

[0038] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the coded video sequence. The categories of these symbols include information used to manage the operation of the video decoder (510) and information for controlling rendering devices, such as a render device (512) (e.g., a display screen), which are not essential parts of the electronic device (530) but can be coupled to the electronic device (530), as shown in Figure 5. The control information for rendering devices may take the form of Supplementary Enhancement Information (SEI) messages or Video Usability Information (VUI) parameter set fragments (not shown). The parser (520) may parse / entropidecode the incoming coded video sequence. The coding of the coded video sequence may follow video coding techniques or standards and may follow various principles, including variable-length coding, Huffman coding, context-dependent or independent arithmetic coding, etc. The parser(520) may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder, based on at least one parameter corresponding to that group. Subgroups may include groups of pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CU), blocks, transform units (TU), predictive units (PU), and so on. The parser(520) may also extract from the coded video sequence information, such as quantization parameter values, motion vectors, and so on, including transform coefficients.

[0039] The parser (520) may perform entropy decoding / parsening operations on the video sequence received from the buffer memory (515) in order to generate symbols (521).

[0040] The reconstruction of symbol (521) can have a wide variety of units depending on the type of coded video picture or part thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. How each unit is included may be controlled by subgroup control information parsed by parser (520) from the coded video sequence. The flow of such subgroup control information between parser (520) and the following multiple units is not shown for clarity.

[0041] Beyond the functional blocks already described, the video decoder (510) can be conceptually subdivided into numerous functional units, as described below. In actual implementations operating under commercial constraints, many of these units may interact closely with each other and be at least partially integrated with one another. However, for the purpose of illustrating the subject matter disclosed, the conceptual subdivision into functional units described below is appropriate.

[0042] The first unit is the scaler / inverse unit (551). The scaler / inverse unit (551) receives control information from the parser (520) as symbols (521), along with the quantized transformation coefficients, including what to transform for use, block size, quantization coefficients, and quantization scaling matrix. The scaler / inverse unit (551) can output a block containing sample values ​​that can be input to the aggregator (555).

[0043] In some cases, the output samples of the scaler / inverse converter (551) may relate to intracoded blocks, i.e., blocks that do not use predictive information from previously reconstructed pictures but can use predictive information from previously reconstructed portions of the current picture. Such predictive information may be supplied by an intrapicture predictive unit (552). In some cases, the intrapicture predictive unit (552) generates a block of the same size and shape as the block being reconstructed, using surrounding already reconstructed information fetched from the current picture buffer (558). The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. In some cases, the aggregator (555) adds the predictive information generated by the intrapredictive unit (552) to the output sample information supplied by the scaler / inverse converter (551), sample by sample.

[0044] In other cases, the output samples of the scaler / inverse unit (551) may relate to an interconnected and potentially motion-compensated block. In such cases, the motion-compensated prediction unit (553) may access the reference picture memory (557) to fetch samples to be used for prediction. After motion-compensating the fetched samples according to the symbols (521) relating to the block, those samples may be added by the aggregator (555) to the output of the scaler / inverse unit (551) (referred to in this case to residual samples or residual signal) to generate output sample information. The address in the reference picture memory (557) from which the motion-compensated prediction unit (553) fetches the prediction samples may be controlled by a motion vector that can be made available to the motion-compensated prediction unit (553) in the form of a symbol (521) which may have X, Y and reference picture components, for example. Motion compensation may also include interpolation of sample values ​​fetched from reference picture memory (557) when the precise motion vectors of subsamples are used, as well as motion vector prediction mechanisms.

[0045] The output samples of the aggregator (555) can undergo various loop filtering techniques in the loop filter unit (556). The video compression technique may include an in-loop filtering technique. This technique is contained in the coded video sequence (also called the coded video bitstream) and is controlled by parameters made available to the loop filter unit (556) as symbols (521) from the parser (520), but can also respond to metadata obtained during decoding of the previous part (in the decoding order) of the coded picture or coded video sequence, and further can respond to previously configured loop-filtered sample values.

[0046] The output of the loop filter unit (556) can be a sample stream that is output to the render device (512) and can also be stored in reference picture memory (557) for use in future interpicture prediction.

[0047] A particular coded picture, once fully reconfigured, may be used as a reference picture for future predictions. For example, once the coded picture corresponding to the current picture is fully reconfigured and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) may become part of the reference picture memory (557), and any unused current picture buffer may be reallocated before the reconfiguration of subsequent coded pictures begins.

[0048] The video decoder (510) may perform decoding operations according to a specified video compression technique in a standard such as the ITU-T recommended H.265. The coded video sequence may conform to the syntax defined by the video compression technique or standard in use, in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, a profile may select a particular tool from all the tools available in the video compression technique or standard as the best tool available for use under that profile. Furthermore, the complexity of the coded video sequence must be within the boundaries defined by the level of the video compression technique or standard in order to comply. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples / second), maximum reference picture size, etc. The limits set by the level may, in some cases, be further limited through the Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management notified in the coded video sequence.

[0049] In an embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may also be included as parts of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may take the form of, for example, a time, space, or signal-to-noise ratio (SNR) enhancement layer, redundant slices, redundant pictures, forward error correction codes, etc.

[0050] Figure 6 shows a block diagram of a video encoder (603) according to an embodiment of the present disclosure. The video encoder (603) is contained within an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) may be used in place of the video encoder (403) in the example of Figure 4.

[0051] The video encoder (603) may receive video samples from a video source (601) (not part of the electronic device (620) in the example in Figure 6) that can capture video images to be coded by the video encoder (603). In other examples, the video source (601) is part of the electronic device (620).

[0052] The video source (601) may supply a source video sequence to be coded by the video encoder (603) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCB, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (601) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be supplied as a series of individual pictures that give motion when viewed sequentially. The picture itself may be organized as a spatial array of pixels, each pixel may have one or more samples depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. This specification will focus on samples below.

[0053] In accordance with the embodiments, the video encoder (603) may encode and compress the pictures of a source video sequence into a coded video sequence (643) in real time or under any other time constraints required by the application. Enforcing an appropriate coding speed is a function of the controller (650). In some embodiments, the controller (650) controls and is functionally coupled to other functional units, such as those described below. Couplings are not shown for clarity. Parameters set by the controller (650) may include parameters related to rate control (e.g., picture skip, quantizer, lambda value of rate distortion optimization technique), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other appropriate functions related to the video encoder (603) optimized for a particular system design.

[0054] In some embodiments, the video encoder (603) is configured to operate in a coding loop. In an overly simplified description, in the example, the coding loop may include a source coder (630) (which is involved in generating symbols, such as a symbol stream, based, for example, an input picture to be coded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to generate sample data in a similar manner to that which a (remote) decoder would also generate (meaning that any compression between the symbols and the coded video stream is reversible in the video compression techniques considered in the disclosed subject). The reconstructed sample stream (sample data) is input to the reference picture memory (634). Since decoding the symbol stream yields a bit-exact result that is independent of the decoder's location (local or remote), the contents in the reference picture memory (634) are also bit-exact between the local encoder and the remote encoder. In other words, the predictive part of the encoder "sees" the exact same sample values ​​as the reference picture samples that the decoder will "see" when using the predictions during decoding. This fundamental principle of reference picture synchronicity (and the resulting drift when synchronicity cannot be maintained, for example, due to channel errors) is also used in several related technologies.

[0055] The operation of the “local” decoder (633) can be the same as that of the “remote” decoder, such as the video decoder (510), which has already been described in detail earlier with reference to Figure 5. Referring temporarily to Figure 5, however, the entropy-decoding portion of the video decoder (510), including the buffer memory (515) and parser (520), does not have to be fully implemented in the local decoder (633), provided that symbols are available and the encoding / decoding of symbols to the coded video sequence by the entropycoder (645) and parser (520) can be reversible.

[0056] An observation that can be made at this point is that any decoder technology present in the decoder, other than parsing / entropy decoding, does not necessarily need to be present in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter focuses on the operation of the decoder. Descriptions of encoder technology can be omitted, as they are the inverse of the decoder technology described comprehensively. More detailed explanations are required only in certain areas and are given below.

[0057] During operation, the source coder (630) may perform motion-compensated predictive coding. This predictively codes the input picture by referencing one or more previously coded pictures from a video sequence designated as “reference pictures”. In this way, the coding engine (632) codes the difference between the pixel blocks of the reference picture that may be selected as predictive references for the input picture and the pixel blocks of the input picture.

[0058] A local video decoder (633) may decode the coded video data of a picture that may be designated as a reference picture, based on symbols generated by the source coder (630). The operation of the coding engine (632) may, advantageously, be an irreversible process. When the coded video data can be decoded by a video decoder (not shown in Figure 6), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder (633) may recreate the decoding process that may be performed by the video decoder on the reference picture, so that the reconstructed reference picture is stored in the reference picture cache (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference picture that has the same content as the reconstructed reference picture that will be obtained by the far-end video decoder (without transmission errors).

[0059] The predictor (635) may perform a predictive search for the coding engine (632). That is, in order for a new picture to be coded, the predictor (635) may search the reference picture memory (634) for specific metadata or sample data (as candidate reference pixel blocks), such as reference picture motion vectors, block shapes, etc., which may serve as appropriate predictive criteria for the new picture. The predictor (635) may operate on a sample block-by-pixel block basis to find appropriate predictive criteria. In some cases, the input picture may have predictive criteria drawn from multiple reference pictures stored in the reference picture memory (634), as determined by the search results obtained by the predictor (635).

[0060] The controller (650) may manage the coding operation of the source coder (630), including, for example, setting parameters and subgroup parameters used to encode video data.

[0061] The outputs of all the above functional units may undergo entropic coding in the entropicorder (645). The entropicorder (645) converts the symbols generated by the various functional units into coded video sequences by lossless compression of the symbols according to techniques such as Huffman coding, variable-length coding, and arithmetic coding.

[0062] The transmitter (640) may buffer the coded video sequence generated by the entropicorder (645) to prepare it for transmission over the communication channel (660). The communication channel (660) may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (640) may merge the coded video data from the videocoder (603) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0063] The controller (650) may manage the operation of the video encoder (603). During coding, the controller (650) may assign each coded picture to a specific coded picture type that may affect the coding techniques that may be applied to each picture. For example, a picture may often be assigned to one of the following picture types:

[0064] An intra-picture (I-picture) may be a picture that can be coded and decoded without using any other pictures in the sequence as a source of prediction. Some video codecs allow various types of intra-pictures, including, for example, independent decoder refresh (IDR) pictures. Those skilled in the art will be aware of such variations of I-pictures and their respective applications and characteristics.

[0065] A predictive picture (P-picture) may be a picture that can be coded and decoded by intra-prediction or inter-prediction using at most one motion vector and reference index to predict the sample values ​​of each block.

[0066] A bidirectionally predictive picture (B-picture) may be a picture that can be coded and decoded by intra-prediction or inter-prediction using at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple-predictive picture(s) may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0067] A source picture may generally be spatially subdivided into multiple sample blocks (e.g., 4x4, 8x8, 4x8, or 16x16 sample blocks, respectively), and each block may be coded. Blocks may be coded predictively by referring to other (already coded) blocks determined by the coding assignment applied to each picture in the block. For example, blocks of picture I may be coded non-predictively, or they may be coded predictively by referring to already coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of picture P may be coded predictively by spatial prediction or temporal prediction by referring to one previously coded reference picture. Blocks of picture B may be coded predictively by spatial prediction or temporal prediction by referring to one or two previously coded reference pictures.

[0068] The video encoder (603) may perform coding operations in accordance with a specified video coding technique or standard, such as ITU-T Recommended H.265. During these operations, the video encoder (603) may perform various compression operations, including predictive coding operations that utilize temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to the syntax defined by the video coding technique or standard being used.

[0069] In an embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source coder (630) may include such data as part of the coded video sequence. The additional data may include time / space / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages or VUI parameter set fragments, and the like.

[0070] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) utilizes spatial correlations within a given picture, while inter-picture prediction utilizes (temporal or other) correlations between pictures. In the example, a particular picture being encoded / decoded, called the current picture, is divided into blocks. If a block within the current picture is similar to a reference block in a previously coded, still-buffering reference picture within the video, that block in the current picture may be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and may have a third dimension to identify the reference picture if multiple reference pictures are used.

[0071] In some embodiments, a dual prediction technique may be used in interpicture prediction. According to the dual prediction technique, two reference pictures are used, for example, a first reference picture and a second reference picture, both of which precede the current picture in the video in the decoding order (but may be past and future, respectively, in the display order). A block in the current picture may be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. That block is predictable by a combination of the first and second reference blocks.

[0072] Furthermore, merge mode techniques can be used in interpicture prediction to improve coding efficiency.

[0073] According to some embodiments of this disclosure, predictions such as interpicture prediction and intrapicture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU contains three coding tree blocks (CTBs), which are one lumen CTB and two chroma CTBs. Each CTU can be recursively quadtree-divided into one or more coding units (CUs). For example, a 64x64 pixel CTU can be divided into one 64x64 pixel CU, or four 32x32 pixel CUs, or sixteen 16x16 pixel CUs. In the example, each CU is parsed to determine a prediction type for the CU, such as an inter-prediction type or an intra-prediction type. The CU is divided into one or more prediction units (PUs) depending on its temporal and / or spatial predictability. Generally, each PU includes one lumar prediction block (PB) and two chroma PBs. In embodiments, the prediction operation in coding (encoding / decoding) is performed in units of prediction blocks. Using a lumar prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., lumar values), such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0074] Figure 7 shows a diagram of a video encoder (703) according to another embodiment of the present disclosure. The video encoder (703) may be configured to receive a processing block (e.g., a prediction block) of sample values ​​in the current video picture that are included in a sequence of video pictures, and to encode the processing block into a coded picture which is a part of a coded video sequence. In the example, the video encoder (703) is used instead of the video encoder (403) in the example of Figure 4.

[0075] In the HEVC example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as an 8x8 sample prediction block. The video encoder (703) determines, for example, using rate-distortion optimization, whether the processing block is best coded in intra-mode, inter-mode, or bi-prediction mode. If the processing block should be coded in intra-mode, the video encoder (703) may use the intra-prediction technique to encode the processing block into a coded picture; if the processing block should be coded in inter-mode or bi-prediction mode, the video encoder (703) may use the inter-prediction or bi-prediction technique, respectively, to encode the processing block into a coded picture. In certain video coding techniques, the merge mode may be an inter-picture prediction submode in which motion vectors are derived from one or more motion vector predictors without benefiting from coded motion vector components outside the predictors. In certain other video coding techniques, there may be motion vector components applicable to the target block. In the example, the video encoder (703) includes other components such as a mode determination module (not shown) that determines the mode of the processing block.

[0076] In the example shown in Figure 7, the video encoder (703) includes an interencoder (730), an intraencoder (722), a residual calculation unit (723), a switch (726), a residual encoder (724), a general-purpose controller (721), and an entropy encoder (725), all coupled together as shown in Figure 7.

[0077] The interencoder (730) is configured to receive a sample of the current block (e.g., a processing block), compare that block with one or more reference blocks in the reference picture (e.g., blocks in the previous and subsequent pictures), generate interprediction information (e.g., a description of residual information, motion vectors, and merge mode information according to the interencoding technique), and compute an interprediction result (e.g., a predicted block) based on the interprediction information using some appropriate technique. In some examples, the reference picture is a decoded reference picture that is decoded based on encoded video information.

[0078] The intra encoder (722) is configured to receive a sample of the current block (e.g., a processing block), and in some cases compare it with a block already coded in the same picture to generate the transformed quantized coefficients, and in some cases also to generate intra prediction information (e.g., intra prediction direction information following one or more intra encoding techniques). In the example, the intra encoder (722) also calculates an intra prediction result (e.g., a prediction block) based on the intra prediction information and a reference block in the same picture.

[0079] The general-purpose controller (721) is configured to determine general-purpose control data and control other components of the video encoder (703) based on the general-purpose control data. In the example, the general-purpose controller (721) determines the mode of the block and supplies control signals to the switch (726) based on the mode. For example, if the mode is intra-mode, the general-purpose controller (721) controls the switch (726) to select the intra-mode result for use by the residual calculation unit (723), and then controls the entropy encoder (725) to select the intra-prediction information and include the intra-prediction information in the bitstream. If the mode is inter-mode, the general-purpose controller (721) controls the switch (726) to select the inter-prediction result for use by the residual calculation unit (723), and then controls the entropy encoder (725) to select the inter-prediction information and include the inter-prediction information in the bitstream.

[0080] The residual calculation unit (723) is configured to calculate the difference (residual data) between the received block and the prediction result selected from the intra-encoder (722) or inter-encoder (730). The residual encoder (724) is configured to operate on the residual data to encode the residual data to generate conversion coefficients. In an example, the residual encoder (724) is configured to convert the residual data from the spatial domain to the frequency domain and generate conversion coefficients. The conversion coefficients are then quantized to obtain quantized conversion coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be appropriately used by the intra-encoder (722) and inter-encoder (730). For example, an interencoder (730) can generate a decoded block based on decoded residual data and interprediction information, and an intraencoder (722) can generate a decoded block based on decoded residual data and intraprediction information. The decoded block is appropriately processed to generate a decoded picture, which is buffered in a memory circuit (not shown) and can be used as a reference picture in some examples.

[0081] The entropy encoder (725) is configured to format the bitstream to include the encoded blocks. The entropy encoder (725) is configured to include various information according to an appropriate standard such as the HEVC standard. In the example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information in the bitstream. Note that residual information is not present when coding blocks in either inter-mode or bi-prediction mode merge submode, according to the disclosed subject matter.

[0082] Figure 8 shows a diagram of a video decoder (810) according to another embodiment of the present disclosure. The video decoder (810) is configured to receive a coded picture, which is a portion of a coded video sequence, and to decode the coded picture to produce a reconstructed picture. In the example, the video decoder (810) is used instead of the video decoder (410) in the example of Figure 4.

[0083] In the example shown in Figure 8, the video decoder (810) includes an entropy decoder (871), an interdecoder (880), a residual decoder (873), a reconfiguration module (874), and an intradecoder (872), which are coupled together as shown in Figure 8.

[0084] The entropy decoder (871) may be configured to reconstruct specific symbols representing syntax elements from the coded picture, from which the coded picture is composed. Such symbols may include, for example, the mode in which the block is coded (e.g., intra-mode, or inter-mode or bi-prediction mode in a merge sub-mode or other sub-mode), prediction information (e.g., intra-prediction information or inter-prediction information) that can identify specific samples or metadata used for prediction by the intra-decoder (872) or inter-decoder (880), respectively, and residual information, for example, in the form of quantized transformation coefficients. In the example, if the prediction mode is inter- or bi-prediction mode, inter-prediction information is supplied to the inter-decoder (880), and if the prediction type is intra-prediction type, intra-prediction information is supplied to the intra-decoder (872). The residual information may undergo inverse quantization and be supplied to the residual decoder (873).

[0085] The interdecoder (880) is configured to receive interprediction information and generate interprediction results based on the interprediction information.

[0086] The intra decoder (872) is configured to receive intra prediction information and generate prediction results based on the intra prediction information.

[0087] The residual decoder (873) is configured to perform inverse quantization to extract inversely quantized conversion coefficients, process the inversely quantized conversion coefficients, and convert the residual from the frequency domain to the spatial domain. The residual decoder (873) may also request certain control information (to include quantization parameters (QP)), which may be supplied by the entropy decoder (871) (this is only low-capacity control information, and the data path is not shown).

[0088] The reconstruction module (874) is configured to combine the residuals output by the residual decoder (873) and the prediction results (which may be output by the inter or intra prediction module) in the spatial domain to form a reconstructed block. The reconstructed block may be a portion of a reconstructed picture, and the reconstructed picture may be a portion of a reconstructed video. It is known that other appropriate operations, such as deblocking operations, may be performed to improve visual quality.

[0089] It is known that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) can be implemented by any suitable technique. In one embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) can be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) can be implemented using one or more processors that execute software instructions.

[0090] [II. Conversion Processing Technology] [1. Block partitioning structures including quadtree partitioning] A block partitioning structure can be called a coding tree. In some embodiments, by using a quadtree structure, coding tree units (CTUs) are divided into coding units (CUs) to adapt to various local features. Discussions about whether to code a picture area using interpicture (temporal) or intrapicture (spatial) prediction take place at the CU level. Each CU can be further divided into one, two, or four prediction units (PUs) according to the PU partitioning type. Within a single PU, the same prediction process is applied, and relevant information is sent to the decoder PU-based.

[0091] After obtaining residual blocks by applying a prediction process based on the PU partition type, the CU can be partitioned into transformation units (TUs) according to other quadtree structures. As is evident, there are numerous partition concepts, including CU, PU, ​​and TU. In some embodiments, CU or TU can only be square in shape, while PU can be square or rectangular. In some embodiments, one coding block may be further partitioned into four square subblocks, and the transformation is performed on each block, i.e., the TU. Each TU may be recursively further partitioned into smaller Tus using a quadtree structure called a Residual Quad-Tree (RQT).

[0092] At picture boundaries, in some embodiments, implicit quadtree partitioning may be used, so that the block continues to quadtree partition until its size fits the picture boundary.

[0093] [2. Quadrarrow-plus-binary tree (QTBT) block partitioning structure] In some embodiments, a Quad-Tree plus Binary Tree (QTBT) structure is used. The QTBT structure eliminates the concept of multiple partition types (CT, PU, ​​and TU concepts) and supports greater flexibility for CU partition shapes. In a QTBT block structure, CU can have either a square or rectangular shape.

[0094] Figure 9A shows CTU(910) partitioned using the QTBT structure(920) shown in Figure 9B. CTU(910) is first partitioned by a quadtree structure. The quadtree leaf nodes are further partitioned by binary or quadtree structures. There are two types of binary tree partitions: symmetric horizontal partitioning and symmetric vertical partitioning. The binary tree leaf nodes are called CUs, which can be used for prediction and transformation processing without further partitioning. Thus, CUs, PUs, and TUs have the same block size in the QTBT coding block structure.

[0095] In some embodiments, a CU may include coding blocks (CBs) for different color components. For example, one CU may include one lumen CB and two chromen CBs in the case of P and B slices in a 4:2:0 chroma format. A CU may also include a CB for a single color component. For example, one CU may include just one lumen CB or just two chromen CBs in the case of an I slice.

[0096] The following parameters are defined for the QTBT partitioning scheme in some embodiments: • CTUsize: The size of the root node in a quadtree, the same concept as seen in HEVC, for example. • MinQTSize: Minimum allowable quadtree leaf node size. • MaxBTSize: Maximum allowable bifurcation tree root node size. • MaxBTDepth: Maximum allowable binary tree depth. • MinBTSize: Minimum allowable binary tree leaf node size.

[0097] In an example of a QTBT partitioning structure, CTUsize is set as a 128x128 chroma sample with two corresponding 64x64 blocks of chroma samples, MinQTSize is set as 16x16, MaxBTSize is set as 64x64, MinBTSize (for both width and height) is set as 4x4, and MaxBTDepth is set as 4. The quadtree partitioning is first applied to the CTU to generate quadtree leaf nodes. The quadtree leaf nodes can have sizes ranging from 16x16 (i.e., MinQTSize) to 128x128 (i.e., CTUsize). If a quadtree leaf node is 128x128, it will not be further partitioned by a binary tree because its size exceeds MaxBTSize (i.e., 64x64). Otherwise, the quadtree leaf node may be further partitioned by a binary tree. Therefore, a quadtree leaf node is also the root node for a binary tree, and it has a binary tree depth of 0.

[0098] When the binary tree depth reaches MaxBTSize (i.e., 4), no further partitioning is possible. If a binary tree node has a width equal to MinBTSize (i.e., 4), no further horizontal partitioning is possible. Similarly, if a binary tree node has a height equal to MinBTSize, no further vertical partitioning is possible. The leaf nodes of the binary tree are further processed by prediction and transformation processes without further partitioning. In this embodiment, the maximum CTU size is 256 × 256 lumens.

[0099] In Figures 9A and 9B, solid lines represent quadtree partitions, and dashed lines represent binary tree partitions. Each partition (i.e., non-leaf) node in a binary tree is flagged to indicate which partition type (i.e., horizontal or vertical) is used. For example, 0 indicates a horizontal partition, and 1 indicates a vertical partition. For quadtree partitions, there is no need to indicate the partition type because quadtree partitions always divide the block both horizontally and vertically to produce four subblocks of equal size.

[0100] In some embodiments, the QTBT scheme supports flexibility for lumens and chromians to have separate QTBT structures. For example, for P and B slices, the lumens and chromians within a single CTU share the same QTBT structure. However, for I slices, the lumens CTB is divided into CUs by the QTBT structure, and the chromian block is divided into chromian CUs by the other QTBT structure. Thus, in I slices, the CU consists of a coding block for the lumens or a coding block for two chromians, while in P or B slices, the CU consists of a coding block for all three color components.

[0101] In some embodiments, interpretation for small blocks is limited to reduce memory access for motion compensation. For example, bipretation is not supported for 4x8 and 8x4 blocks, and interpretation is not supported for 4x4 blocks.

[0102] [3. Ternary (TT) Block Partitioning Structure] In some embodiments, a Multi-Type-Tree (MTT) structure is used to partition a picture. The MTT structure is a more flexible tree structure than the QTBT structure. In addition to quadtrees and binary trees, the MTT uses horizontal center-side ternary trees and vertical center-side ternary trees, as shown in Figures 9C and 9D, respectively. Ternary partitioning can complement quadtree and binary tree partitioning. For example, ternary partitioning can capture objects located at the center of a block, while quadtrees and binary trees can partition across the center of a block. The width and height of partitions by ternary trees are powers of 2, thus eliminating the need for additional transformation partitions.

[0103] In the example, the design of a two-level tree is primarily motivated by complexity reduction. For example, the complexity of traversing the tree is T. D Here, T represents the number of partition types and D is the depth of the tree.

[0104] [4. Example of a linear transformation] In some embodiments, as seen in HEVC, for example, 4-point, 8-point, 16-point, and 32-point DCT-2 transforms are used as primary transforms. Figures 10A–10D show the transform core matrices for 4-point, 8-point, 16-point, and 32-point DCT-2, respectively. The elements of these transform core matrices can be represented using 8-bit integers, and thus these transform core matrices are called 8-bit transform cores. As shown, the transform core matrices of smaller DCT-2 are parts of those of larger DCT-2.

[0105] The DCT-2 core matrix exhibits symmetry / antisymmetry. Therefore, so-called "partial butterfly" executions can be supported to reduce the number of operation counts (multiplication, addition / subtraction, shift). The same result of matrix multiplication can be obtained using partial butterfly executions.

[0106] [5. Examples of further linear transformations] In some embodiments, in addition to the 4-point, 8-point, 16-point, and 32-point DCT-2 transforms described above, further 2-point and 64-point DCT-2 transforms are used. Figures 11A to 11E show the 64 × 64 transform core matrices for the 64-point DCT-2 transform.

[0107] In some embodiments, in addition to the DCT-2 and 4×4DST-7 transforms, an Adaptive Multiple Transform (AMT) (also known as Enhanced Multiple Transform (EMT) or Multiple Transform Selection (MTS)) is used for residual coding of both the intercoded and intercoded blocks. The AMT uses the transform core matrices of multiple selected transforms from the Discrete Cosine Transform (DCT) / Discrete Sine Transform (DST) family, such as the DS-7 or DCT-8 transform, in addition to the DCT-2 transform. Figure 12 shows the transform fundamental functions of the selected DST / DCT transforms.

[0108] In some embodiments, the DST / DCT transformation core matrix used in AMT is represented in an 8-bit format. In some embodiments, AMT is applied to CUs where both width and height are 32 or less. Whether or not AMT should be applied can be controlled by a flag (e.g., mts_flag). For example, when mts_flag is equal to 0, only DCT-2 is applied to code the residual block. When mts_flag is equal to 1, an index (e.g., mts_idx) may be further indicated with two bins to specify the horizontal and vertical transformations to be used.

[0109] Figure 13 shows Table (1300), which represents the mapping relationship between the index (e.g., mts_idx) value and each horizontal or vertical transformation. Row (1301) where the value of mts_idx is -1 corresponds to a scenario where the flag (e.g., mts_flag) is equal to 0, and the DCT-2 transformation is used. Rows (1302) to (1305) where the value of mts_idx is 0, 1, 2, or 3 corresponds to a scenario where mts_flag is equal to 1. In the two rightmost columns of Table (1300), 0 represents the DCT-2 transformation type, 1 represents the DST-7 transformation type, and 2 represents the DCT-8 transformation type.

[0110] Figures 14A to 14D show the transformation core matrices for DST-7 transformation. Figures 15A to 15D show the transformation core matrices for DCT-8 transformation.

[0111] [6. Intra-subpartition (ISP) coding mode] In some embodiments, an intra-sub-partition (ISP) coding mode is used. In ISP coding mode, a rumanetra prediction block can be divided vertically or horizontally into two or four subpartitions. The number of subpartitions may depend on the size of the block. Figure 16 shows the number of subpartitions depending on the block size. Figure 17 shows an example where a 4x8 or 8x4 block is divided into two subpartitions. Figure 18 shows an example where a block larger than 4x8 or 8x4 is divided into four subpartitions. In the examples, all subpartitions satisfy the condition that they have at least 16 samples. In the examples, ISP is not applied to the chroma component.

[0112] In the example, for each subpartition separated from the coding block, a residual signal is generated by entropi-decoding each coefficient sent from the encoder, and then inverse quantization and inverse transformation of these coefficients. Next, the first subpartition within the subpartition is intra-predicted to generate a prediction signal. The prediction signal is added to the residual signal of each of the first subpartitions to obtain the corresponding reconstructed sample. Subsequently, the reconstructed sample values ​​of the first subpartition are available to generate predictions for the second subpartition within the subpartition. This process can be repeated on a subpartition-by-subpartition basis until all subpartitions from the coding block have been reconstructed. In the example, all subpartitions share the same intra-mode.

[0113] In this embodiment, the ISP coding mode is tested only by the intra-mode, which is part of the Most Probable Mode (MPM) list. Therefore, it can be inferred that the MPM flag is 1 if a block uses an ISP. Furthermore, when an ISP is used for a particular block, the respective MPM list is then modified to prioritize the horizontal intra-mode for ISP horizontal splitting and the vertical intra-mode for ISP vertical splitting, with the exception of the DC mode.

[0114] In ISP coding mode, conversion and reconfiguration are performed individually for each subpartition, so each subpartition can be considered a TU (Unit Partition).

[0115] Figures 19A-19B show examples of syntax elements (1900) that are indicated for ISP coding mode. As shown within the box (1910), the syntax element, e.g., intra_subpartitions_mode_flag, indicates whether ISP is used or not. The syntax element, e.g., intra_subpartitions_split_flag, indicates the partition direction (vertical or horizontal).

[0116] [7. Subblock Conversion (SBT)] In some embodiments, a Sub-Block Transform (SBT), also called a Spatially Varying Transform (SVT), is used. SBTs can be applied to interpredictive residuals. In some examples, the residual block is contained within and smaller than the coding block. Therefore, the transformation size in SBT is smaller than the coding block size. For regions not covered by the residual block, zero residuals may be assumed, and therefore no transformation is performed.

[0117] Figures 20A–20D show the subblock types (SVT-H, SVT-V) (e.g., horizontal or vertical division), size, and partitions (e.g., left half, left quarter, right half, right quarter, upper half, upper quarter, lower half, lower quarter). The shaded areas indicated by the letter "A" are assumed to be transformed residual blocks, while the remaining areas are assumed to be untransformed zero residuals.

[0118] As an example, Figures 21A–21I show changes to the specification text of a video coding standard (e.g., VVC) when SBT is used. The added text is shown in the boxes (2101) through (2113). As shown, additional syntax elements, such as the additional overhead bits cu_sbt_flag, cu_sbt_quad_flag, cu_sbt_horizontal_flag, and cu_sbt_pos_flag, may be notified to indicate the subblock type (horizontal or vertical), size (half or quarter), and position (left, right, top or bottom), respectively.

[0119] [8. YUV format] Figure 22 shows various YUV formats used in several embodiments (e.g., 4:4:4, 4:2:2, 4:1:1, and 4:2:0). In the example, the cross-component linear model intra-prediction is used for the 4:2:0 format. A 6-tap interpolation filter can be applied to the downsampled lumar sample corresponding to the chroma sample, as shown in Figure 22. In the example, the downsampled lumar sample Rec'L[x,y] is reconstructed from the nearest nearby lumar sample (Rec L It can be calculated from [x,y], which is represented by:

number

[0120] [9. Virtual Pipeline Data Unit (VPDU)] Virtual Pipeline Data Units (VPDUs) can be defined as non-overlapping M×M-lumen (L) / N×N-chromen (C) units within a picture. In some hardware decoder implementations, consecutive VPDUs are processed simultaneously by multiple pipeline stages. Different stages process different VPDUs simultaneously. The VPDU size can be roughly proportional to the buffer size in the pipeline stage, thereby making it desirable to keep the VPDUs at a specific size (e.g., 64×64 or less). In certain decoders, the VPDU size is set to the maximum translation unit (TU) size. Increasing the maximum TU size from 32×32-L / 16×16-C in HEVC to 64×64-L / 32×32-C in current VVC can result in coding gains, which are expected to quadruple the VPDU size compared to HEVC. However, the BT and TT structures employed in VVC to achieve further coding gains can be recursively applied to 128×128-L / 64×64-C coding tree blocks, resulting in a VPDU size 16 times larger (128×128-L / 64×64-C) compared to HEVC.

[0121] Figure 23 shows specific TT and BT partitioning that are not permitted.

[0122] To keep the VPDU size at 64x64 lumens, certain partition limitations (with syntax signaling changes) are applied in some embodiments: TT splitting is not permitted for CUs where either the width or height, or both the width and height, are equal to 128. For a CU of size 128 × N (width equal to 128 and height less than 128), where N ≤ 64, horizontal BT is not permitted. For N ≤ 64, vertical BT is not allowed for N × 128 CUs (where the height is equal to 128 and the width is less than 128).

[0123] [III. Conversion Block Partitioning and Processing Techniques] In some embodiments, a fixed maximum allowable conversion unit (TU) size or maximum TU size (e.g., 64 × 64 pixels or samples) is used. In some embodiments, a controllable or configurable maximum TU size is used because the maximum TU size may affect hardware complexity (e.g., pipeline intermediate buffer size, number of multipliers, etc.) for the encoder implementation. For example, in addition to the 64 × 64 sample size, the maximum TU size can be other sizes such as 32 × 32 samples, 16 × 16 samples, etc.

[0124] Certain video standards may use SBT and ISP. For example, in SBT, an SPS flag, e.g., sps_sbt_max_size_64_flag, is communicated to indicate whether the maximum SBT size is 32 or 64 lengths. If sps_sbt_max_size_64_flag is true (i.e., the maximum SBT size is 64 lengths) and the maximum TU size is 32 points, an encoder crash may be triggered. Generally, L-length or L-point refers to the maximum dimensions of CU, TU, CB, TB, VPDU, etc. For example, when the maximum TU size is 32 points or 32 lengths, the width and height of the TU are 32 or less.

[0125] In some embodiments, the ISP mode is permitted for various CU sizes, but a conflict may arise when the maximum TU size is set to less than 64: whether implicit conversion partitioning is performed or explicit conversion partitioning using ISP with signaling is performed. For example, when the maximum TU size is 16, for example, for a 64×16CU, without ISP, the CU may be implicitly partitioned into four 16×16TUs. With ISP, the 64×16CU may be partitioned by vertical ISP and thus partitioned into four 16×16TUs, but using signaling.

[0126] When the maximum TU is less than 64x64, the TU processing order is required to be aligned with the execution of the VPDU.

[0127] The embodiments described herein may be used separately or in any order. Furthermore, the embodiments may be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits) such as encoders and decoders. In one example, one or more processors execute a program stored in a non-temporary computer-readable medium.

[0128] In this disclosure, High-Level Syntax (HLS) elements may refer to video parameter sets (VPS), sequence parameter sets (SPS), picture parameter sets (PPS), slice headers, tile headers, tile group headers, etc. A CTU header may refer to a syntax element notified about a CUT, for example, as header information. In the example, the CTU size is the maximum CU size.

[0129] Generally, when the lumar size (represented by lumar samples) of a particular unit (e.g., TU, CU) is known, a corresponding chroma size can be obtained, specified by the number of chroma samples. In the example, if the YUV format is 4:2:0, CU has a CU size of 64×64 lumar samples (i.e., 64×64-L). Therefore, CU has a CU size of 32×32 chroma samples (i.e., 32×32-C). The CU size can be called 64×64-L, 32×32-C, or 64×64-L / 32×32-C. Similarly, TU has a TU size of 64×64 lumar samples (i.e., 64×64-L). Therefore, TU has a TU size of 32×32 chroma samples (i.e., 32×32-C). TU sizes may be referred to as 64×64-L, 32×32-C, or 64×64-L / 32×32-C. For example, a TU includes one lumar conversion block (TB) and two chroma TBs. The lumar TB has a size of 64×64-L, and each of the chroma TBs has a size of 32×32-C. In general, embodiments and methods described for CU or TU can be appropriately adapted for CB and TB, respectively.

[0130] A CU can include a 64×64-L lumen block and two 32×32-C chroma blocks. In the following description, the TU size is expressed using lumen samples within the TU. For example, the maximum TU size for M samples refers to the maximum TU size for M lumen samples. Similarly, other sizes such as VPDU size and CU size are expressed using the respective lumen samples within their corresponding units, such as VPDU and CU. Naturally, TU size, VPDU size, CU size, etc., can be expressed using chroma samples or combinations of lumen and chroma samples.

[0131] The unit size may refer to the width, height, or area of ​​the unit. For example, the maximum TU size may refer to the width, height, or area of ​​the largest TU. In general, TUs, CUs, VPDUs, etc., can have any suitable shape, including rectangular, square, L-shaped, or any other suitable shape. When the shape of the unit is irregular, such as L-shaped, the unit size may specify the area of ​​the unit.

[0132] In some embodiments, the VPDU size and / or maximum TU size may be communicated in the coded video bitstream, for example, in the SPS and PPS. As described above, the VPDU size and / or maximum TU size may be communicated with respect to the lumens sample. Alternatively, the VPDU size and / or maximum TU size may be communicated with respect to the chromens sample.

[0133] In some embodiments, the VPDU size and / or maximum TU size may be stored in the encoder and / or decoder, so the VPDU size and / or maximum TU size are not communicated. In one example, the VPDU size and / or maximum TU size may be stored in the profile and / or level definition. The VPDU size and / or maximum TU size may be stored in relation to the lumens or chroma samples.

[0134] In some embodiments, the VPDU may have different shapes while sharing the same size. For example, when the VPDU size is 4096 with respect to the lumens sample, the VPDU may have a 64 × 64 square shape or a 32 × 128 rectangular shape. The VPDU may also have other shapes, such as an L-shape, as long as the VPDU size is 4096 with respect to the lumens sample. The above description is also applicable to specific TUs.

[0135] [1. Example A] Depending on the aspect of the disclosure, the maximum allowable TU size (also called the maximum TU size) is M samples (e.g., the size of M × M samples). In the example, the maximum width and height of the TU are M. In the example, the maximum area of ​​the TU is M × M. The processing data unit size (e.g., the VPDU size) is K samples (e.g., the size of K × K samples). In the example, the maximum width and height of the processing data unit size are K. In the example, the maximum area of ​​the processing data unit size is K × K. A W × H CU has a width of W samples and a height of H samples. A CU can be divided into a number of subunits called sub-processing units (SPUs) based on the CU size and the processing data unit size K. A CU can be divided into SPUs using any suitable partitioning structure or any combination of suitable partitioning structures, such as QTBT, QT, BT, TT, or a combination thereof. SPUs may have the same size or different sizes.

[0136] In some embodiments, a CU is divided into SPUs if its width W or height H is greater than K. In the example, the SPUs have the same size (i.e., SPU size), and each SPU has a sample size of Min(W,K) × Min(H,K). Thus, the width of each SPU is the smallest of W and K, and the height of its SPU is the smallest of H and K. In some examples, whether or not to divide a CU can be determined based on the size of the CU and the processing data unit size K before dividing the CU.

[0137] A sample size (SPU) within a CU can be further divided into TUs, for example, having a size of M × M samples. In some examples, an SPU can be divided into TUs having a size of Min(W,K,M) × Min(H,K,M). In some examples, whether an SPU should be divided can be determined based on the size of the SPU and the maximum TU size M before the SPU is divided. An SPU can be divided using any suitable partitioning structure or any combination of suitable partitioning structures, such as QTBT, QT, BT, TT, or a combination thereof. Depending on the aspect of the disclosure, one or more partitioning structures for dividing an SPU can be determined based on the size of the SPU and the maximum TU size M. In the example, an SPU can be recursively divided into TUs using the determined one or more partitioning structures.

[0138] In the example, when the width and height of the SPU are greater than the maximum TU size M, the SPU is partitioned into M×M TUs using a quadtree partitioning structure. The SPU can be recursively partitioned into TUs using a quadtree partitioning structure.

[0139] In the example, when the width of the SPU is greater than M and the height of the SPU is equal to M, the SPU is partitioned into M × M TUs using a vertical binary tree partitioning structure. For example, if M is 32 and the SPU has a size of 64 × 32, then the width of the SPU is 64 and the height of the SPU is 32. Thus, a vertical binary tree partitioning structure can be used to partition the SPU into two 32 × 32 TUs. The SPU can be recursively partitioned into TUs using a vertical binary tree partitioning structure.

[0140] In the example, when the width of the SPU is greater than M and the height of the SPU is less than M, the SPU can be partitioned into TUs using a vertical binary tree partitioning structure, in which case the width of the TU is M and the height of the TU is equal to the height of the SPU.

[0141] In the example, when the height of the SPU is greater than M and the width of the SPU is equal to M, the SPU is partitioned into M × M TUs using a horizontal binary tree partitioning structure. The SPU can be recursively partitioned into TUs using a horizontal binary tree partitioning structure. For example, if M is 32 and the SPU has a size of 32 × 64, then the width of the SPU is 32 and the height of the SPU is 64. Thus, a horizontal binary tree partitioning structure can be used to partition the SPU into two 32 × 32 TUs.

[0142] In the example, when the height of the SPU is greater than M and the width of the SPU is less than M, the SPU can be partitioned into TUs using a horizontal binary tree partitioning structure, in which case the height of the TU is M and the width of the TU is equal to the width of the SPU.

[0143] The transformation tree syntax in Figure 24 shows an example of the processing sequence used to handle SPU partitioning and TU processing.

[0144] In the example, a CU of size W×H can be divided in two steps. In the first step, the CU is divided into SPUs, each SPU having a size of Min(W,K)×Min(H,K). Then, in the second step, each SPU is further divided into TUs, each TU having a size of M×M.

[0145] When processing TUs within a CU, SPUs within the CU may be scanned and processed in a first scan order (also called the first order). Furthermore, within each SPU, TUs may be scanned and processed in a second scan order (also called the second order).

[0146] In various embodiments, the first order for processing SPUs can be a raster scan order, a vertical scan order (e.g., scanning SPUs column by column from left to right or vice versa), a zigzag order, a diagonal scan order, and the like.

[0147] In various embodiments, the second order for processing TUs within each SPU can be a raster scan order, a vertical scan order (e.g., scanning TUs column by column from left to right or vice versa), a zigzag order, a diagonal scan order, and so on.

[0148] The first and second sequences can be the same or different in different embodiments. For example, the first sequence for processing SPUs and the second sequence for processing TUs within each SPU are both raster scan sequences in one embodiment.

[0149] [2.Example B] Figure 25 shows a CU(2510) with a W×H sample size, where W=128 and H=64. The maximum TU size M is 32 samples. The processing data unit size K, such as the VPDU size, is 64 samples. The CU(2510) is first divided into a first 64×64 SPU(2520) and a second 64×64 SPU(2530). The first SPU(2520) and the second SPU(2530) can then be further divided into TU0-7, each having an M×M sample size. TU0-3 are included in the first SPU(2520), and TU4×7 are included in the second SPU(2530).

[0150] According to the first sequence, the first SPU(2520) may be processed first, followed by the second SPU(2530). Within the first SPU(2520) or the second SPU(2530), the second sequence used to process TU0-3 or 4-7 is the raster scan sequence. Thus, TU0-7 are processed in the order indicated by the arrow (2551). The first and / or second sequence may be determined explicitly (e.g., via signaling from encoder to decoder) or implicitly.

[0151] In some examples, as described above, splitting the CU into SPUs, with each SPU further containing a TU, improves coding efficiency. Referring to Figure 25, in the example, the first SPU(2520) is the first VPDU, and the second SPU(2530) is the second VPDU. The first VPDU (or first SPU(2520)) and the second VPDU (or second SPU(2530)) can sequentially pass through a multi-stage pipeline that includes a first stage (e.g., entropy decoding), a second stage (e.g., inverse quantization), a third stage (inverse transform), and / or similar. According to the first sequence shown in Figure 25, the first SPU(2520) should be processed before the second SPU(2530), so the first SPU(2520) is processed by the first stage and then proceeds to the second stage. In the example, when the first SPU(2520) is processed by the second stage, the second SPU(2530) is processed by the first stage to improve coding efficiency. Subsequently, the first SPU(2520) can proceed to the third stage, and the second SPU(2530) can move to the second stage. When the first SPU(2520) is processed by the third stage, the second SPU(2530) may be processed by the second stage. The above explanation is given using a VPDU and a multi-stage pipeline as examples and can be appropriately adapted to other architectures or video coding methods. The above explanation can be applied when the first SPU(2520) is included in the first VPDU and the second SPU(2530) is included in the second VPDU. At least some of the SPU processing at different stages is performed simultaneously.

[0152] As described above, when the SPU size is larger than the TU size, multiple TUs within a CU may be grouped into processing data units such as SPUs (or VPDUs), and the SPUs may be processed in a multi-stage pipeline that enables parallel (or simultaneous) processing of consecutive SPUs. In some examples, the explanation may be modified as follows: The CU is divided into a first unit. Furthermore, each of the first units may be divided into a second unit. Each of the second units may be divided into a third unit. In the example, the size of the first unit is larger than the size of the second unit, and the size of the second unit is larger than the size of the third unit. Such partitioning may be advantageous when the first multi-stage pipeline is nested within the second multi-stage pipeline.

[0153] [3.Example C] Figure 26A shows a CU(2610A) with a size of W × H samples, where W = 128 and H = 32. The maximum TU size M is 16 samples. The processing data unit size K, such as the VPDU size, is 64 samples. The smallest of W and K is 64, while the smallest of H and K is 32. Therefore, the size of the SPU can be determined to be 64 × 32 samples, for example, to align the transformation block with the VPDU. The CU(2610A) can be divided into a left SPU(2620A) and a right SPU(2630A), each having a size of 64 × 32 samples. The two SPUs (2620A) and (2630A) can be scanned and processed in a left-to-right order.

[0154] Each of the two SPUs (2620A) and (2630A) can be further divided into TUs, each having a maximum TU size of 16 × 16 samples. As shown, the left SPU (2620A) is divided into TUs 0-7, while the right SPU (2630A) is divided into TUs 8-15. In SPU (2620A), TUs 0-7 can be processed in raster scan order. In SPU (2630A), TUs 8-15 can be processed in raster scan order. Thus, TUs 0-15 can be scanned and processed in the order indicated by the arrow (2651A), in which case TU0 is processed first and TU15 is processed after TUs 0-15 have been processed.

[0155] [4.Example D] Figure 26B shows a CU(2610B) with a sample size of W×H, where W=128 and H=32. The maximum TU size M is 16 samples. The processing data unit size K, such as the VPDU size, is 64 samples. In a similar manner to the example in Figure 25, CU(2610B) can be divided into two SPUs (2620B) and (2630B), each SPU can be further divided into TUs. SPUs (2620B) and (2630B) can be processed from left to right in the same order as in Figure 25. However, unlike the example in Figure 25, TUs 0-7 in SPU(2620B) are processed in a zigzag order, and TUs 8-15 in SPU(2630B) are processed in a zigzag order.

[0156] [5.Example E] Figure 27 shows a flowchart illustrating a conversion block partitioning and processing process (2700) according to an embodiment of the disclosure. Process (2700) may be used in the reconfiguration of blocks coded in intra-mode or inter-mode. In various embodiments, process (2700) is performed by processing circuits such as processing circuits in terminal devices (210), (220), (230) and (240), processing circuits performing the functions of a video encoder (403), processing circuits performing the functions of a video decoder (510), processing circuits performing the functions of a video decoder (410), processing circuits performing the functions of a video encoder (603), and so on. In some embodiments, process (2700) is carried out by software instructions, so when a processing circuit executes a software instruction, the processing circuit executes process (2700). The process starts at (S2701) and proceeds to (S2710).

[0157] In (S2710), coded information of the CU in the picture can be decoded from the coded video bitstream. The coded information can indicate the width of W samples and the height of H samples of the CU.

[0158] In (S2720), for example, if at least one of the width W and height H of the CU is greater than the processing data unit size K, as explained with reference to Figures 24-26, the CU can be divided into SPUs. The size of the SPU can be determined based on the processing data unit size K and the size of the CU. The width of the SPU can be the smallest of W and K, and the height of the SPU can be the smallest of H and K. Thus, the CU can be divided into SPUs, each having the determined width and height. For example, if W is 128, H is 64, and K is 64, the CU can be divided into a 64x64 first SPU and a 64x64 second SPU. For example, if the processing data unit is a VPDU, then K can be the VPDU size.

[0159] In (S2730), one or more partitioning structures for dividing each SPU may be determined, for example, based on one or a combination of the width and height of the SPU and the maximum TU size of the M samples. In the example, at least one of the width and height of the SPU is greater than M.

[0160] As described above, any appropriate partitioning structure can be used to partition or divide each of the SPUs. In the example, when the width and height of the SPU are greater than M, one or more partitioning structures are determined to be quadtree partitioning structures. In the example, when the width of the SPU is greater than M and the height of the SPU is not greater than M, one or more partitioning structures are determined to be vertical binary tree partitioning structures. In the example, when the height of the SPU is greater than M and the width of the SPU is not greater than M, one or more partitioning structures are determined to be horizontal binary tree partitioning structures.

[0161] In (S2740), each SPU may be divided into TUs based on one or more determined partitioning structures. In the example, each SPU may be recursively divided into TUs using one or more determined partitioning structures.

[0162] In (S2750), the TUs of the SPU are processed according to the processing order. For example, as described above, the SPU may be processed according to the first order, and the TUs in each SPU may be processed according to the second order. The residual data of each TU may be determined by various decoding operations (e.g., entropy decoding of the transformation coefficients, inverse quantization or dequantization, inverse transformation, and / or similar). Process (2700) can proceed to (S2799) and terminate.

[0163] Process (2700) is explained using CU as an example. Process (2700) can be appropriately applied to CBs such as rumabloc and chromabloc. For brevity, a description of CBs is omitted.

[0164] Process (2700) can be appropriately adapted. For example, one or more steps can be modified, omitted, or combined. For instance, steps (S2730) and (S2740) can be combined into a single step. Additional steps can also be added. The order in which process (2700) is executed can also be changed.

[0165] [IV. Computer Systems] The above technology can be implemented as computer software that uses computer-readable instructions and is physically stored on one or more computer-readable media. For example, Figure 28 shows a computer system (2800) suitable for implementing a particular embodiment of the subject of disclosure.

[0166] Computer software can be coded in any suitable machine code or computer language that can follow mechanisms such as assembly, compilation, and linking to generate code containing instructions that can be executed directly or through interpretation, microcode execution, etc., by one or more central processing units (CPUs), graphics processing units (GPUs), etc.

[0167] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, game consoles, and devices for the Internet of Things.

[0168] The components shown in Figure 28 with respect to the computer system (2800) are illustrative in nature and are not intended to imply any limitation on the scope of use or functionality of computer software implementing embodiments of the present disclosure. The configuration of the components should not be construed as having any dependence or requirement on any one or combination of components described in the exemplary embodiments of the computer system (2800).

[0169] The computer system (2800) may include certain human interface input devices. Such human interface input devices may respond to input from one or more users, for example, through haptic input (e.g., keyboard, swipe, dataglobe motion), voice input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). The human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as sound (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images taken from a still camera), or video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0170] An input human interface device may include one or more of the following: keyboard (2801), mouse (2802), trackpad (2803), touchscreen (2810), data glove (not shown), joystick (2805), microphone (2806), scanner (2807), and camera (2808) (only one of each is shown).

[0171] The computer system (2800) may also include certain human interface output devices that can stimulate the senses of one or more users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen (2810), data glove (not shown), or joystick (2805), although haptic feedback devices that do not function as input devices may also exist), audio output devices (e.g., speakers (2809), headphones (not shown)), visual output devices (e.g., CRT screens, LCD screens, plasma screens, OLED screens, each with or without touchscreen input functionality and each with or without haptic feedback functionality, some of which are screens (2810) capable of outputting two-dimensional visual output or output in more than three dimensions by means such as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0172] The computer system (2800) may also include human-accessible storage devices and their associated media, such as CD / DVD ROM / RW (2820) with CD / DVD or similar media (2821), thumb drives (2822), removable hard disks or solid-state drives (2823), legacy magnetic media, such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD-based devices, such as security dongles (not shown), and the like.

[0173] Those skilled in the art will understand that the term “computer-readable medium” as used in relation to the subject matter currently disclosed does not include transmission media, carrier waves, or other transient signals.

[0174] The computer system (2800) may also include interfaces to one or more communication networks. These networks can be, for example, wireless, wireline, or optical. They can also be local, wide-area, metropolitan, vehicle and industrial, real-time, latency-tolerant, etc. Examples of networks include local area networks such as Ethernet®, cellular networks including wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., TV wireline or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial TV, and vehicle and factory networks including CAN bus. Certain networks generally require external network interface adapters attached to specific general-purpose digital ports or peripheral buses (2849) (e.g., USB ports on the computer system (2800)). Others are generally integrated into the core of the computer system (2800) by attachment to system buses as described below (e.g., Ethernet networks to PC computer systems, or cellular network interfaces to smartphone computer systems). Using any of these networks, the computer system (2800) can communicate with other entities. Such communications can be unidirectional and receive-only (e.g., broadcast TV) or unidirectional and transmit-only (e.g., a CAN bus to a specific CAN bus device), or they can be bidirectional to other computer systems using, for example, a local or wide-area digital network. Specific protocols or protocol stacks are available for use with each of the aforementioned networks and network interfaces.

[0175] The above-mentioned human interface devices, human-accessible memory devices, and network interfaces may be attached to the core (2840) of the computer system (2800).

[0176] A core (2840) may include one or more central processing units (CPUs) (2841), graphics processing units (GPUs) (2842), dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) (2843), hardware accelerators (2844) for specific tasks, etc. These devices may be connected via a system bus (2848) along with read-only memory (ROM) (2845), random access memory (RAM) (2846), internal mass storage devices such as internal user-inaccessible hard drives, SSDs, etc. (2847). In some computer systems, the system bus (2848) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripherals may be attached directly to the core's system bus (2848) or via a peripheral bus (2849). Architectures for peripheral buses include PCI, USB, etc.

[0177] The CPU (2841), GPU (2842), FPGA (2843), and accelerator (2844) are capable of executing specific instructions that can be combined to constitute the computer code described above. This computer code can be stored in ROM (2845) or RAM (2846). Temporary data can also be stored in RAM (2846), while persistent data can be stored, for example, in a built-in mass storage device (2847). High-speed storage and retrieval to any of the memory devices can be enabled by the use of cache memory. Cache memory may be closely associated with one or more CPUs (2841), GPUs (2842), mass storage devices (2847), ROMs (2845), RAM (2846), etc.

[0178] Computer-readable media may contain computer code for performing various computer implementation operations. The media and computer code may be specifically designed and configured for the purposes of this disclosure, or they may be of a type commonly known and available to those with ordinary skill in computer software technology.

[0179] For example, and not as an limitation, a computer system having architecture (2800), specifically core (2840), can provide functionality as a result of a processor (including CPU, GPU, FPGA, accelerator, etc.) that runs software embodied in one or more tangible computer-readable media. Such computer-readable media can be media related to user-accessible mass storage devices introduced earlier, in addition to specific storage devices of core (2840) that are non-transient in nature, such as core-integrated mass storage (2847) or ROM (2845). Software implementing various embodiments of this disclosure is stored in such devices and is executable by core (2840). The computer-readable media may include one or more memory devices or chips, depending on the specific needs. The software may cause the core (2840) and, specifically, the processor within it (including CPUs, GPUs, FPGAs, etc.) to execute a particular process or a particular part of a particular process as described herein, including defining data structures stored in RAM (2846) and modifying such data structures according to a process defined by the software. Additionally, or alternatively, a computer system may provide functionality as a result of logic implemented or otherwise embodied in hardware within the circuit (e.g., an accelerator (2844)) that can operate in place of or with the software to execute a particular process or a particular part of a particular process as described herein. References to software may, as necessary, include logic, and vice versa. References to computer-readable media may, as necessary, include circuitry storing software for execution (e.g., an integrated circuit (IC)), circuitry embodying logic for execution, or both. This disclosure also encompasses any suitable combination of hardware and software.

[0180] Appendix A: Acronyms ASIC:Application-Specific Integrated Circuit BMS:benchmark set CANBus:Controller Area Network Bus CBF:Coded Block Flag CD:Compact Disc CPU:Central Processing Unit(s) CRT:Cathode Ray Tube CTB:Coding Tree Block(s) CTU:Coding Tree Unit(s) CU:Coding Unit DVD:Digital Video Disc FPGA:Field Programmable Gate Area(s) GOP:Group of Picture(s) GPU:Graphics Processing Unit(s) GSM:Global System for Mobile communications HEVC:High Efficiency Video Coding HRD:Hypothetical Reference Decoder ISP:Intra Sub-Partitions IC:Integrated Circuit JEM:Joint Exploration Model LAN:Local Area Network LCD:Liquid-Crystal Display LTE:Long-Term Evolution MPM:Most Probable Mode MV:Motion Vector OLED:Organic Light-Emitting Diode PB: Prediction Block(s) PCI:Peripheral Component Interconnect PLD: Programmable Logic S PU: Prediction Unit(s) RAM: Random Access Memory ROM: Read-Only Memory SBT: Sub-Block Transform SEI:Supplementary Enhancement Information SNR: Signal Noise Ratio SSD: Solid-State Drive TU: Transform Unit(s) USB: Universal Serial Bus VPDU:Virtual Pipeline Data Unit VUI:Video Usability Information VVC: Versatile Video Coding

[0181] While this disclosure has described several exemplary embodiments, there are various alternatives, substitutions, and equivalents within the scope of this disclosure. Therefore, as will be apparent, those skilled in the art can conceive of numerous systems and methods that embody the principles of this disclosure and thus fall within its spirit and scope, even if not explicitly illustrated or described herein.

[0182] [References] This disclosure claims priority to U.S. Patent Application No. 16 / 823831, filed on March 19, 2020, titled "Method and Apparatus for Video Coding," through U.S. Patent Provisional Application No. 62 / 822787, filed on March 22, 2019, titled "Modified VPDU Compatible Max Transform Control." The full disclosures of these prior applications are incorporated herein by reference in their entirety.

Claims

1. A method for video encoding performed by an encoder, The step of obtaining information about coding units (CUs) within a picture, wherein the information is expressed as W samples representing the width of the CU and H samples representing the height of the CU, and the step of obtaining the information, The steps include: obtaining a unit having a size of Min(W,K) × Min(H,K) samples from the CU based on the width W and height H of the CU and the processing data unit size K; and determining which partitioning structure from a plurality of partitioning structures is selected to partition the unit based on the size of the Min(W,K) × Min(H,K) samples and the maximum conversion unit (TU) size which is width M and height M, wherein the plurality of partitioning structures include a vertical binary tree partitioning structure and a horizontal binary tree partitioning structure; The steps include: dividing the unit into multiple TUs recursively based on a partitioning structure determined to be selected from the plurality of partitioning structures, until the size of each TU is M x M or less; The steps include: performing encoding processing on the plurality of TUs; A method of having.

2. When Min(W,K) is greater than the width M of the maximum TU size, and Min(H,K) is less than or equal to the height M of the maximum TU size, the determined partitioning structure is the vertical binary tree partitioning structure that divides the unit into two partitions having a size of (Min(W,K) / 2) × Min(H,K). The method according to claim 1.

3. When Min(H,K) is greater than the height M of the maximum TU size, and Min(W,K) is less than or equal to the width M of the maximum TU size, the determined partitioning structure is a horizontal binary tree partitioning structure that divides the unit into two partitions having a size of Min(W,K) × (Min(H,K) / 2). The method according to claim 1 or 2.

4. At least one of the width W or the height H is 64 or less. The method according to any one of claims 1 to 3.

5. A device for video encoding, A non-temporary computer-readable medium that stores a program, One or more processors coupled to the aforementioned non-temporary computer-readable medium and It has, When the program is read and executed by one or more processors, it causes one or more processors to perform the method described in any one of claims 1 to 4. Device.

6. A method for video encoding performed by an encoder, The steps include encoding a bitstream that contains a picture, The step of storing the encoded bitstream, The step of encoding the bitstream is: The step of acquiring information of coding units (CUs) in the picture, wherein the information is shown as W samples representing the width of the CU and H samples representing the height of the CU, The steps include: obtaining a unit having a size of Min(W,K) × Min(H,K) samples from the CU based on the width W and height H of the CU and the processing data unit size K; and determining which partitioning structure from a plurality of partitioning structures is selected to partition the unit based on the size of the Min(W,K) × Min(H,K) samples and the maximum conversion unit (TU) size which is width M and height M, wherein the plurality of partitioning structures include a vertical binary tree partitioning structure and a horizontal binary tree partitioning structure; The steps include: dividing the unit into multiple TUs recursively based on a partitioning structure determined to be selected from the plurality of partitioning structures, until the size of each TU is M x M or less; The steps include: performing encoding processing on the plurality of TUs; Methods that include...

Images