Method and apparatus for video coding

Abstract

Aspects of the disclosure provide methods and apparatuses. The apparatus includes processing circuitry to decode, from a coded video bitstream, coding information of a coding block (CB) in a picture. The coding information indicates a width W and a height H of the CB. The processing circuitry partitions the CB into sub-processing units (SPUs) having a width of a minimum of W and K and a height of a minimum of H and K. At least one of the width W and the height H is greater than a processing data unit size K. The processing circuitry determines a partition structure to partition the SPUs based on the width, the height, and a maximum transform unit (TU) size M. At least one of the width and the height is greater than M. The processing circuitry partitions each of the SPUs into MxM TUs based on the partition structure.

Description

[0001] INCORPORATION BY REFERENCE

[0002] This application claims priority to U.S. Patent Application No. 16 / 823,831, filed March 19, 2020, entitled “Method and Apparatus for Video Coding,” which claims priority to U.S. Provisional Application No. 62 / 822,787, filed March 22, 2019, entitled “Modified VPDU Compatible Max Transform Control,” the entire contents of all of the preceding applications are incorporated herein by reference. TECHNICAL FIELD

[0003] Embodiments described herein generally relate to video coding. BACKGROUND

[0004] The background technique provided herein is intended for purposes of general

[0005] Video coding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a series of pictures, each picture having a spatial dimension of, for example, 1920 x 1080 luminance samples and associated chrominance samples. The series of pictures can have a fixed or variable picture rate (informally also known as frame rate), of, for example 60 pictures per second (picture rate). Uncompressed video has very high bit rate requirements. For example, 1080p604:2:0 video at 8 bit per sample and 60 Hz frame rate requires 1.5 Gbit / s bandwidth. An hour of such video requires more than 600 GBytes of storage space.

[0006] One purpose of video encoding and decoding can be to reduce redundancy in the input video signal through compression. Compression can help reduce the aforementioned bandwidth or storage space requirements, in some cases by two orders of magnitude or more. Both lossless compression and lossy compression, as well as a combination thereof, can be employed. Lossless compression refers to techniques where an exact copy of the original signal can be reconstructed from the compressed original signal. When using lossy compression, the reconstructed signal can not be identical to the original signal, but the distortion between original and reconstructed signal is small enough to make the reconstructed signal useful for the intended application. In the case of video, lossy compression is widely used. The amount of distortion that can be tolerated varies depending on the application - for example, users of certain consumer streaming applications can tolerate more distortion than users of a television distribution application. The achievable compression ratio can reflect that a higher tolerable / acceptable distortion can result in a higher compression ratio.

[0007] Video encoders and video decoders can utilize a wide variety of techniques, including, for example, motion compensation, transform, quantization, and entropy coding.

[0008] Video codec techniques can include techniques named intra-coding. In intra-coding, sample values are represented without reference to other, previously reconstructed sample values from a reference picture. In certain video codecs, a picture is spatially sub-divided into blocks of samples. When all blocks of a picture are coded in intra-mode, that picture can be an intra picture. Intra pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and can therefore be used as the first picture in a coded video bitstream and a video session, or as a still image. Samples of an intra block can be subjected to a transform, and the transform coefficients can be quantized before entropy coding. Intra prediction can be a technique that minimizes sample values in the pre-transform domain. In certain cases, the smaller the DC value after a transform and the smaller the AC coefficients, the fewer the bits needed to represent the entropy-coded block at a given quantization step size.

[0009] Traditional intra coding, such as known from e.g. MPEG-2 generation coding technologies, does not use intra prediction. However, some newer video compression technologies include techniques that attempt to use, for example, spatially neighboring and previously decoded sample data and / or metadata. Such techniques are hereinafter referred to as "intra prediction" techniques. Note that, at least in some cases, intra prediction only uses reference data from the current picture that is being reconstructed, and not reference data from reference pictures.

[0010] There can be many different forms of intra prediction. When more than one such technique can be used in a given video coding technology, the one in use can be signaled in an intra prediction mode. In some cases, a mode can have sub modes and / or parameters, and those can be signaled individually or included in the mode codeword. Which codeword to use for a given mode / sub mode / parameter combination can have an impact on the coding efficiency gains through intra prediction, and so can the entropy coding techniques used to translate the codewords to the bitstream.

[0011] H.264 introduced a certain form of intra prediction and it was improved in H.265 and further improved in newer coding techniques, such as the Joint Exploration Model (JEM), the Versatile Video Coding (VVC) and the Benchmark Set (BMS). The predictor block can be formed using neighboring sample values that belong to already coded lines. Depending on the direction, the sample values of the neighboring samples are copied into the predictor block. The reference to the used direction can be coded in the bitstream or can be predicted itself.

[0012] Reference Figure 1A The subset of nine predictor directions known from the 33 possible predictor directions (corresponding to the 33 angular modes) of H.265's Intra prediction is depicted in the lower right. The point where the arrows converge is the sample (101) that is being predicted. The arrows indicate the direction from which the sample (101) is predicted. E.g. the arrow (102) indicates that sample (101) is predicted from a sample or samples to the right and above, at a 45 degree angle to the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from a sample or samples to the bottom left, at a 22.5 degree angle to the horizontal.

[0013] Still referring to Figure 1AIn the top-left corner, a 4x4 square block of samples (104) is depicted (indicated by the thick, dashed line). The square block (104) contains 16 samples, each of which is labeled using "S" and its position, e.g., row index, in the Y dimension and its position, e.g., column index, in the X dimension. For example, sample S21 is the second sample in the Y dimension (from the top) and the first sample in the X dimension (from the left). Similarly, sample S44 is the fourth sample in the Y dimension and the X dimension in the block (104). Since the block is 4x4 samples in size, S44 is in the bottom-right corner. Also shown in FIG. 1 are the reference samples, which follow a similar numbering scheme. The reference samples are labeled using "R" and their Y position (e.g., row index) and X position (column index) relative to the block (104). In both H.264 and H.265, the prediction samples are all adjacent to the block being reconstructed and, therefore, no negative values are needed.

[0014] Intra picture prediction can work by copying reference sample values from the neighboring samples that are occupied by the prediction direction that is signaled. Assume, for example, that the coded video bitstream includes signaling that, for this block, indicates a prediction direction consistent with arrow (102), that is, samples are predicted from a right above diagonal one or more prediction samples. In that case, sample S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0015] In certain cases, the values of multiple reference samples can be combined, e.g., by interpolation, in order to arrive at a reference sample, especially when the direction cannot be evenly divided among the reference samples.

[0016] The number of possible directions has increased as video coding technology has developed. In H.264 (2003), nine different direction could be represented. That number increased to 33 in H.265 (2013), and at the time of this disclosure, JEM / VVC / BMS can support as many as 65 directions. Experiments have been conducted to identify the most likely directions, and certain techniques in the entropy coding are used to represent those likely directions in a small number of bits, accepting a certain penalty for less likely directions. Further, the directions themselves can sometimes be predicted from neighboring directions used in neighboring, already decoded, blocks.

[0017] Figure 1B A diagram (180) illustrates the 65 intra prediction directions according to JEM to show the increasing number of prediction directions over time.

[0018] The mapping of intra prediction directions, which are represented by intra prediction direction bits in the coded video bitstream, can be different from video coding technology to video coding technology, and can range, for example, from simple direct mappings of prediction directions to intra prediction modes, to codewords, to complex adaptive schemes involving most probable modes and like techniques. In all cases, however, there can be certain directions that are statistically likely to occur less often than certain other directions in video content. Since the goal of video compression is the reduction of redundancy, those less likely directions will, in a well functioning video coding technology, be represented by a larger number of bits than are those likely directions.

[0019] Motion compensation can be a lossy compression technique, and can involve a block of sample data from a previously reconstructed picture or a portion thereof (reference picture) being used to predict a newly constructed picture or picture portion after spatially offsetting the block in a direction indicated by a motion vector (MV). In some cases, the reference picture can be the same as the picture currently under reconstruction. The MV can have two dimensions, X and Y, or three dimensions, with the third dimension indicating the reference picture in use (the latter indirectly can be a temporal dimension).

[0020] In some video compression techniques, an MV applicable to a certain area of sample data can be predicted from other MVs, for example from those related to another area of sample data spatially adjacent to the area under reconstruction, and previously reconstructed. Doing so can substantially reduce the amount of data needed to represent the MV, thereby removing redundancy and increasing compression. MV prediction can work effectively, for example, because of the statistical likelihood that certain MVs found in adjacent areas of sample data, when coding video obtained from cameras (known as natural video), point in a similar direction, and therefore can be used as a good predictor. In some cases, MV prediction can be an example of lossless compression, since the predictor can be obtained without any error introduced by the compression process. In other cases, MV prediction itself can be lossy, for example because of rounding errors introduced during the calculation of the predictor from the surrounding MVs, or because of aliasing, when the spatial resolution of the MVs is lower than that of the sample data.

[0021] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding," December 2016). Among the many MV prediction mechanisms provided by H.265 are techniques described below, which we refer to as "spatial merge."

[0022] Referring to FIG. 2, a current block (201) includes samples that have been found by an encoder in a motion search process that can be predicted from a same-sized previous block that has been spatially offset. Rather than coding the MV directly, the MV is derived from metadata associated with one or more reference pictures, e.g., using the MV associated with any of the five surrounding samples labeled A0, Al, and B0, Bl, B2 (corresponding to 202-206, respectively), the MV is derived (in decoding order) from metadata of the most recent reference picture. In H.265, MV prediction can use predictors of the same reference picture that neighboring blocks are also using. SUMMARY

[0023] Aspects of the disclosure provide methods and apparatuses of video encoding / video decoding. In some examples, an apparatus of video decoding includes processing circuitry. The processing circuitry is configured to decode, from a coded video bitstream, coding information of a coding block (CB) in a picture. The coding information indicates a width of W samples of the CB and a height of H samples of the CB. The processing circuitry can divide the CB into sub-processing units (SPUs) whose width is a minimum of W and K and whose height is a minimum of H and K. At least one of the width W and the height H of the CB is greater than a processing data unit size K. The processing circuitry can determine a partition structure to further divide the SPU based on the width and the height of the SPU and a size of a maximum transform unit (TU) of M samples. At least one of the width and the height of the SPU is greater than M. The processing circuitry can divide each of the SPU into MxM TUs based on the determined partition structure.

[0024] In one embodiment, the width and the height of the SPU are greater than M. The processing circuitry can determine the partition structure to be a quad-tree partition structure. The processing circuitry can divide the SPU into TUs based on the quad-tree partition structure.

[0025] In one embodiment, the width of the SPU is greater than M and the height is equal to M. The processing circuitry can determine the partition structure to be a vertical binary tree partition structure. The processing circuitry can divide the SPU into TUs based on the vertical binary tree partition structure.

[0026] In one embodiment, the height of the SPU is greater than M and the width of the SPU is equal to M. The processing circuitry can determine the partitioning structure to be a horizontal binary tree partitioning structure. The processing circuitry can partition the SPU into TUs based on the horizontal binary tree partitioning structure.

[0027] In one embodiment, the processing circuitry can recursively partition one of the SPUs into TUs based on the partitioning structure.

[0028] In one embodiment, the processing circuitry can process the SPUs according to a first scan order and process the TUs in each of the SPUs according to a second scan order. In one example, at least one of the first scan order and the second scan order is one of: (i) a raster scan order; (ii) a vertical scan order; (iii) a zigzag order; and (iv) a diagonal scan order. In one example, the first scan order and the second scan order are raster scan orders. In one example, W is 128, H is 64, K is 64, and M is 32. The first scan order is a left-to-right scan order and the second scan order is a raster scan order.

[0029] In one embodiment, the processing data unit size K indicates a size of a virtual pipeline data unit (VPDU). A first one of the SPUs is included in a first VPDU in the picture and a second one of the SPUs is included in a second VPDU in the picture. After processing the first VPDU in a first stage of a multi-stage pipeline, the processing circuitry can concurrently process the first VPDU in a second stage of the multi-stage pipeline and process the second VPDU in the first stage. BRIEF DESCRIPTION OF DRAWINGS

[0030] Other features, objects, and advantages of the disclosed subject matter will be apparent from the following detailed description and drawings, in which:

[0031] Figure 1A is an exemplary subset of intra prediction modes.

[0032] Figure 1B is an illustration of exemplary intra prediction directions.

[0033] FIG. 2 is a diagram of a current block and spatial merge candidates around it in one example.

[0034] Figure 3 is a diagram of a simplified block diagram of a communication system in accordance with one embodiment.

[0035] Figure 4 is a diagram of a simplified block diagram of a communication system in accordance with one embodiment.

[0036] Figure 5is a schematic illustration of a simplified block diagram of a decoder according to one embodiment.

[0037] Figure 6 is a schematic illustration of a simplified block diagram of an encoder according to one embodiment.

[0038] Figure 7 A block diagram of an encoder according to another embodiment is shown.

[0039] Figure 8 A block diagram of a decoder according to another embodiment is shown.

[0040] Figure 9A A CTU partitioned with a quad-tree plus binary-tree (QTBT) structure is shown.

[0041] Figure 9B A QTBT structure is shown.

[0042] Figure 9C A horizontal center-side ternary tree is shown.

[0043] Figure 9D A vertical center-side ternary tree is shown.

[0044] Figures 10A-10D Transform kernel matrices for a 4-point DCT-2 transform, an 8-point DCT-2 transform, a 16-point DCT-2 transform, and a 32-point DCT-2 transform, respectively, are shown.

[0045] Figures 11A-11E A 64x64 transform kernel matrix for a 64-point DCT-2 transform is shown.

[0046] Figure 12 Transform basis functions for a discrete sine transform (DST) / discrete cosine transform (DCT) selected by an adaptive multi-kernel transform (AMT) are shown.

[0047] Figure 13 A mapping relationship table between mts_idx values and corresponding horizontal or vertical transforms is shown.

[0048] Figures 14A-14D A transform kernel matrix for a DST-7 transform is shown.

[0049] Figures 15A-15D A transform kernel matrix for a DST-8 transform is shown.

[0050] Figure 16 A number of sub-partitions depending on block size is shown.

[0051] Figure 17 An example of intra-sub-partition (ISP) is shown.

[0052] Figure 18An example of an ISP is shown.

[0053] Figures 19A-19B An example of syntax elements for an ISP coding mode is shown.

[0054] Figures 20A-20D An example of sub-block transform (SBT) is shown.

[0055] Figures 21A-21I An example of specification text of a video coding standard when SBT is used is shown.

[0056] Figure 22 Different YUV formats used in some embodiments are shown.

[0057] Figure 23 An example of forbidden triple tree (TT) and binary tree (BT) partitions is shown.

[0058] Figure 24 An example of transform tree syntax is shown.

[0059] Figure 25 An example of a coding block with 128x64 sample size is shown.

[0060] Figure 26A An example of a coding block with 128x32 sample size is shown.

[0061] Figure 26B An example of a coding block with 128x32 sample size is shown.

[0062] Figure 27 A flowchart outlining a process according to embodiments of the disclosure is shown.

[0063] Figure 28 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION

[0064] I. Encoder and decoder for video coding

[0065] Figure 3 A simplified block diagram of a communication system (300) according to one embodiment of the present disclosure is shown. The communication system (300) includes a plurality of terminal devices that can communicate with one another, e.g., via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). The terminal devices (310) and (320) can exchange data, e.g., coded video data, audio data, and so on. The Figure 3In the example of FIG. 3, the first pair of terminal devices (310) and (320) performs unidirectional transmission of coded video data from terminal device (310) to terminal device (320). Similarly, the second pair of terminal devices (330) and (340) also can perform unidirectional transmission of coded video data from terminal device (330) to terminal device (340). In the example of FIG. 3, the terminal devices (310) and (320) can be considered as source and destination terminal devices, respectively, for the coded video data transmitted from terminal device (310) to terminal device (320). Similarly, the terminal devices (330) and (340) can be considered as a source and destination terminal devices, respectively, for the coded video data transmitted from terminal device (330) to terminal device (340). The implementations of FIG. 3 are but one example. Other unidirectional transmission of coded video data can occur from other terminal devices to other terminal devices, such as from terminal device (310) to terminal device (340), terminal device (320) to terminal device (330), terminal device (310) to terminal device (330), and terminal device (320) to terminal device (340).

[0066] In another example, the communication system (300) includes a pair of terminal devices (330) and (340) that perform bidirectional transmission of coded video data that can occur, for example, during a videoconferencing session. For bidirectional transmission of coded video data, in an example, each terminal device of the terminal devices (330) and (340) can code video data (e.g., of a stream of video pictures that are captured by the terminal device) for transmission to the other terminal device of the terminal devices (330) and (340) via the network (350). Additionally, each terminal device of the terminal devices (330) and (340) can receive coded video data transmitted by the other terminal device of the terminal devices (330) and (340), and can decode the coded video data to recover video pictures and can display video pictures according to the recovered video data on a display accessible to the terminal device.

[0067] In Figure 3 In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) can be illustrated as servers, personal computers and smartphones but the principles of the present disclosure can not be limited to these particular computing devices. The embodiments of the present disclosure find application with laptop computers, tablet computers, media players and / or dedicated video conferencing devices. The network (350) represents any number of networks that transmit coded video data among the terminal devices (310), (320), (330), and (340), including for example wire-based (wired) and / or wireless communication networks. The communication network (350) can exchange data in circuit-switched and / or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks and / or the Internet. For the purposes of the present disclosure, the architecture and topology of the network (350) can be immaterial to the operation of the disclosed subject matter unless otherwise explained herein.

[0068] As an example of the application of the disclosed subject matter, Figure 4The placement of video encoder and video decoder in a streaming environment is shown. The disclosed subject matter can be equally applicable to other video enabled applications, including for example video conferencing, digital TV, storing of compressed video on digital media including CD, DVD, memory stick etc. and so on.

[0069] A streaming system can include a capture subsystem (413) that can include a video source (401), for example a digital camera, that creates an uncompressed video picture stream (402). In an example, the video picture stream (402) includes samples taken by the digital camera. The video picture stream (402), depicted as a bold line to emphasize the high data volume compared to encoded video data (404) (or encoded video bitstreams), can be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter as described in more detail below. The encoded video data (404) (or encoded video bitstream (404)), depicted as a thin line to emphasize the lower data volume compared to the video picture stream (402), can be stored for future use on a streaming server (405). One or more streaming client subsystems, such as client subsystems (406) and (408) in Figure 4

[0070] It is noted that the electronic devices (420) and (430) can include other components (not shown). For example, the electronic device (420) can include a video decoder (not shown) and the electronic device (430) can include a video encoder (not shown) as well.

[0071] Figure 5 ​A block diagram of a video decoder (510) is shown according to an embodiment of the present disclosure. The video decoder (510) can be included in an electronic device (530). The electronic device (530) can include a receiver (531) (e.g., receiving circuitry). The video decoder (510) can be used in place of the video decoder (410) in the example of Figure 4

[0072] The receiver (531) can receive one or more coded video sequences to be decoded by the video decoder (510); in the same or another embodiment, one coded video sequence at a time, with the decoding of each coded video sequence being independent of the other coded video sequences. The coded video sequences can be received from a channel (501), which can be a hardware / software link to a storage device that stores the encoded video data. The receiver (531) can receive encoded video data, which can be forwarded to their respective using entities (not shown), along with other data, e.g., coded audio data and / or ancillary data streams. The receiver (531) can separate the coded video sequences from the other data. To protect against network jitter, a buffer memory (515) can be coupled in between the receiver (531) and the entropy decoder / parsen (520) (hereinafter “parsen (520)”). In certain applications, the buffer memory (515) is part of the video decoder (510). In other cases, the buffer memory (415) can be located (not shown) outside the video decoder (510). In yet other cases, a buffer memory (not shown) is located outside the video decoder (510), e.g., to protect against network jitter, and another buffer memory (515) is configured inside the video decoder (510), e.g., to handle playout timing. When the receiver (531) receives data from a store-and-forward device or from an isosychronous network that has sufficient bandwidth and control, it can not be necessary to configure the buffer memory (515), or it can be small. To be used on a service packet network, such as the Internet, the buffer memory (515) can be relatively large and can advantageously be of adaptive size, and can be at least partially implemented in an operating system or a similar element (not shown) outside the video decoder (510).

[0073] The video decoder (510) can include a parser (520) to reconstruct symbols (521) from the coded video sequence. Categories of those symbols include information to manage operation of the video decoder (510), and potentially information to control a display (512) (e.g., a display screen) or similar display device, which can or can not be an integral part of the electronic device (530), but to which data is outputted by the video decoder (510), e.g., for presentation of content. The parser (520) can parse the coded video sequence into the symbols (521), which can include information such as intra- slice or intra-picture prediction parameters, transform quantization parameters, motion vector parameters, and other information considered useful by the video decoder (510) to decode the video slice or picture into a video frame or field based on a video coding / encoding scheme. Figure 5 ​The control information for the display device can be Supplemental Enhancement Information (SEI messages) or parameter sets for Video Usability Information (VUI) fragments (not depicted). The parser (520) can parse / entropy-decode the received coded video sequence. The coding of the coded video sequence can be in accordance with a video coding technology or standard, and can follow various principles, including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, and so forth. The parser (520) can extract from the coded video sequence, a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder, based on at least one parameter corresponding to the group. The subgroups can include Groups of Pictures (GOPs), pictures, tiles, slices, macroblocks, Coding Units (CUs), blocks, Transform Units (TUs), Prediction Units (PUs) and so forth. The parser (520) can also extract from the coded video sequence, information such as transform coefficients, quantizer parameter values, motion vectors and so forth.

[0074] The parser (520) can perform entropy-decoding / parsing operation on the video sequence received from the buffer memory (515), creating symbols (521).

[0075] The reconstruction of symbols (521) can involve a number of different units, depending on the type of coded video picture or portion thereof (e.g., inter and intra picture, inter and intra block), and other factors. Which units are involved, and the manner of the involvement, can be controlled by subgroup control information parsed by the parser (520) from the coded video sequence. For the sake of brevity, such subgroup control information flow between the parser (520) and the following number of units is not depicted.

[0076] Beyond the functional blocks already mentioned, the video decoder (510) can be conceptually subdivided into a number of functional units as described below. In practical implementations, many of these units interact closely with each other, and may, at least partly, be integrated into each other's operation. For the purpose of describing the disclosed subject matter, however, a conceptual subdivision into the functional units below is appropriate.

[0077] The first unit is a scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives quantized transform coefficients as symbols (521) and control information, including which transform to use, block size, quantization factor, quantization scaling matrices, etc., from the parser (520). The scaler / inverse transform unit (551) can output a block comprising sample values, which can be input into the aggregator (555).

[0078] In some cases, the output samples of the scaler / inverse transform unit (551) can belong to an intra coded block; i.e., a block that does not use predictive information from previously reconstructed pictures, but can use predictive information from previously reconstructed parts of the current picture. Such predictive information can be provided by an intra picture prediction unit (552). In some cases, the intra picture prediction unit (552) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information extracted from the current picture buffer (558). For example, the current picture buffer (558) buffers partially reconstructed current pictures and / or fully reconstructed current pictures. In some cases, the aggregator (555) adds, on a per sample basis, the predictive information generated by the intra prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551).

[0079] In other cases, the output samples of the scaler / inverse transform unit (551) can belong to an inter coded and potentially motion compensated block. In this case, the motion compensation prediction unit (553) can access the reference picture memory (557) to fetch samples for prediction. After motion compensation of the fetched samples according to the symbols (521) pertaining to the block, these samples can be added by the aggregator (555) to the output of the scaler / inverse transform unit (551), referred to as residual samples or residual signal in this case, generating the output sample information. The fetching of the prediction samples by the motion compensation prediction unit (553) from addresses within the reference picture memory (557) can be controlled by motion vectors, which are made available to the motion compensation prediction unit (553) in the form of symbols (521), which can have, for example, X, Y, and reference picture components. Motion compensation can also include interpolation of sample values fetched from the reference picture memory (557), motion vector prediction mechanisms, etc., when sub-sample precision motion vectors are used.

[0080] The output samples of the aggregator (555) can be subject to various loop filtering techniques in the loop filter unit (556). Video compression technologies can include in-loop filter technologies that are controlled by parameters included in the coded video sequence (also referred to as coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520), however, the video compression technologies can also be responsive to meta-information obtained during the decoding of previous (in decoding order) parts of the coded picture or coded video sequence, as well as responsive to previously reconstructed and loop-filtered sample values.

[0081] The output of the loop filter unit (556) can be a stream of samples that can be output to the display device (512) and to the reference picture memory (557) for future inter-picture prediction.

[0082] Once fully reconstructed, certain coded pictures can be used as reference pictures for future prediction. For example, once a coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified (by, for example, the parser (520)) as a reference picture, the current picture buffer (558) can become part of the reference picture memory (557), and a new current picture buffer can be reallocated before commencing the reconstruction of the following coded picture.

[0083] The video decoder (510) can perform decoding operations according to a predetermined video compression technology in a standard, such as ITU-T H.265. In the sense that the coded video sequence adheres to the syntax of the video compression technology or standard, and to the configuration file as documented in the video compression technology or standard. Specifically, the configuration file can select, from all the tools available in the video compression technology or standard, certain tools as the only tools to use under that configuration file. Also for compliance, the complexity of the coded video sequence is required to be within limits as defined by the level of the video compression technology or standard. In some cases, the limits set by the level can be further restricted through Hypothetical Reference Decoder (HRD) specifications and metadata for HRD buffer management signaled in the coded video sequence.

[0084] In this embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. This additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or more accurately reconstruct the original video data. The additional data may take the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant images, forward error correction codes, etc.

[0085] Figure 6 A block diagram of a video encoder (603) according to an embodiment disclosed in this application is shown. The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., transmission circuitry). The video encoder (603) can be used in place of... Figure 4 The example video encoder (403).

[0086] The video encoder (603) can obtain data from the video source (601) (not) Figure 6 In the example, a portion of the electronic device (620) receives a video sample, the video source of which can capture video images to be encoded by a video encoder (603). In another embodiment, the video source (601) is a portion of the electronic device (620).

[0087] A video source (601) can provide a sequence of source video samples to be encoded by a video encoder (603) in the form of a digital video sample stream, which can have any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601YCrCb, RGB, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (601) can be a storage device storing previously prepared video. In a video conferencing system, the video source (601) can be a camera that captures local image information as a video sequence. Video data can be provided as multiple individual pictures, which are given motion when viewed in sequence. The pictures themselves can be constructed as spatial pixel arrays, where each pixel can include one or more samples depending on the sampling structure, color space, etc., used. Those skilled in the art can easily understand the relationship between pixels and samples. The following focuses on describing samples.

[0088] According to an embodiment, the video encoder (603) can code and compress the pictures of the source video sequence into a coded video sequence (643) in real time or under any other time constraints as required by the application. Enforcing appropriate coding speed is one function of a controller (650). In some embodiments, the controller (650) controls other functional units as described below and is functionally coupled to the other functional units. For brevity, couplings are not depicted as lines in the figures. Parameters set by the controller (650) can include rate control related parameters (picture skip, quantizer, lambda value of rate-distortion optimization techniques, etc.), picture size, group of pictures (GOP) layout, maximum motion vector search range, and so on. The controller (650) can be configured to have other suitable functions that pertain to the video encoder (603) optimized for a certain system design.

[0089] In some embodiments, the video encoder (603) is configured to operate in a coding loop. As a simple description, in an example, the coding loop can include a source coder (630) (e.g., responsible for creating symbols, e.g., a symbol stream, based on input pictures to be coded, and reference pictures) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create the sample data in a similar manner as a (remote) decoder would create sample data from the symbols (as any compression between symbols and coded video bitstream is lossless in the video compression technologies considered by the disclosed subject matter). The reconstructed sample stream (sample data) is input to the reference picture buffer (634). As the decoding of symbols results in a bit-exact outcome irrespective of the decoder location (local or remote), the content in the reference picture buffer (634) is also bit exact between the local encoder and the remote encoder. In other words, the prediction part of the encoder sees the same reference picture samples as would be seen by the decoder when using the samples for prediction during decoding. This fundamental principle of reference picture synchronicity (and the drift that occurs if synchronicity cannot be maintained, e.g., due to channel errors) is used by some related arts as well.

[0090] The operation of the “local” decoder (633) can be the same as the “remote” decoder detailed above with respect to the video decoder (510). However, additional brief reference is made to the video decoder (510) as follows. Figure 5 The operation of the “local” decoder (633) can be the same as the “remote” decoder detailed above with respect to the video decoder (510). However, additional brief reference is made to the video decoder (510) as follows. Figure 5 When symbols are available and the entropy encoder (645) and parser (520) are capable of losslessly encoding / decoding the symbols into the coded video sequence, the entropy decoding portion of the video decoder (510), including the buffer (515) and the parser (520), can not be fully implemented in the local decoder (633).

[0091] At this point it can be observed that any decoder technology, except for the parsing / entropy decoding that is present in the decoder, must also be present in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter focuses on the decoder operation. The description of the encoder technology can be simplified because the encoder technology is reciprocal to the fully described decoder technology. More detailed description is required only in certain areas, and is provided below.

[0092] During operation, in some examples, the source coder (630) can perform motion compensated predictive coding of the input picture data. This motion

[0093] The local video decoder (633) can decode coded video data of pictures that can be designated as reference pictures based on symbols created by the source coder (630). Operations of the coding engine (632) can advantageously be lossy processes. When the coded video data can be decoded at a video decoder (not shown), the reconstructed video sequence can typically be a replica of the source video sequence with some errors. Figure 6 The local video decoder (633) replicates decoding processes that can be performed by a video decoder on reference pictures and can cause reconstructed reference pictures to be stored in the reference picture cache (634). In this manner, the video encoder (603) can store copies of reconstructed reference pictures locally that have common content as the reconstructed reference pictures that will be obtained by a far-end video decoder (absent transmission errors).

[0094] The predictor (635) can perform a prediction search for the coding engine (632). That is, for each new picture to be coded, the predictor (635) can search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, and others, that can serve as an appropriate prediction reference for the new picture. The predictor (635) can operate on a sample block-by-pixel block basis to find suitable prediction references. In some cases, as determined by search results obtained by the predictor (635), an input picture can have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).

[0095] The controller (650) can manage coding operations of the source coder (630), including, for example, setting of parameters and subgroup parameters used to code the video data.

[0096] The outputs of all the above-described functional units can be entropy encoded in an entropy encoder (645). The entropy encoder (645) losslessly compresses the symbols generated by the various functional units according to, for example, Huffman coding, variable length coding, arithmetic coding, and so on, thereby converting the symbols into an encoded video sequence.

[0097] The transmitter (640) can buffer the encoded video sequence(s) generated by the entropy coder (645) in preparation for transmission via a communication channel (660), which can be a hardware / software link to a storage device that will store the encoded video data. The transmitter (640) can merge encoded video data from the video coder (603) with other data to be transmitted, for example, audio data and / or ancillary data streams (sources not shown).

[0098] The controller (650) can manage operation of the video encoder (603). During coding, the controller (650) can assign to each coded picture a certain coded picture type, which can affect the coding techniques that can be applied to the corresponding picture. For example, pictures often can be assigned as one of the following picture types:

[0099] An Intra Picture (I picture) that can be coded and decoded without using any other picture in the sequence as a source of prediction. Some video codecs allow different types of Intra Picture, including, for example, Independent Decoder Refresh (“IDR”) Pictures. A person of skill in the art is aware of the variants of I pictures and their respective applications and features.

[0100] A predictive picture (P picture) that can be coded and decoded using either intra prediction or inter prediction that uses at most one motion vector and reference index to predict sample values for each block.

[0101] A bi-directionally predictive picture (B picture) that can be coded and decoded using either intra prediction or inter prediction that uses at most two motion vectors and reference indices to predict sample values for each block. Similarly, multiple predictive pictures can use more than two reference pictures and associated metadata for reconstructing a single block.

[0102] A source picture can generally be spatially subdivided into a plurality of blocks of samples (e.g., 4x4, 8x8, 4x8, or 16x16 samples), and encoded block by block. These blocks can be predictively encoded with reference to other (already encoded) blocks, determined by the coding assignment of the respective picture to which the block applies. For example, blocks of an I picture can be non-predictively encoded, or the blocks can be predictively encoded with reference to already encoded blocks of the same picture (spatial or intra prediction). Blocks of pixels of a P picture can be predictively encoded with reference to one previously encoded reference picture, either by spatial prediction or by temporal prediction. Blocks of a B picture can be predictively encoded with reference to one or two previously encoded reference pictures, either by spatial prediction or by temporal prediction.

[0103] The video encoder (603) can perform encoding operations in accordance with a predetermined video coding technology or standard, such as ITU-T H.265. In its operation, the video encoder (603) can perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancies in the input video sequence. The coded video data, therefore, can conform to a syntax specified by the video coding technology or standard being used.

[0104] In an embodiment, the transmitter (640) can transmit additional data with the encoded video. The source coder (630) can include such data as part of the coded video sequence. Additional data can comprise temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, and so on.

[0105] Collected video can be in the form of a plurality of source pictures (video pictures) in temporal sequence. Intra-picture prediction (often simply referred to as intra prediction) exploits spatial correlation in a given picture, while inter-picture prediction exploits correlation between pictures (temporal or other correlation). In an example, a particular picture being encoded / decoded is partitioned into blocks, and the particular picture being encoded / decoded is referred to as the current picture. A block in the current picture can be encoded by a vector, referred to as a motion vector, when the block in the current picture is similar to a reference block in a reference picture that has been coded previously and remains buffered. The motion vector points to the reference block in the reference picture, and can have a third dimension identifying the reference picture in cases where multiple reference pictures are used.

[0106] In some embodiments, bi-prediction techniques can be used in inter-picture prediction. According to bi-prediction techniques, two reference pictures are used, e.g., a first reference picture and a second reference picture that are both before the current picture in decoding order (but can be past and future, respectively, in display order) in the video. A block in the current picture can be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. Specifically, the block can be predicted by a combination of the first reference block and the second reference block.

[0107] Furthermore, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0108] According to some embodiments of the disclosure, predictions, e.g., inter-picture predictions and intra-picture predictions, are performed in the unit of blocks. For example, according to the HEVC standard, a picture in a video sequence is partitioned into coding tree units (CTUs) for compression, the CTUs in a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. In general, a CTU includes three coding tree blocks (CTBs), which are one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree- split into one or multiple coding units (CUs). For example, a CTU of 64x64 pixels can be split into one CU of 64x64 pixels, or 4 CUs of 32x32 pixels, or 16 CUs of 16x16 pixels. In an example, each CU is analyzed to determine a prediction type for the CU, such as an inter prediction type or an intra prediction type. The CU is split into one or more prediction units (PUs) depending on the temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PB), and two chroma PBs. In an embodiment, a prediction operation in encoding (encoding / decoding) is performed in the unit of a prediction block. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values (e.g., luma values) for pixels, such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, and so on.

[0109] Figure 7 A diagram of a video encoder (703) according to another embodiment of the disclosure is shown. The video encoder (703) is configured to receive a processing block (such as a prediction block) of sample values within a current video picture in a video sequence, and encode the processing block into a coded picture that is part of a coded video sequence. In one example, the video encoder (703) is used in place of the video encoder (703) as shown in FIG. 7. The video encoder (703) includes a picture store (703a), a mode select unit (703b), a motion estimation unit (703c), a motion Figure 4The video encoder (403) in the example.

[0110] In HEVC embodiments, the video encoder (703) receives a matrix of sample values for a processing block, such as a prediction block of 8x8 samples or the like. The video encoder (703) uses, for example, rate-distortion (RD) optimization to determine whether to best code the processing block using intra, inter, or bi-prediction modes. When coding the processing block in the intra mode, the video encoder (703) can use an intra-prediction technique to code the processing block into the coded picture; and when coding the processing block in the inter or bi-prediction mode, the video encoder (703) can use an inter-prediction or bi-prediction technique, respectively, to code the processing block into the coded picture. In certain video coding techniques, a merge mode can be an inter-picture prediction submode in which a motion vector component is derived from one or more motion vector predictors without the benefit of a coded motion vector component outside the predictor. In certain other video coding techniques, there can be a motion vector component applicable to the subject block. In an embodiment, the video encoder (703) includes other components, such as a mode decision module (not shown) for determining a mode of the processing block.

[0111] In Figure 7 In an example, the video encoder (703) includes an inter encoder (730), an intra encoder (722), a residue calculator (723), a switch (726), a residue encoder (724), a general controller (721), and an entropy encoder (725) coupled together as shown in Figure 7

[0112] The inter encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in reference pictures (e.g., blocks in previous and future pictures), generate inter-prediction information (e.g., redundancy information descriptions according to inter coding techniques, motion vectors, merge mode information), and calculate an inter-predicted result (e.g., a predicted block) based on the inter-prediction information using any suitable technique. In some examples, the reference pictures are decoded reference pictures based on coded video information.

[0113] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), compare the block to coded blocks in the same picture in some cases, generate quantized coefficients after a transform, and also generate intra-prediction information (e.g., intra-prediction direction information according to one or more intra coding techniques) in some cases. In an example, the intra encoder (722) also calculates an intra-predicted result (e.g., a predicted block) based on the intra-prediction information and a reference block in the same picture.

[0114] ​The general controller (721) is configured to determine general control data and control other components of the video encoder (703) based on the general control data. In an example, the general controller (721) determines a mode of a block and provides a control signal to the switch (726) based on the mode. For example, when the mode is an intra mode, the general controller (721) controls the switch (726) to select the intra mode result for use by the residue calculator (723) and controls the entropy encoder (725) to select the intra prediction information and add the intra prediction information in the bitstream; and when the mode is an inter mode, the general controller (721) controls the switch (726) to select the inter prediction result for use by the residue calculator (723) and controls the entropy encoder (725) to select the inter prediction information and add the inter prediction information in the bitstream.

[0115] The residue calculator (723) is configured to calculate a difference (residual data) between the received block and the prediction result selected from the intra encoder (722) or the inter encoder (730). The residue encoder (724) is configured to operate based on the residual data to encode the residual data to generate transform coefficients. In an example, the residue encoder (724) is configured to convert the residual data from a spatial domain to a frequency domain and generate the transform coefficients. The transform coefficients are then subject to quantization processing to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residue decoder (728). The residue decoder (728) is configured to perform inverse transform and generate decoded residual data. The decoded residual data can be used by the intra encoder (722) and the inter encoder (730) as appropriate. For example, the inter encoder (730) can generate a decoded block based on the decoded residual data and the inter prediction information and the intra encoder (722) can generate a decoded block based on the decoded residual data and the intra prediction information. The decoded block is processed as appropriate to generate a decoded picture and in some examples, the decoded picture can be buffered in a memory circuit (not shown) and used as a reference picture.

[0116] The entropy encoder (725) is configured to format the bitstream to produce an encoded block. The entropy encoder (725) includes various information according to a suitable standard, such as the HEVC standard. In an example, the entropy encoder (725) is configured to include the general control data, the selected prediction information (e.g., the intra prediction information or the inter prediction information), the residual information, and other suitable information in the bitstream. It is noted that according to the disclosed subject matter, there is no residual information when a block is encoded in an inter mode or a merge submode of a bi-predictive mode.

[0117] Figure 8A diagram illustrates a video decoder (810) according to another embodiment of the disclosure. The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence, and decode the coded pictures to generate reconstructed pictures. In an example, the video decoder (810) is used in place of Figure 4 The video decoder (410) in an example.

[0118] In Figure 8 In an example, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residue decoder (873), a reconstruction module (874), and an intra decoder (872) coupled together as shown in FIG. 11. Figure 8

[0119] The entropy decoder (871) can be configured to reconstruct, from the coded picture, certain symbols that represent the syntax elements of which the coded picture is made up. Such symbols can include, for example, the mode in which a block is coded (such as intra, inter, bi-predictive mode, the latter two of which are sub-modes of the inter mode, or another sub-mode), prediction information (such as intra prediction information or inter prediction information) that can identify certain samples or metadata that are used by the intra decoder (872) or the inter decoder (880), respectively, for prediction, residual information in the form of, for example, quantized transform coefficients, and the like. In an example, when the prediction mode is inter or bi-predictive mode, the inter prediction information is provided to the inter decoder (880); and when the prediction type is intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information can be subject to inverse quantization and is provided to the residue decoder (873).

[0120] The inter decoder (880) is configured to receive the inter prediction information and generate an inter prediction result based on the inter prediction information.

[0121] The intra decoder (872) is configured to receive the intra prediction information and generate a prediction result based on the intra prediction information.

[0122] The residue decoder (873) is configured to perform inverse quantization to extract de-quantized transform coefficients and process the de-quantized transform coefficients to convert residual from the frequency domain to the spatial domain. The residue decoder (873) can also require certain control information (to include the quantizer parameter (QP)), and this information can be provided by the entropy decoder (871) (data path not depicted as this is low volume control information only).

[0123] ​The reconstruction module (874) is configured to combine, in the spatial domain, the residual output by the residue decoder (873) and the prediction results (as output by the inter or intra prediction modules) to form a reconstructed block, that can be part of the reconstructed picture, which in turn can be part of the reconstructed video. It should be noted that other suitable operations, such as a deblocking operation and the like, can be performed in order to improve the visual quality.

[0124] It should be noted that the video encoder (403), the video encoder (603), and the video encoder (703), and the video decoder (410), the video decoder (510), and the video decoder (810) can be implemented using any suitable technique. In an embodiment, the video encoder (403), the video encoder (603), and the video encoder (603), and the video decoder (410), the video decoder (510), and the video decoder (810) can be implemented using one or more integrated circuits. In another embodiment, the video encoder (403), the video encoder (603), and the video encoder (603), and the video decoder (410), the video decoder (510), and the video decoder (810) can be implemented using one or more processors that execute software instructions.

[0125] II. Transform processing techniques

[0126] 1. Block partitioning structure including quad-tree partitioning structure

[0127] The block partitioning structure can be referred to as a coding tree. In some embodiments, a quad-tree structure is used to partition a coding tree unit (CTU) into coding units (CUs) to accommodate various local features. A decision is made at the CU level whether to use inter-picture (temporal) or intra-picture (spatial) prediction to code the image region. Depending on the partition type of a prediction unit (PU), each CU can be further partitioned into one, two, or four PUs. The same prediction process is applied within one PU, and the relevant information is transmitted to the decoder on a PU basis.

[0128] After obtaining a residual block by applying the prediction process based on the partition type of the PU, the CU can be partitioned into transform units (TUs) according to another quad-tree structure. As such, there are multiple partition concepts, including CUs, PUs, and TUs. In some embodiments, a CU or TU can be square only, while a PU can be square or rectangular. In some embodiments, a coding block can be further partitioned into four square sub-blocks, and a transform is performed for each sub-block (i.e., TU). Each TU is further recursively partitioned into smaller TUs using a quad-tree structure referred to as a residual quad-tree (RQT).

[0129] In some embodiments, implicit quadtree splitting can be employed at picture boundaries, such that a block remains quadtree split until its size fits a picture boundary.

[0130] 2. Quadtree plus binary tree (QTBT) block split structure

[0131] In some embodiments, the QTBT structure is employed. The QTBT structure dispenses with the concept of multiple split types (the concepts of CU, PU, and TU) and supports more flexible CU split shapes. In the QTBT block structure, a CU can be square or rectangular in shape.

[0132] Figure 9A An example of using Figure 9B A CTU (910) split using the QTBT structure (920) shown. The CTU (910) is first split with a quadtree structure. Further, the quadtree leaf nodes are split with either a binary tree structure or a quadtree structure. In binary tree splitting, there can be two split types: symmetric horizontal split and symmetric vertical split. The binary tree leaf nodes are referred to as CUs, which can be used for prediction and transform processing without any further splitting. Thus, CUs, PUs, and TUs have the same block size in the QTBT coding block structure.

[0133] In some embodiments, a CU can include coding blocks (CBs) for different color components. For example, in the case of P and B slices in 4:2:0 chroma format, one CU includes one luma CB and two chroma CBs. A CU can include CBs for a single color component. For example, in the case of I slices, one CU includes only one luma CB or only two chroma CBs.

[0134] In some embodiments, the following parameters are defined for the QTBT split scheme:

[0135] - CTU size: The root node size of the quadtree, e.g., the same concept as in HEVC.

[0136] - MinQTsize: The minimum value of allowed quadtree leaf node size.

[0137] - MaxBTsize: The maximum value of allowed binary tree root node size.

[0138] - MaxBTDepth: The maximum depth of allowed binary tree.

[0139] - MinBTsize: The minimum value of allowed binary tree leaf node size.

[0140] In one example of the QTBT partitioning structure, the CTU size is set to 128x128 luma samples and two corresponding 64x64 blocks of chroma samples, the MinQTSize is set to 16x16, the MaxBTsize is set to 64x64, the MinBTsize (for both width and height) is set to 4x4, and the MaxBTDepth is set to 4. The quad-tree partitioning is applied first to the CTU to generate quad-tree leaf nodes. The size of the quad-tree leaf nodes can range from 16x16 (i.e., MinQTSize) to 128x128 (i.e., CTU size). If the quad-tree leaf node is 128x128, no further partitioning of the quad-tree leaf node with binary trees is performed since the size exceeds the MaxBTSize (i.e., 64x64). Otherwise, the quad-tree leaf node is further partitioned with binary trees. Thus, the quad-tree leaf node can also be the root node of a binary tree and the depth of the binary tree is 0.

[0141] When the depth of the binary tree reaches the MaxBTDepth (i.e., 4), no further splitting is considered. When the width of the binary tree node is equal to the MinBTSize (i.e., 4), no further horizontal splitting is considered. Similarly, when the height of the binary tree node is equal to the MinBTSize, no further vertical splitting is considered. The leaf nodes of the binary tree are further processed by prediction and transform processing without any further partitioning of the leaf nodes of the binary tree. In one embodiment, the maximum CTU size is 256x256 luma samples.

[0142] In Figure 9A and Figure 9B , the solid lines represent quad-tree splitting and the dashed lines represent binary tree splitting. In each splitting (i.e., non-leaf) node of the binary tree, a flag is signaled to indicate which splitting type (i.e., horizontal or vertical) is used. For example, 0 indicates horizontal splitting and 1 indicates vertical splitting. For quad-tree splitting, no splitting type indication is needed since the quad-tree splitting always splits a block horizontally and vertically into 4 equal-sized sub-blocks.

[0143] In some embodiments, the QTBT scheme supports the flexibility of having separate QTBT structures for luma and chroma. For example, for P and B slices, the luma and chroma blocks in a CTU share the same QTBT structure. However, for I slices, the luma CTBs are partitioned into CUs by a QTBT structure and the chroma blocks are partitioned into chroma CUs by another QTBT structure. Thus, a CU in an I slice can include a coding block of luma components or coding blocks of two chroma components, while a CU in a P or B slice can include coding blocks of all three color components.

[0144] In some embodiments, inter prediction for small blocks is restricted to reduce memory access for motion compensation. For example, bi-prediction is not supported for 4x8 and 8x4 blocks, and inter prediction is not supported for 4x4 blocks.

[0145] 3. Triple tree (TT) block partitioning structure

[0146] In some embodiments, a multi-type tree (MTT) structure is used to partition a picture. The MTT structure is a more flexible tree structure than the QTBT structure. In MTT, in addition to quad-tree and binary tree, a triple tree is adopted Figure 9C and Figure 9D horizontal center-side triple tree and vertical center-side triple tree shown in FIGS. 1 and 2, respectively. Triple tree partitioning can complement quad-tree partitioning and binary tree partitioning. For example, triple tree partitioning can capture objects located at the center of a block, while quad-tree and binary tree can split across the block center. The width and height of a triple tree partitioned block are powers of two, so no additional transform partitioning is needed.

[0147] In one example, the two-level tree design is mainly for complexity reduction. For example, the traversal complexity of a tree is T D where T denotes the number of split types, and D is the depth of the tree.

[0148] 4. Primary transform examples

[0149] In some embodiments, such as in HEVC, 4-point DCT-2 transform, 8-point DCT-2 transform, 16-point DCT-2 transform, and 32-point DCT-2 transform are used as primary transforms. Figures 10A-10D The transform core matrices of 4-point DCT-2 transform, 8-point DCT-2 transform, 16-point DCT-2 transform, and 32-point DCT-2 transform are shown in FIGS. 3, 4, 5, and 6, respectively. The elements of those transform core matrices can be represented using 8-bit integers, so those transform cores are referred to as 8-bit transform cores. As shown, the transform core matrix of a smaller DCT-2 is a part of the transform core matrix of a larger DCT-2.

[0150] The DCT-2 core matrices show a symmetric / anti-symmetric property. Therefore, a so-called “partial butterfly” implementation can be supported to reduce the number of operations (multiplication, addition / subtraction, shift). A partial butterfly implementation can be used to obtain the same result as matrix multiplication.

[0151] 5. Additional primary transform examples

[0152] In some embodiments, in addition to the 4-point DCT-2 transform, 8-point DCT-2 transform, 16-point DCT-2 transform, and 32-point DCT-2 transform described above, additional 2-point DCT-2 and 64-point DCT-2 are used. Figures 11A-11E A 64x64 transform kernel matrix of the 64-point DCT-2 transform is shown.

[0153] In some embodiments, in addition to the DCT-2 and 4x4 DST-7 transforms, an adaptive multi-kernel transform (AMT) (also referred to as an enhanced multi-kernel transform (EMT) or multi-kernel transform selection (MTS)) is used to residue encode inter-coded blocks and intra-coded blocks. In addition to the DCT-2 transform, the AMT uses multiple transforms selected from the discrete cosine transform (DCT) / discrete sine transform (DST) family, such as the transform kernel matrix of the DST-7 or DCT-8 transform. Figure 12 A transform basis function of the selected DST / DCT transform is shown.

[0154] In some embodiments, the DST / DCT transform kernel matrices used in the AMT are represented with 8-bit representation. In some embodiments, the AMT is applied to CUs with width and height less than or equal to 32. Whether to apply the AMT can be controlled by a flag (e.g., mts_flag). For example, when mts_flag is equal to 0, only DCT-2 is applied to encode the residual block. When mts_flag is equal to 1, a 2bin can be further used to signal an index (e.g., mts_idx) to specify the horizontal transform and vertical transform to be used.

[0155] Figure 13 A mapping table (1300) between the index (e.g., mts_idx) values and the corresponding horizontal or vertical transform is shown. The row (1301) with mts_idx value of -1 corresponds to the case when the flag (e.g., mts_flag) is equal to 0 and the DCT-2 transform is used. The rows (1302)-(1305) with mts_idx values of 0, 1, 2, or 3 correspond to the case when mts_flag is equal to 1. In the right two columns of the table (1300), 0 indicates the DCT-2 transform type, 1 indicates the DST-7 transform type, and 2 indicates the DCT-8 transform type.

[0156] Figures 14A-14D A transform kernel matrix of the DST-7 transform is shown. Figures 15A-15D A transform kernel matrix of the DCT-8 transform is shown.

[0157] 6. Intra sub-partition (ISP) coding mode

[0158] In some embodiments, an ISP coding mode is employed. In ISP coding mode, the luma intra-prediction block can be divided vertically or horizontally into 2 or 4 sub-partitions. The number of sub-partitions can depend on the block size. Figure 16 The number of sub-partitions depends on the block size. Figure 17 An example of dividing a 4×8 or 8×4 block into two sub-partitions is shown. Figure 18 An example is shown where a block larger than 4×8 or 8×4 is divided into four sub-partitions. In one example, all sub-partitions satisfy the condition of having at least 16 samples. In one example, ISP is not applied to the chroma component.

[0159] In one example, for each sub-partition within a sub-partition divided from the coded block, a residual signal is generated by entropy decoding of the coefficients sent by the encoder, followed by inverse quantization and inverse transform of these coefficients. Intra-frame prediction is then performed on the first sub-partition to generate a prediction signal. This prediction signal is added to the corresponding residual signal of the first sub-partition to obtain the corresponding reconstructed sample. The reconstructed sample values ​​of the first sub-partition can then be used to generate predictions for the second sub-partition. This process can be repeated sub-partition by sub-partition until all sub-partitions of the coded block have been reconstructed. In one example, all sub-partitions share the same intra-frame mode.

[0160] In one embodiment, the ISP coding mode is tested only using intra-frame modes that are part of the most probable mode (MPM) list. Therefore, if a block uses ISP, the MPM flag can be inferred to be 1. Furthermore, when ISP is used for a block, the corresponding MPM list is modified to exclude DC modes, and the horizontal intra-frame modes split horizontally by ISP and the vertical intra-frame modes split vertically by ISP are prioritized.

[0161] In ISP encoding mode, each sub-partition can be treated as a TU because the transformation and reconstruction are performed separately for each sub-partition.

[0162] Figures 19A-19B An example of a syntax element (1900) for ISP encoding mode is shown. As shown in box (1910), a syntax element (e.g., intra_subpartitions_mode_flag) indicates whether ISP is used. A syntax element (e.g., intra_subpartitions_split_flag) indicates the direction of the split (vertical or horizontal).

[0163] 7. Sub-block transform (SBT)

[0164] In some embodiments, SBT, also known as spatially varying transform (SVT), is employed. SBT can be applied to inter prediction residual. In some examples, the residual block is included in the coding block and is smaller than the coding block. Thus, the transform size in SBT is smaller than the coding block size. The area not covered by the residual block can be assumed to be zero residual, thus no transform processing is applied to that area.

[0165] Figures 20A-20D The supported sub-block types (SVT-H, SVT-V) in SBT (e.g., horizontal split or vertical split), sizes, and positions (e.g., left half, left quarter, right half, right quarter, top half, top quarter, bottom half, bottom quarter) are shown. The shaded area marked with the letter “A” is the transformed residual block, while the other areas are assumed to be zero residual with no transform applied.

[0166] As an example, Figures 21A-21I Changes to the specification text of the video coding standard (i.e., VVC) when SBT is used are shown. The added text is shown in boxes (2101) to (2113). As shown, additional syntax elements, such as the additional bits cu_sbt_flag, cu_sbt_quad_flag, cu_sbt_horizontal_flag, and cu_sbt_pos_flag, can be signaled to indicate the sub-block type (horizontal or vertical), size (half or quarter), and position (left, right, top, or bottom), respectively.

[0167] 8. YUV format

[0168] Figure 22 Different YUV formats (e.g., 4:4:4, 4:2:2, 4:1:1, and 4:2:0) used in some embodiments are shown. In one example, cross component linear model intra prediction is used for 4:2:0 format. As shown, Figure 22 A six-tap interpolation filter can be applied to obtain the downsampled luma sample corresponding to the chroma sample, in one example, the downsampled luma sample Rec L [x,y] represents) is calculated as follows: L [x,y] represents) is calculated as follows:

[0169] Rec L [x,y] = (2 x Rec L [2x,2y] + 2 x Rec L [2x,2y+1] +

[0170] Rec L [2x-1,2y]+Rec L [2x+1,2y]+

[0171] Rec L [2x-1,2y+1]+Rec L [2x+1,2y+1]+4)>>3

[0172] Downsampled luma samples Rec' L [x,y] can be used to predict chroma samples by using the inter-component linear model mode.

[0173] 9. Virtual Pipeline Data Unit (VPDU)

[0174] A Virtual Pipeline Data Unit (VPDU) can be defined as a non-overlapping MxM-luma (L) / NxN-chroma (C) unit in a picture. In some hardware decoder implementations, consecutive VPDU are processed simultaneously by multiple pipeline stages. Different stages process different VPDU simultaneously. The size of a VPDU can be roughly proportional to the buffer size in the pipeline stage, with the expectation that the VPDU size is kept to a certain size (e.g., 64x64 or smaller). In certain decoders, the VPDU size is set to the maximum Transform Unit (TU) size. The expansion of the maximum TU size from 32x32-L / 16x16-C in HEVC to 64x64-L / 32x32-C in current VVC brings coding gain, which is expected to result in 4x VPDU size compared to HEVC. However, the BT and TT structures adopted in VVC for achieving additional coding gain can be recursively applied to 128x128-L / 64x64-C coding tree blocks, which results in 16x VPDU size (128x128-L / 64x64-C) compared to HEVC.

[0175] Figure 23 Certain disallowed TT and BT partitions are shown.

[0176] In order to keep the VPDU size to 64x64 luma samples, certain (with syntax signaling modification) partitioning constraints are applied in some embodiments.

[0177] - TT split is not allowed for CUs with width or height, or both width and height equal to 128.

[0178] - For 128xN CUs with N < 64 (i.e., width equal to 128 and height less than 128), horizontal BT split is not allowed.

[0179] - For Nx128 CUs with N < 64 (i.e., height equal to 128 and width less than 128), no vertical BT split is allowed.

[0180] III. Transform block partitioning and processing techniques

[0181] In some embodiments, a fixed maximum allowed transform unit (TU) size or maximum TU size (e.g., 64x64 pixels or samples) is used. In some embodiments, a controllable or configurable maximum TU size is used because the maximum TU size affects hardware complexity, e.g., affects encoder implementation (e.g., pipeline intermediate buffer size, number of multipliers, etc.). For example, in addition to a 64x64 sample size, the maximum TU size can be other sizes, e.g., 32x32 sample size, 16x16 sample size, etc.

[0182] In some video standards, SBT and ISP can be used. For example, in SBT, a SPS flag (e.g., sps_sbt_max_size_64_flag) is signaled to indicate whether the maximum SBT size is 32- or 64-length. When sps_sbt_max_size_64_flag is true (i.e., the maximum SBT size is 64-length) and the maximum TU size is 32-point, an encoder crash can be triggered. In general, L-length or L-point size refers to the maximum size of a CU, a TU, a CB, a TB, a VPDU, etc. For example, when the maximum TU size is 32-point or 32-length, the width and height of a TU is less than or equal to 32.

[0183] In some embodiments, various CU sizes are allowed to have ISP mode, however, when the maximum TU size is set to be less than 64, a conflict occurs whether to perform implicit transform split or to perform explicit transform split using ISP with signaling. For example, when the maximum TU size is 16, for a 64x16 CU, without using ISP, the CU can be implicitly split into four 16x16 TUs. With ISP, a vertical ISP can be used to partition the 64x16 CU, thus the CU can be split into four 16x16 TUs with signaling.

[0184] When the maximum TU size is less than 64x64, the TU processing order needs to be aligned with the implementation of VPDU.

[0185] The embodiments described herein can be used alone or in any combination in any order. Furthermore, these embodiments can be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits) in an encoder, a decoder, etc. In one example, the one or more processors execute a program that is stored in a non-transitory computer-readable medium.

[0186] In this disclosure, a high-level syntax (HLS) element can refer to a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a slice header, a tile header, a tile group header, etc. A CTU header can refer to one or more signaled syntax elements for a CTU, e.g., as header information. In one example, a CTU size is a maximum CU size.

[0187] Generally, when a luma size (specified by luma samples) of a certain unit (e.g., a TU, a CU) is known, a corresponding chroma size specified by multiple chroma samples can be obtained. In one example, a YUV format of 4:2:0 is used, and a size of a CU is 64x64 luma samples (or 64x64-L). Thus, a size of the CU is 32x32 chroma samples (or 32x32-C). The CU size can be 64x64-L, 32x32-C, or 64x64-L / 32x32-C. Similarly, a size of a TU is 64x64 luma samples (or 64x64-L). Thus, a size of the TU is 32x32 chroma samples (or 32x32-C). The TU size can be 64x64-L, 32x32-C, or 64x64-L / 32x32-C. For example, a TU includes one luma transform block (TB) and two chroma TBs. The luma TB has a size of 64x64-L, and each of the chroma TBs has a size of 32x32-C. Generally, the embodiments and methods described for a CU or a TU can apply to CBs and TBs, respectively.

[0188] A CU can include one 64x64-L luma block and two 32x32-C chroma blocks. In the following description, a TU size is represented using luma samples in the TU. For example, a maximum TU size of M samples refers to a maximum TU size of M luma samples. Similarly, other sizes such as a VPDU size and a CU size are also represented using corresponding luma samples in the respective units (e.g., VPDU and CU). Of course, a TU size, a VPDU size, a CU size, etc. can also be represented using chroma samples or a combination of luma samples and chroma samples.

[0189] The size of a unit can refer to the width, height, or area of the unit. For example, the maximum TU size can refer to the width, height, or area of the maximum TU. In general, TUs, CUs, VPDU, and the like can have any suitable shape, including a rectangular shape, a square shape, an "L" shape, or any suitable shape. When the shape of a unit is irregular (e.g., an "L" shape), the unit size can refer to the area of the unit.

[0190] In some embodiments, the VPDU size and / or the maximum TU size can be signaled in the coded video bitstream (e.g., in the SPS and PPS). As described above, the VPDU size and / or the maximum TU size can be signaled in the form of luma samples. Alternatively, the VPDU size and / or the maximum TU size can be signaled in the form of chroma samples.

[0191] In some embodiments, the VPDU size and / or the maximum TU size can be stored in the encoder and / or the decoder, and thus the VPDU size and / or the maximum TU size is not signaled. In one example, the VPDU size and / or the maximum TU size can be stored in the profile and / or tier definition. The VPDU size and / or the maximum TU size can be stored in the form of luma samples or chroma samples.

[0192] In some embodiments, multiple VPDU share the same size, but can have different shapes. For example, when the VPDU size is 4096 in the form of luma samples, the shape of the VPDU can be a square shape of 64x64 or a rectangular shape of 32x128. The VPDU can also have other shapes, such as an L shape, as long as the VPDU size is 4096 in the form of luma samples. The above description also applies to certain TUs.

[0193] 1. Example A

[0194] According to aspects of this disclosure, the maximum allowable TU size (also referred to as the maximum TU size) is M samples (e.g., a size of MxM samples). In one example, the maximum width and maximum height of a TU is M. In one example, the maximum area of a TU is MxM. The processing data unit size (e.g., the VPDU size) is K samples (e.g., a size of KxK samples). In one example, the maximum width and maximum height of the processing data unit size is K. In one example, the maximum area of the processing data unit size is KxK. A CU of WxH has a width of W samples and a height of H samples. The CU can be partitioned into multiple sub-units (referred to as sub-processing units (SPUs)) based on the CU size and the processing data unit size K. The CU can be partitioned into the SPU using any suitable partitioning structure or a combination of any suitable partitioning structures (e.g., QTBT, QT, BT, TT, or a combination thereof). The SPU can have the same size or different sizes.

[0195] In one embodiment, when the width W or the height H of a CU is greater than K, the CU is partitioned into multiple SPU. In one example, the SPU has the same size (i.e., SPU size) and the size of each SPU is Min(W, K) x Min(H, K) samples. Thus, the width of each SPU is the minimum of W and K, and the height of the SPU is the minimum of H and K. In some examples, before partitioning a CU, a determination can be made whether to partition the CU based on the size of the CU and the processing data unit size K.

[0196] The SPU in a CU can be further partitioned into TUs whose size is, for example, MxM samples. In some examples, the SPU can be partitioned into TUs whose size is Min(W, K, M) x Min(H, K, M). In some examples, before partitioning a SPU, a determination can be made whether to partition the SPU based on the size of the SPU and the maximum TU size M. The SPU can be partitioned using any suitable partitioning structure or a combination of any suitable partitioning structures (e.g., QTBT, QT, BT, TT, or a combination thereof). According to aspects of this disclosure, one or more partitioning structures for partitioning a SPU can be determined based on the size of the SPU and the maximum TU size M. In one example, the SPU can be recursively split into TUs using the determined one or more partitioning structures.

[0197] In one example, when the width and height of a SPU is greater than the maximum TU size M, the SPU is split into MxM TUs using a quad-tree partitioning structure. The SPU can be recursively split into TUs using the quad-tree partitioning structure.

[0198] In one example, when the width of the SPU is greater than M and the height of the SPU is equal to M, the SPU is split into MxM TUs using a vertical binary tree splitting structure. For example, M is 32 and the size of the SPU is 64x32. Thus, the width of the SPU is 64 and the height of the SPU is 32. Thus, the SPU can be split into two 32x32 TUs using the vertical binary tree splitting structure. The SPU can be recursively split into TUs using the vertical binary tree splitting structure.

[0199] In one example, when the width of the SPU is greater than M and the height of the SPU is less than M, the SPU can be split into TUs using a vertical binary tree splitting structure, where the width of the TUs is M and the height is equal to the height of the SPU.

[0200] In one example, when the height of the SPU is greater than M and the width of the SPU is equal to M, the SPU is split into MxM TUs using a horizontal binary tree splitting structure. The SPU can be recursively split into TUs using the horizontal binary tree splitting structure. For example, M is 32 and the size of the SPU is 32x64. Thus, the width of the SPU is 32 and the height of the SPU is 64. Thus, the SPU can be split into two 32x32 TUs using the horizontal binary tree splitting structure.

[0201] When the height of the SPU is greater than M and the width of the SPU is less than M, the SPU can be split into TUs using a horizontal binary tree splitting structure, where the height of the TUs is M and the width is equal to the width of the SPU.

[0202] Figure 24 The transform tree syntax in VVC shows an example of splitting a SPU and the processing order for processing the TUs.

[0203] In one example, a CU of WxH can be split in two steps. In the first step, the CU is split into SPUs, where each SPU has a size of Min(W, K) x Min(H, K). Subsequently, in the second step, each SPU is further split into TUs, where each TU has a size of MxM.

[0204] When processing the TUs in a CU, the SPUs in the CU can be scanned and processed in a first scan order (also referred to as a first order). Further, within each SPU, the TUs can be scanned and processed in a second scan order (also referred to as a second order).

[0205] In various embodiments, the first order for processing the SPUs can be a raster scan order, a vertical scan order (e.g., scanning the SPUs in columns from left to right or scanning the SPUs in columns from right to left), a zig-zag order, a diagonal scan order, etc.

[0206] In various embodiments, the second order for processing the TUs in each SPU can be a raster scan order, a vertical scan order (e.g., scanning the TUs from left to right by column or from right to left by column), a zigzag order, a diagonal scan order, etc.

[0207] In different embodiments, the first order and the second order can be the same or different. For example, in one embodiment, the first order for processing the SPUs and the second order for processing the TUs in each SPU are both a raster scan order.

[0208] 2、Example B

[0209] Figure 25 A CU (2510) with a sample size of WxH is shown, where W=128 and H=64. The maximum TU size M is 32 samples. The processing data unit size K (e.g., a VPDU size, etc.) is 64 samples. The CU (2510) is first split into a first 64x64 SPU (2520) and a second 64x64 SPU (2530). The first SPU (2520) and the second SPU (2530) can then be further divided into TUs 0-7, each with a MxM sample size. TUs 0-3 are included in the first SPU (2520) and TUs 4-7 are included in the second SPU (2530).

[0210] The first SPU (2520) can be processed first according to a first order, and then the second SPU (2530) is processed. Within the first SPU (2520) or the second SPU (2530), a second order for processing TUs 0-3 or TUs 4-7 is a raster scan order. Thus, TUs 0-7 are processed according to the order indicated by arrows (2551). The first order and / or the second order can be determined explicitly (e.g., through signaling from an encoder to a decoder) or implicitly.

[0211] In some examples, dividing a CU into SPUs, where each of the SPUs further includes TUs as described above, can improve coding efficiency. Referring to Figure 25 In one example, the first SPU (2520) is a first VPDU and the second SPU (2530) is a second VPDU. Each of the first VPDU (or the first SPU (2520)) and the second VPDU (or the second SPU (2530)) can sequentially pass through a multi-stage pipeline including a first stage (e.g., entropy decoding), a second stage (e.g., dequantization), a third stage (e.g., inverse transform), etc. According to the multi-stage pipeline, the first SPU (2520) and the second SPU (2530) can be processed in parallel. Figure 25The first SPU (2520) is processed before the second SPU (2530) in the first order shown. Thus, the first SPU (2520) is processed by the first stage and then enters the second stage. In one example, while the second stage processes the first SPU (2520), the first stage processes the second SPU (2530) to improve encoding efficiency. Subsequently, the first SPU (2520) enters the third stage, and the second SPU (2530) can move to the second stage. While the third stage processes the first SPU (2520), the second stage can process the second SPU (2530). The above description is given using VPDUs and a multi-stage pipeline as an example, and can be adapted to other architectures or video encoding methods. The above description can be adaptively modified when the first SPU (2520) is included in a first VPDU and the second SPU (2530) is included in a second VPDU. At least a portion of the SPU processing in different stages can be performed concurrently.

[0212] As described above, when the SPU size is larger than the TU size, multiple TUs in a CU can be grouped into processing data units such as SPUs (or VPDUs), which can be processed in a multi-stage pipeline that allows parallel processing (or concurrent processing) of consecutive SPUs. In some examples, the above description can be modified as follows: a CU is divided into first units. In addition, each unit in the first units can be divided into second units. Each unit in the second units can be divided into third units. In one example, the size of the first units is larger than the size of the second units, and the size of the second units is larger than the size of the third units. Such division is advantageous when the first multi-stage pipeline is nested within the second multi-stage pipeline.

[0213] 3. Example C

[0214] Figure 26A A CU (2610A) with a size of WxH samples is shown, where W=128 and H=32. The maximum TU size M is 16 samples. The processing data unit size K (e.g., VPDU size, etc.) is 64 samples. The minimum of W and K is 64, and the minimum of H and K is 32. Thus, the size of the SPUs can be determined to be 64x32 samples to, for example, align the transform blocks with the VPDUs. The CU (2610A) can be divided into a left SPU (2620A) and a right SPU (2630A), each with a size of 64x32 samples. The two SPUs (2620A, 2630A) can be scanned and processed in a left-to-right order.

[0215] Each of the two SPUs (2620A, 2630A) can be further divided into TUs, each having a maximum TU size of 16x16 samples. As shown, the left SPU (2620A) is divided into TUs 0-7, and the right SPU (2630A) is divided into TUs 8-15. In SPU (2620A), TUs 0-7 can be processed in raster scan order. In SPU (2630A), TUs 8-15 can be processed in raster scan order. Thus, TUs 0-15 can be scanned and processed in the order shown by arrows (2651A), where TU 0 is processed first, and TU 15 is processed after TUs 0-14 are processed.

[0216] 4. Example D

[0217] Figure 26B A CU (2610B) having a WxH sample size is shown, where W=128 and H=32. The maximum TU size M is 16 samples. The processing data unit size K (e.g., VPDU size, etc.) is 64 samples. In a manner similar to that of Figure 25 Example D, the CU (2610B) can be divided into two SPUs (2620B, 2630B), each of which can be further divided into TUs. The SPUs (2620B, 2630B) can be processed in the same order from left to right as in Figure 25 Example D. However, unlike in Figure 25 Example D, TUs 0-7 in SPU (2620B) are processed in a zigzag scan order, and TUs 8-15 in SPU (2630B) are processed in a zigzag scan order.

[0218] 5. Example E

[0219] Figure 27 A flowchart outlining a transform block division and processing process (2700) according to embodiments of the present disclosure is shown. The process (2700) can be used to reconstruct a block coded in intra mode or inter mode. In various embodiments, the process (2700) is performed by processing circuitry, such as the processing circuitry in the terminal device (210), the terminal device (220), the terminal device (230), and the terminal device (240), processing circuitry performing functions of the video encoder (403), processing circuitry performing functions of the video decoder (310), processing circuitry performing functions of the video decoder (410), processing circuitry performing functions of the video encoder (603), and the like. In some embodiments, the process (2700) is implemented in software instructions, and thus, when the processing circuitry executes the software instructions, the processing circuitry performs the process (2700). The process starts at (S2701), and continues to (S2710).

[0220] At (S2710), encoded information of a CU in a picture can be decoded from an encoded video bitstream. The encoded information is used to indicate a width of W samples and a height of H samples of the CU.

[0221] At (S2720), the CU can be divided into SPUs, for example, when at least one of the width W and the height H of the CU is greater than a processing data unit size K, for example, as described with reference to Figure 24 -26. A size of the SPU can be determined based on the processing data unit size K and a size of the CU. A width of the SPU can be a minimum of W and K, and a height of the SPU can be a minimum of H and K. Thus, the CU can be divided into SPUs, where each SPU has the determined width and height. For example, when W is 128, H is 64, and K is 64, the CU can be split into a first SPU of 64x64 and a second SPU of 64x64. For example, the processing data unit can be a VPDU, and thus K can be a VPDU size.

[0222] At (S2730), one or more partition structures to partition each of the SPUs can be determined, for example, based on one or a combination of the width and the height of the SPU and a maximum TU size of M samples. In one example, at least one of the width and the height of the SPU is greater than M.

[0223] As described above, any suitable partition structure can be used to partition or split each of the SPUs. In one example, when the width and the height of the SPU are greater than M, the one or more partition structures are determined to be quad-tree partition structures. In one example, when the width of the SPU is greater than M and the height of the SPU is not greater than M, the one or more partition structures are determined to be vertical binary-tree partition structures. In one example, when the height of the SPU is greater than M and the width of the SPU is not greater than M, the one or more partition structures are determined to be horizontal binary-tree partition structures.

[0224] At (S2740), each of the SPUs can be partitioned into TUs based on the determined one or more partition structures. In one example, the respective SPU can be recursively partitioned into TUs using the determined one or more partition structures.

[0225] At (S2750), the TUs of the SPUs are processed according to a processing order. For example, the SPUs can be processed according to a first order as described above, and the TUs in each SPU can be processed according to a second order. Residual data of each TU can be determined through various decoding operations, such as entropy decoding of transform coefficients, inverse quantization or dequantization, inverse transform, etc. The process (2700) can continue to (S2799) and end.

[0226] The process (2700) is described using a CU as an example. The process (2700) can also be applied to a CB, e.g., a luma block, a chroma block, etc. For brevity, description about a CB is omitted.

[0227] The process (2700) can be modified adaptively. For example, one or more operations therein can be modified, omitted, or combined. For example, operations (S2730) and (S2740) can be combined into a single operation. One or more further operations can also be added. The execution order of the process (2700) can also be modified.

[0228] IV. Computer System

[0229] The above-described techniques can be implemented as computer software using computer-readable instructions and physically stored in one or more computer-readable media. For example, Figure 28 A computer system (2800) suitable for implementing certain embodiments of the disclosed subject matter is shown.

[0230] The computer software can be coded using any suitable machine code or computer language that can be subject to assembly, compilation, linking, or similar

[0231] The instructions can be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, internet of things devices, and the like.

[0232] Figure 28 The components of the computer system (2800) shown are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. Neither should the configuration of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of a computer system (2800).

[0233] Computer system (2800) can include certain human interface input devices. Such a human interface input device can be responsive to user input data being in the form of physical motion, physical interaction, and / or physical sounds. Accordingly, computer system (2800) can include a keyboard (2801), mouse (2802), touchpad (2803), microphone (2806), camera (2808), and / or display screen (2810) as certain human interface input devices.

[0234] Input human interface devices can include one or more of: a keyboard (2801), mouse (2802), trackpad (2803), touchscreen (2810), data g love (not depicted), joystick (2805), microphone (2806), scanner (2807), camera (2808).

[0235] Computer system (2800) can also include certain human interface output devices. Such human interface output devices can be stimulating one or more of the human senses of sight, touch, taste, smell, and / or hearing. For example, such human interface output devices can include a display screen (2810) (which can also function as an input device) of the computer system (2800), a speaker (2809) (which can also function as an input device), a printer (not depicted), and / or tactile feedback devices (not depicted). Such human interface output devices can also include non-visual indicators, such as audio commands, and / or tactile feedback indicators.

[0236] Computer system (2800) can also include human accessible storage devices and their associated media and / or computer readable media, such as hard disk drives, floppy disks, read-only memory, and read / write memory, as well as input / output interfaces (not depicted) for those devices. Optical media, such as compact discs and magnetic media, such as magnetic tape and floppy disks, are other examples of storage devices and their associated media.

[0237] Those skilled in the art will further appreciate that the term“computer-readable medium” used in connection with the presently disclosed subject matter does not encompass transitory signals per se.

[0238] Computer system (2800) can also include an interface to one or more communication networks. Networks can for example be wireless networks, wire-line networks, optical networks. Networks can further be local networks, wide-area networks, metropolitan area networks, vehicular and industrial networks, real-time networks, delay-tolerant networks, and others. Examples of networks include local area networks such as Ethernet networks, wireless LANs, cellular networks to include GSM, 3G, 4G, 5G, LTE and the like, TV wireline or wireless wide area digital networks to include cable networks, satellite TV and terrestrial TV networks, vehicular and industrial to include CANBus, and others. Some networks commonly require various permissions, licenses, subscriptions and / or fees to access them. Networks can be directed to access various services, including internet access; wired or wireless telephone or telecommunication networks; and the like. Networks can be private, open to certain subscribers or groups of subscribers, or public.

[0239] The above human interface devices, human-accessible storage devices, and network interfaces can be attached to the core (2840) of computer system (2800).

[0240] The core (2840) can include one or more Central Processing Units (CPU) (2841), Graphics Processing Units (GPU) (2842), specialized programmable processing units in the form of Field Programmable Gate Areas (FPGA) (2843), hardware accelerators for certain tasks (2844), and so forth. These devices, along with Read-only memory (ROM) (2845), Random-access memory (2846), internal mass storage such as internal non-user accessible hard drives, SSDs, and so forth (2847), can be connected through a system bus (2848). In some computer systems, the system bus (2848) can be accessible through one or more physical plugs to enable extensions by additional CPUs, GPU, and so forth. The peripheral devices can be attached either directly to the core’s system bus (2848) or through a peripheral bus (2849). Architectures for a peripheral bus include PCI, USB, and so forth.

[0241] CPUs (2841), GPUs (2842), FPGAs (2843), and accelerators (2844) can execute certain instructions that, in combination, can make up the aforementioned computer code. That computer code can be stored in ROM (2845) or RAM (2846). Transitional data for the CPU (2841) or GPU (2842) can be stored in RAM (2846), while permanent data can be stored for example, in the internal mass storage (2847). Fast storage and retrieval can be provided to any of these by using cache memory, which can be closely associated with one or more CPU (2841), GPU (2842), mass storage (2847), ROM (2845), RAM (2846), and so forth.

[0242] The computer software can be implemented as computer code that is stored on a computer readable medium, which can be any medium that is readable by a machine. The computer software can be implemented in a variety of programming languages, including but not limited to assembly language, machine language, high-level procedural, high-level object-oriented, and so forth. The computer software can be implemented in compiled or interpreted form. The computer software can be implemented using a variety of media that is readable by a machine, including but not limited to volatile or non-volatile media, removable or non-removable media, and so forth. The computer software can be stored on a computer readable medium that is internal or external to a computer.

[0243] As a non-limiting example, a computer system with architecture (2800), and specifically the core (2840), can provide functionality as a result of processor(s) (including CPUs, GPUs, FPGA, accelerators, and the like) executing software embodied in one or more tangible, computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as introduced above, as well as certain high-speed caches (e.g., that are typically internal to the core (2840) and / or accessible only on a high-speed bus such as a system bus or an internal memory bus). Such software can include instructions that, when executed, enable the core (2840) to perform a process or implement a portion of a process, including a process or a portion of a process as described herein. Additionally or alternatively, such software can include instructions that, when executed, enable a processor to generate electrical signals that can ultimately cause a computing device to perform a process or implement a portion of a process, including a process or a portion of a process as described herein. Of course, combinations of hardware and software can be used to implement the processes described herein. It should also be noted that such software can when executed, enable the computer system 2800 to be transformed from a general-purpose computing device into a special-purpose computing device that is customized to facilitate the practice of a process as described herein.

[0244] Appendix A: Abbreviations

[0245] ASIC: application specific integrated circuit

[0246] BMS: base set

[0247] CANBus: controller area network bus

[0248] CBF: coding block flag

[0249] CD: compact disc

[0250] CPU: central processing unit

[0251] CRT: cathode ray tube

[0252] CTB: coding tree block

[0253] CTU: coding tree unit

[0254] CU: coding unit

[0255] DVD: digital video disc

[0256] FPGA: field programmable gate area

[0257] GOP: group of pictures

[0258] GPU: graphics processing unit

[0259] GSM: global system for mobile communications

[0260] HEVC: high efficiency video coding

[0261] HRD: hypothetical reference decoder

[0262] ISP: intra sub-partition

[0263] IC: integrated circuit

[0264] JEM: joint exploration model

[0265] LAN: local area network

[0266] LCD: liquid crystal display

[0267] LTE: long term evolution

[0268] MPM: most probable mode

[0269] MV: motion vector

[0270] OLED: organic light emitting diode

[0271] PB: prediction block

[0272] PCI: peripheral component interconnect

[0273] PLD: programmable logic device

[0274] PU: prediction unit

[0275] RAM: random access memory

[0276] ROM: read only memory

[0277] SBT: sub-block transform

[0278] SEI: supplemental enhancement information

[0279] SNR: signal to noise ratio

[0280] SSD: solid state drive

[0281] TU: transform unit

[0282] USB: universal serial bus

[0283] VPDU: virtual pipeline data unit

[0284] VUI: video usability information

[0285] VVC: Versatile Video Coding

[0286] While the present disclosure has described a number of exemplary embodiments, there are alterations, modifications, and various further equivalents, which fall within the scope of the present disclosure. Accordingly, it should be understood that the systems and methods, while the best mode presently known for carrying out the present disclosure, are susceptible to modification, change and alteration. Therefore, the foregoing description should be taken as illustrative and not in a limiting sense.

Claims

1. A method for video decoding, comprising: Decode the encoding information of coded blocks in an image from an encoded video stream, wherein the encoding information indicates that the width of the coded block includes W samples and the height includes H samples; The coded block is divided into sub-processing units, the width of which is the minimum of W and K, and the height of which is the minimum of H and K. At least one of the width W and height H of the coded block is greater than the processing data unit size K. The sub-processing unit is a Virtual Pipeline Data Unit (VPDU). Get the first syntax element IntraSubPartSplitType; When it is determined that the first syntax element IntraSubPartSplitType is equal to the value NO_ISP_SPLIT used to represent the non-intra-frame sub-partition ISP partitioning mode, it is determined whether at least one of the width and height of the sub-processing unit is greater than 64. When at least one of the width and height of the sub-processing unit is determined to be greater than 64, the maximum transformation unit size M is determined to be 64; When the width and height of the sub-processing unit are determined to be no greater than 64, the maximum transformation unit size M is determined to be MaxTbSizeY, where MaxTbSizeY can take values ​​of 32 or 16. Based on the width and height of the sub-processing unit and the size M of the maximum transformation unit, a partitioning structure for partitioning the sub-processing unit is determined from multiple partitioning structures; Based on the determined partitioning structure, the sub-processing unit is divided into multiple transformation units.

2. The method according to claim 1, characterized in that, Based on the determined partitioning structure, the sub-processing unit is recursively divided into multiple transformation units, including: When the width and height of the sub-processing unit are both greater than M, the determined partitioning structure is a quadtree partitioning structure. When the width of the sub-processing unit is greater than M and the height of the sub-processing unit is not greater than M, the determined partitioning structure is a vertical binary tree partitioning structure. When the width of the sub-processing unit is not greater than M and the height of the sub-processing unit is greater than M, the determined partitioning structure is a horizontal binary tree partitioning structure; and Based on the determined partitioning structure, the sub-processing unit is recursively divided into multiple transformation units until the size of each transformation unit is less than or equal to M×M.

3. The method according to claim 1, wherein, The MaxTbSizeY is 32; The width of the sub-processing unit is 64, and the height of the sub-processing unit is equal to M; Determining the partitioning structure includes: defining the partitioning structure as a vertical binary tree partitioning structure; and Recursively dividing the sub-processing unit into multiple transformation units includes: dividing the sub-processing unit into two transformation units based on the vertical binary tree partitioning structure, with each transformation unit having a size of 32x32.

4. The method according to claim 1, wherein: The MaxTbSizeY is 32; The height of the sub-processing unit is 64, and the width of the sub-processing unit is equal to M; Determining the partitioning structure includes: defining the partitioning structure as a horizontal binary tree partitioning structure; and Recursively dividing the sub-processing unit into multiple transformation units includes: dividing the sub-processing unit into two transformation units based on the horizontal binary tree partitioning structure, with each transformation unit having a size of 32x32.

5. The method according to claim 1, wherein, The plurality of transformation units are processed according to one of the following scanning sequences: (i) raster scanning sequence; (ii) vertical scanning sequence; (iii) zigzag sequence; and (iv) diagonal scanning sequence.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: The first syntax element intra_subpartitions_mode_flag is received, indicating that the coded block is decoded using the intra-subpartitions mode; Upon receiving the first syntax element intra_subpartitions_mode_flag, a second syntax element intra_subpartitions_split_flag is further received to indicate whether the sub-processing unit is to be vertically or horizontally partitioned under the intra-subpartition mode. Determine the mapping relationship table between the first index value mts_idx, the first flag mts_flag and the corresponding horizontal or vertical transformation, wherein the first flag mts_flag is used to indicate whether to use adaptive multi-kernel transformation; When the first flag mts_flag is equal to 0, both the horizontal and vertical transformations are of the Discrete Cosine Transform (DCT-2) type. When the first flag mts_flag is equal to 1, the types of the horizontal transformation and the vertical transformation include at least one of the following: Both the horizontal transformation and the vertical transformation are of the DST-7 type. The horizontal transformation is of type DST-8 transformation, and the vertical transformation is of type DST-7 transformation; The horizontal transformation is of type DST-7 transformation, and the vertical transformation is of type DST-8 transformation; Both the horizontal and vertical transformations are of the DST-8 type, and the DST-8 transformation is implemented through a transformation core matrix, which includes any of the following types: Where {a, b, c, d} = {84, 74, 55, 29}; , where {a, b, c, d, e, f, g, h} = {86, 85, 78, 71, 60, 46, 32, 17}.

7. A video encoding method for generating a video bitstream, the method comprising: Determine the width of W samples and the height of H samples of the coded block in the image; The coded block is divided into sub-processing units, the width of which is the minimum of W and K, and the height of which is the minimum of H and K. At least one of the width W and height H of the coded block is greater than the processing data unit size K. The sub-processing unit is a Virtual Pipeline Data Unit (VPDU). Set the first syntax element IntraSubPartSplitType; When the first syntax element IntraSubPartSplitType is equal to the value NO_ISP_SPLIT used to represent the non-intra-frame sub-partition ISP partitioning mode, determine whether at least one of the width and height of the sub-processing unit is greater than 64. When it is determined that at least one of the width and height of the sub-processing unit is greater than 64, the maximum transformation unit size M is set to 64; When the width and height of the sub-processing unit are determined to be no greater than 64, the maximum transformation unit size M is set to MaxTbSizeY, where MaxTbSizeY can take values ​​of 32 or 16. Based on the width and height of the sub-processing unit and the maximum transformation unit size M, a partitioning structure for partitioning the sub-processing unit is set from multiple partitioning structures; Based on the established partitioning structure, the sub-processing unit is recursively divided into multiple transformation units.

8. A video decoding apparatus, comprising a processing circuit, characterized in that, The processing circuit is configured to perform decoding using the decoding method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform decoding according to any one of claims 1 to 6, or to generate a video stream according to the video encoding method of claim 7.

10. A method for storing a bitstream, characterized in that, A video stream is stored on a non-transitory computer-readable storage medium, the video stream being generated by the video encoding method according to claim 7.

11. A method for transmitting video streams, characterized in that, The video encoding method of claim 7 is used to generate a video stream and transmit the video stream.

Images