Cross-channel prediction based on multiple prediction modes

Abstract

This disclosure describes a set of advanced video coding techniques. More specifically, the disclosed techniques include cross-channel prediction, in particular chroma from luma (CfL) prediction. In some example implementations, samples of chroma channels are predicted based on a weighted sum or weighted average of AC prediction contributions derived based on reconstructed samples of the luma channel and DC prediction contributions generated based on intra prediction of samples of the chroma channels based on reconstructed samples of the chroma channels, the intra prediction being performed based on an intra prediction mode other than the DC mode.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. non-provisional patent application No. 18 / 053,150, entitled "Cross-Channel Prediction Based on Multiple Prediction Modes," filed on November 7, 2022, which in turn claims the benefit of priority to U.S. provisional patent application No. 63 / 299,221, entitled "Multiple Prediction Mode Based CfL Mode," filed on January 13, 2022, each of which is incorporated by reference in its entirety herein.

[0002] This disclosure describes a set of advanced video coding techniques. More specifically, the techniques disclosed involve cross-channel prediction, in particular chroma from luma (CfL) prediction. [Background technology]

[0003] This background discussion provided herein is intended to generally present the context of the present disclosure. The work of the inventors cited herein is not admitted, expressly or impliedly, as prior art to the present disclosure, as are aspects of the description that may not have been admitted as prior art at the time of filing of this application, to the extent that their work is described in this background section.

[0004] Video encoding and decoding may be performed using inter-picture prediction with motion compensation. Uncompressed digital video may include a sequence of pictures, each having spatial dimensions of, for example, 1920x1080 luma samples and associated full subsampled or subsampled chroma samples. The sequence of pictures may have a fixed or variable picture rate (alternatively called frame rate), for example, 60 pictures / second or 60 frames / second. Uncompressed video has specific bit rate requirements for streaming or data processing. For example, a video with a pixel resolution of 1920x1080, a frame rate of 60 frames / second, and 4:2:0 chroma subsampling with 8 bits per pixel per color channel requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires more than 600 gigabytes of storage space.

[0005] One objective of video encoding and decoding may be to reduce redundancy in an uncompressed input video signal due to compression. Compression may help reduce the aforementioned bandwidth and / or storage space requirements, in some cases by more than two orders of magnitude. Both lossless and lossy compression, as well as combinations thereof, may be employed. Lossless compression refers to techniques where an exact copy of the original signal can be reconstructed from the compressed original signal via a decoding process. Lossy compression refers to a coding / decoding process where the original video information is not fully preserved during coding and cannot be fully restored during decoding. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals will be small enough to make the reconstructed signal useful for its intended application, even with some information loss. For video, lossy compression has been widely adopted in many applications. The amount of tolerable distortion depends on the application. For example, a user of a particular consumer video streaming application may tolerate higher distortion than a user of a movie or television broadcast application. The compression ratio achievable by a particular coding algorithm can be selected or adjusted to reflect different distortion tolerances, and generally, higher distortion tolerances allow for coding algorithms that result in higher losses and higher compression ratios.

[0006] Video encoders and decoders can utilize techniques from a number of broad categories and steps, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

[0007] Video codec technology can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. If all blocks of samples are coded in intra mode, the picture may be called an intra-picture. Intra-pictures and their derived pictures, such as independent decoder refresh pictures, may be used to reset the decoder state and thus may be used as the first picture in a coded video bitstream and video session or as a still image. Samples of the block after intra prediction may then be transformed to the frequency domain, and the transform coefficients so generated may be quantized before entropy coding. Intra-prediction refers to a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after transformation and the smaller the AC coefficients, the fewer bits are needed at a given quantization step size to represent the block after entropy coding.

[0008] Conventional intra-coding, for example as known from MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to encode / decode a block based on surrounding sample data and / or metadata that precedes the block of data being intra-coded or intra-decoded in decoding order, e.g., obtained during encoding and / or decoding of spatial neighbors. Such techniques are hereinafter referred to as "intra-prediction" techniques. It should be noted that in at least some cases, intra-prediction uses reference data only from the current picture being reconstructed, and not from other reference pictures.

[0009] Intra prediction may take many different forms. When more than one such technique is available in a given video coding technique, the technique used may be referred to as an intra prediction mode. One or more intra prediction modes may be provided in a particular codec. In certain cases, a mode may have sub-modes and / or may be associated with various parameters, and the mode / sub-mode information and intra coding parameters of a block of video may be coded individually or collectively included in the codeword of the mode. Which codeword is used for a given mode, sub-mode, and / or parameter combination may affect coding efficiency gains via intra prediction and may also affect the entropy coding technique used to convert the codeword into a bitstream.

[0010] Certain modes of intra prediction were introduced in H.264, improved in H.265, and further refined in newer coding techniques such as the joint exploration model (JEM), versatile video coding (VVC), and benchmark sets (BMS). In general, in intra prediction, the predictor block may be formed using neighborhood sample values ​​that become available. For example, available values ​​of a particular set of neighborhood samples along a particular direction and / or line may be copied to the predictor block. The reference to the direction in use may be coded in the bitstream or may itself be predicted.

[0011] Referring to FIG. 1A, at the bottom right, a subset of nine predictor directions specified in the 33 possible intra predictor directions of H.265 (corresponding to the 33 angle modes out of the 35 intra modes specified in H.265) is shown. The point 101 where the arrows converge represents the sample to be predicted. The arrows represent the direction in which neighboring samples are used to predict the sample at 101. For example, arrow 102 indicates that sample 101 is predicted from the neighboring sample to the top right at an angle of 45 degrees from the horizontal. Similarly, arrow 103 indicates that sample 101 is predicted from the neighboring sample to the bottom left of sample 101 at an angle of 22.5 degrees from the horizontal.

[0012] Still referring to FIG. 1A, a square block 104 of 4×4 samples (indicated by a thick dashed line) is depicted at the top left. The square block 104 contains 16 samples, each labeled with "S" and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample of the block 104 in both the Y and X dimensions. Since the block is 4×4 samples in size, S44 is at the bottom right. An exemplary reference sample is also shown that follows a similar numbering scheme. The reference sample is labeled R, its Y position (e.g., row index) and X position (column index) relative to the block 104. In both H.264 and H.265, predictive samples neighboring the block being reconstructed are used.

[0013] Intra-picture prediction of block 104 can be started by copying reference sample values ​​from neighboring samples according to the signaled prediction direction. For example, assume that the coded video bitstream includes signaling for this block 104 indicating the prediction direction of arrow 102, i.e., the sample is predicted from the top right predicted sample at an angle of 45 degrees from the horizontal. In such a case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then sample S44 is predicted from reference sample R08.

[0014] In certain cases, to calculate a reference sample, especially when the orientation is not evenly divisible by 45 degrees, the values ​​of multiple reference samples may be combined, for example by interpolation.

[0015] The number of possible directions has increased as video coding technology continues to develop. For example, in H.264 (2003), nine different directions are available for intra prediction. This increases to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions at the time of this disclosure. Experimental studies have been conducted to help identify the most suitable intra prediction directions, and those most suitable directions can be encoded with a small number of bits using certain techniques of entropy coding, accepting certain bit penalties for the directions. Furthermore, the direction itself may in some cases be predictable from the neighboring directions used in the intra prediction of the decoded neighboring blocks.

[0016] FIG. 1B shows a schematic diagram 180 depicting 65 intra prediction directions according to JEM to illustrate the increase in the number of prediction directions in various encoding techniques developed over time.

[0017] The manner in which bits representing intra-prediction directions in a coded video bitstream are mapped to prediction directions may vary across video coding techniques, ranging from simple direct mapping of prediction directions to intra-prediction modes, to complex adaptation schemes involving codewords, most probable modes, and similar techniques. In all cases, however, there may be certain directions of intra-prediction that are statistically less likely to occur in the video content than certain other directions. Since the goal of video compression is to reduce redundancy, in a properly designed video coding technique, less likely directions may be represented with more bits than more likely directions.

[0018] Inter-picture prediction, or inter-prediction, may be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or part thereof (reference picture) may be used to predict a newly reconstructed picture or picture part (e.g., block) after being spatially shifted in a direction indicated by a motion vector (hereinafter, MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions X and Y, or three dimensions, with the third dimension being an indication of the reference picture in use (similar to the temporal dimension).

[0019] In some video compression techniques, a current MV applicable to a particular area of ​​sample data can be predicted from other MVs, e.g., other MVs related to other areas of sample data that are spatially adjacent to the area being reconstructed and that precede the current MV in decoding order. Doing so can substantially reduce the overall amount of data required to code the MV by relying on the removal of redundancy of correlated MVs, thereby increasing compression efficiency. MV prediction can work effectively because, for example, when coding an input video signal derived from a camera (known as natural video), areas larger than the area where a single MV is applicable have a statistical likelihood to move in a similar direction in the video sequence, and therefore, in some cases, can be predicted using similar motion vectors derived from MVs of nearby areas. As a result, the actual MV of a given area is similar or identical to the MV predicted from the surrounding MVs. Such MVs can be represented with fewer bits after entropy coding than would be used if the MVs were directly coded instead of predicted from nearby MVs. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from an original signal (i.e., sample stream). In other cases, the MV prediction itself may be non-lossy, for example due to rounding errors when computing the predictor from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms specified by H.265, the one described below is a technique hereafter referred to as "spatial merging".

[0021] Specifically, referring to FIG. 2, a current block (201) contains samples that are detected by the encoder during the motion search process as predictable from a spatially shifted previous block of the same size. Instead of coding its MV directly, the MV may be derived from metadata associated with one or more reference pictures, e.g., from the most recent reference picture (in decoding order), using MVs associated with any one of five surrounding samples, denoted A0, A1, and B0, B1, B2 (202-206, respectively). In H.265, MV prediction may use predictors from the same reference picture that neighboring blocks use. Summary of the Invention [Means for solving the problem]

[0022] This disclosure describes a set of advanced video coding techniques. More specifically, the techniques disclosed involve cross-channel prediction, in particular chroma from luma (CfL) prediction.

[0023] In some implementations, a method for video processing is disclosed, the method including: determining, based on at least one reconstructed sample from a second color channel of a received bitstream, to apply a cross-channel prediction mode to a current sample of a first color channel in a current coding block; deriving an alternating current (AC) contribution to a predictor of the current sample of the first color channel based on the at least one reconstructed sample from the second color channel; performing intra prediction of the current sample of the first color channel based on the at least one reconstructed sample of the first color channel under an intra prediction mode other than a direct current intra prediction mode, to derive a direct current (DC) contribution to the predictor of the current sample of the first color channel; calculating a weighted sum or weighted average of the AC contribution and the DC contribution to generate a predictor of the current sample of the first color channel; and reconstructing the current sample of the first color channel based on at least the predictor of the current sample of the first color channel.

[0024] In the above implementation, the first color channel includes a chroma channel and the second color channel includes a luma channel.

[0025] In any one of the above embodiments, the method may further include a step of subsampling a luma channel of the current coding block to obtain a subsampled luma block corresponding to a chroma block of a chroma channel of the current coding block.

[0026] In any one of the above embodiments, the intra prediction mode for performing intra prediction to derive a DC contribution to a predictor of a current sample of a first color channel includes one of SMOOTH, SMOOTH-H, SMOOTH-V, PAETH, directional filtering, recursive filtering, or an offset-based improved intra prediction mode.

[0027] In any one of the above embodiments, an intra prediction mode for performing intra prediction to derive a DC contribution to a predictor of a current sample of a chroma channel is predetermined.

[0028] In any one of the above implementations, an intra prediction mode for performing intra prediction to derive a DC contribution to a predictor of a current sample of a chroma channel is selected from a predetermined subset of intra prediction modes according to explicit signaling in the received bitstream.

[0029] In any one of the above embodiments, an intra prediction mode for performing intra prediction to derive a DC contribution to a predictor of a current sample of a chroma channel is derived based on a reference intra prediction mode associated with a luma sample in the subsampled luma block, the luma sample being co-located with the current sample of the chroma channel.

[0030] In any one of the above embodiments, an intra prediction mode for performing intra prediction to derive a DC contribution to a predictor of a current sample of a chroma channel is derived from a reference intra prediction mode associated with a neighboring chroma block of the current coding block.

[0031] In any one of the above implementations, an intra prediction mode for performing intra prediction to derive a DC contribution to a predictor of a current sample of a chroma channel is derived according to a size of a current coding block.

[0032] In any one of the above embodiments, the step of deriving the AC contribution to the predictor of the current sample of the chroma channel includes a step of subtracting a sample average of the subsampled and reconstructed luma block from a reconstructed luma sample that is co-located with the current sample of the chroma channel.

[0033] In any one of the above embodiments, the step of deriving an AC contribution to the predictor of the current sample of the chroma channel includes a step of subtracting a prediction of a luma sample co-located with the current sample of the chroma channel from the reconstructed luma sample.

[0034] In any one of the above implementations, the prediction of the luma sample is generated under an intra prediction mode.

[0035] In any one of the above embodiments, the intra prediction mode for deriving a prediction of a luma sample includes one of SMOOTH, SMOOTH-H, SMOOTH-V, PAETH, directional filtering, recursive filtering, or an offset-based improved intra prediction mode.

[0036] In any one of the above embodiments, the intra-prediction mode for generating the prediction of the luma samples is explicitly signaled in the received bitstream.

[0037] In any one of the above embodiments, the intra prediction mode for generating the prediction of the luma sample is derived implicitly.

[0038] In any one of the above embodiments, the intra prediction mode for generating the prediction of the luma sample is implicitly derived as the same as the intra prediction mode for generating the DC contribution to the predictor of the current sample of the chroma channel.

[0039] In any one of the above embodiments, the intra prediction mode for generating the prediction of the luma sample is derived as a reference intra prediction mode applied to a luma sample co-located with the current sample of a chroma channel.

[0040] In any one of the above embodiments, the intra prediction mode for generating the prediction of the luma sample is derived from a reference intra prediction mode associated with a neighboring luma block of the current coding block.

[0041] In some other embodiments, a device for processing video information is disclosed. The device may include circuitry configured to perform any one of the method embodiments described above.

[0042] Aspects of the present disclosure also provide a non-transitory computer-readable medium having stored thereon instructions that, when executed by a computer, cause the computer to perform a method for video decoding and / or encoding.

[0043] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0044] [Figure 1A] FIG. 13 is a schematic diagram of an example subset of intra-prediction directional modes. [Figure 1B] FIG. 2 illustrates an exemplary intra-prediction direction. [Diagram 2] FIG. 2 is a schematic diagram of a current block and its surrounding spatial merging candidates for motion vector prediction in one example. [Diagram 3] FIG. 3 is a schematic diagram illustrating a simplified block diagram of a communication system 300 in accordance with an exemplary embodiment. [Figure 4] FIG. 4 is a schematic diagram illustrating a simplified block diagram of a communication system 400 in accordance with an example embodiment. [Diagram 5] FIG. 2 is a schematic diagram illustrating a simplified block diagram of a video decoder according to an exemplary embodiment. [Figure 6] FIG. 2 is a schematic diagram illustrating a simplified block diagram of a video encoder according to an example embodiment. [Figure 7] FIG. 2 is a block diagram of a video encoder according to another example embodiment. [Figure 8] FIG. 4 is a block diagram of a video decoder according to another exemplary embodiment. [Figure 9] FIG. 2 illustrates a coding block partitioning scheme according to an exemplary embodiment of the present disclosure. [Figure 10] FIG. 13 illustrates another scheme for coding block partitioning according to an exemplary embodiment of the present disclosure. [Figure 11] FIG. 13 illustrates another scheme for coding block partitioning according to an exemplary embodiment of the present disclosure. [Figure 12] FIG. 2 illustrates an exemplary partitioning of a base block into coding blocks according to an exemplary partitioning scheme. [Figure 13] FIG. 2 illustrates an exemplary division-of-thirds scheme. [Figure 14] FIG. 2 illustrates an exemplary quadtree / binary tree coding block partitioning scheme. [Figure 15] 4 illustrates a scheme for splitting a coding block into multiple transform blocks and the coding order of the transform blocks according to an exemplary embodiment of the present disclosure. [Figure 16] 4 illustrates another scheme for splitting a coding block into multiple transform blocks and the coding order of the transform blocks according to an example embodiment of the present disclosure. [Figure 17] FIG. 13 illustrates another scheme for splitting a coding block into multiple transform blocks, according to an exemplary embodiment of the present disclosure. [Figure 18] FIG. 1 illustrates an example fine angle in directional intra prediction. [Figure 19] FIG. 13 is a diagram showing nominal angles in directional intra prediction. [Figure 20] FIG. 1 illustrates the top, left, and top-left positions of the PAETH mode of a coding block. [Figure 21] FIG. 13 illustrates an example of a recursive intra-filtering mode. [Figure 22] FIG. 13 is a block diagram of a luma-to-chroma (CfL) prediction unit configured to generate prediction samples of chroma blocks based on reconstructed luma samples. [Diagram 23] FIG. 1 is a flow diagram of an example CfL prediction process. [Figure 24] 1 is a flowchart of an example of a CfL prediction process. [Diagram 25] FIG. 1 is a schematic diagram of a computer system according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0045] FIG. 3 illustrates a simplified block diagram of a communication system 300 according to an embodiment of the present disclosure. The communication system 300 includes a plurality of terminals capable of communicating with each other, for example, via a network 350. For example, the communication system 300 includes a first pair of terminals 310 and 320 interconnected via the network 350. In the example of FIG. 3, the first pair of terminals 310 and 320 may perform unidirectional transmission of data. For example, the terminal 310 may code video data (e.g., of a stream of video pictures captured by the terminal 310) for transmission to another terminal 320 via the network 350. The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal 320 may receive the coded video data from the network 350, decode the coded video data to reconstruct the video pictures, and display the video pictures according to the reconstructed video data. The unidirectional data transmission may be implemented in media serving applications, etc.

[0046] In another example, the communication system 300 includes a second pair of terminals 330 and 340 for bidirectional transmission of coded video data, which may be implemented, for example, during video conferencing applications. For bidirectional transmission of data, in one example, each of the terminals 330 and 340 may code video data (e.g., of a stream of video pictures captured by the terminal) for transmission to the other of the terminals 330 and 340 over the network 350. Also, each of the terminals 330 and 340 may receive coded video data transmitted by the other of the terminals 330 and 340, may decode the coded video data to recover the video pictures, and may display the video pictures on an accessible display device according to the recovered video data.

[0047] In the example of FIG. 3, terminals 310, 320, 330, and 340 may be implemented as a server, a personal computer, and a smartphone, although applicability of the underlying principles of the present disclosure may not be so limited. Embodiments of the present disclosure may be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, and the like. Network 350 represents any number or type of network that conveys coded video data between terminals 310, 320, 330, and 340, including, for example, wireline and / or wireless communication networks. Communications network 350 may exchange data over circuit-switched, packet-switched, and / or other types of channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of network 350 may not be important to the operation of the present disclosure unless explicitly described herein.

[0048] 4 shows an arrangement of video encoders and video decoders in a video streaming environment as one example for an application of the disclosed subject matter. The disclosed subject matter may be equally applicable to other video applications including, for example, video conferencing, digital television broadcasting, gaming, virtual reality, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0049] The video streaming system may include a video source 401, e.g., a video capture subsystem 413 that may include a digital camera, for creating a stream of uncompressed video pictures or images 402. In one example, the stream of video pictures 402 includes samples recorded by the digital camera of the video source 401. The stream of video pictures 402, shown as a thick line to emphasize the large amount of data compared to the encoded video data 404 (or coded video bitstream), may be processed by an electronic device 420 that includes a video encoder 403 coupled to the video source 401. The video encoder 403 may include hardware, software, or a combination thereof for realizing or implementing aspects of the disclosed subject matter, as described in more detail below. The encoded video data 404 (or coded video bitstream 404), shown as a thin line to emphasize the small amount of data compared to the uncompressed video picture stream 402, may be stored in a streaming server 405 for future use or directly in a downstream video device (not shown). One or more streaming client subsystems, such as client subsystems 406 and 408 of FIG. 4, can access the streaming server 405 to retrieve copies 407 and 409 of the encoded video data 404. The client subsystem 406 may include a video decoder 410, for example, within the electronic device 430. The video decoder 410 decodes the incoming copy of the encoded video data 407 and creates an outgoing stream of video pictures 411 that are uncompressed and may be rendered on a display 412 (e.g., a display screen) or other rendering device (not shown). The video decoder 410 may be configured to perform some or all of the various functions described in this disclosure. In some streaming systems, the encoded video data 404, 407, and 409 (e.g., a video bitstream) may be encoded according to a particular video coding / compression standard.Examples of these standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC and other video coding standards.

[0050] It should be noted that electronic devices 420 and 430 may include other components (not shown). For example, electronic device 420 may include a video decoder (not shown), and electronic device 430 may likewise include a video encoder (not shown).

[0051] 5 shows a block diagram of a video decoder 510 according to any embodiment of the present disclosure below. The video decoder 510 can be included in an electronic device 530. The electronic device 530 can include a receiver 531 (e.g., a receiving circuit). The video decoder 510 can be used in place of the video decoder 410 of the example of FIG. 4.

[0052] The receiver 531 may receive one or more coded video sequences to be decoded by the video decoder 510. In the same or another embodiment, one coded video sequence may be decoded at a time, and the decoding of each coded video sequence is independent of the other coded video sequences. Each video sequence may be associated with multiple video frames or images. The coded video sequences may be received from a channel 501, which may be a storage device that stores the coded video data or a hardware / software link to a streaming source that transmits the coded video data. The receiver 531 may receive the coded video data along with other data, such as coded audio data and / or auxiliary data streams, and may be forwarded to respective processing circuits (not shown). The receiver 531 may separate the coded video sequences from the other data. To address network jitter, a buffer memory 515 may be disposed between the receiver 531 and the entropy decoder / parser 520 (hereinafter "parser 520"). In certain applications, the buffer memory 515 may be implemented as part of the video decoder 510. In other applications, the buffer memory 515 may be external to and separate from the video decoder 510 (not shown). In still other applications, there may be a buffer memory (not shown) external to the video decoder 510, for example to deal with network jitter, and there may be another additional buffer memory 515 internal to the video decoder 510, for example to handle playback timing. If the receiver 531 is receiving data from a store / forward device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory 515 may not be needed or may be small. For use with best-effort packet networks such as the Internet, a sufficient size of the buffer memory 515 may be required, and the size may be relatively large.Such a buffer memory may be implemented with an adaptive size and may be implemented at least in part in an operating system or similar element (not shown) external to the video decoder 510.

[0053] The video decoder 510 may include a parser 520 for reconstructing symbols 521 from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder 510 and potentially information for controlling a rendering device such as a display 512 (e.g., a display screen) that may or may not be an integral part of the electronic device 530, but may be coupled to the electronic device 530, as shown in FIG. 5. The control information for the rendering device may be in the form of a supplemental enhancement information (SEI message) or a video usability information (VUI) parameter set fragment (not shown). The parser 520 may parse / entropy decode the coded video sequence received by the parser 520. The entropy coding of the coded video sequence may be in accordance with a video coding technique or standard and may follow various principles including variable length coding, Huffman coding, arithmetic coding with or without context sensitivity, etc. The parser 520 may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the subgroup. The subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. Parser 520 may also extract information from the coded video sequence, such as transform coefficients (e.g., Fourier transform coefficients), quantizer parameter values, motion vectors, etc.

[0054] Parser 520 may perform entropy decoding / parsing operations on the video sequence received from buffer memory 515 to produce symbols 521 .

[0055] The reconstruction of symbols 521 may involve a number of different processing or functional units, depending on the type of coded video picture or portion thereof (inter-picture and intra-picture, inter-block and intra-block, etc.) and other factors. The units involved and how they are involved may be controlled by subgroup control information parsed from the coded video sequence by parser 520. The flow of such subgroup control information between parser 520 and the following processing or functional units is not shown for the sake of simplicity.

[0056] Besides the functional blocks already mentioned, the video decoder 510 may be conceptually subdivided into several functional units, as described below. In an actual implementation operating under commercial constraints, many of these functional units may closely interact with each other and may be at least partially integrated with each other. However, for the purpose of clearly describing the various functions of the disclosed subject matter, a conceptual subdivision into functional units is adopted in the following disclosure.

[0057] The first unit may include a scalar / inverse transform unit 551. The scalar / inverse transform unit 551 may receive quantized transform coefficients and control information from the parser 520, including information indicating which type of inverse transform to use, block size, quantization coefficients / parameters, quantization scaling matrices, and states (lies) as symbols 521. The scalar / inverse transform unit 551 may output a block including sample values ​​that may be input to an aggregator 555.

[0058] In some cases, the output samples of the scalar / inverse transform 551 may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by the intra-picture prediction unit 552. In some cases, the intra-picture prediction unit 552 may generate a block of the same size and shape as the block being reconstructed using neighboring block information already reconstructed and stored in the current picture buffer 558. The current picture buffer 558 may, for example, buffer the partially reconstructed current picture and / or the fully reconstructed current picture. In some implementations, the aggregator 555 may add the prediction information generated by the intra-prediction unit 552 to the output sample information provided by the scalar / inverse transform unit 551 on a sample-by-sample basis.

[0059] In other cases, the output samples of the scalar / inverse transform unit 551 may relate to an inter-coded and potentially motion-compensated block. In such a case, the motion compensation prediction unit 553 may access the reference picture memory 557 to fetch samples used for inter-picture prediction. After motion compensating the fetched samples according to the symbols 521 related to the block, these samples may be added to the output of the scalar / inverse transform unit 551 (the output of the unit 551 may be referred to as a residual sample or a residual signal) by the aggregator 555 to generate output sample information. The addresses in the reference picture memory 557 from which the motion compensation prediction unit 553 fetches prediction samples available to the motion compensation prediction unit 553 in the form of the symbols 521, which may have, for example, X, Y components (shift), and reference picture components (time), may be controlled by the motion vector. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory 557 when a sub-sample accurate motion vector is used, and may also be associated with a motion vector prediction mechanism, etc.

[0060] In loop filter unit 556, the output samples of aggregator 555 may be subjected to various loop filtering techniques. Video compression techniques may include in-loop filter techniques controlled by parameters included in the coded video sequence (also called coded video bitstream) and made available to loop filter unit 556 as symbols 521 from parser 520, but may also be responsive to meta-information obtained during decoding of a previous portion (in decoding order) of the coded picture or coded video sequence, or to previously reconstructed, loop filtered sample values. As described in more detail below, several types of loop filters may be included as part of loop filter unit 556, in various orders.

[0061] The output of the loop filter unit 556 may be a sample stream that may be output to the rendering device 512 as well as stored in a reference picture memory 557 for use in future inter-picture prediction.

[0062] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future inter-picture prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by parser 520), the current picture buffer 558 can become part of reference picture memory 557, and a new current picture buffer can be reallocated before beginning reconstruction of a subsequent coded picture.

[0063] The video decoder 510 may perform decoding operations according to a given video compression technique adopted in a standard, such as ITU-T Rec. H.265. The coded video sequence may conform to a syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and to a profile documented in the video compression technique or standard. Specifically, a profile may select a particular tool from all tools available in the video compression technique or standard as the only tool available for use under that profile. To conform to a standard, the complexity of the coded video sequence may be within a range defined by a level of the video compression technique or standard. In some cases, the level limits a maximum picture size, a maximum frame rate, a maximum reconstructed sample rate (e.g., measured in megasamples per second), a maximum reference picture size, etc. The limits set by the level may be further limited in some cases by metadata for HRD buffer management and a Hypothetical Reference Decoder (HRD) specification signaled in the coded video sequence.

[0064] In some example embodiments, the receiver 531 may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder 510 to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0065] 6 shows a block diagram of a video encoder 603 according to an exemplary embodiment of the present disclosure. The video encoder 603 may be included in an electronic device 620. The electronic device 620 may further include a transmitter 640 (e.g., a transmitting circuit). The video encoder 603 may be used in place of the video encoder 403 of the example of FIG. 4.

[0066] The video encoder 603 may receive video samples from a video source 601 (which is not part of the electronic device 620 in the example of FIG. 6) that may capture video images to be coded by the video encoder 603. In another example, the video source 601 may be implemented as part of the electronic device 620.

[0067] The video source 601 may provide a source video sequence to be coded by the video encoder 603 in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCb, RGB, XYZ ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media presentation system, the video source 601 may be a storage device capable of storing pre-prepared videos. In a video conferencing system, the video source 601 may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual pictures or images that give motion when viewed in sequence. The pictures themselves may be organized as a spatial array of pixels, each pixel may contain one or more samples depending on the sampling structure, color space, etc. being used. Those skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0068] According to some example embodiments, the video encoder 603 can code and compress pictures of a source video sequence into a coded video sequence 643 in real time or under other time constraints required by the application. Enforcing an appropriate coding rate constitutes one function of the controller 650. In some embodiments, the controller 650 can be functionally coupled to and control other functional units, as described below. For the sake of brevity, couplings are not shown. Parameters set by the controller 650 can include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller 650 can be configured with other appropriate functions related to the video encoder 603 optimized for a particular system design.

[0069] In some example embodiments, the video encoder 603 may be configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop may include a source coder 630 (responsible for creating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture, for example) and a (local) decoder 633 embedded in the video encoder 603. The decoder 633 reconstructs the symbols to create sample data in a similar manner as a (remote) decoder would create them, even if the embedded decoder 633 processes a video stream coded by the source coder 630 without entropy coding (since any compression between the symbols and the coded video bitstream in entropy coding may be lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory 634. Since the decoding of the symbol stream produces bit-exact results regardless of the location of the decoder (local or remote), the content in the reference picture memory 634 is also bit-exact between the local and remote encoders. In other words, the prediction part of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder will "see" when using prediction during decoding. This fundamental principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, e.g., due to channel error) is used to improve coding quality.

[0070] The operation of the "local" decoder 633 may be the same as that of a "remote" decoder, such as the video decoder 510 already described in detail above in connection with Figure 5. However, with brief reference also to Figure 5, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder 645 and parser 520 may be lossless, the entropy decoding portion of the video decoder 510, including the buffer memory 515, and the parser 520 may not be fully implemented in the local decoder 633 within the encoder.

[0071] At this point, it can be said that any decoder technology, except for parsing / entropy decoding, which may only exist in the decoder, may also necessarily need to exist in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter may focus on the decoder operation, which is similar to the decoding part of the encoder. Thus, the description of the encoder technology may be omitted, since it is the reverse of the decoder technology described in general. Only in certain areas or aspects, a more detailed description of the encoder is provided below.

[0072] During operation in some example implementations, the source coder 630 may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as "reference pictures." In this manner, the coding engine 632 codes color channel differences (or residuals) between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as predictive references for the input picture. The terms "residual" and its adjective form "residual" may be used interchangeably.

[0073] The local video decoder 633 may decode the coded video data of the pictures that may be designated as reference pictures based on the symbols created by the source coder 630. The operation of the coding engine 632 may advantageously be a lossy process. When the coded video data is decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder 633 may replicate the decoding process that may be performed by the video decoder on the reference pictures and store the reconstructed reference pictures in the reference picture cache 634. In this way, the video encoder 603 may locally store copies of reconstructed reference pictures that have common content with the reconstructed reference pictures obtained by the far-end (remote) video decoder (without transmission errors).

[0074] The predictor 635 may perform a prediction search for the coding engine 632. That is, for a new picture to be coded, the predictor 635 may search the reference picture memory 634 for sample data (as candidate reference pixel blocks) that may serve as suitable prediction references for the new picture, or specific metadata, such as reference picture motion vectors, block shapes, etc. The predictor 635 may operate on a sample block-by-sample block basis to find a suitable prediction reference. In some cases, as determined by the search results obtained by the predictor 635, the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory 634.

[0075] The controller 650 may manage the coding operations of the source coder 630, including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0076] The output of all the aforementioned functional units may be subjected to entropy coding in the entropy coder 645. The entropy coder 645 converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0077] The transmitter 640 may buffer the coded video sequence created by the entropy coder 645 to prepare it for transmission over a communication channel 660, which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter 640 may merge the coded video data from the video coder 603 with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0078] The controller 650 may manage the operation of the video encoder 603. During coding, the controller 650 may assign a particular coded picture type to each coded picture, which may affect the coding techniques that may be applied to the respective picture. For example, pictures may often be assigned as one of the following picture types:

[0079] An intra picture (I-picture) may be a picture that can be coded and decoded without using other pictures in a sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures, as well as their respective uses and characteristics.

[0080] A predictive picture (P picture) may be a picture that can be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict sample values ​​for each block.

[0081] A bidirectionally predicted picture (B-picture) may be a picture that can be coded and decoded using intra- or inter-prediction that uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, a multi-predicted picture can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0082] A source picture may generally be spatially subdivided into multiple sample coding blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and coded block by block. A block may be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to the respective picture of the block. For example, a block of an I picture may be non-predictively coded or predictively coded with reference to already coded blocks of the same picture (spatial prediction or intra prediction). A pixel block of a P picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. A block of a B picture may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures. A source picture or an intermediate processed picture may be subdivided into other types of blocks for other purposes. The division of coding blocks and other types of blocks may or may not follow the same scheme, as described in more detail below.

[0083] The video encoder 603 may perform coding operations in accordance with a given video coding technique or standard, such as ITU-T Rec. H.265. In its operations, the video encoder 603 may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. The coded video data may therefore conform to a syntax specified by the video coding technique or standard being used.

[0084] In some example embodiments, the transmitter 640 can transmit additional data along with the coded video. The source coder 630 can include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0085] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation in a given picture, while inter-picture prediction exploits temporal or other correlation between pictures. For example, a particular picture being encoded / decoded, called the current picture, may be divided into blocks. If a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, it may be coded by a vector, called a motion vector. A motion vector points to a reference block in a reference picture, and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0086] In some exemplary embodiments, bi-prediction techniques may be used for inter-picture prediction. According to such bi-prediction techniques, two reference pictures, such as a first reference picture and a second reference picture, are used, both of which advance the current picture in the video in decoding order (but may be in the past or future, respectively, in display order). A block in the current picture may be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. A block may be jointly predicted by a combination of the first reference block and the second reference block.

[0087] Furthermore, a merge mode technique may be used to improve coding efficiency in inter-picture prediction.

[0088] According to some exemplary embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, a picture in a sequence of video pictures is divided into coding tree units (CTUs) for compression, and the CTUs in a picture may have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU may include three parallel coding tree blocks (CTBs), namely, one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree partitioned into one or more coding units (CUs). For example, a CTU of 64×64 pixels may be partitioned into one CU of 64×64 pixels, or four CUs of 32×32 pixels. Each of one or more of the 32×32 blocks may be further partitioned into four CUs of 16×16 pixels. In some exemplary embodiments, each CU may be analyzed during encoding to determine the prediction type of the CU from among various prediction types, such as an inter prediction type or an intra prediction type. A CU may be divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. In general, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. The division of a CU into PUs (or PBs of different color channels) may be performed in various spatial patterns. A luma PB or a chroma PB may include a matrix of sample values ​​(e.g., luma values), such as, for example, 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 samples, etc.

[0089] 7 shows a diagram of a video encoder 703 according to another example embodiment of this disclosure. The video encoder 703 is configured to receive a processed block (e.g., a predictive block) of sample values ​​in a current video picture in a sequence of video pictures and to encode the processed block into a coded picture that is part of a coded video sequence. The example video encoder 703 may be used in place of the example video encoder (403) of FIG. 4.

[0090] For example, the video encoder 703 receives a matrix of sample values ​​of a processing block, such as a prediction block of 8x8 samples. The video encoder 703 then determines whether the processing block is best coded using intra-mode, inter-mode, or bi-prediction mode, for example, using rate-distortion optimization (RDO). If the processing block is determined to be coded in intra-mode, the video encoder 703 may code the processing block into a coded picture using intra-prediction techniques, and if the processing block is determined to be coded in inter-mode or bi-prediction mode, the video encoder 703 may code the processing block into a coded picture using inter-prediction techniques or bi-prediction techniques, respectively. In some exemplary embodiments, a merge mode may be used as a sub-mode of inter-picture prediction, in which motion vectors are derived from one or more motion vector predictors without benefit of coded motion vector components outside the predictors. In some other exemplary embodiments, there may be motion vector components applicable to the current block. Thus, the video encoder 703 may include components not explicitly shown in FIG. 7, such as a mode decision module, to determine the permutation mode of a processing block.

[0091] In the example of FIG. 7, the video encoder 703 includes an inter-encoder 730, an intra-encoder 722, a residual calculator 723, a switch 726, a residual encoder 724, an overall controller 721, and an entropy encoder 725 coupled to each other as shown in the exemplary configuration of FIG. 7.

[0092] The inter-encoder 730 is configured to receive a sample of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., blocks in previous and subsequent pictures in display order), generate inter-prediction information (e.g., a description of redundant information due to inter-coding techniques, motion vectors, merge mode information), and calculate an inter-prediction result (e.g., a prediction block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture that has been decoded based on the encoded video information using a decoding unit 633 incorporated in the example encoder 620 of FIG. 6 (depicted as a residual decoder 728 of FIG. 7, as described in more detail below).

[0093] The intra encoder 722 is also configured to receive samples of a current block (e.g., a processing block), compare the block with blocks already encoded in the same picture, generate quantized coefficients after transformation, and possibly generate intra prediction information (e.g., intra prediction direction information by one or more intra coding techniques). The intra encoder 722 can calculate an intra prediction result (e.g., a prediction block) based on the intra prediction information and a reference block in the same picture.

[0094] The overall controller 721 may be configured to determine overall control data and control other components of the video encoder 703 based on the overall control data. In one example, the overall controller 721 determines a prediction mode of the block and provides a control signal to the switch 726 based on the prediction mode. For example, if the prediction mode is an intra mode, the overall controller 721 controls the switch 726 to select an intra mode result used by the residual calculator 723 and controls the entropy encoder 725 to select intra prediction information and include the intra prediction information in the bitstream. Also, if the prediction mode of the block is an inter mode, the overall controller 721 controls the switch 726 to select an inter prediction result used by the residual calculator 723 and controls the entropy encoder 725 to select inter prediction information and include it in the bitstream.

[0095] The residual calculator 723 may be configured to calculate a difference (residual data) between the received block and a prediction result of the block selected from the intra encoder 722 or the inter encoder 730. The residual encoder 724 may be configured to encode the residual data to generate transform coefficients. For example, the residual encoder 724 may be configured to transform the residual data from a spatial domain to a frequency domain to generate transform coefficients. Then, the transform coefficients are subjected to a quantization process to obtain quantized transform coefficients. In various exemplary embodiments, the video encoder 703 also includes a residual decoder 728. The residual decoder 728 is configured to perform an inverse transform and generate decoded residual data. The decoded residual data may be appropriately used in the intra encoder 722 and the inter encoder 730. For example, the inter encoder 730 may generate a decoded block based on the decoded residual data and the inter prediction information, and the intra encoder 722 may generate a decoded block based on the decoded residual data and the intra prediction information. The decoded blocks are appropriately processed to generate a decoded picture, which may be buffered in a memory circuit (not shown) and used as a reference picture.

[0096] The entropy encoder 725 may be configured to format a bitstream to include the coded block and perform entropy coding. The entropy encoder 725 is configured to include various information in the bitstream. For example, the entropy encoder 725 may be configured to include overall control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other suitable information in the bitstream. When coding a block in a merged sub-mode of either an inter mode or a bi-prediction mode, the residual information may not be present.

[0097] 8 shows a diagram of an example video decoder 810 according to another embodiment of the present disclosure. The video decoder 810 is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder 810 may be used in place of the video decoder 410 of the example of FIG. 4.

[0098] In the example of FIG. 8, the video decoder 810 includes an entropy decoder 871, an inter decoder 880, a residual decoder 873, a reconstruction module 874, and an intra decoder 872 coupled together as shown in the exemplary configuration of FIG.

[0099] The entropy decoder 871 may be configured to reconstruct from the coded picture certain symbols representing syntax elements that make up the coded picture. Such symbols may include, for example, prediction information (e.g., intra- or inter-prediction information) that may identify the mode in which the block is coded (e.g., intra- or bi-prediction mode, merged submode or another submode), certain samples or metadata used for prediction by the intra decoder 872 or the inter decoder 880, residual information in the form of, for example, quantized transform coefficients, etc. In one example, if the prediction mode is an inter- or bi-prediction mode, the inter-prediction information is provided to the inter decoder 880, and if the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra decoder 872. The residual information is dequantized and provided to the residual decoder 873.

[0100] The inter decoder 880 may be configured to receive the inter prediction information and generate inter prediction results based on the inter prediction information.

[0101] The intra decoder 872 may be configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0102] The residual decoder 873 may be configured to perform inverse quantization to extract inverse quantized transform coefficients and process the inverse quantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder 873 may also utilize certain control information (to include quantizer parameters (QP)) that may be provided by the entropy decoder 871 (a data path is not depicted since this is likely to be low data volume control information).

[0103] The reconstruction module 874 may be configured to combine, in the spatial domain, the residual output by the residual decoder 873 and a prediction result (possibly output by an inter-prediction module or an intra-prediction module) to form a reconstructed block that forms part of the reconstructed picture as part of the reconstructed video. It should be noted that other suitable operations, such as a deblocking operation, may also be performed to improve visual quality.

[0104] It should be noted that the video encoders 403, 603, and 703 and the video decoders 410, 510, and 810 may be implemented using any suitable technique. In some exemplary embodiments, the video encoders 403, 603, and 703 and the video decoders 410, 510, and 810 may be implemented using one or more integrated circuits. In another embodiment, the video encoders 403, 603, and 603 and the video decoders 410, 510, and 810 may be implemented using one or more processors executing software instructions.

[0105] Turning to block partitioning for coding and decoding, a general partitioning may start from a base block and may follow a predefined set of rules, a specific pattern, a partitioning tree, or any partitioning structure or scheme. The partitioning may be hierarchical and recursive. After isolating or dividing the base block according to any of the exemplary partitioning procedures described below or other procedures, or combinations thereof, a final set of partitions or coding blocks may be obtained. Each of these partitions may be at one of various partitioning levels in the partitioning hierarchy and may be partitions of various shapes. Each of the partitions may be referred to as a coding block (CB). In various exemplary partitioning implementations described further below, each resulting CB may be a CB of any of the allowed sizes and partitioning levels. Such partitions are referred to as coding blocks because they may form a unit for which some basic coding / decoding decisions may be made and coding / decoding parameters may be optimized, determined, and signaled in the encoded video bitstream. The highest or deepest level in the final partition represents the depth of the coding block partitioning tree structure. The coding block may be a luma coding block or a chroma coding block. The CB tree structure for each color is sometimes called a coding block tree (CBT).

[0106] The coding blocks of all color channels may be collectively referred to as a coding unit (CU). The hierarchical structure of all color channels may be collectively referred to as a coding tree unit (CTU). The partitioning pattern or structure of various color channels within a CTU may or may not be the same.

[0107] In some implementations, the partition tree scheme or structure used for the luma channel and the chroma channel may not need to be the same. In other words, the luma channel and the chroma channel may have separate coding tree structures or patterns. Furthermore, whether the luma channel and the chroma channel use the same coding partition tree structure or different coding partition tree structures, and the actual coding partition tree structure to be used may depend on whether the slice being coded is a P slice, a B slice, or an I slice. For example, for an I slice, the chroma channel and the luma channel may have separate coding partition tree structures or coding partition tree structure modes, while for a P slice or a B slice, the luma channel and the chroma channel may share the same coding partition tree scheme. When separate coding partition tree structures or modes are applied, the luma channel may be partitioned into CB by one coding partition tree structure, and the chroma channel may be partitioned into chroma CB by another coding partition tree structure.

[0108] In some exemplary implementations, a predefined partitioning pattern may be applied to the base block. As shown in FIG. 9, an exemplary four-way partitioning tree may start at a first predefined level (e.g., 64×64 block level or other size as the base block size), and the base block may be partitioned hierarchically down to a predefined lowest level (e.g., 4×4 level). For example, the base block may follow four predefined partitioning options or patterns shown by 902, 904, 906, and 908, and the partition designated as R allows for recursive partitioning in that the same partitioning options shown in FIG. 9 may be repeated at lower scales down to the lowest level (e.g., 4×4 level). In some implementations, additional restrictions may be added to the partitioning scheme of FIG. 9. In the implementation of FIG. 9, rectangular partitions (e.g., 1:2 / 2:1 rectangular partitions) may be allowed but not recursive, and square partitions are allowed to be recursive. Subsequent partitioning of FIG. 9 by recursion generates a final set of coding blocks, if necessary. A coding tree depth may be further defined to indicate the division depth from a root node or root block. For example, the coding tree depth for a root node or root block, e.g., a 64x64 block, may be set to 0, and the coding tree depth increases by 1 after the root block is further divided one time according to FIG. 9. The maximum or deepest level from the 64x64 base block to the 4x4 minimum partition is 4 (starting from level 0) in the above scheme. Such a division scheme may be applied to one or more of the color channels. Each color channel may be independently divided according to the scheme of FIG. 9 (e.g., a division pattern or option in the predefined pattern may be determined independently for each of the color channels at each hierarchical level). Alternatively, two or more color channels may share the same hierarchical pattern tree of FIG. 9 (e.g., the same division pattern or option in the predefined pattern may be selected for two or more color channels at each hierarchical level).

[0109] FIG. 10 illustrates another exemplary predefined partitioning pattern that allows recursive partitioning to form a partitioning tree. As shown in FIG. 10, an exemplary 10-way partitioning structure or pattern may be predefined. The root block may start from a predefined level (e.g., from a base block at a 128×128 level or a 64×64 level). The exemplary partitioning structure in FIG. 10 includes various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. The partition type with three subpartitions shown at 1002, 1004, 1006, and 1008 in the second column of FIG. 10 may be referred to as a “T” partition. The “T” partitions 1002, 1004, 1006, and 1008 may be referred to as a left T, an upper T, a right T, and a lower T. In some exemplary implementations, none of the rectangular partitions in FIG. 10 may be further refined. A coding tree depth may be further defined to indicate the partition depth from the root node or root block. For example, the coding tree depth for a root node or block, e.g., a 128x128 block, may be set to 0, and after the root block is further split one time according to FIG. 10, the coding tree depth increases by 1. In some implementations, only the square partitions of 1010 may allow recursive splitting to the next level of the split tree following the pattern of FIG. 10. In other words, recursive splitting may not be possible for the square partitions in the T-shaped patterns 1002, 1004, 1006, and 1008. If necessary, the splitting procedure following FIG. 10 by recursion generates a final set of coding blocks. Such a scheme may be applied to one or more of the color channels. In some implementations, more flexibility may be added to the use of partitions less than 8x8 levels. For example, in some cases, 2x2 chroma inter prediction may be used.

[0110] In some other exemplary implementations of coding block partitioning, a quadtree structure may be used to partition a base block or an intermediate block into quadtree partitions. Such quadtree partitioning may be applied hierarchically and recursively to any square partition. Whether a base block or an intermediate block or partition is further quadtree partitioned may be adapted to various local characteristics of the base block or intermediate block / partition. The quadtree partitioning at the picture boundary may be further adapted. For example, an implicit quadtree partitioning may be performed at the picture boundary such that a block continues to be quadtree partitioned until its size fits into the picture boundary.

[0111] In some other exemplary implementations, a hierarchical bisection from the base block may be used. For such a scheme, the base block or mid-level block may be divided into two partitions. The bisection may be either horizontal or vertical. For example, a horizontal bisection may divide the base block or mid-block into equal left and right partitions. Similarly, a vertical bisection may divide the base block or mid-block into equal top and bottom partitions. Such bisection may be hierarchical and recursive. It may be determined at each of the base block or mid-block whether the bisection scheme should continue and, if the scheme continues further, whether a horizontal or vertical bisection should be used. In some implementations, further division may stop at a predefined minimum partition size (in one or both dimensions). Alternatively, further division may stop when a predefined division level or depth from the base block is reached. In some implementations, the aspect ratio of the partitions may be limited. For example, the aspect ratio of the partitions may not be smaller than 1:4 (or larger than 4:1). Therefore, a vertical strip partition, having a vertical to horizontal aspect ratio of 4:1, can only be further divided vertically into upper and lower partitions, each having a vertical to horizontal aspect ratio of 2:1.

[0112] In yet some other examples, a ternary division scheme may be used to divide the base block or any intermediate blocks, as shown in FIG. 13. The ternary pattern may be implemented vertically, as shown at 1302 in FIG. 13, or horizontally, as shown at 1304 in FIG. 13. The exemplary division ratio in FIG. 13 is shown as 1:2:1, either vertically or horizontally, but other ratios may be predefined. In some implementations, two or more different ratios may be predefined. Such a ternary division scheme may be used to complement a quadtree or binary tree partitioning structure, in that such a ternary tree division can capture objects located at block centers in one contiguous partition, whereas quadtrees and binary trees always divide along block centers, thus dividing objects into separate partitions. In some implementations, the width and height of the partitions of the exemplary ternary tree are always powers of two to avoid further transformations.

[0113] The above partitioning schemes may be combined in any manner at different partitioning levels. As an example, the above-mentioned quadtree and bipartitioning schemes may be combined to partition the base block into a quadtree-bipartition (QTBT) structure. In such a scheme, the base block or intermediate blocks / partitions may be either quadtree-partitioned or bipartitioned, if specified, according to a set of predefined conditions. A specific example is shown in FIG. 14. In the example of FIG. 14, the base block is first quadtree-partitioned into four partitions, as shown by 1402, 1404, 1406, and 1408. Each of the resulting partitions is then either quadtree-partitioned into four further partitions (such as 1408) or bipartitioned into two further partitions at the next level (e.g., either horizontally or vertically, such as 1402 or 1406, both of which are symmetric), or not partitioned (such as 1404). Bisection or quadtree partitioning may be recursively enabled for square partitions as shown by the overall exemplary partition pattern in 1410 and the corresponding tree structure / representation in 1420, where solid lines represent quadtree partitioning and dashed lines represent bisection. A flag may be used for each bisection node (non-leaf binary partition) to indicate whether the bisection is horizontal or vertical. For example, flag "0" may represent horizontal bisection and flag "1" may represent vertical bisection, as shown in 1420, which matches the partition structure in 1410. In the case of quadtree partitioning, there is no need to indicate the partition type, since quadtree partitioning always splits a block or partition both horizontally and vertically to generate four sub-blocks / partitions of equal size. In some implementations, flag "1" may represent horizontal bisection and flag "0" may represent vertical bisection.

[0114] In some example implementations of QTBT, the quadtree and bisection rule sets may be represented by the following predefined parameters and their associated corresponding functions: -CTU size: quadtree root node size (base block size) -MinQTSize: The minimum allowable quadtree leaf node size. -MaxBTSize: Maximum allowed binary tree root node size -MaxBTDepth: Maximum allowed binary tree depth -MinBTSize: The minimum allowable binary tree leaf node size. In some exemplary implementations of the QTBT partitioning structure, the CTU size may be set as 128×128 luma samples with two corresponding 64×64 blocks of chroma samples (when exemplary chroma subsampling is considered and used), the MinQTSize may be set as 16×16, the MaxBTSize may be set as 64×64, the MinBTSize may be set as 4×4 (for both width and height), and the MaxBTDepth may be set as 4. A quadtree partition may be first applied to the CTU to generate quadtree leaf nodes. The quadtree leaf nodes may have a size from its minimum allowed size of 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If a node is 128×128, it will not be split by the binary tree first because its size exceeds MaxBTSize (i.e., 64×64). Otherwise, a node that does not exceed MaxBTSize may be split by the binary tree. In the example of FIG. 14, the base block is 128×128. The base block can only be quadtree split according to a predefined set of rules. The base block has a depth of 0. Each of the resulting four partitions is 64×64, not exceeding MaxBTSize, and may be further quadtree split or bintree split at level 1. The process continues. When the bintree depth reaches MaxBTDepth (i.e., 4), no further splits may be considered. When the width of a bintree node is equal to MinBTSize (i.e., 4), no further horizontal splits may be considered. Similarly, when the height of a bintree node is equal to MinBTSize, no further vertical splits may be considered.

[0115] In some example implementations, the above QTBT scheme may be configured to support flexibility for luma and chroma to have the same QTBT structure or separate QTBT structures. For example, for P slices and B slices, the luma CTB and chroma CTB in one CTU may share the same QTBT structure. However, for I slices, the luma CTB may be divided into CBs by a QTBT structure, and the chroma CTB may be divided into chroma CBs by another QTBT structure. This means that CUs may be used to refer to different color channels in an I slice, e.g., an I slice may consist of a coding block of a luma component or a coding block of two chroma components, and a CU in a P slice or B slice may consist of coding blocks of all three color components.

[0116] In some other implementations, the QTBT scheme may be complemented with the ternary scheme described above. Such implementations may be referred to as multi-type tree (MTT) structures. For example, in addition to bisection of the nodes, one of the trisection patterns of FIG. 13 may be selected. In some implementations, only square nodes may undergo trisection. An additional flag may be used to indicate whether the trisection is horizontal or vertical.

[0117] The design of two-level or multi-level trees, such as the QTBT implementation and the QTBT implementation complemented by trisection, may be motivated primarily by reducing complexity. In theory, the complexity of traversing a tree is T D where T denotes the number of split types and D is the depth of the tree. A tradeoff may be made by using multitypes (T) while reducing the depth (D).

[0118] In some implementations, the CB may be further divided. For example, the CB may be further divided into multiple prediction blocks (PBs) for the purpose of intra-frame or inter-frame prediction during the coding and decoding process. In other words, the CB may be further divided into different sub-partitions, where individual prediction decisions / configurations may be made. In parallel, the CB may be further divided into multiple transform blocks (TBs) for the purpose of describing the level at which the transformation or inverse transformation of the video data is performed. The division scheme of the CB into PBs and TBs may be the same or different. For example, each division scheme may be performed using its own procedure, for example, based on various characteristics of the video data. The division schemes of the PBs and TBs may be independent in some exemplary implementations. The division schemes and boundaries of the PBs and TBs may be correlated in some other exemplary implementations. In some implementations, for example, the TBs may be divided after the PB division, and in particular, each PB may be further divided into one or more TBs after being determined following the division of the coding block. For example, in some implementations, the PB may be divided into one, two, four, or other number of TBs.

[0119] In some implementations, the luma and chroma channels may be processed differently to split the base blocks into coding blocks and further into prediction and / or transform blocks. For example, in some implementations, splitting of coding blocks into prediction and / or transform blocks may be allowed for the luma channel, but such splitting of coding blocks into prediction and / or transform blocks may not be allowed for the chroma channels. In such implementations, the transformation and / or prediction of luma blocks may therefore only be performed at the coding block level. In another example, the minimum transform block size of the luma and chroma channels may be different, e.g., the coding blocks of the luma channel may be allowed to be split into smaller transform and / or predictive blocks than the chroma channels. In yet another example, the maximum depth of the splitting of coding blocks into transform and / or predictive blocks may differ between the luma and chroma channels, e.g., the coding blocks of the luma channel may be allowed to be split into deeper transform and / or predictive blocks than the chroma channels. As a specific example, a luma coding block may be divided into transform blocks of multiple sizes that can be represented by a recursive partitioning down by up to two levels, and transform block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4, as well as transform block sizes from 4×4 to 64×64 may be allowed. However, for chroma blocks, only the largest possible transform block designated for the luma block may be allowed.

[0120] In some example implementations for partitioning a coding block into PBs, the depth, shape, and / or other characteristics of the PB partition may depend on whether the PB is intra-coded or inter-coded.

[0121] The division of the coding block (or prediction block) into transform blocks may be performed in various exemplary manners, including but not limited to quadtree division and predefined pattern division, recursively or non-recursively, further considering transform blocks at the boundaries of the coding block or prediction block. In general, the resulting transform blocks may be at different division levels, may not be the same size, and may not need to be square in shape (e.g., they may be rectangular with some allowed size and aspect ratio). Further examples are described in more detail below with respect to Figures 15, 16, and 17.

[0122] However, in some other implementations, the CB obtained via any of the above partitioning schemes may be used as a basic or minimum coding block for prediction and / or transformation. In other words, no further partitioning is performed for the purpose of performing inter / intra prediction and / or transformation. For example, the CB obtained from the above QTBT scheme may be used as a unit for performing prediction as is. Specifically, such a QTBT structure removes the concept of multiple partition types, i.e., removes the separation of CU, PU, ​​and TU, and supports more flexibility for CU / CB partition shapes as described above. In such a QTBT block structure, the CU / CB can have either a square or a rectangle. The leaf nodes of such a QTBT are used as units for prediction and transformation processing without further partitioning. This means that the CU, PU, ​​and TU have the same block size in such an exemplary QTBT coding block structure.

[0123] The various CB division schemes described above, as well as further division of the CB into PB and / or TB (including no PB / TB division), may be combined in any manner. The following specific embodiments are provided as non-limiting examples.

[0124] Specific exemplary implementations of the division of coding blocks and transform blocks are described below. In such exemplary implementations, base blocks may be divided into coding blocks using recursive quadtree division or predefined division patterns described above (such as the division patterns of Figures 9 and 10). At each level, whether further quadtree division of a particular partition should be continued may be determined by local video data characteristics. The resulting CBs may be at different quadtree division levels and of different sizes. The decision on whether to code a picture area using inter-picture (temporal) prediction or intra-picture (spatial) prediction may be made at the CB level (or at the CU level for all three color channels). Each CB may be further divided into one, two, four, or other number of PBs according to a predefined PB division type. Within one PB, the same prediction process may be applied, and related information may be transmitted to the decoder on a PB basis. After obtaining the residual blocks by applying the prediction process based on the PB partition type, the CBs may be divided into TBs according to another quadtree structure similar to the coding tree for CBs. In this particular implementation, the CB or TB need not be limited to a square. Furthermore, in this particular example, the PB may be a square or rectangular shape for inter prediction, and may only be a square for intra prediction. A coding block may be divided, for example, into four square TBs. Each TB may be further divided recursively (using quadtree partitioning) into smaller TBs called residual quadtrees (RQTs).

[0125] Another exemplary embodiment for splitting a base block into CB, PB, and / or TB is further described below. For example, rather than using multiple partition unit types such as the types shown in FIG. 9 or FIG. 10, a quadtree with nested multi-type trees using bipartition and tripartition segmentation structures (e.g., QTBT or QTBT with tripartition as described above) may be used. Separation of CB, PB, and TB (i.e., splitting CB into PB and / or TB, and splitting PB into TB) may be abandoned except when necessary for CBs with sizes too large for the maximum transform length, when such CBs require further splitting. This exemplary splitting scheme may be designed to support further flexibility on CB partition shapes, such that both prediction and transformation may be performed at the CB level without further splitting. In such coding tree structures, the CBs may have either a square or a rectangle. Specifically, the coding tree block (CTB) may first be split by a quadtree structure. The quadtree leaf nodes may then be further split by a nested multi-type tree structure. An example of a nested multi-type tree structure using bisection or trisection is shown in Figure 11. Specifically, the exemplary multi-type tree structure in Figure 11 includes four split types called vertical bisection (SPLIT_BT_VER) 1102, horizontal bisection (SPLIT_BT_HOR) 1104, vertical trisection (SPLIT_TT_VER) 1106, and horizontal trisection (SPLIT_TT_HOR) 1108. Then, CB corresponds to the leaf of the multi-type tree. In this exemplary implementation, as long as CB is not too large for the maximum transform length, this segmentation is used for both prediction and transform processes without further splitting. This means that in most cases, CB, PB, and TB have the same block size in the quadtree with nested multi-type tree coding block structure. An exception occurs when the maximum supported transform length is smaller than the width or height of the color components of CB.In some implementations, in addition to bisection or trisection, the nested pattern of FIG. 11 may further include a quadtree division.

[0126] One specific example of quadtree with nested multi-type tree coding block structure of block partitions (including quadtree, binary tree, and ternary tree splitting options) for one base block is shown in FIG. 12. More specifically, FIG. 12 shows that a base block 1200 is quadtree split into four square partitions 1202, 1204, 1206, and 1208. A decision to further use the multi-type tree structure and quadtree of FIG. 11 for further splitting is made for each of the quadtree split partitions. In the example of FIG. 12, partition 1204 is not split any further. Partitions 1202 and 1208 each adopt another quadtree split. For partition 1202, the second level quadtree split top-left, top-right, bottom-left, and bottom-right partitions adopt third level splits of quadtree, horizontal binary split 1104 of FIG. 11, non-split, and horizontal tri-split 1108 of FIG. 11, respectively. Partition 1208 adopts another quadtree partitioning, and the second level quadtree partitioned top-left, top-right, bottom-left, and bottom-right partitions adopt third level partitioning of vertical trisection 1106, unsplit, unsplit, and horizontal bisection 1104 of FIG. 11, respectively. Two of the subpartitions of the top-left partition of the third level of 1208 are further split according to horizontal bisection 1104 and horizontal trisection 1108 of FIG. 11, respectively. Partition 1206 adopts the second level partitioning pattern into two partitions following vertical bisection 1102 of FIG. 11, and the two partitions are further split at the third level according to horizontal trisection 1108 and vertical bisection 1102 of FIG. 11, respectively. A fourth level partitioning is further applied to one of them according to horizontal bisection 1104 of FIG. 11.

[0127] In the above specific example, the maximum luma transform size may be 64×64, and the maximum supported chroma transform size may be different from the luma, e.g., 32×32. Even if the above example CB of FIG. 12 is not generally further divided into smaller PBs and / or TBs, when the width or height of a luma coding block or a chroma coding block is larger than the maximum transform width or maximum transform height, the luma coding block or the chroma coding block may be automatically divided in the horizontal and / or vertical directions to satisfy the transform size restrictions in that direction.

[0128] In the specific example of splitting the base block into CBs above, as mentioned above, the coding tree scheme can support the ability for luma and chroma to have separate block tree structures. For example, for P slices and B slices, the luma CTB and chroma CTB in one CTU can share the same coding tree structure. For I slices, for example, luma and chroma can have separate coding block tree structures. When separate block tree structures are applied, the luma CTB may be split into luma CBs by one coding tree structure, and the chroma CTB is split into chroma CBs by another coding tree structure. This means that a CU in an I slice may be composed of a coding block of a luma component or a coding block of two chroma components, and a CU in a P slice or B slice is always composed of coding blocks of all three color components, unless the video is monochrome.

[0129] When a coding block is further divided into multiple transform blocks, the transform blocks therein may be ordered in the bitstream according to various orders or scanning schemes. Exemplary implementations for dividing a coding block or a prediction block into transform blocks and the coding order of the transform blocks are described in further detail below. In some exemplary implementations, as described above, the transform partitioning can support transform blocks of multiple shapes, e.g., 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, with transform block sizes ranging from 4×4 to 64×64, for example. In some implementations, if the coding block is smaller than or equal to 64×64, the transform block partitioning may be applied only to the luma component, such that for chroma blocks, the transform block size is the same as the coding block size. Otherwise, if the width or height of the coding block is greater than 64, both the luma coding block and the chroma coding block may be implicitly partitioned into a multiple of min(W,64)×min(H,64) transform blocks and min(W,32)×min(H,32) transform blocks, respectively.

[0130] In some exemplary implementations of transform block partitioning, for both intra-coded and inter-coded blocks, a coding block may be further partitioned into multiple transform blocks with a partition depth up to a predefined number of levels (e.g., two levels). The transform block partition depth and size may be related. In some exemplary implementations, the mapping from the transform size of the current depth to the transform size of the next depth is shown in Table 1 below.

[0131] [Table 1]

[0132] Based on the example mapping in Table 1, for a 1:1 square block, the next level transform partition can create four 1:1 square sub-transform blocks. The transform partition may stop at, for example, 4×4. Thus, the transform size of the current depth of 4×4 corresponds to the same size of 4×4 of the next depth. In the example of Table 1, for a 1:2 / 2:1 non-square block, the next level transform partition can create two 1:1 square sub-transform blocks, while for a 1:4 / 4:1 non-square block, the next level transform partition can create two 1:2 / 2:1 sub-transform blocks.

[0133] In some example implementations, further restrictions may be applied on transform block partitioning for the luma components of intra-coded blocks. For example, for each level of transform partitioning, all sub-transform blocks may be restricted to have equal size. For example, for a 32×16 coding block, level 1 transform partitioning creates two 16×16 sub-transform blocks, and level 2 transform partitioning creates eight 8×8 sub-transform blocks. In other words, to keep the transform units equal in size, a second level partitioning must be applied to all first level sub-blocks. An example of a transform block partitioning of an intra-coded square block according to Table 1 is shown in FIG. 15 with the coding order indicated by the arrows. Specifically, 1502 shows a square coding block. The first level partitioning according to Table 1 into four equal-sized transform blocks is shown in 1504 with the coding order indicated by the arrows. The second level partitioning of all first level equal-sized blocks according to Table 1 into 16 equal-sized transform blocks is shown in 1506 with the coding order indicated by the arrows.

[0134] In some exemplary implementations, the above restrictions on intra-coding may not apply to the luma components of an inter-coded block. For example, after the first level of transform partitioning, any one of the sub-transform blocks may be further partitioned independently at another level. Thus, the resulting transform blocks may or may not be of the same size. An exemplary partitioning of an inter-coded block into transform blocks according to their coding order is shown in FIG. 16. In the example of FIG. 16, an inter-coded block 1602 is partitioned into transform blocks at two levels according to Table 1. At the first level, the inter-coded block is partitioned into four transform blocks of equal size. Then, only one of the four transform blocks (but not all of them) is further partitioned into four sub-transform blocks, as indicated by 1604, resulting in a total of seven transform blocks with two different sizes. An exemplary coding order of these seven transform blocks is indicated by the arrows at 1604 in FIG. 16.

[0135] In some example implementations, for chroma components, some additional restrictions on the transform blocks may be applied, e.g., for chroma components, the transform block size may be as large as the coding block size, but cannot be smaller than a predefined size, e.g., 8×8.

[0136] In some other example implementations, for coding blocks with either width (W) or height (H) greater than 64, both luma coding blocks and chroma coding blocks may be implicitly divided into multiples of min(W,64)×min(H,64) and min(W,32)×min(H,32) transform units, respectively, where in this disclosure "min(a,b)" may return the smaller value between a and b.

[0137] FIG. 17 further illustrates another alternative exemplary scheme for splitting a coding block or a predictive block into transform blocks. As illustrated in FIG. 17, instead of using a recursive transform partition, a set of predefined partition types may be applied to the coding block according to the transform type of the coding block. In the particular example illustrated in FIG. 17, one of six exemplary partition types may be applied to split the coding block into a varying number of transform blocks. Such a scheme for generating transform block partitions may be applied to either the coding block or the predictive block.

[0138] More specifically, the partitioning scheme of FIG. 17 provides up to six exemplary partition types for any given transform type (transform type refers to the type of first-order transform, such as ADST, for example). In this scheme, every coding block or predictive block may be assigned a transform partition type, for example, based on a rate-distortion cost. In one example, the transform partition type assigned to a coding block or predictive block may be determined based on the transform type of the coding block or predictive block. As shown by the six transform partition types illustrated in FIG. 17, a particular transform partition type may correspond to a transform block partition size and pattern. The correspondence between various transform types and various transform partition types may be predefined. An example is shown below, where the capitalized labels indicate transform partition types that may be assigned to a coding block or predictive block based on a rate-distortion cost. ·PARTITION_NONE: Allocate the transformation size equal to the block size. ·PARTITION_SPLIT: Allocates a height of 1 / 2 the block size and a height transformation size of 1 / 2 the block size. ·PARTITION_HORZ: Allocates a transformation size with width equal to the block size and height equal to 1 / 2 the block size. ·PARTITION_VERT: Allocates a transformation size with a width half the block size and a height equal to the block size. ·PARTITION_HORZ4: Allocates a transformation size with width equal to the block size and height 1 / 4 of the block size. ·PARTITION_VERT4: Allocates a transformation size with a width of 1 / 4 of the block size and a height equal to the block size.

[0139] In the above example, all transform partition types as shown in Figure 17 include uniform transform sizes for the partitioned transform blocks. This is merely an example and not a limitation. In some other implementations, mixed transform block sizes may be used for the partitioned transform blocks in a particular partition type (or pattern).

[0140] A video block (PB or CB, also referred to as PB when not further divided into multiple predictive blocks) may be predicted in various manners rather than being directly coded, thereby exploiting various correlations and redundancies in the video data to improve compression efficiency. Correspondingly, such prediction may be performed in various modes. For example, a video block may be predicted via intra prediction or inter prediction. In particular, in an inter prediction mode, a video block may be predicted by one or more other reference blocks or inter predictor blocks from one or more other frames via either single reference inter prediction or mixed reference inter prediction. To perform inter prediction, a reference block may be specified by its frame identifier (the temporal location of the reference block) and a motion vector indicating a spatial offset between a current block being encoded or decoded and the reference block (the spatial location of the reference block). The reference frame identification and the motion vector may be signaled in the bitstream. The motion vector as a spatial block offset may be directly signaled or may itself be predicted by another reference motion vector or a predictor motion vector. For example, the current motion vector may be predicted directly by a reference motion vector (e.g., of a candidate neighboring block) or by a combination of the reference motion vector and a motion vector difference (MVD) between the current motion vector and the reference motion vector. The latter is sometimes called merge mode with motion vector difference (MMVD). The reference motion vector may be identified in the bitstream, for example, as a pointer to a spatially neighboring block of the current block or a temporally neighboring but spatially co-located block.

[0141] Returning to the intra prediction process, samples in a block (e.g., luma or chroma prediction block, or coding block if not further divided into prediction blocks) are predicted by samples of neighboring lines, then neighboring lines, or one or more other lines, or combinations thereof, to generate a prediction block. The residual between the actual block being coded and the prediction block may then be processed through a transform after quantization. Various intra prediction modes may be made available, and parameters related to intra mode selection and other parameters may be signaled in the bitstream. Various intra prediction modes may relate, for example, to one or more line positions for predicting samples, the direction in which the prediction samples are selected from predicting one or more lines, and other special intra prediction modes.

[0142] For example, the set of intra-prediction modes (interchangeably referred to as "intra modes") may include a predefined number of directional intra-prediction modes. As described above with respect to the exemplary embodiment of FIG. 1, these intra-prediction modes may correspond to a predefined number of directions to follow when selecting a sample outside the block as a destination for a sample being predicted within a particular block. In another particular exemplary implementation, eight main directional modes may be supported and predefined, corresponding to angles between 45 degrees and 207 degrees with respect to the horizontal axis.

[0143] In some other implementations of intra prediction, the directional intra modes may be further expanded to a set of angles with finer granularity to further exploit more diverse spatial redundancy in the directional texture. For example, the above 8-angle implementation may be configured to provide eight nominal angles called V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED, as shown in FIG. 19, and a predefined number (e.g., seven) of finer angles may be added to each nominal angle. Such expansion may result in a larger total number of directional angles (e.g., 56 in this example) that may be available for intra prediction, corresponding to the same number of predefined directional intra modes. The prediction angle may be represented by the nominal intra angle + angle delta. In the above specific example with seven finer angle directions for each nominal angle, the angle delta may be -3 to 3 times the step size of 3 degrees. Other exemplary angle schemes may be used, as shown in FIG. 18 with 65 different prediction angles.

[0144] In some implementations, instead of or in addition to the above directional intra modes, a predefined number of non-directional intra prediction modes may also be predefined and made available. For example, five non-directional intra modes called smooth intra prediction modes may be specified. These non-directional intra mode prediction modes may be specifically called DC, PAETH, SMOOTH, SMOOTH_V, and SMOOTH_H intra modes. Prediction of samples of a particular block under these exemplary non-directional modes is shown in FIG. 20. As an example, FIG. 20 shows that a 4×4 block 2002 is predicted by samples from an upper neighboring line and / or a left neighboring line. A particular sample 2010 in the block 2002 may correspond to a sample 2004 immediately above the sample 2010 of the upper neighboring line of the block 2002, a sample 2006 above and to the left of the sample 2010 as the intersection of the upper neighboring line and the left neighboring line, and a sample 2008 immediately to the left of the sample 2010 of the left neighboring line of the block 2002. In the exemplary DC intra prediction mode, the average of the left and top neighboring samples 2008 and 2004 may be used as a predictor for sample 2010. For the exemplary PAETH intra prediction mode, the top, left, and top left reference samples 2004, 2008, and 2006 may be fetched, and then the closest of these three reference samples to (top+left-top left) may be set as the predictor for sample 2010. For the exemplary SMOOTH_V intra prediction mode, sample 2010 may be predicted by quadratic interpolation in the vertical direction of the top left neighboring sample 2006 and the left neighboring sample 2008. For the exemplary SMOOTH_H intra prediction mode, sample 2010 may be predicted by quadratic interpolation in the horizontal direction of the top left neighboring sample 2006 and the top neighboring sample 2004. For the exemplary SMOOTH intra prediction mode, sample 2010 may be predicted by an average of quadratic interpolation in the vertical and horizontal directions. The above non-directional intra-mode implementations are provided as non-limiting examples only.Other neighboring line and other non-directional sample selections are also contemplated, as well as other schemes for combining prediction samples to predict a particular sample within a prediction block.

[0145] The selection of a particular intra-prediction mode by the encoder from the above directional or non-directional modes at various coding levels (picture, slice, block, unit, etc.) may be signaled in the bitstream. In some exemplary implementations, the eight exemplary nominal directional modes may be signaled first, along with the five smooth modes without angles (a total of 13 options). Then, if the signaled mode is one of the eight nominal angle intra-modes, an index indicating the selected angle delta for the corresponding signaled nominal angle is further signaled. In some other exemplary implementations, all intra-prediction modes may be indexed together for signaling (e.g., 56 directional modes plus 5 non-directional modes to generate 61 intra-prediction modes).

[0146] In some exemplary implementations, the exemplary 56 or other number of directional intra-prediction modes may be implemented with a unified directional predictor that projects each sample of a block to a reference subsample position and interpolates the reference samples, for example, by a 2-tap bilinear filter.

[0147] In some implementations, additional filter modes, called FILTER INTRA modes, may be designed to capture the decaying spatial correlation with the references on the edges. In these modes, in addition to samples outside the block, samples predicted within the block may be used as intra prediction reference samples for some patches within the block. These modes may, for example, be predefined and made available for intra prediction of at least the luma block (or only the luma block). A predefined number (e.g., 5) of filter intra modes may be predesigned, each of which is represented by a set of n-tap filters (e.g., 7-tap filters) that reflect the correlation between a sample in a 4×2 patch and its adjacent n neighbors. In other words, the weighting coefficients of the n-tap filters may depend on the position. Taking an 8×8 block, a 4×2 patch, and 7-tap filtering as examples, an 8×8 block 2002 may be divided into eight 4×2 patches, as shown in FIG. 21. These patches are denoted by B0, B1, B1, B3, B4, B5, B6, and B7 in FIG. 21. For each patch, its seven neighbors, denoted by R0-R6 in FIG. 21, may be used to predict samples in the current patch. For patch B0, all neighbors may already be reconstructed. However, for other patches, some of the neighbors may not be reconstructed because they are in the current block, in which case the predicted values ​​of the immediate neighbors are used as reference. For example, all neighbors of patch B7 as shown in FIG. 21 have not been reconstructed, so the predicted samples of the neighbors are used instead.

[0148] In some implementations of intra prediction, a color component may be predicted using one or more other color components, which may be any one of the components of the YCrCb color space, the RGB color space, the XYZ color space, etc.

[0149] One type of intra prediction, which predicts one color component using one or more other color components, is called Chroma from Luma (CfL) prediction. In CfL prediction, a chroma component is predicted based on a luma component. The predicted chroma component may include a chroma block, which may include a sample (or, specifically, a chroma sample). The predicted sample is referred to as a prediction sample. Also, the predicted chroma block may correspond to a luma block. In this specification, unless otherwise specified, the correspondence between a luma block and a chroma block refers to the chroma block being co-located with the luma block.

[0150] Also, as used herein and described in more detail below, the luma components used to predict chroma components may include luma samples. The luma samples may include luma samples of the corresponding or co-located chroma block itself and / or may include neighboring luma samples that are luma samples of one or more neighboring luma blocks in the vicinity of the co-located luma block corresponding to the predicted chroma block. Additionally, in at least some implementations, the luma samples used in the CfL prediction process are reconstructed luma samples, which may be copies of the original luma samples derived or reconstructed from a compressed version of the original luma samples using a decoding process.

[0151] In some implementations, an encoder (e.g., any of the encoders 403, 603, 703) and / or a decoder (e.g., any of the decoders 410, 510, 810) may be configured to perform CfL prediction via a CfL prediction process. Also, in at least some of these implementations, the encoder and / or the decoder may be configured in a CfL prediction mode to perform the CfL prediction process. As described in more detail below, the encoder and / or the decoder may be operable in at least one of a number of different CfL prediction modes. In the different CfL prediction modes, the encoder and / or the decoder may perform different respective CfL processes to generate chroma prediction samples. The chroma prediction samples may alternatively be referred to as chroma predictors.

[0152] Referring to FIG. 22, the encoder and / or decoder may include a CfL prediction unit 2202 configured to perform a CfL prediction process. In various implementations, the CfL prediction unit 2202 may be a standalone unit or a component or subunit of another unit of the encoder or decoder. For example, in any of the various implementations, the CfL prediction unit 2202 may be a component or subunit of the intra prediction unit 552, the intra encoder 722, or the intra decoder 872 described above. The CfL prediction unit 2202 may also be configured to perform various or multiple operations or functions to perform or implement the CfL process. For simplicity, the CfL prediction unit 2202 is described as being a component of the encoder and / or decoder that performs each of these operations or functions. However, in any of the various embodiments, the CfL prediction unit 2202 may be further configured or organized into multiple subunits each configured to perform one or more of the operations or functions of the CfL prediction process, or one or more units other than the CfL prediction unit 2202 may be configured to perform one or more operations of the CfL prediction process. Also, in any of the various embodiments, the CfL prediction unit 2202 may be implemented in hardware or a combination of hardware or software to perform and / or execute the operations or functions of the CfL process. For example, the CfL prediction unit may be implemented as an integrated circuit, or a processor configured to execute software or firmware stored in a memory, or a combination thereof. Also, in any of the various embodiments, a non-transitory computer-readable storage medium may store computer instructions executable by a processor to perform the functions or operations of the CfL prediction unit 2202.

[0153] In the case of CfL prediction, the encoder and / or decoder may determine to apply a CfL prediction mode to a chroma sample or block of a received encoded bitstream, such as via the CfL prediction unit 2202. Such a decision may be extracted or derived from the bitstream in the form of explicit signaling. The CfL prediction unit 2202 may then perform CfL prediction of the chroma sample or chroma block according to the CfL prediction procedure determined to be applied. Correspondingly, the encoder and / or decoder may reconstruct a chroma sample or block corresponding to at least one or more luma samples by applying CfL prediction, such as via the CfL prediction unit 2202.

[0154] FIG. 23 illustrates a flow diagram of an example method 2300 of a CfL prediction process that the CfL prediction unit 2202 may perform. In some implementations, a CfL prediction process such as that illustrated in FIG. 23 generates a number of chroma prediction samples based on alternating current (AC) contributions of reconstructed luma samples and direct current (DC) contributions of chroma samples. Each of the AC and DC contributions may function as a prediction contribution to a chroma component, and are also referred to as the AC and DC prediction contributions. In certain of these implementations, each of the chroma prediction samples is modeled as a linear function of the luma AC and DC prediction contributions, such as according to the following equation: CfL(α)=α×L AC +DC(1) Here, L AC denotes the AC prediction contribution to a chroma sample predicted from a luma component (luma sample), α denotes a scaling parameter of the linear model, and DC denotes a DC prediction contribution to a chroma sample from a chroma component. Also, in at least some implementations, the AC prediction contribution and the DC prediction contribution are obtained for each predicted chroma sample in the block.

[0155] In certain of these implementations as shown in FIG. 23, in block 2302, luma samples in a current block may be reconstructed (either in a decoder or in a loop decoding unit in an encoder), and multiple already reconstructed luma samples of the current luma block may be subsampled (or downsampled) to a chroma resolution (e.g., 4:2:0, 4:2:2, or 4:4:4), especially if the luma components have a higher resolution than the chroma components. In block 2304, the subsampled luma samples may be averaged to generate a luma average (or DC of already reconstructed luma samples in the current block). In block 2306, the luma average may be subtracted from a reconstructed luma sample co-located with the current chroma sample to be predicted to generate an AC portion of the luma component as an AC prediction contribution to the current chroma sample. In block 2308, the AC prediction contribution to the current chroma sample may be multiplied by a scaling parameter α to generate a scaled AC prediction contribution to the current chroma sample. In block 2310, the DC prediction contribution from the chroma components to the current chroma sample may be added to the AC prediction contribution according to the linear model of equation (1) to generate a chroma prediction sample for the current chroma sample. In at least some implementations, the scaling parameter α may be based on the original chroma sample and may be signaled in the bitstream. This may reduce decoder complexity and result in a more accurate prediction.

[0156] In some exemplary implementations, the DC prediction contribution to a current chroma sample or pixel may be calculated using a prediction of the current chroma sample using an intra DC mode. In other words, the DC prediction contribution to the current chroma sample may be generated by averaging the already reconstructed top chroma sample of the current block's upper neighboring chroma line and the already reconstructed left chroma sample of the current block's left neighboring chroma line. As shown in FIG. 20 in particular, the DC prediction contribution to a current pixel 2010 in chroma may be generated by averaging the already reconstructed neighboring block chroma samples 2004 and 2008. In other words, the DC intra mode is used to predict the DC prediction contribution to the predictor of the current chroma sample. The other part of the prediction of the current chroma sample comes from the AC prediction component based on the luma component.

[0157] In some other exemplary implementations, the above DC prediction contribution (the DC term or second term in equation (1)) may be generated in other prediction manners (referred to as other DC prediction modes) instead of the DC intra prediction mode.

[0158] For example, a set of prediction modes may be available to generate a DC prediction contribution for a current chroma sample or pixel. The set of prediction modes available to generate the DC prediction contribution may be predefined. Alternatively, the set of prediction modes available to generate the DC prediction contribution may be configured and signaled in the bitstream. The set of prediction modes for generating the DC prediction contribution may be a subset of a larger set of prediction modes. The larger set of prediction modes may include, for example, various intra-prediction modes. The various intra-prediction modes of the larger set of prediction modes may include, but are not limited to, various directional intra-prediction modes and non-directional intra-prediction modes. The larger set of prediction modes may further include, for example, recursive filtering modes. The larger set of prediction modes may also include offset-based prediction modes, for example using a multi-tap offset filter based on neighboring samples of the current chroma pixel. Such offset-based prediction may be further combined with other intra-prediction modes, for example in the form of offset-based refinement of intra-prediction mode (ORIP). In particular, in some example implementations, a chroma sample that is intra predicted using any of the intra prediction modes may be further filtered before being used to generate a DC prediction contribution for the current chroma pixel or sample. Such a filter may be implemented to add an offset to the DC prediction contribution for the current chroma pixel, for example, using ORIP. Such composite modes may be included within a larger set of prediction modes available for generating a DC prediction contribution for the current chroma sample or pixel.

[0159] The set of prediction modes available for generating the DC prediction contribution may be selected from a larger set of prediction modes and may be any combination of various directional intra prediction modes, non-directional intra prediction modes, recursive filtering modes, offset-based prediction modes, and other prediction modes. The directional intra prediction modes may include generating intra predictions of samples or pixels using various directional samples, for example, as shown in Figures 1B, 18, and 19. The non-directional intra prediction modes may include, but are not limited to, the DC, SMOOTH, SMOOTH-V, SMOOTH-H, and PAETH intra prediction modes described above. The selection of components of the set of prediction modes available for generating the DC prediction contribution for the current chroma sample may be signaled at various levels of the bitstream. The prediction modes used within the set of available prediction modes may be further selected via optimization by the encoder at the chroma sample level or higher, and such specific selection may be signaled in the bitstream at these corresponding levels. The selected prediction mode for the DC prediction contribution may be referred to as "Mode A".

[0160] In some example implementations, the set of prediction modes available for generating the DC prediction contribution may include one or more of the SMOOTH, SMOOTH-V, SMOOTH-H, or PAETH intra prediction modes.

[0161] In some exemplary implementations, the set of prediction modes available for generating the DC prediction contribution may include only one prediction mode. Such a prediction mode may be, for example, any of the prediction modes mentioned above other than the DC intra prediction mode. Specifically, the one prediction mode available for generating the DC prediction contribution may be a non-directional intra prediction mode instead of the DC mode. For example, the one prediction mode available for generating the DC prediction contribution may be one of the non-directional intra prediction modes including SMOOTH, SMOOTH-V, SMOOTH-H, or PAETH intra prediction modes.

[0162] In some exemplary implementations, the set of prediction modes available for generating the DC prediction contribution may include only two prediction modes. The two prediction modes may be determined from any two of the larger sets of prediction modes listed above. For example, the two prediction modes may include two non-directional intra prediction modes, such as any combination of two prediction modes among DC, SMOOTH, SMOOTH-V, SMOOTH-H, and PAETH intra prediction modes. For example, the two prediction modes may include two non-directional intra prediction modes, such as any combination of two prediction modes among SMOOTH, SMOOTH-V, SMOOTH-H, and PAETH intra prediction modes.

[0163] In some exemplary implementations, the prediction mode selected to generate the DC prediction contribution for the current chroma sample or pixel may be signaled or implicitly derived. For example, the set of prediction modes available to generate the DC prediction contribution for the current chroma sample or pixel may include two predefined prediction modes as described above, and one bit of explicit signaling may be implemented to indicate which of the two prediction modes is used to generate the DC prediction contribution for the current chroma sample or pixel. In some other exemplary implementations, a single prediction mode may be predefined and thus implicitly derived as the prediction mode for generating the DC prediction contribution for the current chroma sample or pixel. The single predefined prediction mode may be, for example, a SMOOTH intra prediction mode.

[0164] In some other example implementations, the prediction mode selected to generate the DC prediction contribution for the current chroma sample or pixel may be implicitly indicated by other coding information, including but not limited to a block size, a co-located luma prediction mode, and a neighboring block prediction mode. For example, a correspondence between multiple prediction modes and block sizes may be predefined or configured, and the prediction mode for generating the DC prediction contribution for the current chroma sample or pixel may be derived from the current block size according to such correspondence. In another example, the prediction mode for generating the DC prediction contribution for the current chroma sample or pixel may be derived as the same prediction mode used by a luma sample co-located with the chroma sample or by a neighboring block (selected from neighboring blocks, such as, for example, an upper neighboring block, then a left neighboring block, at a predefined position or in a predefined scan order).

[0165] Returning to the AC prediction contribution from the luma sample to the current chroma sample or pixel (the AC term or the first term in Equation (1)), rather than subtracting an average of already constructed luma samples in the current coding block (as shown by 2304 in Figure 23 and described above), other collective forms of luma samples representing DC levels of luma samples co-located or located near the current chroma sample or pixel may be subtracted from the subsampled luma sample (2302 in Figure 23) and co-located with the chroma sample or pixel. For example, the AC prediction contribution to the current chroma sample or pixel may be calculated by subtracting a prediction sample of the co-located luma sample from the co-located luma sample. Such a prediction sample of the co-located luma sample may be based on one of various prediction modes, including, but not limited to, a directional intra prediction mode, a non-directional intra prediction mode, a recursive filtering prediction mode, an offset-based prediction mode, etc. The predicted samples of the co-located luma samples may be referred to as the luma average equivalent (e.g., equivalent to luma average 2304 in Figure 23). The prediction mode selected to generate the luma average equivalent may be referred to as "Mode B." The luma samples used to generate the luma average equivalent may include chroma samples or reconstructed neighboring luma samples of the luma samples co-located with the pixel.

[0166] For example, mode B may be one of the non-directional intra prediction modes of DC, SMOOTH, SMOOTH-H, SMOOTH-V, PAETH modes. Mode B may be one of the non-directional intra prediction modes of SMOOTH, SMOOTH-H, SMOOTH-V, PAETH modes. Mode B may be one of the directional intra prediction modes described above in connection with Figures 1B, 18, and 19. Mode B may be a recursive filtering mode or an offset-based prediction mode (such as ORIP).

[0167] In some example implementations, the luma average equivalent may be generated using the same prediction mode as the chroma prediction mode used to generate the DC prediction contribution for the chroma sample or pixel, as described above. In other words, mode B may be selected as mode A or may be the same as mode A by default. As such, mode B may be derived implicitly according to an indication or signaling of mode A.

[0168] In some other example implementations, mode B may be derived in other ways. For example, mode B may be implicitly set to the intra-prediction mode of one of the neighboring blocks, such as the neighboring block to the left or above the current block. In some example implementations, the prediction mode of the co-located luma sample may be implicitly used as mode B.

[0169] The various exemplary implementations above are described in the context of CfL according to equation (1), where the AC prediction contributions to chroma samples are derived from luma samples, and the DC prediction contributions to chroma samples are derived from predictions of chroma samples. The basic principles of these implementations are not so limited. For example, a chroma sample only represents a sample of one of the color channels, and a luma sample only represents a sample of another color channel. The basic principles described above generally apply to cross-channel prediction between any two color channels. Such predictions may be referred to as cross-channel predictions.

[0170] FIG. 24 illustrates a flowchart of an example method 2400 of an inter-channel prediction process (such as a CfL process) that may be implemented by the prediction unit 2202 of FIG. 22. At 2401, the cross-channel prediction process starts. At 2410, it is determined to apply a cross-channel prediction mode to a current sample of a chroma channel in a current coding block based on at least one reconstructed sample from a luma channel of a received bitstream. At 2420, an alternating current (AC) contribution to a predictor of a current sample of a chroma channel is generated based on at least one reconstructed sample from the luma channel. At 2430, a direct current (DC) contribution to a predictor of a current sample of a first color channel is generated by applying intra prediction of the current sample of the first color channel based on at least one sample of the first color channel under an intra prediction mode other than a DC intra prediction mode. At 2440, a predictor of the current sample of the first color channel is generated by calculating a weighted average of the AC contribution and the DC contribution. At 2450, the current sample of the first color channel is reconstructed based on the predictor of at least the current sample of the first color channel. At 2499, the cross-channel prediction process ends.

[0171] The embodiments in this disclosure may be used separately or combined in any order. Furthermore, each of the methods (or embodiments), the encoder, and the decoder may be implemented by a processing circuit (e.g., one or more processors, or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium. The embodiments of this disclosure may be applied to luma blocks or chroma blocks.

[0172] The techniques described above may be implemented as computer software using computer readable instructions and physically stored on one or more computer readable media. For example, FIG. 25 illustrates a computer system 2500 suitable for implementing certain embodiments of the disclosed subject matter.

[0173] Computer software can be coded using any suitable machine code or computer language that is amenable to mechanisms such as assembly, compilation, linking, etc., to create code that includes instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc. directly, or via interpretation, microcode execution, etc.

[0174] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming consoles, Internet of Things devices, and the like.

[0175] 25 for computer system 2500 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of computer system 2500.

[0176] The computer system 2500 may include certain human interface input devices. Such human interface input devices may respond to input by one or more human users, for example, via tactile input (such as keystrokes, swipes, data glove movements), audio input (such as voice, clapping), visual input (such as gestures), or olfactory input (not depicted). Human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (such as voice, music, ambient sounds), images (such as scanned images, photographic images obtained from a still camera), and video (such as two-dimensional video, three-dimensional video including stereoscopic video).

[0177] The input human interface devices may include one or more of a keyboard 2501, a mouse 2502, a trackpad 2503, a touch screen 2510, a data glove (not shown), a joystick 2505, a microphone 2506, a scanner 2507, and a camera 2508 (only one of each is depicted).

[0178] The computer system 2500 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen 2510, data gloves (not shown), or joystick 2505, although there may also be haptic feedback devices that do not function as input devices), audio output devices (speakers 2509, headphones (not shown), etc.), visual output devices (screens 2510 including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capabilities, each with or without haptic feedback capabilities, some of which may be capable of outputting two-dimensional visual output or output in more than three dimensions through means such as stereo output), virtual reality glasses (not shown), holographic displays and smoke tanks (not shown), and printers (not shown).

[0179] The computer system 2500 may also include human accessible storage devices and their associated media, such as optical media including a CD / DVD ROM / RW 2520 having media 2521 such as CDs / DVDs, thumb drives 2522, removable hard drives or solid state drives 2523, legacy magnetic media such as tapes and floppy disks (not shown), and dedicated ROM / ASIC / PLD based devices (not shown) such as security dongles.

[0180] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0181] The computer system 2500 may also include an interface 2554 to one or more communication networks 2555. The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, cellular networks including wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., cable or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicular and industrial including CAN bus, etc. Certain networks generally require an external network interface adapter attached to a particular general-purpose data port or peripheral bus 2549 (e.g., a USB port of the computer system 2500, etc.), while other networks are generally integrated into the core of the computer system 2500 by attachment to the system bus, as described below (e.g., an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (2500) may communicate with other entities. Such communications may be one-way receive only (e.g., television broadcast), one-way transmit only (e.g., from the CANbus to a particular CANbus device), or two-way, for example, to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used in each of these networks and network interfaces, as described above.

[0182] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to core 2540 of computer system 2500.

[0183] The core 2540 may include one or more central processing units (CPUs) 2541, graphics processing units (GPUs) 2542, dedicated programmable processing units in the form of field programmable gate areas (FPGAs) 2543, hardware accelerators 2544 for specific tasks, graphics adapters 2550, etc. These devices may be connected via a system bus 2548 along with read only memory (ROM) 2545, random access memory 2546, internal mass storage devices 2547 such as internal hard drives, SSDs, etc. that are not accessible to the user. In some computer systems, the system bus 2548 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 2548 or via a peripheral bus 2549. In one example, a screen 2510 may be connected to the graphics adapter 2550. Peripheral bus architectures include PCI, USB, etc.

[0184] The CPU 2541, GPU 2542, FPGA 2543, and accelerator 2544 can execute certain instructions that can combine to constitute the aforementioned computer code. That computer code can be stored in ROM 2545 or RAM 2546. Transient data can also be stored in RAM 2546, while persistent data can be stored, for example, in internal mass storage device 2547. The use of cache memory, which can be closely associated with one or more of the CPU 2541, GPU 2542, mass storage device 2547, ROM 2545, RAM 2546, etc., can enable fast storage and retrieval from any of the memory devices.

[0185] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the available kind well known to those skilled in the computer software arts.

[0186] As a non-limiting example, the architecture 2500, and specifically a computer system having the cores 2540, can provide functionality as a result of the processors (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage devices as introduced above, as well as specific storage devices of a particular core 2540 that are non-transient in nature, such as mass storage device 2547 internal to the core or ROM 2545. Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the cores 2540. The computer-readable media can include one or more memory devices or chips, depending on the particular needs. The software can cause the cores 2540, and specifically the processors therein (including CPUs, GPUs, FPGAs, etc.) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM 2546 and modifying such data structures according to the processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 2544) that may operate in place of or in conjunction with software to perform certain processes or certain portions of certain processes described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that store software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0187] In addition to the features mentioned in each of the independent aspects listed above, some examples may exhibit, alone or in combination, any features mentioned in the dependent aspects and / or disclosed in the above description and illustrated in the figures.

[0188] While this disclosure describes several exemplary embodiments, there are alterations, substitutions, and various alternative equivalents that fall within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art can devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure. Appendix A: Acronyms JEM Joint Exploration Model VVC Versatile Video Coding BMS Benchmark Set MV Motion Vector HEVC High Efficiency Video Coding SEI Supplemental Extension Information VUI Video Usability Information GOP Groups of Pictures TU Conversion Unit PU Prediction Unit CTU Coding Tree Unit CTB coding tree block PB Predicted Block HRD Hypothetical Reference Decoder SNR Signal to Noise Ratio CPU Central Processing Unit GPU Graphics Processing Unit CRT Braun tube LCD Liquid Crystal Display OLED Organic Light Emitting Diode CD Compact Disc DVD Digital Video Disc ROM Read Only Memory RAM Random Access Memory ASIC Application Specific Integrated Circuit PLD Programmable Logic Device LAN Local Area Network GSM Global System for Mobile Communications LTE Long Term Evolution CANBus Controller Area Network Bus USB Universal Serial Bus PCI Peripheral Component Interconnect FPGA Field Programmable Gate Area SSD Solid State Drive IC Integrated Circuit HDR High Dynamic Range SDR Standard Dynamic Range JVET Joint Video Exploration Team MPM Most Accurate Mode WAIP Wide-angle Intra Prediction CU coding unit PU Prediction Unit TU Conversion Unit CTU Coding Tree Unit PDPC Position-dependent prediction combination ISP Intra Subpartition SPS sequence parameter settings PPS Picture Parameter Set APS Adaptation Parameter Set VPS Video Parameter Set DPS Decoding Parameter Set ALF Adaptive Loop Filter SAO Sample Adaptive Offset CC-ALF Cross-Component Adaptive Loop Filter CDEF Constrained Directional Extension Filter CCSO Cross Component Sample Offset LSO Local Sample Offset LR Loop Recovery Filter AV1 AOMedia Video 1 AV2 AOMedia Video 2 LFNST Low Frequency Nonseparable Transform IST Intra Quadratic Transform Chroma from CfL Luma [Explanation of symbols]

[0189] 101 sample, 102 arrow, 103 arrow, 104 block, 300 communication system, 310 terminal, 320 terminal, 330 terminal, 350 communication network, 400 communication system, 401 video source, 402 stream, 403 video encoder, 404 video bitstream, 405 streaming server, 406 client subsystem, 407 copy of video data, 410 video decoder, 411 video picture, 412 display, 413 video capture subsystem, 420 electronic device, 430 electronic device, 501 channel, 510 video decoder, 512 display, 515 buffer memory, 520 parser, 521 symbol, 530 electronic device, 531 receiver, 551 inverse transform unit, 552 intra-picture prediction unit, 553 motion compensation prediction unit, 555 aggregator, 556 Loop filter unit, 557, reference picture memory, 558, picture buffer, 601, video source, 603, video encoder, 620, encoder, 630, source coder, 632, coding engine, 633, decoder, 633, local video decoder, 634, reference picture memory, 635, predictor, 640, transmitter, 643, video sequence, 645, entropy coder, 650, controller, 660, communication channel, 703, video encoder, 721, general controller, 722, intra encoder, 723, residual calculator, 724, residual encoder, 725, entropy encoder, 726, switch, 728, residual decoder, 730, inter encoder, 810, video decoder, 871, entropy decoder, 872, intra decoder, 873, residual decoder, 874, reconstruction module, 880, inter decoder, 1002, partition, 1004 partition, 1006 partition, 1102 vertical halves, 1104 horizontal halves, 1106 vertical thirds, 1108 horizontal thirds, 1200 base block, 1202 partition, 1204 partition, 1206 partition, 1208 partition, 1602 block, 2002 block, 2004 sample, 2006 top left sample, 2008sample, 2010 sample, 2202 prediction unit, 2302 block, 2304 block, 2306 block, 2308 block, 2310 block, 2500 computer system, 2501 keyboard, 2502 mouse, 2503 trackpad, 2505 joystick, 2506 microphone, 2507 scanner, 2508 camera, 2509 speaker, 2510 screen, 2521 medium, 2522 thumb drive, 2523 solid state drive, 2540 core, 2541 central processing unit (CPU), 2542 graphics processing unit (GPU), 2543 field programmable gate area (FPGA), 2544 accelerator, 2545 read only memory (ROM), 2546 random access memory (RAM), 2547 mass storage device, 2548 System bus, 2549 peripheral bus, 2550 graphics adapter, 2554 interface, 2555 communication network

Claims

1. 1. A method for video processing, the method comprising: determining, based on at least one reconstructed sample from a luma channel of the received bitstream, a chroma from luma (CfL) prediction mode to apply to a current sample of a chroma channel in a current coding block; deriving an AC (AC) contribution to a predictor of the current sample of the chroma channel based on the at least one reconstructed sample from the luma channel; performing intra prediction of the current sample of the chroma channel based on at least one reconstructed sample of the chroma channel under an intra prediction mode other than a direct current (DC) intra prediction mode to derive a DC contribution to the predictor of the current sample of the chroma channel; calculating a weighted average of the AC contribution and the DC contribution to generate the predictor for the current sample of the chroma channel; reconstructing the current sample of the chroma channel based on the predictor of the current sample of at least the chroma channel.

2. The method of claim 1 , further comprising: subsampling the luma channel of the current coding block to obtain subsampled luma blocks that correspond to chroma blocks of the chroma channels of the current coding block.

3. 3. The method of claim 2, wherein the intra prediction mode for performing the intra prediction to derive the DC contribution to the predictor of the current sample of the chroma channel comprises one of SMOOTH, SMOOTH-H, SMOOTH-V, PAETH, directional filtering, recursive filtering, or an offset-based refined intra prediction mode.

4. 4. The method according to claim 1, wherein the intra prediction mode for performing the intra prediction to derive the DC contribution to the predictor of the current sample of the chroma channel is predetermined.

5. 4. The method of claim 1, wherein the intra prediction mode for performing the intra prediction to derive the DC contribution to the predictor of the current sample of the chroma channel is selected from a predetermined subset of intra prediction modes according to explicit signaling in the received bitstream.

6. 4. The method of claim 1, wherein the intra prediction mode for performing the intra prediction to derive the DC contribution to the predictor of the current sample of the chroma channel is derived based on a reference intra prediction mode associated with a luma sample in the subsampled luma block, the luma sample being co-located with the current sample of the chroma channel.

7. 4. The method of claim 1, wherein the intra prediction mode for performing the intra prediction to derive the DC contribution to the predictor of the current sample of the chroma channel is derived from a reference intra prediction mode associated with a neighboring chroma block of the current coding block.

8. 4. The method of claim 1, wherein the intra prediction mode for performing the intra prediction to derive the DC contribution to the predictor of the current sample of the chroma channel is derived according to a size of the current coding block.

9. 4. The method of claim 1, wherein deriving the AC contribution to the predictor of the current sample of the chroma channel comprises subtracting a sample average of a subsampled reconstructed luma block from a reconstructed luma sample co-located with the current sample of the chroma channel.

10. 4. The method of claim 1, wherein deriving the AC contribution to the predictor of the current sample of the chroma channel comprises subtracting a prediction of a luma sample co-located with the current sample of the chroma channel from the reconstructed luma sample.

11. The method of claim 10 , wherein the prediction of the luma sample is generated under an intra prediction mode.

12. 12. The method of claim 11, wherein the intra prediction mode for generating the prediction of the luma sample includes one of a SMOOTH, a SMOOTH-H, a SMOOTH-V, a PAETH, a directional filtering, a recursive filtering, or an offset-based refinement intra prediction mode.

13. The method of claim 12, wherein the intra-prediction mode for generating the prediction of the luma samples is explicitly signaled in the received bitstream.

14. The method of claim 11 , wherein the intra-prediction mode for generating the prediction of the luma sample is derived implicitly.

15. 15. The method of claim 14, wherein the intra prediction mode for generating the prediction of the luma sample is implicitly derived as the same as the intra prediction mode for deriving the DC contribution to the predictor of the current sample of the chroma channel.

16. The method of claim 14 , wherein the intra prediction mode for generating the prediction of the luma sample is derived as a reference intra prediction mode applied to the luma sample co-located with the current sample of the chroma channel.

17. The method of claim 14 , wherein the intra prediction mode for generating the prediction of the luma sample is derived from a reference intra prediction mode associated with a neighboring luma block of the current coding block.

18. 1. An electronic device for video processing, comprising: determining, based on the at least one reconstructed sample from a luma channel of the received bitstream, to apply a cross-channel prediction mode to a current sample of a chroma channel in the current coding block; deriving an AC (AC) contribution to a predictor of the current sample of the chroma channel based on the at least one reconstructed sample from the luma channel; performing intra prediction of the current sample of the chroma channel based on at least one reconstructed sample of the chroma channel under an intra prediction mode other than a direct current (DC) intra prediction mode to derive a DC contribution to the predictor of the current sample of the chroma channel; calculating a weighted average of the AC contribution and the DC contribution to generate the predictor for the current sample of the chroma channel; 11. An electronic device comprising: a memory for storing instructions and a processor for executing instructions to cause the current sample of the chroma channel to be reconstructed based on the predictor of at least the current sample of the chroma channel.

19. 20. The electronic device of claim 18, wherein the intra prediction mode for performing the intra prediction to derive the DC contribution to the predictor of the current sample of the chroma channel includes one of SMOOTH, SMOOTH-H, SMOOTH-V, PAETH, directional filtering, recursive filtering, or an offset-based improved intra prediction mode.

20. A non-transitory computer readable storage medium for storing computer instructions, the computer instructions, when executed by a process or electronic device, causing the electronic device to: determining, based on at least one reconstructed sample from a luma channel of the received bitstream, to apply a chroma from luma (CfL) prediction mode to a current sample of a chroma channel in a current coding block; deriving an AC (AC) contribution to a predictor of the current sample of the chroma channel based on the at least one reconstructed sample from the luma channel; performing intra prediction of the current sample of the chroma channel based on at least one reconstructed sample of the chroma channel under an intra prediction mode other than a direct current (DC) intra prediction mode to derive a DC contribution to the predictor of the current sample of the chroma channel; calculating a weighted average of the AC contribution and the DC contribution to generate the predictor for the current sample of the chroma channel; 13. A non-transitory computer-readable storage medium configured to cause the current sample of the chroma channel to be reconstructed based on the predictor of the current sample of at least the chroma channel.

Images