Chroma-from-Luma Prediction Using Nearby Luma Samples

Abstract

The present disclosure relates to video processing, including a video processing device, the video device determining that a Chroma from Luma (CfL) prediction mode should be applied to a luma block of a received coded bitstream; generating a nearby luma average for the luma block by averaging a set of reconstructed luma samples, the set of reconstructed luma samples including a plurality of reconstructed nearby luma samples in at least one nearby luma block in a vicinity of the luma block; generating alternating current (AC) contributions of a plurality of predictive samples of a chroma block co-located with the luma block based on the plurality of luma samples in the luma block and the nearby luma average; and reconstructing the chroma block by at least applying the CfL prediction mode based on the AC contributions.

Description

[Technical field]

[0001]

[0001] Incorporation by Reference This application claims priority to U.S. Non-provisional Application No. 17 / 951,911, entitled "Chroma-From-Luma Prediction Using Nearby Luma Samples," filed on September 23, 2022, and U.S. Provisional Application No. 63 / 330,706, entitled "Improved Chroma-From-Luma Intra Prediction Modes," filed on April 13, 2022, each of which is incorporated by reference in its entirety.

[0002]

[0002] Technical Field This disclosure describes a set of advanced video coding techniques. More specifically, the disclosed techniques relate to chroma-from-luma prediction. [Background technology]

[0003] This background discussion provided herein is intended to generally present the context of the present disclosure. Work undertaken by the current inventor is not admitted, expressly or impliedly, as prior art to the present disclosure to the extent that such work is described in this background section or in a descriptive manner that may not otherwise qualify as prior art as of the filing date of this case.

[0004]

[0002] Video coding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a sequence of pictures, each having spatial dimensions of, for example, 1920x1080 luminance samples and associated full or subsampled chrominance samples. The sequence of pictures can have a fixed or variable picture rate (alternatively called frame rate), for example, 60 pictures per second or 60 frames per second. Uncompressed video has specific bitrate requirements for streaming or data processing. For example, a video with a pixel resolution of 1920x1080, a frame rate of 60 frames per second, and 4:2:0 chroma subsampling at 8 bits per pixel per color channel requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 Gbytes of storage space.

[0005]

[0003] One of the goals of video coding and decoding can be said to be the reduction of redundancy in the uncompressed input video signal by compression. Compression can help reduce the aforementioned bandwidth and / or storage space requirements, in some cases by more than one order of magnitude. Both lossless and non-lossless compression, as well as combinations thereof, can be used. Lossless compression refers to techniques where an exact copy of the original signal can be reconstructed from the compressed original signal by the decoding process. Non-lossless compression refers to coding / decoding processes where the original video information is not fully preserved during coding and is not fully recoverable during decoding. When using non-lossless compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is small enough to make the reconstructed signal useful for the intended application despite some information loss. For video, non-lossless compression is widely used in many applications. The amount of distortion that is tolerated depends on the application. For example, a user of a particular consumer video streaming application may be able to tolerate higher distortion than a user of a movie or television broadcast application. The compression ratio achievable by a particular coding algorithm can be selected or adjusted to reflect different distortion tolerances: higher tolerable distortion generally allows for coding algorithms that result in higher loss and higher compression ratios.

[0006]

[0004] Video encoders and decoders may utilize techniques from a number of broad categories and steps, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

[0007]

[0005] Video codec techniques may include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. If all blocks of samples are coded in intra mode, the picture may be referred to as an intra picture. Intra pictures and their derivatives (e.g., independent decoder refresh pictures) may be used to reset the decoder state and thus may be used as the first picture in a coded video bitstream and video session or as a still image. The samples of the block after intra prediction may be subjected to a transform to the frequency domain, and the transform coefficients so generated may be quantized before entropy coding. Intra prediction refers to a technique that minimizes the sample values ​​in the pre-transform domain. In some cases, the smaller the transformed DC value and the smaller the AC coefficients, the fewer bits needed for a given quantization step size to represent the block after entropy coding.

[0008]

[0006] Traditional intra-coding, such as that known from MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to code / decode a block based on, for example, spatially adjacent surrounding sample data and / or metadata available during encoding / decoding that precede the block of data being coded or decoded in decoding order. Such techniques are hereafter referred to as "intra-prediction" techniques. Note that, at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed, and not from other reference pictures.

[0009]

[0007] There may be many different forms of intra prediction. When more than one such technique may be used in a given video coding technique, the technique used may be referred to as an intra prediction mode. One or more intra prediction modes may be provided in a particular codec. In some cases, a mode may have sub-modes and / or may be associated with various parameters, and the mode / sub-mode information and intra coding parameters for a block of video may be coded individually or collectively included in a mode codeword. The codeword used for a given mode, sub-mode, and / or parameter combination may affect the coding efficiency gain from intra prediction, and therefore may also affect the entropy coding technique used to convert the codeword into a bitstream.

[0010]

[0008] Certain modes of intra prediction were introduced in H.264, improved in H.265, and further improved in new coding techniques such as the Joint Search Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). In general, for intra prediction, a prediction block can be formed using available neighboring sample values. For example, available values ​​of a particular set of neighboring samples along a particular direction and / or line can be copied to a predictor block. The reference to the direction to use can be coded in the bitstream or can be predicted itself.

[0011]

[0009] Referring to FIG. 1A, shown at the bottom right is a subset of 9 predictor directions specified among the 33 possible intra predictor directions of H.265 (corresponding to the 33 angle modes of the 35 intra modes specified in H.265). The point 101 where the arrows converge represents the sample to be predicted. The arrows indicate the direction from which nearby samples are used to predict the sample at 101. For example, arrow 102 indicates that sample 101 is predicted from one or more nearby samples at an angle of 45 degrees from the horizontal to the upper right. Similarly, arrow 103 indicates that sample 101 is predicted from one or more nearby samples at an angle of 22.5 degrees from the horizontal to the lower left of sample 101.

[0012]

[0010] With further reference to FIG. 1A, a square block (104) of 4x4 samples is shown at the top left (indicated by a thick dashed line). The square block 104 contains 16 samples, each of which is labeled with "S" and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in the block 104 in both the Y and X dimensions. Since the block is 4x4 samples in size, S44 is at the bottom right. Also shown are exemplary reference samples following a similar numbering scheme. The reference samples are labeled with R and their Y position (e.g., row index) and X position (column index) relative to the block 104. In both H.264 and H.265, nearby prediction samples that are adjacent to the block being reconstructed are used.

[0013]

[0011] Intra-picture prediction of block 104 may start by copying reference sample values ​​from nearby samples according to the signaled prediction direction. For example, assume that the coded video bitstream includes signaling for this block 104 indicating a prediction direction of arrow 102 - i.e., the sample is predicted from one or more prediction samples pointing to the upper right at an angle of 45 degrees from the horizontal. Then samples S41, S32, S23, and S14 are predicted from the same reference sample R05. And sample S44 is predicted from reference sample R08.

[0014] In some cases, values ​​of multiple reference samples may be combined, for example by interpolation, to calculate a reference sample; particularly when the orientation is not evenly divisible by 45 degrees.

[0015]

[0013] The number of possible directions is increasing as video coding technology continues to develop. For example, in H.264 (2003), nine different directions were available for intra prediction. This increased to 33 in H.265 (2013), and JEM / VVC / BMS at the time of this disclosure can support up to 65 directions. Experimental studies have been conducted to help identify the most likely intra prediction direction, and certain techniques in entropy coding can be used to encode the most likely direction with fewer bits, while accepting a certain bit penalty for the direction. Furthermore, the direction itself can often be predicted from nearby directions used in intra prediction of nearby blocks being decoded.

[0016] FIG. 1B shows a diagram 180 illustrating the 65 intra prediction directions according to the JEM, illustrating the increasing number of prediction directions in different coding techniques that have evolved over time.

[0017]

[0015] The mapping of bits representing intra-prediction directions to prediction directions in the coded video bitstream may vary from one video coding technique to another; for example, it may range from a simple direct mapping of prediction directions to complex adaptation schemes involving intra-prediction modes, codewords, most-probable modes, and similar techniques. However, in any given case, there may be certain directions of intra-prediction that are statistically less likely to occur in the video content than certain other directions. Since the goal of video compression is to reduce redundancy, these unlikely directions can be represented with more bits than the more likely directions in a well-designed video coding technique.

[0018]

[0016] Inter-picture prediction or inter-prediction can be based on motion compensation, in which sample data from a previously reconstructed picture or part of it (reference picture) can be used to predict a newly reconstructed picture or part of a picture (e.g., a block) after spatial shifting in the direction indicated by a motion vector (hereinafter MV). In some cases, the reference picture can be the same as the picture currently being reconstructed. MV can have two dimensions X and Y, or three dimensions, the third dimension being an indication of the reference picture in use (the latter is analogous to the temporal dimension).

[0019]

[0017] In some video compression techniques, the current MV applicable to a particular area of ​​sample data can be predicted from other MVs, e.g., from other MVs related to other areas of sample data spatially adjacent to the area being reconstructed and preceding the current MV in decoding order. In this way, the overall amount of data required to code the MV can be significantly reduced by relying on the reduction of redundancy in the interrelated MVs, thereby increasing the compression efficiency. MV prediction can work effectively because, for example, when coding an input video signal derived from a camera (known as natural video), there is a statistical likelihood that areas larger than the area to which a single MV is applicable move in a similar direction in the video sequence and can therefore possibly be predicted using similar motion vectors derived from MVs of nearby areas. This results in an actual MV for a given area that is similar or identical to the MV predicted from the surrounding MVs. Such MVs can be represented with fewer bits than would be used if the MVs were coded directly, rather than predicted from nearby MVs, after entropy coding. In some cases, MV prediction may be an example of lossless compression of a signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, the MV prediction itself may be non-lossless, for example due to rounding errors when computing the predictor from some surrounding MVs.

[0020]

[0018] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms specified by H.265, the one described below is a technique hereafter called "spatial merge".

[0021]

[0019] Specifically, referring to Fig. 2, a current block (201) contains samples that have been found by the encoder during the motion search process to be predictable from a previous block of the same size but spatially shifted. Instead of coding its MV directly, the MV can be derived from metadata associated with one or more reference pictures, for example, from the most recent reference picture (in decoding order) using MVs associated with any of the five surrounding samples (202 to 206, respectively) marked A0, A1, and B0, B1, B2. In H.265, MV prediction can use predictors from the same reference picture that nearby blocks use. Summary of the Invention

[0022]

[0020] Aspects of the disclosure provide a method and apparatus for Chroma from Luma (CfL) prediction.

[0023] In some implementations, a method for video processing includes: determining that a Chroma from Luma (CfL) prediction mode should be applied to a luma block of the received coded bitstream; generating a nearby luma average for a luma block by averaging a set of reconstructed luma samples, the set of reconstructed luma samples including a plurality of nearby reconstructed luma samples in at least one nearby luma block in a vicinity of the luma block; generating alternating current (AC) contributions of a plurality of prediction samples of a chroma block co-located with the luma block based on a plurality of luma samples in the luma block and a nearby luma average; and Reconstructing chroma blocks by applying at least the CfL prediction modes based on AC contributions.

[0024]

[0022] In some other implementations, a device for processing video information is disclosed. The device may include circuitry configured to perform any of the implementations of the methods described above.

[0025]

[0023] An aspect of the present disclosure also provides a non-transitory computer-readable storage medium storing instructions that, when executed by a computer for video encoding and / or decoding, cause the computer to perform the above-mentioned method for video encoding and / or decoding. [Brief description of the drawings]

[0026] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1A] FIG. 1A shows a schematic diagram of an example subset of intra-prediction direction modes. [Figure 1B] FIG. 1B shows an example of an exemplary intra prediction direction. [Diagram 2]

[0027] FIG. 2 shows a schematic diagram of a current block and surrounding spatial merging candidates for motion vector prediction in one example. [Diagram 3]

[0028] FIG. 3 illustrates a simplified block diagram schematic of a communication system 300 in accordance with an example embodiment. [Figure 4]

[0029] FIG. 4 illustrates a simplified block diagram schematic of a communication system 400 in accordance with an example embodiment. [Diagram 5]

[0030] FIG. 5 shows a schematic diagram of a simplified block diagram of a video decoder according to an exemplary embodiment. [Figure 6]

[0031] FIG. 6 shows a schematic diagram of a simplified block diagram of a video encoder according to an exemplary embodiment. [Figure 7]

[0032] FIG. 7 shows a block diagram of a video encoder according to another exemplary embodiment. [Figure 8]

[0033] FIG. 8 shows a block diagram of a video decoder according to another exemplary embodiment. [Figure 9]

[0034] FIG. 9 illustrates a coding block partitioning scheme according to an exemplary embodiment of the present disclosure. [Figure 10]

[0035] FIG. 10 illustrates another scheme of coding block partitioning according to an exemplary embodiment of the present disclosure. [Figure 11]

[0036] FIG. 11 illustrates another scheme of coding block partitioning according to an exemplary embodiment of the present disclosure. [Figure 12]

[0037] FIG. 12 illustrates an exemplary partitioning of base blocks into coding blocks according to an exemplary partitioning scheme. [Figure 13]

[0038] Figure 13 illustrates an exemplary ternary partitioning scheme. [Figure 14]

[0039] FIG. 14 illustrates an exemplary quad-tree binary tree coding block partitioning scheme. [Figure 15]

[0040] FIG. 15 illustrates a scheme for partitioning a coding block into multiple transform blocks and a coding order for the transform blocks, according to an example embodiment of the present disclosure. [Figure 16]

[0041] FIG. 16 illustrates another scheme for partitioning a coding block into multiple transform blocks and the coding order of the transform blocks, in accordance with an example embodiment of the present disclosure. [Figure 17]

[0042] FIG. 17 illustrates another scheme for partitioning a coding block into multiple transform blocks, in accordance with an example embodiment of this disclosure. [Figure 18]

[0043] FIG. 18 shows an example fine angle in directional intra prediction. [Figure 19]

[0044] FIG. 19 shows the nominal angles for directional intra prediction. [Figure 20]

[0045] FIG. 20 shows the top, left and top-left positions for the PAETH mode for a block. [Figure 21]

[0046] Figure 21 illustrates an exemplary recursive intra-filtering mode. [Figure 22A]

[0047] FIG. 22A shows a block diagram of a chroma-from-luma (CfL) prediction unit configured to generate prediction samples of a chroma block based on luma samples of a luma block. [Figure 22B]

[0048] FIG. 22B shows a block diagram of a CfL prediction unit configured to generate prediction samples of a chroma block based on nearby luma samples of a co-located luma block. [Figure 22C]

[0049] FIG. 22C shows a block diagram of a CfL prediction unit configured to generate prediction samples of a chroma block based on luma samples of a co-located luma block and nearby luma samples. [Figure 23]

[0050] FIG. 23 illustrates a flow diagram of an exemplary CfL prediction process. [Figure 24]

[0051] Figure 24 shows a block diagram of luma samples inside and outside the picture boundary. [Diagram 25]

[0052] Figure 25 shows a schematic diagram of luma samples in the vicinity of a luma block. [Figure 26]

[0053] FIG. 26 shows a flowchart of another example CfL prediction process. [Figure 27]

[0054] FIG. 27 shows a flowchart of another example CfL prediction process. [Figure 28]

[0055] FIG. 28 illustrates a flow diagram of another example CfL prediction process. [Figure 29]

[0056] FIG. 29 shows a schematic diagram of an example four-reference line intra-coding for a chroma block. [Diagram 30]

[0057] FIG. 30 illustrates a schematic diagram of a computer system according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027]

[0058] FIG. 3 illustrates a simplified block diagram of a communication system 300 according to an embodiment of the present disclosure. The communication system 300 includes a plurality of terminal devices capable of communicating with each other, for example, via a network 350. For example, the communication system 300 includes a first pair of terminal devices 310 and 320 interconnected via the network 350. In the example of FIG. 3, the first pair of terminal devices 310 and 320 are capable of unidirectional transmission of data. For example, the terminal device 310 may code video data (e.g., video data of a stream of video pictures captured by the terminal device 310) for transmission to another terminal device 320 via the network 350. The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device 320 may receive the coded video data from the network 350, decode the coded video data to reconstruct the video pictures, and display the video pictures according to the reconstructed video data. Unidirectional data transmission may be implemented in media serving applications, etc.

[0028]

[0059] In another example, the communication system 300 includes a second pair of terminal devices 330 and 340 for bidirectional transmission of coded video data, which may occur, for example, during a video conference application. With respect to the bidirectional transmission of data, for example, each of the terminal devices 330 and 340 may code video data (e.g., video data of a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices 330 and 340 over the network 350. Each of the terminal devices 330 and 340 may also receive the coded video data transmitted by the other of the terminal devices 330 and 340, may decode the coded video data to recover the video pictures, and may display the video pictures on an accessible display device according to the recovered video data.

[0029]

[0060] In the example of FIG. 3, terminal devices 310, 320, 330, and 340 are embodied as a server, a personal computer, and a smartphone, although applicability according to the principles of the present disclosure need not be so limited. Embodiments of the present disclosure may be embodied in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, and / or the like. Network 350 represents any number or type of network that carries coded video data between terminal devices 310, 320, 330, and 340, including, for example, wired (wired) and / or wireless communication networks. Communications network 350 may exchange data over circuit-switched, packet-switched, and / or other types of channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of the present disclosure, the architecture and topology of network 350 may not be important to the operation of the present disclosure, unless expressly described below.

[0030]

[0061] Figure 4 illustrates the arrangement of a video encoder and a video decoder in a video streaming environment as an example application of the disclosed subject matter, which may be equally applicable to other video applications including, for example, video conferencing, digital TV broadcasting, gaming, virtual reality, storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.), etc.

[0031]

[0062] The video streaming system may include a video source 401 and a video capture subsystem 413, which may include, for example, a digital camera, for generating a stream of uncompressed video pictures or images 402. In one example, the stream of video pictures 402 includes samples recorded by the digital camera of the video source 401. The stream of video pictures 402, depicted as a thick line to emphasize the larger amount of data compared to the encoded video data 404 (or coded video bitstream), may be processed by an electronic device 420 including a video encoder 403 coupled to the video source 401. The video encoder 403 may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter as described in detail below. The encoded video data 404 (or encoded video bitstream 404), depicted as thin lines to emphasize the smaller amount of data compared to the stream of uncompressed video pictures 402, can be stored in the streaming server 405 for future use or directly in a downstream video device (not shown). One or more streaming client subsystems, such as client subsystems 406 and 408 of FIG. 4, can access the streaming server 405 to retrieve copies 407 and 409 of the encoded video data 404. The client subsystem 406 can include a video decoder 410, for example within the electronic device 430. The video decoder 410 decodes the incoming copy of the encoded video data 407 and generates an output stream of decompressed video pictures 411 that can be rendered on a display 412 (e.g., a display screen) or other rendering device (not shown). The video decoder 410 may be configured to perform some or all of the various functions described in this disclosure.In some streaming systems, the encoded video data 404, 407, and 409 (e.g., video bitstreams) may be encoded according to a particular video coding / compression standard. Examples of these standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC and other video coding standards.

[0032]

[0063] It should be noted that electronic devices 420 and 430 may include other components (not shown). For example, electronic device 420 may include a video decoder (not shown) and electronic device 430 may include a video encoder (not shown).

[0033]

[0064] 5 illustrates a block diagram of a video decoder 510 according to some embodiments of the present disclosure. The video decoder 510 may be included in an electronic device 530. The electronic device 530 may include a receiver 531 (e.g., a receiving circuit). The video decoder 510 may be used in place of the video decoder 410 in the example of FIG. 4.

[0034]

[0065] The receiver 531 may receive one or more coded video sequences to be decoded by the video decoder 510. In the same or another embodiment, it may receive one coded video sequence at a time, where the decoding of each coded video sequence is independent of the other coded video sequences. Each video sequence may relate to multiple video frames or images. The coded video sequences may be received from a channel 501, which may be a hardware / software link to a storage device that stores the coded video data or a streaming source that transmits the coded video data. The receiver 531 may receive the coded video data together with other data, such as coded audio data and / or auxiliary data streams, which may be forwarded to their respective processing circuits (not shown). The receiver 531 may separate the coded video sequences from the other data. To deal with network jitter, a buffer memory 515 may be placed between the receiver 531 and the entropy decoder / parser 520 (hereafter referred to as the parser 520). In certain applications, the buffer memory 515 may be implemented as part of the video decoder 510. In other applications, it may be separate and external to the video decoder 510 (not shown). In yet other applications, there may be a buffer memory (not shown) external to the video decoder 510, e.g. to deal with network jitter, and there may be another additional buffer memory 515 inside the video decoder 510, e.g. to handle playback timing. If the receiver 531 is receiving data from a store / forward device with sufficient bandwidth and controllability, or from an isochronous network, the buffer memory 515 may not be needed or may be small.For use with best effort packet networks such as the Internet, a buffer memory 515 of sufficient size may be required, and may be relatively large in size. Such a buffer memory may be implemented with an adaptive size and may be implemented at least in part in an operating system or similar element (not shown) outside the video decoder 510.

[0035]

[0066] The video decoder 510 may include a parser 520 to reconstruct symbols 521 from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder 510 and potentially information for controlling a rendering device such as a display 512 (e.g., a display screen), which may or may not be an integral part of the electronic device 530 as shown in FIG. 5, but may be coupled to the electronic device 530. The control information for the rendering device may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser 520 may parse / entropy decode the coded video sequence received by the parser 520. The entropy coding of the coded video sequence may follow a video coding technique or standard and may follow various principles including variable length coding, Huffman coding, arithmetic coding with or without context effects, etc. The parser 520 can extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on the at least one parameter corresponding to the subgroup. The subgroups can include Groups of Pictures (GOP), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser 520 can also extract from the coded video sequence information such as transform coefficients (e.g., Fourier transform coefficients), quantization parameter values, motion vectors, etc.

[0036]

[0067] The parser 520 is capable of performing an entropy decoding / parsing process on the video sequence received from the buffer memory 515 to generate symbols 521 .

[0037]

[0068] The reconstruction of symbols 521 may involve several different processing or functional units depending on the type of coded video picture or portion thereof (inter and intra picture, inter and intra block) and other factors. Which units are involved and how may be controlled by subgroup control information parsed by parser 520 from the coded video sequence. The flow of such subgroup control information between parser 520 and the following several processing or functional units is not depicted for clarity.

[0038]

[0069] Beyond the functional blocks already mentioned, the video decoder 510 may be conceptually subdivided into a number of functional units as described below. In a practical implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing various functions of the disclosed subject matter, a conceptual subdivision into functional units is adopted in the following disclosure.

[0039]

[0070] The first unit may be a scalar / inverse transform unit 551. The scalar / inverse transform unit 551 may receive quantized transform coefficients as well as control information (including what type of inverse transform to use, block size, quantization factors / parameters, quantization scaling matrix, etc.) from the parser 520 as symbols 521. The scalar / inverse transform unit 551 may output blocks containing sample values ​​that may be input to an aggregator 555.

[0040]

[0071] In some cases, the output samples of the scalar / inverse transform 551 may relate to intra-coded blocks, i.e. blocks that do not use prediction information from a previously reconstructed picture, but are able to use prediction information from a previously reconstructed part of the current picture. Such prediction information may be provided by the intra picture prediction unit 552. In some cases, the intra picture prediction unit 552 may generate a block of the same size and shape as the block being reconstructed using information of surrounding blocks already reconstructed and stored in the current picture buffer 558. The current picture buffer 558 may buffer, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator 555 may add, possibly on a sample-by-sample basis, the prediction information generated by the intra prediction unit 552 to the output sample information as provided by the scalar / inverse transform unit 551.

[0041]

[0072] In other cases, the output samples of the scalar / inverse transform unit 551 may relate to a block that may be inter-coded and motion compensated. In such a case, the motion compensated prediction unit 553 may access the reference picture memory 557 to retrieve samples used for inter-picture prediction. After motion compensating the retrieved samples according to the symbols 521 related to the block, these samples are added by the aggregator 555 to the output of the scalar / inverse transform unit 551 (the output of the unit 551 may be called residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory 557 from which the motion compensated prediction unit 553 fetches prediction samples may be controlled by motion vectors available to the motion compensated prediction unit 553, for example in the form of symbols 521 that may have X, Y components (shift) and a reference picture component (time). Motion compensation may also involve interpolation of sample values ​​taken from reference picture memory 557 when sub-sample accurate motion vectors are used, and may also involve motion vector prediction schemes, etc.

[0042]

[0073] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit 556. The video compression techniques may include in-loop filter techniques controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit 556 as symbols 521 from the parser 520, which may be responsive to meta information obtained during decoding of a previous portion of the coded picture or coded video sequence (in decoding order), as well as to previously reconstructed loop filtered sample values. As described in more detail below, several types of loop filters may be included as part of the loop filter unit 556 in various orders.

[0043]

[0074] The output of the loop filter unit 556 may be a sample stream that can be output to the rendering device 512, but also stored in a reference picture memory 557 for use in future inter-picture prediction.

[0044]

[0075] Once a given coded picture is fully reconstructed, it can be used as a reference picture for future inter-picture prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by parser 520), current picture buffer 558 can become part of reference picture memory 557, and a fresh current picture buffer can be reallocated before starting reconstruction of a future coded picture.

[0045]

[0076] The video decoder 510 may perform decoding operations according to a given video compression technique adopted in a standard such as ITU-T Rec. H.265. A coded video sequence may conform to the syntax specified by the video compression technique or standard used in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and the profile as documented in the video compression technique or standard. In particular, a profile may select a particular tool from among all tools available in the video compression technique or standard as the only tool that may be used under that profile. Also, to conform to a standard, the complexity of the coded video sequence may be within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, the maximum frame rate, the maximum reconstruction sample rate (e.g., measured in units of megasamples per second), the maximum reference picture size, etc. The limits set by the levels may in some cases be further constrained by Hypothetical Reference Decoder (HRD) specifications and metadata for HRD buffer management signaled in the coded video sequence.

[0046]

[0077] In some example embodiments, the receiver 531 may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder 510 to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0047]

[0078] 6 illustrates a block diagram of a video encoder 603 according to an exemplary embodiment of the present disclosure. The video encoder 603 may be included in an electronic device 620. The electronic device 620 may include a transmitter (640) (e.g., a transmitting circuit). The video encoder 603 may be used in place of the video encoder 403 in the example of FIG. 4.

[0048]

[0079] Video encoder 603 may receive video samples from a video source 601 (which in the example of FIG. 6 is not part of electronic device 620) capable of capturing video images to be coded by video encoder 603. In another example, video source 601 may be implemented as part of electronic device 620.

[0049]

[0080] The video source 601 may provide a source video sequence to be coded by the video encoder 603 in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCB, RGB, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source 601 may be a storage device capable of storing pre-prepared video. In a video conferencing system, the video source 601 may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual pictures or images that convey motion when viewed in sequence. The picture itself may be organized as a spatial array of pixels, where each pixel may contain one or more samples depending on the sampling structure, color space, etc. in use. Those skilled in the art will readily appreciate the relationship between pixels and samples. The following discussion focuses on examples.

[0050]

[0081] According to some example embodiments, the video encoder 603 can code and compress the pictures of the source video sequence into the coded video sequence 643 in real time or under any other time constraint required by the application. Imposing an appropriate coding rate is one function of the controller 650. In some embodiments, the controller 650 can be functionally coupled to and control other functional units as described below, which couplings are not depicted for clarity. Parameters set by the controller 650 can include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller 650 can be configured to have other suitable functions related to the video encoder 603 optimized for a particular system design.

[0051]

[0082] In some example embodiments, the video encoder 603 can be configured to operate in a coding loop. As a simplified explanation, in one example, the coding loop can include a source coder 630 (e.g., responsible for generating a symbol-like symbol stream based on an input picture to be coded and reference pictures) and a (local) decoder (533) built into the video encoder 603. Even if the built-in decoder 633 processes the video stream coded by the source coder 630 without entropy coding, the decoder 633 reconstructs the symbols to generate sample data in a manner similar to that generated by a (remote) decoder (since any compression between the symbols in entropy coding and the coded video bitstream can be considered lossless in the video compression techniques contemplated in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory 634. Since the decoding of the symbol stream produces bit-exact results independent of the location of the decoder (local or remote), the contents in the reference picture memory 634 are also bit-exact between the local and remote encoders. In other words, the prediction part of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder would "see" if it used prediction during decoding. This basic principle of reference picture synchronization is used to improve coding quality (if synchrony cannot be maintained due to channel errors, for example, drift would result).

[0052]

[0083] The operation of the "local" decoder 633 may be the same as that of a "remote" decoder, such as the video decoder 510 already described in detail above in connection with Figure 5. However, with brief reference also to Figure 5, the entropy decoding portion of the video decoder 510, including the buffer memory 515 and the parser 520, may not be fully implemented in the local decoder 633 at the encoder, since symbols are available and the encoding / decoding of symbols for the coded video sequence by the entropy coder 645 and the parser 520 may be lossless.

[0053]

[0084] An insight that can be made at this point is that any decoder technology, except analysis / entropy decoding, which may only exist in the decoder, may also need to exist in a substantially identical functional form in the corresponding encoder. For this reason, the disclosed subject matter may often focus on the decoder operations associated with the decoder portion of the encoder. A description of the encoder technology may be omitted, since it is the inverse of the decoder technology described generically. Only in certain areas or aspects will a more detailed description of the encoder be provided below.

[0054]

[0085] During operation, in some example implementations, the source coder 630 may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence that have been designated as "reference pictures." In this manner, the coding engine 632 codes differences (or residuals) in chroma channels between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as prediction references for the input picture. The terms "residue" and its derivative "residual" may be used interchangeably.

[0055]

[0086] The local video decoder 633 can decode the coded video data of the pictures that can be designated as reference pictures based on the symbols generated by the source coder 630. The operation of the coding engine 632 can advantageously be a non-lossless process. If the coded video data can be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence can be a replica of the source video sequence, typically with some errors. The local video decoder 633 repeats the decoding process that can be performed by the video decoder with respect to the reference pictures, which can cause the reconstructed reference pictures to be stored in the reference picture cache 634. In this way, the video encoder 603 can locally store copies of reconstructed reference pictures that have common content as reconstructed reference pictures obtained by a far-end (remote) video decoder (assuming there are no transmission errors).

[0056]

[0087] The predictor 635 may perform a prediction search for the coding engine 632. That is, for a new picture to be coded, the predictor 635 may search the reference picture memory 534 for sample data (such as candidate reference pixel blocks) or predefined metadata (such as reference picture motion vectors, block shapes, etc.) that may serve as suitable prediction references for the new picture. The predictor 535 may operate on a sample block-pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor 635, an input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory 634.

[0057]

[0088] The controller 650 may manage the coding operations of the source coder 630, including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0058]

[0089] All outputs of the aforementioned functional units can be subjected to entropy coding in an entropy coder 645, which converts the symbols produced by the various functional units into a coded video sequence by lossless compression of the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0059]

[0090] The transmitter 640 may buffer the coded video sequence as produced by the entropy coder 645 to prepare it for transmission over a communication channel 660, which may be a hardware / software link to a storage device that stores the coded video data. The transmitter 640 may merge the coded video data from the video encoder 603 with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0060]

[0091] The controller 650 may manage the operation of the video encoder 603. During coding, the controller 650 may assign a particular coded picture type to each of the coded pictures, which may affect the coding technique that may be applied to each picture. For example, a picture may often be assigned as one of the following picture types:

[0061]

[0092] An Intra picture (I-picture) can be one that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow different types of Intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0062]

[0093] A predictive picture (P-picture) can be one that can be encoded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0063]

[0094] Bi-directionally predicted pictures (B-pictures) can be coded and decoded using intra- or inter-prediction, which uses at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures can use more than two reference pictures and associated metadata for the reconstruction of a block.

[0064]

[0095] A source picture is usually spatially subdivided into multiple sample coding blocks (e.g., into blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and can be coded block by block. Blocks can be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to the respective picture. For example, blocks of an I picture can be non-predictively coded or they can be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of a P picture can be predictively coded with spatial or temporal prediction with reference to one previously coded reference picture. Blocks of a B picture can be predictively coded with spatial or temporal prediction with reference to one or two previously coded reference pictures. Source pictures or intermediate processing pictures can be subdivided into other types of blocks for other purposes. The division of coding blocks and other types of blocks may or may not follow the same method as described in more detail below.

[0065]

[0096] The video encoder 603 may perform coding operations according to a given video coding technique or standard, such as ITU-T Rec. H.265, in which the video encoder 603 may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. The coded video data may accordingly conform to a syntax specified by the video coding technique or standard used.

[0066]

[0097] In some example embodiments, the transmitter 640 may transmit additional data along with the encoded video. The source coder 630 may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data (redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.).

[0067]

[0098] Video can be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra prediction) exploits spatial correlation in a given picture, while inter-picture prediction exploits temporal or other correlation between pictures. For example, a particular picture under encoding / decoding, referred to as the current picture, may be partitioned into blocks. If a block in the current picture resembles a reference block in a reference picture that was previously coded and is still buffered in the video, it can be coded by a vector called a motion vector. A motion vector points to a reference block in the reference picture and can have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0068]

[0099] In some example embodiments, bi-prediction techniques may be used for inter-picture prediction. According to such bi-prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture, both preceding in decoding order (but possibly past and future, respectively, in display order) a current picture in a video. A block in the current picture may be coded with a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block may be jointly predicted by a combination of the first and second reference blocks.

[0069]

[0100] Furthermore, to improve coding efficiency, it is possible to use merge mode techniques for inter-picture prediction.

[0070]

[0101] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed on a block-by-block basis. For example, a picture in a sequence of video pictures may be partitioned into coding tree units (CTUs) for compression, and the CTUs in a picture may have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. In general, a CTU may include three parallel coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU may be recursively quad-tree partitioned into one or more coding units (CUs). For example, a 64x64 pixel CTU may be partitioned into one CU of 64x64 pixels and four CUs of 32x32 pixels. Each of the one or more 32x32 blocks may be further partitioned into four CUs of 16x16 pixels. In some example embodiments, each CU may be analyzed during encoding to determine a prediction type of the CU among various prediction types, such as an inter prediction type or an intra prediction type. The CU may be divided into one or more prediction units (PUs) depending on temporal and / or spatial predictability. In general, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, the prediction process in coding (encoding / decoding) is performed in units of prediction blocks. The division of the CU into PUs (or PBs of different color channels) may be performed in various spatial patterns. For example, a luma or chroma PB may include a matrix of values ​​(e.g., luma values) for pixels, such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0071]

[0102] 7 shows a diagram of a video encoder 703 according to another embodiment of the present disclosure. The video encoder 703 is configured to receive a processed block of sample values ​​(e.g., a predictive block) in a current video picture in a sequence of video pictures and to encode the processed block into a coded picture that is part of a coded video sequence. For example, the video encoder 703 can be used in place of the example video encoder 403 of FIG. 4.

[0072]

[0103] For example, the video encoder 703 receives a matrix of sample values ​​for the processing block, such as a predicted block of 8x8 samples. The video encoder 703 then determines whether the processing block is best coded using intra-mode, inter-mode, or bi-prediction mode, e.g., using rate-distortion optimization (RDO). If it is determined that the processing block should be coded in intra-mode, the video encoder 703 may use intra-prediction techniques to code the processing block into a coded picture; and if it is determined that the processing block should be coded in inter-mode or bi-prediction mode, the video encoder 703 may use inter-prediction techniques or bi-prediction techniques, respectively, to code the processing block into a coded picture. In some exemplary embodiments, a merge mode may be used as a sub-mode of inter-picture prediction, in which case the motion vector is derived from one or more motion vector predictors without benefit of any coded motion vector components outside the predictors. In some other exemplary embodiments, there may be motion vector components applicable to the current block. Thus, the video encoder 703 may include components not explicitly shown in FIG. 7, such as a mode decision module, to determine the prediction mode of a processing block.

[0073]

[0104] In the example of FIG. 7, the video encoder 703 includes an inter-encoder 730, an intra-encoder 722, a residual calculator 723, a switch 726, a residual encoder 724, a general controller 721, and an entropy encoder 725 coupled together as shown in the exemplary arrangement of FIG. 7.

[0074]

[0105] The inter encoder 730 is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in a reference picture (e.g., blocks in previous and subsequent pictures in display order), generate inter prediction information (e.g., an account of redundant information due to inter coding techniques, motion vectors, merge mode information), and calculate an inter prediction result (e.g., a prediction block) based on the inter prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture that has been decoded based on the encoded video information using a decoding unit 633 (such as shown as a residual decoder 728 in FIG. 7, described in more detail below) incorporated in the example encoder 620 of FIG. 6.

[0075]

[0106] The intra encoder 722 is configured to receive samples of a current block (e.g., a processing block), compare the block with blocks already coded in the same picture, generate quantized coefficients after transformation, and possibly also generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). The intra encoder 722 is capable of computing an intra prediction result (e.g., a prediction block) based on the intra prediction information and reference blocks in the same picture.

[0076]

[0107] The generic controller 721 can be configured to determine generic control data and control other components of the video encoder 703 based on the generic control data. In one example, the generic controller 721 determines a prediction mode of the block and provides a control signal to the switch 726 based on the prediction mode. For example, if the prediction mode is an intra mode, the generic controller 721 controls the switch 726 to select the intra mode result for use by the residual calculator 723 and controls the entropy encoder 725 to select the intra prediction information and include the intra prediction information in the bitstream; and if the prediction mode of the block is an inter mode, the generic controller 721 controls the switch 726 to select the inter prediction result for use by the residual calculator 723 and controls the entropy encoder 725 to select the inter prediction information and include the inter prediction information in the bitstream.

[0077]

[0108] The residual calculator 723 may be configured to calculate a difference (residual data) between the received block and a prediction result for the block selected from the intra encoder 722 or the inter encoder 730. The residual encoder 724 may be configured to encode the residual data to generate transform coefficients. For example, the residual encoder 724 may be configured to transform the residual data from a spatial domain to a frequency domain to generate transform coefficients. The transform coefficients are then subjected to a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder 703 also includes a residual decoder 728. The residual decoder 728 is configured to perform an inverse transform and generate decoded residual data. The decoded residual data may be used by the intra encoder 722 and the inter encoder 730 as appropriate. For example, the inter encoder 730 may generate a decoded block based on the decoded residual data and the inter prediction information, and the intra encoder 722 may generate a decoded block based on the decoded residual data and the intra prediction information. The decoded blocks are appropriately processed to generate a decoded picture, which may be buffered in a memory circuit (not shown) and used as a reference picture.

[0078]

[0109] An entropy encoder (725) may be configured to format a bitstream to include the encoded blocks and perform entropy coding. The entropy encoder 725 may be configured to include various information in the bitstream. For example, the entropy encoder 725 may be configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other suitable information in the bitstream. Residual information may not be present when coding blocks in a merged sub-mode of either the inter mode or the bi-prediction mode.

[0079]

[0110] 8 shows a diagram of an example video decoder 810 according to another embodiment of the present disclosure. The video decoder 810 is configured to receive coded pictures that are part of a coded video sequence and to decode the coded pictures to generate reconstructed pictures. In one example, the video decoder 810 may be used in place of the video decoder 410 in the example of FIG. 4.

[0080]

[0111] In the example of FIG. 8, the video decoder 810 includes an entropy decoder 871, an inter decoder 880, a residual decoder 873, a reconstruction module 874, and an intra decoder 872 coupled together as shown in the exemplary arrangement of FIG. 8.

[0081]

[0112] The entropy decoder 871 may be configured to reconstruct from the coded picture certain symbols that represent syntax elements from which the coded picture is constructed. Such symbols may include, for example, the mode in which the block is coded (e.g., intra mode, inter mode, bi-predictive mode, merged submode, or another submode), prediction information (e.g., intra prediction information or inter prediction information) that may identify certain samples or metadata used for prediction by the intra decoder 872 or the inter decoder 880, residual information (e.g., in the form of quantized transform coefficients), etc. In one example, if the prediction mode is an inter or bi-predictive mode, the inter prediction information is provided to the inter decoder 880; if the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder 872. The residual information may be inverse quantized and provided to the residual decoder 873.

[0082]

[0113] The inter decoder 880 may be configured to receive the inter prediction information and generate inter prediction results based on the inter prediction information.

[0083]

[0114] The intra decoder 872 may be configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0084]

[0115] The residual decoder 873 may be configured to perform inverse quantization to extract unquantized transform coefficients and process the unquantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder 873 may also use certain control information (including a quantization parameter (QP)), which may be provided by the entropy decoder 871 (this may only be a small amount of control information, so a data path is not depicted).

[0085]

[0116] The reconstruction module 874 may be configured to combine, in the spatial domain, the residual as output by the residual decoder 873 and the prediction result (possibly output by an inter or intra prediction module) to form a reconstructed block that forms part of the reconstructed picture as part of the reconstructed video. It should be noted that other suitable processes, such as deblocking processes, may also be performed to improve visual quality.

[0086]

[0117] The video encoders 403, 603, and 703 and the video decoders 410, 510, and 810 may be implemented using any suitable technology. In some example embodiments, the video encoders 403, 603, and 703 and the video decoders 410, 510, and 810 may be implemented using one or more integrated circuits. In other embodiments, the video encoders 403, 603, and 703 and the video decoders 410, 510, and 810 may be implemented using one or more processors executing software instructions.

[0087]

[0118] Turning to block partitioning for encoding and decoding, a general partitioning may start from a base block and may follow a predefined set of rules, a specific pattern, a partition tree, or some partition structure or scheme. The partitioning may be hierarchical and recursive. After dividing or partitioning the base block according to any of the exemplary partitioning procedures described below or other procedures, or a combination thereof, a final set of partitions or coding blocks may be obtained. Each of these partitions may be one of various partitioning levels in a partitioning hierarchy and may be of various shapes. Each partition may be referred to as a coding block (CB). For various exemplary partitioning implementations described further below, each resulting CB may be of any allowed size and partitioning level. Such partitions are referred to as coding blocks because they form a unit for which some basic coding / decoding decisions may be made and coding / decoding parameters may be optimized, determined, and signaled in the encoded video bitstream. The highest or deepest level in the final partition represents the depth of the coding block partition structure of the tree. A coding block may be a luma coding block or a chroma coding block. The CB tree structure for each color may be called a coding block tree (CBT).

[0088]

[0119] The coding blocks of all color channels may be collectively referred to as a coding unit (CU). The hierarchical structure of all color channels may be collectively referred to as a coding tree unit (CTU). The partitioning pattern or structure of the various color channels within a CTU may or may not be the same.

[0089]

[0120] In some implementations, the partition tree schemes or structures used for the luma and chroma channels may not need to be the same. In other words, the luma and chroma channels may have separate coding tree structures or patterns. Furthermore, whether the luma and chroma channels use the same or different coding partition tree structures and actual coding partition tree structures may depend on whether the slice being coded is a P, B, or I slice. For example, for an I slice, the chroma and luma channels may have separate coding partition tree structures or coding partition tree structure modes, while for a P or B slice, the luma and chroma channels may share the same coding partition tree scheme. When separate coding partition tree structures or modes are applied, the luma channel may be partitioned into CBs by one coding partition tree structure and the chroma channels may be partitioned into chroma CBs by another coding partition tree structure.

[0090]

[0121] In some example implementations, a predefined partitioning pattern may be applied to the base block. As shown in FIG. 9, an example 4-way partition tree may start at a first predefined level (e.g., 64x64 block level or other size as the base block size), and the base block may be partitioned hierarchically down to a predefined lowest level (e.g., 4x4 level). For example, the base block may follow four predefined partitioning options or patterns shown at 902, 904, 906, and 908, where a partition designated as R is allowed for recursive partitioning, where the same partition option as shown in FIG. 9 may be repeated at a lower scale down to the lowest level (e.g., 4x4 level). In some implementations, additional restrictions may be applied to the partitioning scheme of FIG. 9. In the implementation of FIG. 9, rectangular partitions (e.g., 1:2 / 2:1 rectangular partitions) may be allowed, but they may not be allowed to be recursive, whereas square partitions are allowed to be recursive. Partitioning according to FIG. 9, with recursion if necessary, generates a final set of coding blocks. A coding tree depth may be further defined to indicate the partition depth from a root node or root block. For example, the coding tree depth of a root node or root block, e.g., a 64x64 block, may be set to 0, and after the root block is further partitioned once according to FIG. 9, the coding tree depth is increased by one. The maximum or deepest level from the 64x64 base block to the smallest partition of 4x4 is 4 (starting from level 0) for the above scheme. Such a partitioning scheme may be applied to one or more color channels.Each color channel may be partitioned independently according to the scheme of Figure 9 (e.g., a partitioning pattern or option among the predefined patterns may be determined independently for each color channel at each hierarchical level), or alternatively, two or more color channels may share the same hierarchical pattern tree of Figure 9 (e.g., the same partitioning pattern or option among the predefined patterns may be selected for two or more color channels at each hierarchical level).

[0091]

[0122] FIG. 10 illustrates another exemplary predefined partitioning pattern that allows recursive partitioning to form a partitioning tree. As illustrated in FIG. 10, an exemplary 10-way partitioning structure or pattern may be predefined. A root block may start from a predefined level (e.g., from a base block at the 128x128 level or the 64s64 level). The exemplary partitioning structure of FIG. 10 includes various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. A partition type with 3 sub-partitions, shown as 1002, 1004, 1006, and 1008 in the second row of FIG. 10, may be referred to as a "T-type" partition. The "T-type" partitions 1002, 1004, 1006, and 1008 may be referred to as a left T-type, an upper T-type, a right T-type, and a lower T-type. In some exemplary implementations, none of the rectangular partitions of FIG. 10 are allowed to be further subdivided. A coding tree depth may be further defined to indicate the partition depth from the root node or root block. For example, the coding tree depth of the root node or root block (e.g., a 128x128 block) may be set to 0, and after the root block is further partitioned once according to FIG. 10, the coding tree depth is increased by one. In some implementations, only full square partitions of 1010 may be allowed for recursive partitioning to the next level of the partitioning tree following the pattern of FIG. 10. In other words, recursive partitioning may not be allowed for square partitions of the T-shaped patterns 1002, 1004, 1006, and 1008. The partitioning procedure according to FIG. 10, with recursion if necessary, generates the final set of coding blocks. Such a scheme may be applied to one or more color channels. In some implementations, more flexibility may be added for the use of partitions below the 8x8 level. For example, 2x2 chroma inter prediction may be used in certain cases.

[0092]

[0123] In some other example implementations for coding block partitioning, a quadtree structure may be used to divide a base block or an intermediate block into quadtree partitions. Such quadtree division may be applied hierarchically and recursively to any square-shaped partitions. Whether a base block or intermediate block or partition is further quadtree divided may be adapted to various local characteristics of the base block or intermediate block / partition. The quadtree partitioning at picture boundaries may be further adapted. For example, an implicit quadtree division may be performed at picture boundaries, so that a block will remain quadtree divided until its size fits the picture boundary.

[0093]

[0124] In some other exemplary implementations, hierarchical binary partitioning from a base block may be used. For such a scheme, a base block or a mid-level block may be divided into two partitions. The binary partitioning may be horizontal or vertical. For example, horizontal binary partitioning may divide a base block or a mid-level block into equal left and right partitions. Similarly, vertical binary partitioning may divide a base block or a mid-level block into equal top and bottom partitions. Such binary partitioning may be hierarchical and recursive. At each base block or mid-level block, a determination may be made whether the binary partitioning scheme should continue and, if the scheme continues further, whether horizontal or vertical binary partitioning should be used. In some implementations, further partitioning may stop at a predefined minimum partition size (in one or both dimensions). Alternatively, further partitioning may stop if a predefined partitioning level or depth from the base block is reached. In some implementations, the aspect ratio of the partitions may be limited. For example, the aspect ratio of a partition may be no smaller than 1:4 (or no larger than 4:1). Thus, a vertical strip partition, which has a vertical_to_horizontal aspect ratio of 4:1, may simply be further divided vertically into an upper partition and a lower partition, each with a vertical_to_horizontal aspect ratio of 2:1.

[0094]

[0125] In yet other examples, a ternary partitioning scheme may be used to partition the base block or any intermediate blocks, as shown in FIG. 13. The ternary pattern may be implemented vertically, as shown in 1302 of FIG. 13, or horizontally, as shown in 1304 of FIG. 13. The exemplary partitioning ratio in FIG. 13 is shown as 1:2:1 for either the vertical or horizontal directions, but other ratios may be predefined. In some implementations, two or more different ratios may be predefined. Such a ternary partitioning scheme may be used to complement a quadtree or binary partitioning structure, where such ternary tree partitioning may capture objects located at the block center in one unbroken partition, whereas quadtrees and binary trees always split along the block center, thus splitting such objects into separate partitions. In some implementations, to avoid additional transformations, the width and height of the partitions in the exemplary ternary tree are always powers of two.

[0095]

[0126] The above partitioning schemes can be combined in any manner at different partitioning levels. As an example, the above mentioned quadtree and binary tree schemes may be combined to partition the base block into a quadtree-binary-tree (QTBT) structure. In such a scheme, the base block or intermediate blocks / partitions may be either quadtree-partitioned or binary tree-partitioned according to a set of predefined conditions, if specified. A specific example is shown in FIG. 14. In the example of FIG. 14, first the base block is quadtree-partitioned into four partitions, as shown at 1402, 1404, 1406, 1408. Then, each resulting partition is either quadtree-partitioned into four further partitions (as at 1408), binary-partitioned into two further partitions (e.g., split horizontally or vertically, both symmetric, as at 1402 or 1406), or unpartitioned (as at 1404). Binary or quad-tree partitioning may be allowed recursively for square-shaped partitions, as shown by the overall exemplary partition pattern in 1410 and the corresponding tree structure / representation in 1420 (solid lines represent quad-tree partitioning and dashed lines represent quad-tree partitioning). A flag may be used for each binary tree partition node (non-leaf binary tree partition) to indicate whether the binary tree partition is horizontal or vertical. For example, as shown in 1420, flag "0" may represent horizontal bisection and flag "1" may represent vertical bisection, consistent with the partitioning structure in 1410. For quad-tree partitioning, there is no need to indicate the partition type, since a quad-tree partition always splits a block or partition both horizontally and vertically to generate four sub-blocks / partitions of equal size. In some implementations, flag "1" may represent horizontal bisection and flag "0" may represent vertical bisection.

[0096]

[0127] In some example implementations of QTBT, the quadtree and binary tree splitting ruleset can be expressed by the following pre-defined parameters and their associated corresponding functions: - CTU size: quadtree root node size (size of base block) - MinQTSize: The minimum allowed quadtree leaf node size. - MaxBTSize: The maximum allowed binary tree root node size. - MaxBTDepth: Maximum allowed binary tree depth - MinBTSize: The minimum allowable binary tree leaf node size. In some exemplary implementations of the QTBT partitioning structure, the CTU size may be set as 128x128 luma samples with two corresponding 64x64 blocks of chroma samples (if exemplary chroma subsampling is considered and used), MinQTSize may be set as 16x16, MaxBTSize may be set as 64x64, MinBTSize (for both width and height) may be set as 4x4, and MaxBTDepth may be set as 4. Quad-tree partitioning may be applied to the CTU first to generate quad-tree leaf nodes. The quad-tree leaf nodes may have sizes ranging from its minimum allowed size of 16x16 (i.e., MinQTSize) to 128x128 (i.e., CTU size). If the node is 128x128, it will not be initially split by the binary tree since its size exceeds MaxBTSize (i.e., 64x64). Otherwise, nodes that do not exceed MaxBTSize can be partitioned by binary tree. In the example of FIG. 14, the base block is 128x128. The base block can only be quadtree partitioned according to a predefined set of rules. The base block has a partition depth of 0. Each of the resulting four partitions is 64x64, which does not exceed MaxBTSize, and can be further quadtree or binary tree partitioned at level 1. The process continues. When the binary tree depth reaches MaxBTDepth (i.e., 4), no further partitions may be considered. If a binary tree node has a width equal to MinBTSize (i.e., 4), no further horizontal partitions may be considered. Similarly, if a binary tree node has a height equal to MinBTSize, no further vertical partitions may be considered.

[0097]

[0128] In some example implementations, the above QTBT scheme can be configured to support flexibility for luma and chroma to have the same QTBT structure or separate QTBT structures. For example, for P and B slices, the luma and chroma CTBs in one CTU can share the same QTBT structure. However, for an I slice, the luma CTB may be partitioned into CBs by a QTBT structure, and the chroma CTB may be partitioned into chroma CBs by another QTBT structure. This means that CUs may be used to indicate different color channels of an I slice; for example, an I slice may consist of a coding block of a luma component or a coding block of two chroma components, and a CU of a P or B slice may consist of coding blocks of all three color components.

[0098]

[0129] In some other implementations, the QTBT scheme can be supplemented with the ternary scheme described above. Such implementations may be referred to as multi-type-tree (MTT) structures. For example, in addition to bisection of nodes, one of the ternary partition patterns in Figure 13 may be selected. In some implementations, only square nodes may be subject to trisection. An additional flag may be used to indicate whether the ternary partitioning is horizontal or vertical.

[0099]

[0130] Two-level or multi-level tree designs, such as the QTBT implementation and the QTBT implementation supplemented by trisection, can be primarily motivated by reduced complexity. In theory, the complexity of traversing a tree is T D where T represents the number of split types and D is the depth of the tree. Trade-offs may be made by using multiple types (T) while reducing the depth (D).

[0100]

[0131] In some implementations, the CB may be further partitioned. For example, the CB may be further partitioned into multiple prediction blocks (PBs) for the purpose of intra- or inter-frame prediction during the coding and decoding process. In other words, the CB may be further divided into different sub-partitions, where individual prediction decisions / settings may be made. In parallel, the CB may be further partitioned into multiple transform blocks (TBs) for the purpose of delimiting the levels at which transformation or inverse transformation of the video data is performed. The partitioning scheme of the CB into PBs and TBs may be the same or different. For example, each partitioning scheme may be performed using its own procedure, for example, based on various characteristics of the video data. The partitioning schemes of the PBs and TBs may be independent in some exemplary implementations. The partitioning schemes and boundaries of the PBs and TBs may be interrelated in other exemplary implementations. In some implementations, for example, the TBs may be partitioned after the PB partitioning, and in particular, each PB may be further partitioned into one or more TBs after being determined after the partitioning of the coding blocks. For example, in some implementations, the PB may be divided into one, two, four, or some other number of TBs.

[0101]

[0132] In some implementations, luma and chroma channels may be treated differently for purposes of dividing base blocks into coding blocks and further partitioning into prediction and / or transform blocks. For example, in some implementations, partitioning of coding blocks into prediction and / or transform blocks may be allowed for luma channels, but such partitioning of coding blocks into prediction and / or transform blocks may not be allowed for chroma channels. Thus, in such implementations, transform and / or prediction of luma blocks may be performed only at the coding block level. In another example, the minimum transform block size for luma and chroma channels may be different, e.g., coding blocks of luma channels may be allowed to be partitioned into smaller transform and / or prediction blocks than chroma channels. In yet another example, the maximum depth of partitioning of coding blocks into transform blocks and / or prediction blocks may differ between luma and chroma channels, e.g., coding blocks for luma channels may be allowed to be partitioned into deeper transform and / or prediction blocks than chroma channels. As a specific example, luma coding blocks may be partitioned into transform blocks of multiple sizes that can be represented by a recursive partitioning that goes down at most two levels, and transform block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4, and transform block sizes from 4x4 to 64x64 may be allowed. However, for chroma blocks, only the maximum possible transform block specified for the luma block may be allowed.

[0102]

[0133] In some example implementations for partitioning coding blocks into PBs, the depth, shape, and / or other characteristics of the PB partitioning may depend on whether the PB is intra-coded or inter-coded.

[0134] Partitioning of coding blocks (or prediction blocks) into transform blocks may be performed in various exemplary manners, including but not limited to quad-tree partitioning and predefined pattern partitioning, recursively or non-recursively, and with additional consideration of transform blocks at coding or prediction block boundaries. In general, the resulting transform blocks may be at different partitioning levels, may not be of the same size, and may not be required to be in a square shape (e.g., they may be rectangular with some allowed size and aspect ratio). Further examples are described in more detail below in connection with Figures 15, 16, and 17.

[0103]

[0135] However, in some other implementations, the CB resulting from any of the above partitioning schemes may be used as a basic or smallest coding block for prediction and / or transformation. In other words, no further partitioning is performed for inter / intra prediction purposes and / or transformation purposes. For example, the CB resulting from the above QTBT scheme may be directly used as a unit for performing prediction. Specifically, such a QTBT structure removes the concept of multiple partition types, i.e., removes the separation of CU, PU, ​​and TU, and supports more flexibility for CU / CB partition shapes, as described above. In such a QTBT block structure, the CU / CB can have either a square or rectangular shape. The leaf nodes of such a QTBT are used as units of prediction and transformation processing without any further partitioning. This means that in such an exemplary QTBT coding block structure, the CU, PU, ​​and TU have the same block size.

[0104]

[0136] The various CB partitioning schemes described above, and further partitioning of the CB into PB and / or TB (not including PB / TB partitioning), may be combined in any manner. The following specific implementations are provided as non-limiting examples.

[0105]

[0137] A specific implementation example of coding block and transform block partitioning is described below. In such an exemplary implementation, a base block may be partitioned into coding blocks using a recursive quadtree partition or the above-mentioned predefined partitioning patterns (such as in FIG. 9 and FIG. 10). At each level, whether further quadtree partitioning of a particular partition should continue or not can be determined by local video data characteristics. The resulting CBs may be at different quadtree partitioning levels and of different sizes. Deciding whether to code a picture area using inter-picture (temporal) prediction or intra-picture (spatial) prediction may be done at the CB level (or at the CU level for all three color channels). Each CB may be further partitioned into one, two, four, or other number of PBs according to a predefined PB partition type. Within one PB, the same prediction process may be applied, and related information may be transmitted to the decoder on a PB basis. By applying the prediction process based on the PB partition type, after obtaining the residual block, the CB can be partitioned into TBs according to another quadtree structure similar to the coding tree for CBs. In this particular implementation, the CB or TB may be square shaped, but need not be limited to a square shape. Furthermore, in this particular example, the PB may be square or rectangular shaped for inter prediction, and only square for intra prediction. The coding block may be partitioned, for example, into four square shaped TBs. Each TB may be further partitioned recursively (using quadtree partitioning) into smaller TBs, called residual quadtrees (RQTs).

[0106]

[0138] Another implementation example for partitioning the base block into CBs, PBs, and / or TBs is further described below. For example, instead of using multiple partition unit types as shown in FIG. 9 and FIG. 10, a quadtree (e.g., QTBT or QTBT with 3-way partitioning as described above) with nested multi-type trees using 2-way and 3-way segmentation structures may be used. The separation of CB, PB, and TB (i.e., partitioning of CB into PB and / or TB, and partitioning of PBs into TBs) may be abandoned except when required for CBs with a size that is too large for the maximum transform length (such CBs may require further partitioning). This exemplary partitioning scheme may be designed to support more flexibility for CB partition shapes, so that both prediction and transformation can be performed at the CB level without further partitioning. In such a coding tree structure, the CB may have either a square or rectangular shape. Specifically, the coding tree block (CTB) may be first partitioned by a quadtree structure. The quadtree leaf nodes may then be further partitioned by nested multi-type tree structures. An example of a nested multi-type tree structure using bisection or trisection is shown in Figure 11. Specifically, the example multi-type tree structure of Figure 11: Vertical bisection (SPLIT_BT_VER) (1102), Horizontal bisection (SPLIT_BT_HOR) (1104), Split Vertical Thirds (SPLIT_TT_VER) (1106), and Includes four split types referred to as Horizontal Thirds (SPLIT_TT_HOR) (1108). Thus, CB corresponds to a leaf of the multi-type tree. In this implementation, this segmentation is used for both prediction and transformation without any other partitioning, as long as CB is not too large for the maximum transformation length. This means that in most cases, in a quadtree with a nested multi-type tree coding block structure, CB, PB, and TB have the same block size. An exception occurs when the maximum supported transformation length is smaller than the width or height of the color components of CB. In some implementations, in addition to bisection or trisection, the nested pattern of FIG. 11 may further include a quadtree division.

[0107]

[0139] One example of a quadtree with nested multi-type tree coding block structure of block partitions for one base block is shown in FIG. 12 (including options of quadtree partitioning, binary tree partitioning, and ternary tree partitioning). More specifically, FIG. 12 shows a base block 1200 being quadtree partitioned into four square partitions 1202, 1204, 1206, and 1208. A decision on using the quadtree and multi-type tree structures of FIG. 11 for further partitioning is made for each partition that is quadtree partitioned. In the example of FIG. 12, partition 1204 is not further partitioned. Partitions 1202 and 1208 each employ a different quadtree partitioning. For partition 1202, the second level quadtree partitioned top-left, top-right, bottom-left, and bottom-right partitions respectively adopt the third level quadtree, horizontal binary tree partition 1104 of FIG. 11, no partition, and horizontal ternary tree partition 1108 of FIG. 11. Partition 1208 adopts another quadtree partition, with the second level quadtree partitioned top-left, top-right, bottom-left, and bottom-right partitions respectively adopting the third level vertical ternary tree partition 1106 of FIG. 11, no partition, no partition, and horizontal binary tree partition 1104 of FIG. 11. Two of the sub-partitions of the third level top-left partition of 1208 are further partitioned according to the horizontal binary partition 1104 and horizontal ternary partition 1108 of FIG. 11, respectively. Partition 1206 follows the second level division pattern, vertical bisection 1102 of Figure 11, into two partitions, and those two patterns are further divided at a third level according to horizontal bisection 1108 and vertical bisection 1102 of Figure 11. For one of them, a fourth level division is further applied according to horizontal bisection 1104 of Figure 11.

[0108]

[0140] For the above example, the maximum luma transform size may be 64x64, and the maximum supported chroma transform size may be different from that of the luma, e.g., 32x32. Although the above example CB of Figure 12 is generally not further divided into smaller PBs and / or TBs, still, if the width or height of a luma coding block or a chroma coding block is larger than the maximum transform width or height, the luma coding block or the chroma coding block may be automatically divided horizontally and / or vertically to meet the transform size limitation in that direction.

[0109]

[0141] In the specific example of partitioning base blocks into CBs above and as explained above, the coding tree scheme can support the ability to have separate block tree structures for luma and chroma. For example, for P and B slices, the luma and chroma CTBs in one CTU can share the same coding tree structure. For example, for I slices, luma and chroma can have separate coding block tree structures. When separate block tree structures are applied, the luma CTB can be partitioned into luma CBs by one coding tree structure and the chroma CTBs are partitioned into chroma CBs by another coding tree structure. This means that a CU of an I slice may consist of a coding block of the luma component or a coding block of two chroma components, and a CU of a P or B slice always consists of coding blocks of all three color components, unless the video is monochrome.

[0110]

[0142] When a coding block is further partitioned into multiple transform blocks, the transform blocks therein may be ordered in the bitstream according to various orders or scanning schemes. Exemplary implementations for partitioning coding or prediction blocks into transform blocks and the coding order of the transform blocks are described in further detail below. In some exemplary implementations, as described above, the transform partitioning may support multiple shapes of transform blocks, e.g., 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, with transform block sizes ranging, e.g., from 4x4 to 64x64. In some implementations, when a coding block is 64x64 or smaller, the transform block partitioning may only be applied to the luma component, such that the transform block size matches the coding block size for the chroma blocks. Otherwise, if the width or height of a coding block is greater than 64, both the luma and chroma coding blocks may be implicitly divided into multiples of min(W,64)×min(H,64) and min(W,32)×min(H,32), respectively.

[0111]

[0143] In some example implementations of transform block partitioning, for both intra- and inter-coded blocks, the coding blocks may be further partitioned into multiple transform blocks with partitioning depth up to a preset number of levels (e.g., two levels). The partitioning depth and size of the transform blocks may be related. For some example implementations, the mapping from the transform size of the current depth to the transform size of the next depth is shown in Table 1 below.

[0112] Table 1: Conversion Partition Size Settings

[0113] [Table 1]

[0144] Based on the mapping example in Table 1, for a 1:1 square block, the next level transform partitioning may create four 1:1 square sub-transform blocks. The transform partition may stop at, for example, 4x4. Thus, the transform size of the current depth of 4x4 corresponds to the same size of 4x4 for the next depth. In the example of Table 1, for a 1:2 / 2:1 non-square block, the next level transform partitioning may create two 1:1 square sub-transform blocks, while for a 1:4 / 4:1 non-square block, the next level transform partitioning may create two 1:2 / 2:1 sub-transform blocks.

[0114]

[0145] In some example implementations, for the luma components of intra-coded blocks, additional restrictions may be applied on transform block partitioning. For example, for each level of transform partitioning, all sub-transform blocks may be restricted to have equal size. For example, for a 32x16 coding block, level 1 transform partitioning creates two 16x16 sub-transform blocks, and level 2 transform partitioning creates eight 8x8 sub-transform blocks. In other words, the second level partitioning must be applied to all first level sub-blocks to keep the transform units equal in size. An example of transform block partitioning for an intra-coded square block according to Table 1 is shown in FIG. 15 with the coding order indicated by the arrows. Specifically, 1502 shows a square coding block. The first level partitioning into four equal-sized transform blocks according to Table 1 is shown in 1504 with the coding order indicated by the arrows. A second level split of all first level equal sized blocks into 16 equal sized transform blocks according to Table 1 is shown at 1506 with the coding order indicated by the arrows.

[0115]

[0146] In some example implementations, for the luma components of an inter-coded block, the above restrictions on intra-coding may not apply. For example, after the first level of transform partitioning, any one of the sub-transform blocks may be further partitioned independently at another further level. The resulting transform blocks may or may not be of the same size. An example partitioning of an inter-coded block into transform blocks in their coding order is shown in FIG. 16. In the example of FIG. 16, an inter-coded block 1602 is partitioned into transform blocks at two levels according to Table 1. At the first level, the inter-coded block is partitioned into four transform blocks of equal size. Then, only one of the four transform blocks (but not all of them) is further partitioned into four sub-transform blocks, resulting in a total of seven transform blocks with two different sizes, as shown at 1604. An example coding order of these seven transform blocks is shown by the arrows at 1604 in FIG. 16.

[0116]

[0147] In some example implementations, for chroma components, some additional restrictions on the transform blocks may be applied, e.g., for chroma components, the transform block size may be as large as the coding block size, but may not be smaller than a predefined size (e.g., 8x8).

[0117]

[0148] In some other example implementations, for coding blocks with either width (W) or height (H) greater than 64, both luma and chroma coding blocks may be implicitly divided into multiples of min(W,64)×min(H,64) and min(W,32)×min(H,32) transform units, respectively, where in this disclosure, “min(a,b)” can return the smaller value between a and b.

[0118]

[0149] FIG. 17 further illustrates another alternative example scheme for partitioning coding blocks or prediction blocks into transform blocks. As shown in FIG. 17, instead of using recursive transform partitioning, a predefined set of partitioning types may be applied to coding blocks according to their transform type. In the particular example shown in FIG. 17, one of six example partitioning types may be applied to split the coding block into a varying number of transform blocks. Such a scheme resulting in transform block partitioning may be applied to either coding blocks or prediction blocks.

[0119]

[0150] More specifically, the partitioning scheme of FIG. 17 provides up to six example partition types for any given transform type (transform type refers to primary transform such as ADST and other types). In this scheme, every coding block or prediction block can be assigned to a transform partition type, for example, based on a rate-distortion cost. In one example, the transform partition type assigned to a coding block or prediction block may be determined based on the transform type of the coding block or prediction block. A particular transform partition type may correspond to a transform block partition size and pattern, as illustrated by the six transform partition types illustrated in FIG. 17. The correspondence between various transform types and various transform partition types may be predefined. In the examples illustrated below, uppercase labels indicate transform partition types, which may be assigned to a coding block or prediction block based on a rate-distortion cost: PARTITION_NONE: Allocates the transformation size equal to the block size.

[0120] · PARTITION_SPLIT: Allocates a transformation size of 1 / 2 the block size width and 1 / 2 the block size height.

[0121] · PARTITION_VERT: Allocates a transformation size of 1 / 2 the width of the block size and the same height as the block size.

[0122] PARTITION_HORZ4: Allocates a transformation size with width equal to the block size and 1 / 4 of the block size height.

[0123] · PARTITION_VERT4: Allocates a transformation size of 1 / 4 the width of the block size and the same height as the block size.

[0124]

[0151] In the above example, all of the transform partition types as shown in Figure 17 include uniform transform sizes for the partitioned transform blocks. This is by way of example only and not limitation. In some other implementations, mixed transform block sizes may be used for the partitioned transform blocks of a particular partition type (or pattern).

[0125]

[0152] A video block (PB or CB, sometimes referred to as PB when not further partitioned into multiple predictive blocks) may be predicted in various ways, rather than being directly coded, to exploit various correlations and redundancies in the video data and improve compression efficiency. Accordingly, such prediction may be performed in various modes. For example, a video block may be predicted by intra prediction or inter prediction. In particular, in an inter prediction mode, a video block may be predicted by one or more other reference blocks or inter predictor blocks from one or more other frames, either via single reference or mixed reference inter prediction. In case of an inter prediction implementation, a reference block may be specified by its frame identifier (temporal location of the reference block) and a motion vector indicating a spatial offset (spatial location of the reference block) between the current block being coded or decoded and the reference block. The reference frame identification and the motion vector may be signaled in the bitstream. The motion vector as a spatial block offset may be directly signaled or may itself be predicted by another reference motion vector or a predictive motion vector. For example, the current motion vector may be predicted directly by a reference motion vector (e.g., of a block nearby the candidate), or by a combination of the reference motion vector and the motion vector difference (MVD) between the current motion vector and the reference motion vector. The latter is sometimes called merge mode with motion vector difference (MMVD). The reference motion vector can be specified in the bitstream, for example, as a pointer to a block that is in the spatial neighborhood of the current block, or to a spatially collocated block that is in the temporal neighborhood.

[0126]

[0153] Returning to the description of the intra prediction process; samples in a block (e.g., luma or chroma prediction block, or coding block if not further divided into prediction blocks) are predicted by samples in the neighborhood, the next neighborhood, or one or more other lines, or a combination thereof, to generate a prediction block. The residual between the actual block being coded and the prediction block can then be processed by a transform followed by quantization. Various intra prediction modes may be made available, and parameters related to intra mode selection and other parameters may be signaled in the bitstream. Various intra prediction modes may relate, for example, to one or more line positions for predicting samples, the direction in which prediction samples are selected from one or more prediction lines, and other special intra prediction modes.

[0127]

[0154] For example, the set of intra-prediction modes (interchangeably referred to as "intra modes") may include a predefined number of directional intra-prediction modes. As described above with respect to the example implementation of FIG. 1, these intra-prediction modes may correspond to a predetermined number of directions along which out-of-block samples are selected as predictions for samples predicted in a particular block. In another particular example implementation, eight primary directional modes may be supported and predefined, corresponding to angles from 45 degrees to 207 degrees relative to the horizontal axis.

[0128]

[0155] In some other implementations of intra prediction, the directional intra modes can be further expanded to a set of angles with finer granularity to further exploit more diverse types of spatial redundancy in the directional texture. For example, the eight angle implementation above can be configured to provide eight nominal angles called V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED, as shown in FIG. 19, and for each nominal angle, a predefined number (e.g., seven) of finer angles can be added. With such expansion, a larger total number of directional angles (e.g., 56 in this example) corresponding to the same number of predefined directional intra modes can become available for intra prediction. The prediction angle may be expressed by the nominal intra angle plus an angle delta. In the particular example above with seven finer angle directions for each nominal angle, the angle delta may be -3 to 3 multiplied by a step size of 3 degrees. Any angle scheme may be used, as shown in FIG. 18, which has 65 different prediction angles.

[0129]

[0156] In some implementations, instead of or in addition to the above directional intra modes, a predefined number of non-directional intra prediction modes may also be predefined and made available. For example, five non-directional intra modes, called smooth intra prediction modes, may be specified. These non-directional intra mode prediction modes may be specifically called DC, PAETH, SMOOTH, SMOOTH_V, and SMOOTH_H intra modes. Prediction of samples of a particular block under these exemplary non-directional modes is shown in FIG. 20. As an example, FIG. 20 shows a 4x4 block 2002 predicted by samples from the neighboring lines above and / or the neighboring lines to the left. A particular sample 2010 in block 2002 may correspond to sample 2004 immediately above sample 2010 on the upper neighborhood line of block 2002, sample 2006 above and to the left of sample 2010 as the intersection of the upper and left neighborhood lines, and sample 2008 immediately to the left of sample 2010 on the left neighborhood line of block 2002.

[0130] For the exemplary DC intra prediction mode, the average of the nearby left and top samples 2008 and 2004 may be used as a predictor for sample 2010 .

[0131] For an exemplary PAETH intra-prediction mode, the top, left, and top-left reference samples 2004, 2008, and 2006 may be taken, and then the value of any of these three reference samples closest to (top+left-top-left) may be set as the predictor for sample 2010.

[0132] For the exemplary SMOOTH_V intra-prediction mode, sample 2010 may be predicted by quadratic interpolation in the vertical direction of nearby top-left sample 2006 and nearby left-hand sample 2008.

[0133] For the exemplary SMOOTH_H intra-prediction mode, sample 2010 may be predicted by quadratic interpolation in the horizontal direction of the top-left nearby sample 2006 and the upper nearby sample 2004.

[0134] For the exemplary SMOOTH intra-prediction mode, sample 2010 may be predicted by an average of quadratic interpolations in the vertical and horizontal directions.

[0135] The above implementations of the non-directional intra modes are shown merely as non-limiting examples: other nearby lines and other non-directional selections of samples, as well as schemes of combining prediction samples to predict a particular sample in a prediction block, are also contemplated.

[0136]

[0157] The selection of a particular intra-prediction mode by the encoder from the above directional or non-directional modes at various coding levels (picture, slice, block, unit, etc.) can be signaled in the bitstream. In some example implementations, the example eight nominal directional modes with five non-angle smooth modes (total of 13 options) can be signaled first. Then, if the signaled mode was one of the eight nominal angle intra modes, an index is further signaled to indicate the selected angle delta relative to the corresponding signaled nominal angle. In some other example implementations, all intra-prediction modes can be indexed together for signaling (e.g., 56 directional modes, plus five non-directional modes resulting in 61 intra-prediction modes).

[0137]

[0158] In some example implementations, the example 56 or other number of directional intra-prediction modes can be implemented using a unified directional predictor that projects each sample of a block to a reference sub-sample location and interpolates the reference sample with a 2-tap bilinear filter.

[0138]

[0159] In some implementations, additional filter modes, called FILTER INTRA modes, may be designed to capture the decaying spatial correlation with the reference at the edge. For these modes, in addition to the out-of-block samples, the prediction samples within the block may be used as intra prediction reference samples for some patches within the block. These modes may be predefined and made available for intra prediction, for example, for at least the luma block (or only the luma block). A predefined number (e.g., 5) of filter intra modes may be predesigned, each represented by a set of n-tap filters (e.g., 7-tap filters) that reflect the correlation between a sample in a 4x2 patch and its n neighbors. In other words, the weighting coefficients of the n-tap filters may be position-dependent. Taking an 8x8 block, a 4x2 patch, and 7-tap filtering as an example, the 8x8 block 2002 may be divided into eight 4x2 patches, as shown in FIG. 21. These patches are denoted by B0, B1, B1, B3, B4, B5, B6, and B7 in FIG. 21. For each patch, seven neighbors, denoted by R0-R6 in FIG. 21, may be used to predict samples in the current patch. For patch B0, all neighbors may already be reconstructed. However, for other patches, some neighbors may be in the current block and therefore not reconstructed, and therefore the predicted values ​​of the nearest neighbors are used as reference. For example, all neighbors of patch B7 as shown in FIG. 21 have not been reconstructed, and therefore the predicted samples of the neighbors are used instead.

[0139]

[0160] In some implementations of intra prediction, a color component may be predicted using one or more other color components, which may be any of the components in the YCrCb, RGB, XYZ color spaces, etc.

[0140]

[0161] One type of intra prediction, in which one color component is predicted using one or more other color components, is Chroma from Luma (CfL) prediction. In CfL prediction, a chroma component is predicted based on a luma component. The predicted chroma component may include a chroma block, which may include one or more chroma samples. The predicted samples are called prediction samples. Also, the predicted chroma block may correspond to a luma block. In this application, unless otherwise specified, the correspondence between a luma block and a chroma block refers to the chroma block being co-located with the luma block.

[0141]

[0162] Also, as used herein and as described in more detail below, the luma components used to predict the chroma components may include luma samples that may be luma samples of the corresponding or co-located chroma block itself and / or may include nearby luma samples that are luma samples of one or more nearby luma blocks that are in the vicinity or adjacent to the corresponding co-located luma block of the predicted chroma block. Furthermore, in at least some implementations, the luma samples used in the CfL prediction process are reconstructed luma samples that may be copies of the original luma samples derived or reconstructed from a compressed version of the original luma samples using a decoding process.

[0142]

[0163] In some implementations, an encoder (e.g., any of the encoders 403, 603, 703) and / or a decoder (e.g., any of the decoders 410, 510, 810) may be configured to perform CfL prediction via a CfL prediction process. In at least some of these implementations, the encoder and / or the decoder may be configured in a CfL prediction mode to perform the CfL prediction process. As described in more detail below, the encoder and / or the decoder may be operable in at least one of a number of different CfL prediction modes. In the different CfL prediction modes, the encoder and / or the decoder may perform different respective CfL processes to generate chroma prediction samples.

[0143]

[0164] 22A-22C, the encoder and / or decoder may include a CfL prediction unit 2202 configured to perform a CfL prediction process. In various implementations, the CfL prediction unit 2202 may be a stand-alone unit or may be a component or sub-unit of another unit of the encoder or decoder. For example, in any of various implementations, the CfL prediction unit 2202 may be a component or sub-unit of the intra prediction unit 552, the intra encoder 722, or the intra decoder 872. The CfL prediction unit 2202 may also be configured to perform various or multiple operations or functions to perform or execute the CfL process. For simplicity, the CfL prediction unit 2202 is described as being a component of the encoder and / or decoder that performs each of these operations or functions. However, in any of various implementations, the CfL prediction unit 2202 may be further configured or organized into multiple sub-units (each configured to perform one or more of the operations or functions of the CfL prediction process), or one or more units separate from the CfL prediction unit 2202 may be configured to perform one or more operations of the CfL prediction process. Also, in any of various implementations, the CfL prediction unit 2202 may be implemented in hardware or a combination of hardware and software to perform and / or execute the operations or functions of the CfL process. For example, the CfL prediction unit may be implemented as an integrated circuit, or a processor configured to execute software or firmware stored in a memory, or a combination thereof. Also, in any of various implementations, a non-transitory computer-readable storage medium may store computer instructions executable by a processor to perform the functions or operations of the CfL prediction unit 2202.

[0144]

[0165] In the case of CfL prediction, the encoder and / or decoder may determine, e.g., via the CfL prediction unit 2202, that a CfL prediction mode should be applied to a luma block in the received coded bitstream. Then, the CfL prediction unit 2202 may perform CfL prediction on the luma block according to the determined CfL prediction mode to be applied. Correspondingly, the encoder and / or decoder may reconstruct, e.g., via the CfL prediction unit 2202, a chroma block that corresponds to or is co-located with the luma block, at least in part by application of the CfL prediction mode.

[0145]

[0166] As described above, the CfL prediction unit 2202 may operate in at least one of a plurality of CfL prediction modes. Referring to FIG. 22A, in a first CfL prediction mode (or a first set of one or more CfL prediction modes), the CfL prediction unit 2202 may be configured to generate a plurality of prediction samples of a chroma block corresponding to a luma block based on luma samples of the luma block. As described above, the chroma block corresponding to a luma block is a chroma block co-located with the luma block. Correspondingly, in FIG. 22A, in the first CfL prediction mode, the CfL prediction unit uses luma samples of a luma block co-located with a chroma block for which the prediction samples are generated.

[0146]

[0167] FIG. 23 illustrates a flow diagram of an example method 2300 of a CfL prediction process that the CfL prediction unit 2202 may perform in the first CfL prediction mode. In some implementations, the CfL prediction process as illustrated in FIG. 23 generates a number of chroma prediction samples based on an alternating current (AC) contribution of the luma sample and a direct current (DC) contribution of the chroma sample. Each of the AC and DC contributions may be a prediction of a chroma component, also referred to as a prediction of the AC and DC contributions. In particular in these implementations, the chroma prediction samples are modeled as a linear function of the luma sample, for example according to the following equation: CfL(α)=α×L AC +DC (1) Here, L AC denotes the AC contribution of the luma component (luma sample), α denotes the scaling parameter of the linear model, and DC denotes the DC contribution of the chroma components. Also, in at least some implementations, the AC contributions are taken for each of the samples of the block, while the DC contribution is taken for the entire block.

[0147]

[0168] In particular in these implementations as shown in FIG. 23, in block 2302, multiple luma samples of a luma block may be subsampled (or downsampled) to a chroma resolution (e.g., 4:2:0, 4:2:2, or 4:4:4). In block 2304, the subsampled luma samples may be averaged to generate a luma average. In block 2306, the luma average may be subtracted from the luma samples to generate an AC contribution of the luma component. In block 2308, the AC contribution of the luma component may be multiplied with a scaling parameter α to generate a scaled AC contribution of the luma component. The scaled AC contribution of the luma component may also be an AC contribution prediction of the chroma component. In block 2310, a DC contribution prediction of the chroma component may be added to the AC contribution prediction to generate a chroma prediction sample according to a linear model. In at least some implementations, the scaling parameter α is based on the original chroma samples and may be signaled in the bitstream. This may reduce decoder complexity and result in more accurate prediction. Additionally or alternatively, the DC contributions of the chroma components may be calculated using an intra DC mode in the chroma components in some example implementations.

[0148]

[0169] Further, in some implementations of method 2300 or the first CfL mode, if some luma samples of a comparably positioned luma block are outside the picture boundary, these luma samples may be padded and the padded luma samples may be used to calculate the luma average, such as block 2304. Figure 24 shows a schematic diagram of luma samples inside and outside a picture defined by a picture boundary. In at least some implementations, luma samples of the outer picture may be padded by copying the value of the nearest available sample in the current block.

[0149]

[0170] Additionally or alternatively, in some implementations, when performing CfL prediction, the sub-sampling performed in block 2302 may be combined with the averaging performed in block 2304 and / or the subtraction performed in block 2306. This can eliminate the sub-sampling division and rounding errors while simplifying the linear modeling equation. The following equation (2) corresponds to the combination of both steps, which simplifies to equation (3). Both equations (2) and (3) use integer division. Also, MxN is a matrix of pixels in the luma plane.

[0150]

number

[0171] Based on chroma subsampling, x ×S y ∈{1,2,4}. Also, both M and N may be powers of 2, and M×N is also a power of 2. For example, in a 4:2:0 chroma subsampling situation, instead of applying a box filter, the sum of the four reconstructed luma pixels that match the chroma pixel may be used. Correspondingly, the CfL prediction may be scaled by 2.

[0151]

[0172] Further, as described above with respect to equation (1), the CfL prediction process in FIG. 23 may employ only one linear model between the luma and chroma samples of the entire coded block. However, the only linear model may be less than optimal for some relationships between the luma and chroma samples of the entire coded block. Additionally or alternatively, in the CfL prediction method 2300, the luma samples of the corresponding luma block are used to calculate the average and thus the AC contribution. However, in at least some implementations, the DC contribution may be determined or calculated by averaging nearby luma samples. Inconsistencies between the luma samples used for the AC and DC contributions, respectively, and the nearby luma samples may result in inaccurate or less than optimal predictions. Additionally or alternatively, signaling the scaling value α in the bitstream may undesirably consume some bits. Additionally or alternatively, in the CfL prediction method 2300 in FIG. 23, a chroma block is predicted from a co-located luma block, while luma samples in the vicinity of the co-located luma block are not used.

[0152]

[0173] 22B , in addition to or alternatively to operating in the first CfL prediction mode, in some implementations, the CfL prediction unit 2202 may operate in a second CfL prediction mode (or a second set of one or more CfL prediction modes) to generate multiple prediction samples of a chroma block corresponding to a luma block based on luma samples in the vicinity of the luma block, i.e., the CfL prediction unit 2202 may generate chroma prediction samples using luma samples in the vicinity of the luma block without using luma samples of the luma block. Referring to FIG. 22C, in addition to or alternatively to operating in the first CfL prediction mode and / or the second CfL prediction mode, in some implementations, the CfL prediction unit 2202 may operate in a third CfL prediction mode (or a third set of one or more CfL prediction modes) to generate multiple prediction samples of a chroma block corresponding to a luma block based on luma samples of the luma block and luma samples in the vicinity of the luma block.

[0153]

[0174] FIG. 25 shows a schematic diagram of an example luma block 2502 and its neighboring luma samples. In general, the neighboring luma samples of a given luma block are luma samples of or within neighboring or nearby luma blocks that are adjacent and / or near the given luma block. Each neighboring luma sample may be of or include one of multiple types of neighboring luma samples. Each type may correspond to a relative spatial relationship with the given luma block. Similarly, each neighboring or nearby luma block may have a type that corresponds to one of the types of neighboring luma samples contained therein. In at least some implementations, the multiple types of neighboring luma samples and / or blocks may include: left, top left, top, top right, right, bottom right, bottom, and bottom left. FIG. 25 shows where nearby luma samples may be spatially located for a given luma block 2502, which are: Near-left luma sample 2504 in the near-left luma block, 2506 luma samples near the top left in the near top left luma block, 2508 near-top luma samples in the near-top luma block, 2510 luma samples near the top right in the luma block near the top right, Near-right luma sample 2512 in the near-right luma block, Luma sample 2514 near the bottom right in the luma block near the bottom right, 2516 near-bottom luma samples in the near-bottom luma block, and Contains 2518 luma samples near the bottom left in the near-bottom left luma block. Additionally, the top-left, top-right, bottom-left, and bottom-right neighboring luma samples and blocks may be generally and / or collectively referred to as corner-near luma samples and blocks, respectively. In any of various implementations of the second CfL mode and / or the third CfL mode, the CfL prediction unit 2202 may use at least one of all types or less than all types of the neighboring luma samples when performing CfL prediction.

[0154]

[0175] FIG. 26 illustrates a flowchart of an example method 2600 of a CfL prediction process that the CfL prediction unit 2202 may perform when operating in the second CfL prediction mode (FIG. 22B) and / or the third CfL prediction mode (FIG. 22C). In block 2602, the CfL prediction unit 2202 may determine a plurality of neighboring luma samples of a luma block. In block 2604, the CfL prediction unit 2202 may generate a plurality of prediction samples of a chroma block corresponding to the luma block based on the plurality of neighboring luma samples.

[0155]

[0176] FIG. 27 illustrates a flowchart of another example method 2700 of a CfL prediction process that the CfL prediction unit 2202 may perform when operating in the second CfL prediction mode (FIG. 22B) and / or the third CfL prediction mode (FIG. 22C). In block 2702, the CfL prediction unit 2202 may determine a plurality of neighboring luma samples of a luma block. In block 2704, the CfL prediction unit 2202 may generate AC contributions and DC contributions of a plurality of predicted samples of a chroma block corresponding to the luma block. At least one of the AC contributions or the DC contributions is generated in block 2704 based on a set of luma samples including the plurality of neighboring luma samples determined in block 2702. In block 2706, the CfL prediction unit 2202 may generate a plurality of predicted samples of the chroma block based on the AC contributions and the DC contributions determined in block 2704. In at least some implementations, the CfL prediction unit 2202 may determine, in block 2706, a number of chroma prediction samples using AC and DC contributions according to the linear model of equation (1) above.

[0156]

[0177] In some implementations, the example methods 2600 and 2700 may be combined. For example, block 2602 may include block 2702, and block 2604 may include block 2704 and / or block 2706.

[0157]

[0178] Figure 28 shows a flow diagram of another example method 2800 of a CfL prediction process that the CfL prediction unit 2202 may perform when operating in the third CfL prediction mode (Figure 22C). The CfL prediction process 2800 may be similar to the CfL prediction process 2300 in Figure 23, except that instead of averaging the luma samples of a luma block, the CfL prediction process 2800 averages nearby luma samples to generate a nearby luma average. Thus, the AC contribution is based on both the luma samples of the luma block and the nearby luma samples.

[0158]

[0179] More specifically, in block 2802, multiple luma samples of a luma block may be subsampled to a chroma resolution. In block 2804, multiple neighboring luma samples of a luma block may be subsampled to a chroma resolution. Furthermore, in at least some implementations, the same subsampling (or downsampling) method is used to subsample the luma sample and the neighboring luma samples (i.e., the same subsampling method is applied in both blocks 2802 and 2804). For example, if subsampling is performed according to a 4:2:0 format, Two rows in the upper vicinity area (e.g., area 2508 in FIG. 25 ), Two columns in the nearby area on the left (e.g., area 2504 in FIG. 25), and / or Four pixels in the upper left area (eg, area 2506 in FIG. 25) are subsampled (or downsampled). Correspondingly, when the CfL prediction unit 2202 determines the AC contribution (e.g., blocks 2806 and 2808 below), nearby luma samples can be averaged and subtracted from the reconstructed luma sample value, as shown in FIG. 28 and described further below.

[0159]

[0180] More specifically, in block 2806, the sub-sampled nearby luma samples may be averaged to generate a nearby luma average. In block 2808, the nearby luma average may be subtracted from the sub-sampled luma samples to generate an AC contribution for the luma component. In block 2808, the AC contribution for the luma component may be multiplied with a scaling parameter α to generate a scaled AC contribution for the luma component. The scaled AC contribution for the luma component may also be an AC contribution prediction for the chroma component. In block 2812, the DC contribution prediction for the chroma component may be added to the AC contribution prediction to generate a chroma prediction sample, for example according to the linear model of equation (1) above. In at least some implementations, the scaling parameter α is based on the original chroma samples and may be signaled in the bitstream. This may reduce decoder complexity and result in a more accurate prediction. Additionally or alternatively, the DC contributions of the chroma components may be calculated using an intra DC mode within the chroma components in some example implementations.

[0160]

[0181] In other implementations of example method 2800, the neighboring luma samples may not be subsampled before being used to determine the neighboring luma average, i.e., in these other implementations, block 2804 may be skipped or may not be performed.

[0161]

[0182] Also, in some implementations, all or some of the blocks of method 2800 may be combined with method 2600 and / or method 2700. For example, after the nearby luma samples are determined in blocks 2602 and / or 2702, they may be subsampled in block 2804 and / or averaged in block 2806. Additionally or alternatively, generating a chroma prediction sample based on the nearby luma samples in block 2604 may include one or more of: subsampling the nearby luma samples in block 2804, averaging the (subsampled) nearby luma samples in block 2806, generating a luma AC contribution in block 2808, and / or scaling the luma AC contribution in block 2810. Additionally or alternatively, generating the AC contribution at block 2704 of method 2700 may include one or more of subsampling nearby luma samples at block 2804, averaging the (subsampled) nearby luma samples at block 2806, generating the luma AC contribution at block 2808, and / or scaling the luma AC contribution at block 2810. Additionally or alternatively, generating the chroma prediction sample based on the AC and DC contributions at block 2706 of method 2700 may include adding the DC contribution to the AC contribution at block 2812. Other ways of combining methods 2600, 2700, and / or 2800 may be possible.

[0162]

[0183] Also, in any of the various implementations of the example methods 2600, 2700, 2800, the nearby luma samples that the CfL prediction unit 2202 determines, subsamples, averages, and / or otherwise uses to generate the multiple chroma prediction samples may include all or at least one and less than all of the types of nearby luma samples including at least one of left, top left, top, top right, right, bottom right, bottom, or bottom left. For example, in some implementations, the at least one type may include at least one of left, top, or top left. In some other implementations, the at least one type may include at least one of left, top, right, bottom, top left, top right, bottom left, or bottom right. In some other implementations, the at least one type may include at least one of top right or bottom left. Also, in any of various implementations, including implementations of methods 2600, 2700, and / or 2800, when the CfL prediction unit 2202 determines to use a certain type of nearby luma sample for the CfL prediction process, the CfL prediction unit 2202 may use one or more nearby luma samples having the determined type.

[0163]

[0184] Additionally or alternatively, in some implementations of example methods 2600, 2700, and / or 2800, signaling may be used to explicitly specify or instruct the CfL prediction unit 2202 as to which of the types of nearby luma samples to use for the CfL prediction process. For example, the CfL prediction unit 2202 may receive a signal that is generated internally, such as by another unit or component of the electronic device (e.g., an encoder or decoder) in which the CfL prediction unit 2202 is implemented, or that is generated externally or remotely, such as by another unit or component of a different or separate electronic device (e.g., an encoder or decoder) in which the CfL prediction unit 2202 is implemented. In some other implementations, code information and / or characteristics of the luma block and / or predicted chroma block may be used to implicitly indicate which type of nearby luma samples should be used for the CfL prediction process, non-limiting examples of which include the intra prediction mode, block shape, block size, or block aspect ratio of the luma block and / or chroma block.

[0164]

[0185] Additionally or alternatively, in some implementations of example methods 2600, 2700, and / or 2800, only those types of nearby luma samples that are available may be used for the CfL prediction process. In some implementations, nearby luma samples are available when they are not at a picture boundary or a super-block boundary. For example, CfL prediction unit 2202 may use nearby top luma samples for CfL prediction when only nearby top luma samples are available (e.g., determine nearby top luma samples in block 2602 and / or generate chroma prediction samples based on nearby top luma samples in block 2604). As another example, the CfL prediction unit may use the left-side near luma sample for CfL prediction when only the left-side near luma sample is available (e.g., determining the left-side near luma sample in block 2602 and / or generating a chroma prediction sample based on the left-side near luma sample in block 2604).

[0165]

[0186] Additionally or alternatively, in some implementations of example methods 2600, 2700, and / or 2800, when a certain type of nearby luma samples is determined to be used in the CfL prediction process and all of the certain type of nearby luma samples are not available, the CfL prediction unit 2202 may pad the certain type of nearby luma samples and use the padded nearby luma samples in the CfL prediction process. For example, when the left nearby luma samples are not available, the padding may include copying the left column luma samples of the current luma block as the left nearby luma samples. As another example, when the top nearby luma samples are not available, the padding may include copying the top row luma samples of the current luma block as the top nearby luma samples. For at least some of these implementations, the CfL prediction unit may pad nearby luma samples according to the same padding method used in the intra angle prediction modes.

[0166]

[0187] Additionally or alternatively, in some implementations of example methods 2600, 2700, and / or 2800, when it is determined that the upper nearby luma samples are to be used for the CfL prediction process, the CfL prediction unit 2202 may use only the upper nearby luma samples in the nearest upper reference line in the CfL prediction process. In particular, in these implementations, the CfL prediction unit 2202 may use the upper nearby luma samples in the nearest upper reference line as the nearby luma samples for the CfL prediction process when the co-located luma block is located at a super-block boundary. In general, the AoMediaVideo Model (AVM) uses coding blocks of various sizes, the largest of which is called a super-block. In some implementations, the largest block or super-block is 128×128 pixels or 64×64 pixels. The super-block size is signaled in the sequence header and has a default size of 128 x 128 pixels. The minimum coding block size is 4 x 4.

[0167]

[0188] Additionally or alternatively, in some implementations of example methods 2600, 2700, and / or 2800, the nearby luma samples used in the CfL prediction process may include, and in some implementations may only include, nearby luma samples in one or more nearest neighboring reference lines. At least some of these implementations may be used in combination with one or more of the other conditions or aspects described above. For example, when the CfL prediction unit 2202 determines to use a nearby luma sample of a particular type, the CfL prediction unit 2202 may use a nearby luma sample of a particular type in a nearest neighboring reference line of a particular type. FIG. 29 illustrates a diagram showing upper and left reference lines of co-located chroma blocks. In this example, reference line 0 may be the nearest upper neighboring reference in that it is closer or closer than the other upper reference lines 1, 2, and 3.

[0168]

[0189] Additionally or alternatively, in some implementations of example methods 2600, 2700, and / or 2800, the CfL prediction unit 2202 may use one or more boundary luma samples of a luma block to perform the CfL prediction process. The boundary luma samples may be luma samples that define an outer boundary of a luma block. For example, a luma sample at a left boundary of a given luma block at least partially defines a left boundary of the given luma block. As another example, a top boundary at least partially defines a top or upper boundary of the given luma block. The CfL prediction unit 2202 may use one or more boundary luma samples in combination with nearby luma samples. For example, in an implementation of method 2800, the CfL prediction unit 2202 may determine a nearby luma average by averaging the nearby luma samples in combination with one or more boundary luma samples of a comparably located luma block. In these various implementations, the boundary luma samples used for averaging may be obtained from sub-sampled luma samples before the sub-sampling in block 2802 or from sub-sampled luma samples after the sub-sampling in block 2802. Notably, in these implementations, the one or more boundary luma samples may include one or more upper boundary luma samples and / or one or more left boundary luma samples.

[0169]

[0190] The embodiments of the present disclosure may be used separately or in any order in combination. Furthermore, each method (or embodiment), encoder, and decoder may be implemented by processing circuitry (e.g., one or more processors, or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium. The embodiments of the present disclosure may be applied to luma blocks or chroma blocks.

[0170]

[0191] The techniques described above may be implemented as computer software using computer readable instructions and may be physically stored on one or more computer readable media. For example, Figure 30 illustrates a computer system 3000 suitable for implementing certain embodiments of the disclosed subject matter.

[0171]

[0192] Computer software may be coded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms to produce code that contains instructions that may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or instructions that may be executed via interpretation, microcode execution, etc.

[0172]

[0193] The instructions may be executed in various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0173]

[0194] 30 with respect to computer system 3000 are exemplary in nature and are not intended to suggest any limitations on the scope of use or functionality of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system 3000.

[0174]

[0195] The computer system 3000 may include certain human interface input devices that may be responsive to input by one or more human users, for example, via tactile input (e.g., keystrokes, swipes, data glove movements), auditory input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic pictures).

[0175]

[0196] The input human interface devices may include one or more of the following (only one of each is depicted): a keyboard 3001, a mouse 3002, a trackpad 3003, a touch screen 3010, a data glove (not shown), a joystick 3005, a microphone 3006, a scanner 3007, and a camera 3008.

[0176]

[0197] The computer system 3000 may also include certain human interface output devices that may stimulate one or more of the senses of a human user, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen 3010, data gloves (not shown), joystick 3005, although there may be haptic feedback devices that do not function as input), auditory output devices (e.g., speakers 3009, headphones (not shown)), visual output devices (e.g., screens 3010 including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output, three or more dimensional output by means such as stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0177]

[0198] The computer system 3000 may also include human accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 3020 using media such as CD / DVDs 3021, thumb drives 3022, removable hard drives or solid state drives 3023, legacy magnetic media (not shown) such as tapes and floppy disks (not shown), and specialized ROM / ASIC / PLD based devices such as security dongles (not shown).

[0178]

[0199] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not encompass transmission media, carrier waves, or other transitional signals.

[0179]

[0200] The computer system 3000 may also include a network interface 3054 to one or more communication networks 3055. The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include Ethernet, wireless LANs, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), TV wired or wireless wide area digital networks (including cable TV, satellite TV, and terrestrial TV), vehicular and industrial including CANBus, etc. Particular networks typically require an external network interface adapter attached to a particular general-purpose data port or peripheral bus 3049 (e.g., a USB port on the computer system 3000); others are typically integrated into the core of the computer system 3000 by attaching to a system bus as described below (e.g., an Ethernet interface is integrated in a PC computer system, a cellular network interface is integrated in a smartphone computer system). Using any of these networks, the computer system 3000 can communicate with other entities. Such communications can be one-way receive only (e.g., broadcast television), one-way transmit only (e.g., CANbus to certain CANbus devices), or bi-directional, such as to other computer systems using local or wide area digital networks. Particular protocols and protocol stacks can be used with each of these networks and network interfaces, as described above.

[0180]

[0201] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to the core 3040 of the computer system 3000.

[0181]

[0202] The core 3040 may include one or more central processing units (CPUs) 3041, graphics processing units (GPUs) 3042, specialized programmable processing devices in the form of field programmable gate arrays (FPGAs) 3043, hardware accelerators for specific tasks 3044, graphics adapters 3050, etc. These devices may be connected via a system bus 3048 along with read only memory (ROM) 3045, random access memory 3046, and internal mass storage devices (e.g., internal non-user accessible hard drives, SSDs, etc.) 3047. In some computer systems, the system bus 3048 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 3048 or via a peripheral bus 3049. In one example, a screen 3010 may be connected to the graphics adapter 3050. Peripheral bus architectures include PCI, USB, and the like.

[0182]

[0203] The CPU 3041, GPU 3042, FPGA 3043, and accelerator 3044 may combine to execute certain instructions that may constitute the aforementioned computer code. The computer code may be stored in ROM 3045 or RAM 3046. Temporary data may be stored in RAM 3046 while persistent data may be stored in, for example, internal mass storage 3047. Fast storage and retrieval from any memory device may be enabled by utilizing a cache memory, which may be closely associated with one or more of the CPU 3041, GPU 3042, mass storage 3047, ROM 2045, RAM 2046, etc.

[0183]

[0204] The computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and computer code can be specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those having skill in the computer software arts.

[0184]

[0205] As a non-limiting example, the architecture 3000, and in particular a computer system having the core 3040, may provide the functionality of executing software embodied in one or more tangible computer readable media as a result of the processor (including CPU, GPU, FPGA, accelerator, etc.). Such computer readable media may be media associated with user accessible mass storage as described above, as well as specific storage of the core 3040 of a non-transitory nature, such as mass storage 3047 internal to the core or ROM 3045. Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core 3040. The computer readable media may include one or more memory devices or chips, depending on the particular needs. The software may cause the core 3040, and in particular the processor therein (including CPU, GPU, FPGA, etc.), to perform certain processes or certain parts of certain processes described herein, including defining data structures stored in RAM 3046 and modifying such data structures according to the processes defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 3044) that may execute in place of or in conjunction with software to perform particular processes or portions of particular processes described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry embodying logic for execution, or both, as appropriate. The present disclosure encompasses any appropriate combination of hardware and software.

[0185]

[0206] The subject matter of the present disclosure may also relate to or include, among other aspects, the following:

[0207] In a first aspect, a method for video processing includes: determining that a Chroma from Luma (CfL) prediction mode should be applied to a luma block of the received coded bitstream; generating a nearby luma average for a luma block by averaging a set of reconstructed luma samples, the set of reconstructed luma samples including a plurality of nearby reconstructed luma samples in at least one nearby luma block in a vicinity of the luma block; generating alternating current (AC) contributions of a plurality of prediction samples of a chroma block co-located with the luma block based on a plurality of luma samples in the luma block and a nearby luma average; and Reconstructing the chroma blocks by applying at least a CfL prediction mode based on the AC contribution.

[0186]

[0208] A second aspect includes the first aspect, further comprising: Prior to averaging the set of reconstructed luma samples, the method includes sub-sampling the plurality of reconstructed nearby luma samples to generate a plurality of sub-sampled reconstructed nearby luma samples, wherein the set of reconstructed luma samples includes the plurality of sub-sampled reconstructed nearby luma samples.

[0187]

[0209] A third aspect includes any of the first or second aspects, and further comprising: The step of generating the AC contribution includes the step of subtracting a nearby luma average from the plurality of luma samples.

[0188]

[0210] A fourth aspect includes any of the first to third aspects, and further comprising: The steps to reconstruct a chroma block are: generating a scaled AC contribution by multiplying the AC contribution by a scaling factor; and adding the scaled AC contribution to the DC contribution of a number of predicted samples of the chroma block; This includes the following points.

[0189]

[0211] A fifth aspect includes any of the first to fourth aspects, and further comprising: Multiple reconstructed nearby luma samples are: one or more left reconstructed nearby luma samples, one or more upper reconstructed nearby luma samples, or One or more luma samples in the upper left reconstructed neighborhood The present invention includes at least one of the following items.

[0190]

[0212] A sixth aspect includes any of the first to fourth aspects, and further comprising: Multiple reconstructed nearby luma samples are: one or more upper reconstructed nearby luma samples, and One or more left reconstructed nearby luma samples This includes the fact that it only includes

[0191]

[0213] A seventh aspect includes any of the first to fourth aspects, and further comprising: Multiple reconstructed nearby luma samples are: one or more left reconstructed nearby luma samples, one or more upper reconstructed nearby luma samples, One or more right reconstructed nearby luma samples, One or more lower reconstructed nearby luma samples, or Luma samples near one or more reconstructed corners The present invention includes at least one of the following items.

[0192]

[0214] An eighth aspect includes any of the first to fourth aspects, and further comprising: Multiple reconstructed nearby luma samples are: One or more nearby reconstructed luma samples in the upper right corner, or One or more luma samples near the lower left reconstructed region The present invention includes at least one of the following items.

[0193]

[0215] A ninth aspect includes any of the first to fourth aspects, and further comprising: The number of reconstructed nearby luma samples may include less than all types of reconstructed nearby luma samples.

[0194]

[0216] A tenth aspect includes any of the first to ninth aspects, and further comprising: The reconstructed nearby luma samples used to generate the nearby luma average include one or more types of reconstructed nearby luma samples: Explicitly specified by signalling, or This includes matters that are implicitly specified based on the coded information.

[0195]

[0217] An eleventh aspect includes any of the tenth aspect, further comprising: The coded information may include at least one of: an intra prediction mode, a block shape, a block size, or a block aspect ratio for the luma block.

[0196]

[0218] A twelfth aspect includes any of the first to fourth aspects, and further comprising: The plurality of reconstructed nearby luma samples may include only one or more upper reconstructed nearby luma samples in response to only an upper reconstructed nearby luma sample being available.

[0197]

[0219] A thirteenth aspect includes any of the first to fourth aspects, and further comprising: The plurality of reconstructed nearby luma samples may include only one or more left reconstructed nearby luma samples in response to only a left reconstructed nearby luma sample being available.

[0198]

[0220] A fourteenth aspect includes any of the first to fourth aspects, and further comprising: If it is determined that a certain type of reconstructed nearby luma sample is to be used for the nearby luma average, and if no reconstructed nearby luma sample of that type is available, the method further includes a step of padding one or more reconstructed nearby luma samples of that type, wherein the padding step is performed according to a padding method used in the intra angle prediction mode, and the nearby luma average is determined based on the one or more padded reconstructed nearby luma samples.

[0199]

[0221] A fifteenth aspect includes any of the first to fourth aspects, and further comprising: The plurality of reconstructed nearby luma samples includes one or more upper reconstructed nearby luma samples, and the one or more upper reconstructed nearby luma samples are only those at a nearest upper reference line in accordance with which the luma block is located at a super-block boundary.

[0200]

[0222] A sixteenth aspect includes any of the first to fifteenth aspects, and further comprising: The multiple reconstructed nearby luma samples are included in the nearest adjacent reference line.

[0201]

[0223] A seventeenth aspect includes any of the first or third to sixteenth aspects, and further comprising: The multiple reconstructed nearby luma samples are not subsampled before being averaged to produce the nearby luma average.

[0202]

[0224] An 18th aspect includes any of the 1 to 17 aspects, and further comprising: The set of reconstructed luma samples further includes one or more border luma samples in the luma block.

[0203]

[0225] A 19th aspect includes any of the 18th aspect, further comprising: The one or more boundary luma samples may include at least one of: one or more top boundary luma samples, or one or more left boundary luma samples.

[0204]

[0226] A twentieth aspect includes a video processing device including a memory that stores a set of instructions and a processor configured to execute the set of instructions to implement any of the first to nineteenth aspects.

[0205]

[0227] A twenty-first aspect includes a non-transitory computer-readable medium storing a set of computer instructions that, when executed by a processor of a video processing device, causes the processor to perform any of the first to nineteenth aspects.

[0206]

[0228] In addition to the features mentioned in each of the independent aspects listed above, some examples may exhibit optional features, either alone or in combination, mentioned in dependent aspects and / or disclosed in the above description and illustrated in the drawings.

[0207]

[0229] While this disclosure has described several exemplary embodiments, there are substitutions, permutations, and various alternative equivalents that fall within the scope of this disclosure. Thus, it will be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

[0208]

[0230] Appendix A: Abbreviations JEM: joint exploration model VVC: versatile video coding BMS: benchmark set MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Supplementary Enhancement Information VUI: Video Usability Information GOPs: Groups of Pictures TUs: Transform Units PUs: Prediction Units CTUs: Coding Tree Units CTBs: Coding Tree Blocks PBs: Prediction Blocks HRD: Hypothetical Reference Decoder SNR: Signal Noise Ratio CPUs: Central Processing Units GPUs: Graphics Processing Units CRT: Cathode Ray Tube LCD: Liquid-Crystal Display OLED: Organic Light-Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application-Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile communications; LTE: Long-Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Areas; Field Programmable Gate Array SSD: solid-state drive IC: Integrated Circuit HDR: high dynamic range SDR: standard dynamic range JVET: Joint Video Exploration Team MPM: most probable mode WAIP: Wide-Angle Intra Prediction CU: Coding Unit PU: Prediction Unit TU: Transform Unit CTU: Coding Tree Unit PDPC: Position Dependent Prediction Combination ISP: Intra Sub-Partitions SPS: Sequence Parameter Setting PPS: Picture Parameter Set APS: Adaptation Parameter Set VPS: Video Parameter Set DPS: Decoding Parameter Set ALF: Adaptive Loop Filter SAO: Sample Adaptive Offset CC-ALF: Cross-Component Adaptive Loop Filter CDEF: Constrained Directional Enhancement Filter CCSO: Cross-Component Sample Offset LSO: Local Sample Offset LR: Loop Restoration Filter AV1: AOMedia Video 1;AOMedia Video 1 AV2: AOMedia Video 2;AOMedia Video 2 LFNST: low-Frequency Non-Separable Transform IST: Intra Secondary Transform

Claims

1. 1. A method for video processing comprising: determining that a Chroma from Luma (CfL) prediction mode should be applied to a luma block of the received coded bitstream; generating a nearby luma average for the luma block by averaging a set of reconstructed luma samples, the set of reconstructed luma samples including a plurality of nearby reconstructed luma samples in at least one nearby luma block in a vicinity of the luma block; generating alternating current (AC) contributions of a plurality of prediction samples of a chroma block co-located with the luma block based on a plurality of luma samples in the luma block and the neighboring luma average; and reconstructing the chroma blocks by at least applying the CfL prediction mode based on the AC contributions; The method includes:

2. 13. The method of claim 1 further comprising: sub-sampling the plurality of reconstructed nearby luma samples to generate a plurality of sub-sampled reconstructed nearby luma samples prior to averaging the set of reconstructed luma samples; wherein the set of reconstructed luma samples comprises the plurality of sub-sampled reconstructed neighboring luma samples.

3. 3. The method of claim 1 or 2, wherein generating the AC contribution comprises subtracting the nearby luma average from the plurality of luma samples.

4. 4. The method of claim 1, wherein the step of reconstructing the chroma blocks comprises: multiplying the AC contribution by a scaling factor to generate a scaled AC contribution; and adding the scaled AC contribution to a DC contribution of a plurality of prediction samples of the chroma block; A method comprising:

5. 5. The method of claim 1, wherein the plurality of reconstructed nearby luma samples are: one or more left reconstructed nearby luma samples, one or more upper reconstructed nearby luma samples, or One or more upper left reconstructed nearby luma samples The method includes at least one of the following:

6. 5. The method of claim 1, wherein the plurality of reconstructed nearby luma samples are: one or more upper reconstructed nearby luma samples, and One or more nearby left reconstructed luma samples A method that includes only

7. 5. The method of claim 1, wherein the plurality of reconstructed nearby luma samples are: one or more left reconstructed nearby luma samples, one or more upper reconstructed nearby luma samples; one or more right reconstructed nearby luma samples, one or more lower reconstructed nearby luma samples, or Luma samples near one or more reconstructed corners The method includes at least one of the following:

8. 5. The method of claim 1, wherein the plurality of reconstructed nearby luma samples are: One or more nearby upper right reconstructed luma samples, or One or more nearby lower-left reconstructed luma samples The method includes at least one of the following:

9. 5. The method of claim 1, wherein one or more types of reconstructed nearby luma samples including the plurality of reconstructed nearby luma samples used to generate the nearby luma average include: Explicitly specified by signalling, or The method is implicit based on the coded information.

10. 10. The method of claim 9, wherein the coded information includes at least one of: an intra-prediction mode, a block shape, a block size, or a block aspect ratio of the luma block.

11. 5. The method according to claim 1, wherein the plurality of reconstructed nearby luma samples comprises only one or more upper reconstructed nearby luma samples in response to only the upper reconstructed nearby luma samples being available.

12. 5. The method according to claim 1, wherein the plurality of reconstructed nearby luma samples comprises only one or more left reconstructed nearby luma samples in response to only the left reconstructed nearby luma samples being available.

13. 5. The method according to claim 1, further comprising the step of padding one or more reconstructed nearby luma samples of a certain type if it is decided to use the reconstructed nearby luma samples of that type for the nearby luma average, if no reconstructed nearby luma samples of that type are available, wherein the padding step is performed according to a padding method used in an intra-angle prediction mode, and the nearby luma average is determined based on the one or more padded reconstructed nearby luma samples.

14. 5. The method according to claim 1, wherein the plurality of reconstructed nearby luma samples includes one or more upper reconstructed nearby luma samples, the one or more upper reconstructed nearby luma samples being only those at a nearest upper reference line depending on whether the luma block is located at a super-block boundary.

15. 5. The method of claim 1, wherein the plurality of reconstructed nearby luma samples are in nearest adjacent reference lines.

16. 17. The method of claim 1, wherein the plurality of reconstructed nearby luma samples are not sub-sampled before being averaging to generate the nearby luma average.

17. 17. The method of claim 1, wherein the set of reconstructed luma samples further comprises one or more border luma samples in the luma block.

18. 20. The method of claim 17, wherein the one or more boundary luma samples include at least one of: one or more top boundary luma samples, or one or more left boundary luma samples.

19. a memory storing a set of instructions; and a processor configured to execute said set of instructions; 11. A video processing device comprising: determining that a Chroma from Luma (CfL) prediction mode should be applied to a luma block of the received coded bitstream; averaging a set of reconstructed luma samples to generate a nearby luma average for the luma block, the set of reconstructed luma samples including a plurality of nearby reconstructed luma samples in at least one nearby luma block in a neighborhood of the luma block; generating alternating current (AC) contributions of a plurality of prediction samples of a chroma block co-located with the luma block based on a plurality of luma samples in the luma block and the neighboring luma average; and reconstructing the chroma blocks by at least applying the CfL prediction mode based on the AC contributions; a video processing device configured to:

20. A computer program product causing a processor of a computer to carry out a method, the method comprising: determining that a Chroma from Luma (CfL) prediction mode should be applied to a luma block of the received coded bitstream; averaging a set of reconstructed luma samples to generate a nearby luma average for the luma block, the set of reconstructed luma samples including a plurality of nearby reconstructed luma samples in at least one nearby luma block in a neighborhood of the luma block; generating alternating current (AC) contributions of a plurality of prediction samples of a chroma block co-located with the luma block based on a plurality of luma samples in the luma block and the neighboring luma average; and reconstructing the chroma blocks by at least applying the CfL prediction mode based on the AC contributions; A computer program comprising:

Images