Luma to Chroma Prediction Using Mapping and Different Types

Abstract

The present disclosure relates to a video processing method including: determining that a chroma block should be predicted in a luma-to-chroma (CfL) mode, where the chroma block corresponds to a luma block; mapping luma samples of the luma block to neighboring chroma samples in at least one neighboring chroma block in the vicinity of the chroma block; and performing CfL prediction of the chroma block using the neighboring chroma samples as prediction samples of the chroma block. The present disclosure also relates to a video processing method including: determining a type of a luma-to-chroma (CfL) prediction process from a plurality of different types of CfL prediction processes; and performing the CfL prediction process according to the type of the CfL prediction process.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority to U.S. Patent Application No. 17 / 951,931, entitled "CHROMA FROM LUMA PREDICTION USING MAPPING AND DIFFERENT TYPES," filed September 23, 2022, and U.S. Provisional Patent Application No. 63 / 332,567, entitled "IMPROVED SIGNALING METHOD FOR CHROMA FROM LUMA INTRA PREDICTION MODE," filed April 19, 2022, each of which is incorporated by reference in its entirety herein.

[0002] This disclosure describes a set of advanced video coding techniques. More specifically, the techniques disclosed involve luma-to-chroma prediction. [Background technology]

[0003] This background discussion provided herein is intended to generally present the context of the present disclosure. The work of the inventors cited herein is not admitted, expressly or impliedly, as prior art to the present disclosure, as are aspects of the description that may not have been admitted as prior art at the time of filing of this application, to the extent that their work is described in this background section.

[0004] Video encoding and decoding may be performed using inter-picture prediction with motion compensation. Uncompressed digital video may include a sequence of pictures, each having spatial dimensions of, for example, 1920x1080 luma samples and associated full subsampled or subsampled chroma samples. The sequence of pictures may have a fixed or variable picture rate (alternatively called frame rate), for example, 60 pictures / second or 60 frames / second. Uncompressed video has specific bit rate requirements for streaming or data processing. For example, a video with a pixel resolution of 1920x1080, a frame rate of 60 frames / second, and 4:2:0 chroma subsampling with 8 bits per pixel per color channel requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 gigabytes of storage space.

[0005] One objective of video encoding and decoding may be to reduce redundancy in an uncompressed input video signal due to compression. Compression may help reduce the aforementioned bandwidth and / or storage space requirements, in some cases by more than two orders of magnitude. Both lossless and lossy compression, as well as combinations thereof, may be employed. Lossless compression refers to techniques where an exact copy of the original signal may be reconstructed from the compressed original signal through a decoding process. Lossy compression refers to a coding / decoding process where the original video information is not fully preserved during coding and cannot be fully restored during decoding. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signals is made small enough to make the reconstructed signal useful for its intended application, even with some information loss. For video, lossy compression is widely adopted in many applications. The amount of tolerable distortion depends on the application. For example, a user of a particular consumer video streaming application may tolerate higher distortion than a user of a movie or television broadcast application. The compression ratio achievable by a particular coding algorithm can be selected or adjusted to reflect different distortion tolerances, and generally, higher distortion tolerances allow for coding algorithms that result in higher losses and higher compression ratios.

[0006] Video encoders and decoders can utilize techniques from a number of broad categories and steps, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

[0007] Video codec technology can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. When all blocks of samples are coded in intra mode, the picture can be called an intra-picture. Intra-pictures and their derived pictures, such as independent decoder refresh pictures, can be used to reset the decoder state and thus can be used as the first picture in the coded video bitstream and video session or as still images. The samples of the block after intra prediction can then be transformed to the frequency domain, and the transform coefficients so generated can be quantized before entropy coding. Intra prediction refers to a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after transformation and the smaller the AC coefficients, the fewer bits are needed at a given quantization step size to represent the block after entropy coding.

[0008] Conventional intra-coding, for example as known from MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to code / decode a block based on surrounding sample data and / or metadata, for example obtained during spatially neighboring encoding and / or decoding, that precede in decoding order the block of data being intra-coded or intra-decoded. Such techniques are hereinafter referred to as "intra-prediction" techniques. It should be noted that in at least some cases, intra-prediction uses reference data only from the current picture being reconstructed, and not from other reference pictures.

[0009] Intra prediction may take many different forms. When two or more of such techniques are available in a given video coding technique, the technique used may be referred to as an intra prediction mode. One or more intra prediction modes may be provided in a particular codec. In certain cases, a mode may have sub-modes and / or may be associated with various parameters, and the mode / sub-mode information and intra coding parameters of a block of video may be coded individually or collectively included in the codeword of the mode. Which codeword is used for a given mode, sub-mode, and / or parameter combination may affect coding efficiency gains via intra prediction and may also affect the entropy coding technique used to convert the codeword into a bitstream.

[0010] Certain modes of intra prediction were introduced in H.264, improved in H.265, and further improved in newer coding techniques such as Joint Search Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). In general, in intra prediction, the predictor block may be formed using nearby sample values ​​that become available. For example, available values ​​of a particular set of nearby samples along a particular direction and / or line may be copied to the predictor block. The reference to the direction in use may be coded in the bitstream or may itself be predicted.

[0011] Referring to FIG. 1A, at the bottom right, a subset of nine predictor directions specified in the 33 possible intra predictor directions of H.265 (corresponding to the 33 angle modes of the 35 intra modes specified in H.265) is shown. The point 101 where the arrows converge represents the sample to be predicted. The arrows represent the direction in which neighboring samples are used to predict the sample at 101. For example, arrow 102 indicates that sample 101 is predicted from one or more neighboring samples toward the top right at an angle of 45 degrees from the horizontal. Similarly, arrow 103 indicates that sample 101 is predicted from one or more neighboring samples toward the bottom left of sample 101 at an angle of 22.5 degrees from the horizontal.

[0012] 1A, a square block 104 of 4x4 samples (indicated by a thick dashed line) is depicted at the top left. The square block 104 contains 16 samples, each labeled with "S" and its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample of the block 104 in both the Y and X dimensions. Since the block is 4x4 samples in size, S44 is at the bottom right. An exemplary reference sample is also shown that follows a similar numbering scheme. The reference sample is labeled with R, its Y position (e.g., row index), and X position (column index) relative to the block 104. In both H.264 and H.265, prediction samples from neighboring neighborhoods adjacent to the block being reconstructed are used.

[0013] Intra-picture prediction of block 104 can start by copying reference sample values ​​from neighboring samples according to the signaled prediction direction. For example, assume that the coded video bitstream includes signaling for this block 104 indicating the prediction direction of arrow 102, i.e., the sample is predicted from one or more prediction samples at an angle of 45 degrees from the horizontal direction toward the upper right. In such a case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In certain cases, to calculate a reference sample, especially when the orientation is not evenly divisible by 45 degrees, the values ​​of multiple reference samples may be combined, for example by interpolation.

[0015] The number of possible directions has increased as video coding technology continues to develop. For example, in H.264 (2003), nine different directions are available for intra prediction. This increases to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions at the time of this disclosure. Experimental studies have been conducted to help identify the most suitable intra prediction directions, and those most suitable directions can be coded with a small number of bits using certain techniques of entropy coding, accepting a certain bit penalty for the direction. Furthermore, the direction itself may be predictable, in some cases, from neighboring directions used in intra prediction of the decoded neighboring blocks.

[0016] FIG. 1B shows a schematic diagram 180 depicting 65 intra prediction directions according to JEM to illustrate the increase in the number of prediction directions in various coding techniques developed over time.

[0017] The manner in which bits representing intra-prediction directions in a coded video bitstream are mapped to prediction directions may vary across video coding techniques, ranging from simple direct mapping of prediction directions to intra-prediction modes, to complex adaptation schemes involving codewords, most probable modes, and similar techniques. In all cases, however, there may be certain directions of intra-prediction that are statistically less likely to occur in the video content than certain other directions. Since the goal of video compression is to reduce redundancy, in a properly designed video coding technique, less likely directions may be represented with more bits than more likely directions.

[0018] Inter-picture prediction, or inter-prediction, may be based on motion compensation. In motion compensation, sample data from a previously reconstructed picture or part thereof (reference picture) may be used to predict a newly reconstructed picture or picture part (e.g., block) after being spatially shifted in a direction indicated by a motion vector (hereinafter, MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions X and Y, or three dimensions, with the third dimension being an indication of the reference picture in use (similar to the temporal dimension).

[0019] In some video compression techniques, a current MV applicable to a particular area of ​​sample data can be predicted from other MVs, e.g., other MVs related to other areas of sample data that are spatially adjacent to the area being reconstructed and that precede the current MV in decoding order. Doing so can substantially reduce the overall amount of data required to code the MV by relying on the removal of redundancy of correlated MVs, thereby increasing compression efficiency. MV prediction can work effectively because, for example, when coding an input video signal derived from a camera (known as natural video), areas larger than the area where a single MV is applicable have a statistical likelihood to move in a similar direction in the video sequence, and therefore, in some cases, can be predicted using similar motion vectors derived from MVs of nearby areas. As a result, the actual MV of a given area is similar or identical to the MV predicted from the surrounding MVs. Such MVs can then be represented, after entropy coding, with fewer bits than would be used if the MVs were directly coded instead of predicted from nearby MVs. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from an original signal (i.e., a sample stream). In other cases, the MV prediction itself may be non-lossy, for example due to rounding errors when computing a predictor from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms specified by H.265, the one described below is a technique hereafter referred to as "spatial merging".

[0021] Specifically, referring to FIG. 2, a current block (201) contains samples that are detected by the encoder during the motion search process as predictable from a spatially shifted previous block of the same size. Instead of coding its MV directly, the MV may be derived from metadata associated with one or more reference pictures, e.g., the most recent reference picture (in decoding order), using MVs associated with any one of five surrounding samples, denoted A0, A1, and B0, B1, B2 (202-206, respectively). In H.265, MV prediction may use predictors from the same reference picture that neighboring blocks use. Summary of the Invention [Means for solving the problem]

[0022] Aspects of the present disclosure provide methods and apparatus for luma-to-chroma (CfL) prediction.

[0023] In some implementations, a method for video processing includes determining that a chroma block should be predicted in luma-to-chroma (CfL) mode, where the chroma block corresponds to a luma block; mapping a plurality of luma samples of the luma block to a plurality of neighboring chroma samples in at least one neighboring chroma block in the vicinity of the chroma block; and performing CfL prediction of the chroma block using the plurality of neighboring chroma samples as a plurality of prediction samples of the chroma block.

[0024] In some other implementations, a method for video processing includes determining a type of luma to chroma (CfL) prediction process from a plurality of different types of CfL prediction processes, and performing the CfL prediction process according to the type of CfL prediction process.

[0025] In some other implementations, a device for processing video information is disclosed. The device may include circuitry configured to perform any one of the above method implementations.

[0026] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for video processing for video decoding and / or video encoding.

[0027] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0028] [Figure 1A] FIG. 1 is a schematic diagram of an example subset of intra-prediction directional modes. [Figure 1B] FIG. 2 illustrates an exemplary intra-prediction direction. [Diagram 2] FIG. 2 is a schematic diagram of a current block and its surrounding spatial merging candidates for motion vector prediction in one example. [Diagram 3] FIG. 3 is a schematic diagram illustrating a simplified block diagram of a communication system 300 in accordance with an exemplary embodiment. [Figure 4] FIG. 4 is a schematic diagram illustrating a simplified block diagram of a communication system 400 in accordance with an example embodiment. [Diagram 5] FIG. 2 is a schematic diagram illustrating a simplified block diagram of a video decoder according to an exemplary embodiment. [Figure 6] FIG. 2 is a schematic diagram illustrating a simplified block diagram of a video encoder according to an example embodiment. [Figure 7] FIG. 2 is a block diagram of a video encoder according to another example embodiment. [Figure 8] FIG. 4 is a block diagram of a video decoder according to another exemplary embodiment. [Figure 9]FIG. 2 illustrates a coding block partitioning scheme according to an exemplary embodiment of the present disclosure. [Figure 10] FIG. 13 illustrates another scheme for coding block partitioning according to an exemplary embodiment of the present disclosure. [Figure 11] FIG. 13 illustrates another scheme for coding block partitioning according to an exemplary embodiment of the present disclosure. [Figure 12] FIG. 2 illustrates an exemplary division of a base block into coding blocks according to an exemplary division scheme. [Figure 13] FIG. 2 illustrates an exemplary division-of-thirds scheme. [Figure 14] FIG. 2 illustrates an exemplary quadtree / binary tree coding block partitioning scheme. [Figure 15] 2 illustrates a scheme for splitting a coding block into multiple transform blocks and the coding order of the transform blocks according to an exemplary embodiment of the present disclosure. [Figure 16] 4A-4C are diagrams illustrating another scheme for splitting a coding block into multiple transform blocks and the coding order of the transform blocks according to an exemplary embodiment of the present disclosure. [Figure 17] FIG. 2 illustrates another scheme for splitting a coding block into multiple transform blocks, according to an exemplary embodiment of the present disclosure. [Figure 18] FIG. 1 illustrates an example fine angle in directional intra prediction. [Figure 19] FIG. 13 illustrates nominal angles in directional intra prediction. [Figure 20] FIG. 13 illustrates top, left, and top-left positions for PAETH modes in a block. [Figure 21] FIG. 13 illustrates an example of a recursive intra-filtering mode. [Figure 22A] 1 is a block diagram of a luma-to-chroma (CfL) prediction unit configured to generate prediction samples of a chroma block based on luma samples of the luma block. [Figure 22B]1 is a block diagram of a CfL prediction unit configured to generate prediction samples of a chroma block based on neighboring luma samples of a co-located luma block. [Figure 22C] 1 is a block diagram of a CfL prediction unit configured to generate predicted samples of a chroma block based on luma samples of a co-located luma block and neighboring luma samples. [Diagram 23] FIG. 1 is a flow diagram of an example CfL prediction process. [Figure 24] FIG. 1 is a block diagram of luma samples inside and outside a picture boundary. [Diagram 25] FIG. 2 is a schematic diagram of neighboring luma samples of a luma block. [Figure 26] 13 is another example flowchart of a CfL prediction process. [Figure 27] 13 is another example flowchart of a CfL prediction process. [Figure 28] FIG. 13 is another example flow diagram of a CfL prediction process. [Figure 29] 13 is another example flowchart of a CfL prediction process. [Diagram 30] FIG. 13 is a schematic diagram of luma samples mapped to neighboring chroma samples for CfL prediction of a chroma sample corresponding to the luma sample. [Diagram 31] FIG. 13 is a schematic diagram of patch-based mapping of luma samples to neighboring luma samples. [Diagram 32] FIG. 13 is a schematic diagram of an example four-baseline intra-coding for chroma blocks. [Diagram 33] 13 is another example flowchart of a CfL prediction process. [Diagram 34] FIG. 1 is a schematic diagram of a computer system according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] FIG. 3 illustrates a simplified block diagram of a communication system 300 according to an embodiment of the present disclosure. The communication system 300 includes a plurality of terminal devices capable of communicating with each other, for example, via a network 350. For example, the communication system 300 includes a first pair of terminal devices 310 and 320 interconnected via the network 350. In the example of FIG. 3, the first pair of terminal devices 310 and 320 can perform unidirectional transmission of data. For example, the terminal device 310 can code video data (e.g., of a stream of video pictures captured by the terminal device 310) for transmission to another terminal device 320 via the network 350. The encoded video data can be transmitted in the form of one or more coded video bitstreams. The terminal device 320 can receive the coded video data from the network 350, decode the coded video data to reconstruct the video pictures, and display the video pictures according to the reconstructed video data. The unidirectional data transmission may be implemented in media serving applications, etc.

[0030] In another example, the communication system 300 includes a second pair of terminal devices 330 and 340 for bidirectional transmission of coded video data, which may be implemented, for example, during video conferencing applications. For bidirectional transmission of data, in one example, each terminal device of the terminal devices 330 and 340 can code video data (e.g., of a stream of video pictures captured by the terminal device) for transmission to the other terminal device of the terminal devices 330 and 340 over the network 350. Also, each terminal device of the terminal devices 330 and 340 can receive coded video data transmitted by the other terminal device of the terminal devices 330 and 340, can decode the coded video data to recover the video pictures, and can display the video pictures on an accessible display device according to the recovered video data.

[0031] In the example of FIG. 3, terminal devices 310, 320, 330, and 340 may be implemented as a server, a personal computer, and a smartphone, although applicability of the underlying principles of the present disclosure may not be so limited. Embodiments of the present disclosure may be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated video conferencing equipment, and the like. Network 350 represents any number or type of network that conveys coded video data between terminal devices 310, 320, 330, and 340, including, for example, wired (wired) and / or wireless communication networks. Communication network 350 may exchange data over circuit-switched, packet-switched, and / or other types of channels. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this discussion, the architecture and topology of network 350 may not be important to the operation of the present disclosure unless explicitly described herein.

[0032] 4 illustrates the arrangement of a video encoder and a video decoder in a video streaming environment as an example of an application for the disclosed subject matter. The disclosed subject matter may be equally applicable to other video applications including, for example, video conferencing, digital television broadcasting, gaming, virtual reality, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0033] The video streaming system may include a video source 401, e.g., a video capture subsystem 413 that may include a digital camera, for creating a stream of uncompressed video pictures or images 402. In one example, the stream of video pictures 402 includes samples recorded by the digital camera of the video source 401. The stream of video pictures 402, shown as a thick line to emphasize the large amount of data compared to the encoded video data 404 (or coded video bitstream), may be processed by an electronic device 420 that includes a video encoder 403 coupled to the video source 401. The video encoder 403 may include hardware, software, or a combination thereof for realizing or implementing aspects of the disclosed subject matter, as described in more detail below. The encoded video data 404 (or coded video bitstream 404), shown as a thin line to emphasize the small amount of data compared to the uncompressed video picture stream 402, may be stored in a streaming server 405 for future use or directly in a downstream video device (not shown). One or more streaming client subsystems, such as client subsystems 406 and 408 of FIG. 4, can access the streaming server 405 to retrieve copies 407 and 409 of the encoded video data 404. The client subsystem 406 can include a video decoder 410, for example, within the electronic device 430. The video decoder 410 decodes the incoming copy of the encoded video data 407 and creates an outgoing stream of video pictures 411 that can be uncompressed and rendered on a display 412 (e.g., a display screen) or other rendering device (not shown). The video decoder 410 may be configured to perform some or all of the various functions described in this disclosure. In some streaming systems, the encoded video data 404, 407, and 409 (e.g., a video bitstream) may be encoded according to a particular video coding / compression standard.Examples of these standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC and other video coding standards.

[0034] It should be noted that electronic devices 420 and 430 may include other components (not shown). For example, electronic device 420 may include a video decoder (not shown), and electronic device 430 may likewise include a video encoder (not shown).

[0035] 5 shows a block diagram of a video decoder 510 according to any embodiment of the present disclosure below. The video decoder 510 can be included in an electronic device 530. The electronic device 530 can include a receiver 531 (e.g., a receiving circuit). The video decoder 510 can be used in place of the video decoder 410 of the example of FIG. 4.

[0036] The receiver 531 may receive one or more coded video sequences that are decoded by the video decoder 510. In the same or another embodiment, one coded video sequence may be decoded at a time, and the decoding of each coded video sequence is independent of the other coded video sequences. Each video sequence may be associated with multiple video frames or images. The coded video sequences may be received from a channel 501, which may be a hardware / software link to a storage device that stores the coded video data or a streaming source that transmits the coded video data. The receiver 531 may receive the coded video data along with other data, such as coded audio data and / or auxiliary data streams, and may be forwarded to respective processing circuits (not shown). The receiver 531 may separate the coded video sequences from the other data. To address network jitter, a buffer memory 515 may be disposed between the receiver 531 and the entropy decoder / parser 520 (hereinafter "parser 520"). In certain applications, the buffer memory 515 may be implemented as part of the video decoder 510. In other applications, the buffer memory 515 may be outside and separate from the video decoder 510 (not shown). In still other applications, there may be a buffer memory (not shown) outside the video decoder 510, for example to deal with network jitter, and there may be another additional buffer memory 515 inside the video decoder 510, for example to handle playback timing. If the receiver 531 is receiving data from a store / forward device with sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory 515 may not be needed or may be small. For use with best-effort packet networks such as the Internet, a sufficient size of the buffer memory 515 may be required, and the size may be relatively large.Such a buffer memory may be implemented with an adaptive size and may be implemented at least in part in an operating system or similar element (not shown) external to the video decoder 510.

[0037] The video decoder 510 may include a parser 520 for reconstructing symbols 521 from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder 510 and potentially information for controlling a rendering device such as a display 512 (e.g., a display screen) that may or may not be an integral part of the electronic device 530, but may be coupled to the electronic device 530, as shown in FIG. 5. The control information for the rendering device may be in the form of a supplemental enhancement information (SEI message) or a video usability information (VUI) parameter set fragment (not shown). The parser 520 may parse / entropy decode the coded video sequence received by the parser 520. The entropy coding of the coded video sequence may be in accordance with a video coding technique or standard and may follow various principles including variable length coding, Huffman coding, arithmetic coding with or without context dependency, etc. The parser 520 may extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the subgroup. The subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. Parser 520 may also extract information from the coded video sequence, such as transform coefficients (e.g., Fourier transform coefficients), quantizer parameter values, motion vectors, etc.

[0038] The parser 520 may perform an entropy decoding / parsing operation on the video sequence received from the buffer memory 515 to produce symbols 521 .

[0039] The reconstruction of symbols 521 may involve a number of different processing or functional units, depending on the type of coded video picture or portion thereof (inter-picture and intra-picture, inter-block and intra-block, etc.) and other factors. The units involved and how they are involved may be controlled by subgroup control information parsed from the coded video sequence by parser 520. The flow of such subgroup control information between parser 520 and the following processing or functional units is not shown for the sake of simplicity.

[0040] Besides the functional blocks already mentioned, the video decoder 510 may be conceptually subdivided into several functional units, as described below. In an actual implementation operating under commercial constraints, many of these functional units may closely interact with each other and may be at least partially integrated with each other. However, for the purpose of clearly describing the various functions of the disclosed subject matter, a conceptual subdivision into functional units is adopted in the following disclosure.

[0041] The first unit may include a scalar / inverse transform unit 551. The scalar / inverse transform unit 551 may receive quantized transform coefficients and control information from the parser 520, including information indicating which type of inverse transform to use, block size, quantization coefficients / parameters, quantization scaling matrices, and states as symbols 521. The scalar / inverse transform unit 551 may output blocks including sample values ​​that can be input to an aggregator 555.

[0042] In some cases, the output samples of the scalar / inverse transform 551 may relate to intra-coded blocks, i.e., blocks that do not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by the intra-picture prediction unit 552. In some cases, the intra-picture prediction unit 552 may generate a block of the same size and shape as the block being reconstructed using surrounding block information already reconstructed and stored in the current picture buffer 558. The current picture buffer 558 buffers, for example, the partially reconstructed current picture and / or the fully reconstructed current picture. In some implementations, the aggregator 555 may add, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit 552 to the output sample information provided by the scalar / inverse transform unit 551.

[0043] In other cases, the output samples of the scalar / inverse transform unit 551 may relate to an inter-coded and potentially motion-compensated block. In such a case, the motion compensated prediction unit 553 may access the reference picture memory 557 to fetch samples used for inter-picture prediction. After motion compensating the fetched samples according to the symbols 521 related to the block, these samples may be added to the output of the scalar / inverse transform unit 551 (the output of the unit 551 may be referred to as a residual sample or a residual signal) by the aggregator 555 to generate output sample information. The addresses in the reference picture memory 557 from which the motion compensated prediction unit 553 fetches prediction samples available to the motion compensated prediction unit 553 in the form of the symbols 521, which may have, for example, X, Y components (shift), and reference picture components (time), may be controlled by the motion vector. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory 557 when a sub-sample accurate motion vector is used, and may also be associated with a motion vector prediction mechanism, etc.

[0044] The output samples of aggregator 555 may be subjected to various loop filtering techniques in loop filter unit 556. The video compression techniques may include in-loop filter techniques controlled by parameters included in the coded video sequence (also called coded video bitstream) and made available to loop filter unit 556 as symbols 521 from parser 520, but may also be responsive to meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence, or may be responsive to previously reconstructed and loop filtered sample values. As described in more detail below, several types of loop filters may be included as part of loop filter unit 556, in various orders.

[0045] The output of the loop filter unit 556 may be a sample stream that may be output to the rendering device 512 as well as stored in a reference picture memory 557 for use in future inter-picture prediction.

[0046] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future inter-picture prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by parser 520), the current picture buffer 558 can become part of reference picture memory 557, and a new current picture buffer can be reallocated before beginning reconstruction of a subsequent coded picture.

[0047] The video decoder 510 may perform decoding operations according to a given video compression technique adopted in a standard, such as ITU-T Rec. H.265. The coded video sequence may conform to a syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, the profile may select a particular tool from all tools available in the video compression technique or standard as the only tool available for use under that profile. To conform to a standard, the complexity of the coded video sequence may be within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further limited in some cases by a hypothetical reference decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0048] In some demonstrative embodiments, the receiver 531 may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder 510 to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0049] 6 shows a block diagram of a video encoder 603 according to an exemplary embodiment of the present disclosure. The video encoder 603 may be included in an electronic device 620. The electronic device 620 may further include a transmitter 640 (e.g., a transmitting circuit). The video encoder 603 may be used in place of the video encoder 403 of the example of FIG.

[0050] The video encoder 603 may receive video samples from a video source 601 (which is not part of the electronic device 620 in the example of FIG. 6) that may capture video images to be coded by the video encoder 603. In another example, the video source 601 may be implemented as part of the electronic device 620.

[0051] The video source 601 may provide a source video sequence to be coded by the video encoder 603 in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCb, RGB, XYZ ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source 601 may be a storage device capable of storing pre-prepared video. In a video conferencing system, the video source 601 may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual pictures or images that give motion when viewed in sequence. The picture itself may be organized as a spatial array of pixels, each pixel may contain one or more samples depending on the sampling structure, color space, etc. being used. Those skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0052] According to some exemplary embodiments, the video encoder 603 can code and compress pictures of a source video sequence into a coded video sequence 643 in real time or under other time constraints required by the application. Enforcing an appropriate coding rate constitutes one function of the controller 650. In some embodiments, the controller 650 can be functionally coupled to and control other functional units, as described below. For the sake of brevity, couplings are not shown. Parameters set by the controller 650 can include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller 650 can be configured to have other appropriate functions associated with the video encoder 603 optimized for a particular system design.

[0053] In some exemplary embodiments, the video encoder 603 may be configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop may include a source coder 630 (responsible for creating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture, for example) and a (local) decoder 633 embedded in the video encoder 603. The decoder 633 reconstructs the symbols to create sample data in a similar manner as a (remote) decoder would, even if the embedded decoder 633 processes a video stream coded by the source coder 630 without entropy coding (since any compression between the symbols and the coded video bitstream in entropy coding may be lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory 634. Since the decoding of the symbol stream results in a bit-exact result regardless of the location of the decoder (local or remote), the content in the reference picture memory 634 is also bit-exact between the local and remote encoders. In other words, the prediction part of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder will "see" when using the prediction during decoding. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, e.g., due to channel error) is used to improve coding quality.

[0054] The operation of the "local" decoder 633 may be the same as that of a "remote" decoder, such as the video decoder 510 already described in detail above in connection with Figure 5. However, with brief reference also to Figure 5, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder 645 and parser 520 may be lossless, the entropy decoding portion of the video decoder 510, including the buffer memory 515, and the parser 520 may not be fully implemented in the local decoder 633 within the encoder.

[0055] At this point, it can be said that any decoder technology, except for parsing / entropy decoding, which may only exist in the decoder, may also necessarily need to exist in the corresponding encoder in substantially the same functional form. For this reason, the disclosed subject matter may focus on the decoder operation, which is similar to the decoding part of the encoder. Therefore, the description of the encoder technology may be omitted, since it is the reverse of the decoder technology described in general. Only in certain areas or aspects, a more detailed description of the encoder is provided below.

[0056] During operation in some example implementations, the source coder 630 may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as "reference pictures." In this manner, the coding engine 632 codes color channel differences (or residuals) between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as predictive references for the input picture. The terms "residual" and its adjective form "residual" may be used interchangeably.

[0057] The local video decoder 633 may decode the coded video data of the pictures that may be designated as reference pictures based on the symbols created by the source coder 630. The operation of the coding engine 632 may advantageously be a lossy process. When the coded video data may be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may typically be a copy of the source video sequence with some errors. The local video decoder 633 may replicate the decoding process that may be performed by the video decoder on the reference pictures and store the reconstructed reference pictures in the reference picture cache 634. In this way, the video encoder 603 may locally store copies of reconstructed reference pictures that have common content with the reconstructed reference pictures obtained by the far-end (remote) video decoder (without transmission errors).

[0058] The predictor 635 may perform a predictive search for the coding engine 632. That is, for a new picture to be coded, the predictor 635 may search the reference picture memory 634 for sample data (as candidate reference pixel blocks) or specific metadata, such as reference picture motion vectors, block shapes, etc., that may serve as suitable predictive references for the new picture. The predictor 635 may operate on a sample block-by-sample block basis to find suitable predictive references. In some cases, as determined by the search results obtained by the predictor 635, the input picture may have predictive references drawn from multiple reference pictures stored in the reference picture memory 634.

[0059] The controller 650 may manage the coding operations of the source coder 630, including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0060] The output of all the aforementioned functional units may undergo entropy coding in the entropy coder 645. The entropy coder 645 converts the symbols produced by the various functional units into a coded video sequence by losslessly compressing the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0061] The transmitter 640 may buffer the coded video sequence created by the entropy coder 645 in preparation for transmission over a communication channel 660, which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter 640 may merge the coded video data from the video coder 603 with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0062] The controller 650 may manage the operation of the video encoder 603. During coding, the controller 650 may assign a particular coded picture type to each coded picture, which may affect the coding technique that may be applied to the respective picture. For example, pictures may often be assigned as one of the following picture types:

[0063] An intra picture (I-picture) may be one that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures, as well as their respective uses and characteristics.

[0064] A predictive picture (P picture) may be one that can be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0065] Bidirectionally predicted pictures (B-pictures) may be coded and decoded using intra- or inter-prediction, which uses up to two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multi-predicted pictures may use more than two reference pictures and associated metadata for the reconstruction of a block.

[0066] A source picture may generally be spatially subdivided into multiple sample coding blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples, respectively) and coded block by block. A block may be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to the respective picture of the block. For example, a block of an I picture may be non-predictively coded or predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). A pixel block of a P picture may be predictively coded via spatial prediction or via temporal prediction with reference to one previously coded reference picture. A block of a B picture may be predictively coded via spatial prediction or via temporal prediction with reference to one or two previously coded reference pictures. A source picture or an intermediate processed picture may be subdivided into other types of blocks for other purposes. The division of coding blocks and other types of blocks may or may not follow the same scheme, as described in more detail below.

[0067] The video encoder 603 may perform coding operations in accordance with a given video coding technique or standard, such as ITU-T Rec. H.265. In its operations, the video encoder 603 may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0068] In some example embodiments, the transmitter 640 can transmit additional data along with the encoded video. The source coder 630 can include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0069] Video may be captured in a time sequence as multiple source pictures (video pictures). Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation in a given picture, while inter-picture prediction exploits temporal or other correlation between pictures. For example, a particular picture being coded / decoded, called the current picture, may be divided into blocks. If a block in the current picture is similar to a reference block in a previously coded and still buffered reference picture in the video, it may be coded by a vector, called a motion vector. A motion vector points to a reference block in a reference picture, and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0070] In some exemplary embodiments, bi-prediction techniques may be used for inter-picture prediction. According to such bi-prediction techniques, two reference pictures, such as a first reference picture and a second reference picture, are used, both of which advance the current picture in the video in decoding order (but may be past or future, respectively, in display order). A block in the current picture may be coded by a first motion vector pointing to a first reference block in the first reference picture and a second motion vector pointing to a second reference block in the second reference picture. A block may be jointly predicted by a combination of the first reference block and the second reference block.

[0071] Furthermore, merge mode techniques may be used to improve coding efficiency in inter-picture prediction.

[0072] According to some exemplary embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, a picture in a sequence of video pictures is divided into coding tree units (CTUs) for compression, and the CTUs in a picture can have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU can include three parallel coding tree blocks (CTBs), namely, one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree partitioned into one or more coding units (CUs). For example, a CTU of 64×64 pixels can be partitioned into one CU of 64×64 pixels, or four CUs of 32×32 pixels. Each of one or more of the 32×32 blocks can be further partitioned into four CUs of 16×16 pixels. In some exemplary embodiments, each CU can be analyzed during encoding to determine the prediction type of the CU from among various prediction types, such as an inter prediction type or an intra prediction type. A CU may be divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. In general, each PU includes a luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. The division of a CU into PUs (or PBs of different color channels) may be performed in various spatial patterns. A luma PB or a chroma PB may include a matrix of sample values ​​(e.g., luma values), such as, for example, 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 samples, etc.

[0073] 7 shows a diagram of a video encoder 703 according to another example embodiment of this disclosure. The video encoder 703 is configured to receive a processed block (e.g., a predictive block) of sample values ​​in a current video picture in a sequence of video pictures and to encode the processed block into a coded picture that is part of a coded video sequence. The example video encoder 703 may be used in place of the example video encoder (403) of FIG. 4.

[0074] For example, the video encoder 703 receives a matrix of sample values ​​of a processing block, such as a predictive block of 8×8 samples. The video encoder 703 then determines whether the processing block is best coded using intra mode, inter mode, or bi-predictive mode, for example, using rate-distortion optimization (RDO). If the processing block is determined to be coded in intra mode, the video encoder 703 may use intra prediction techniques to encode the processing block into a coded picture, and if the processing block is determined to be coded in inter mode or bi-predictive mode, the video encoder 703 may use inter prediction techniques or bi-predictive techniques, respectively, to encode the processing block into a coded picture. In some exemplary embodiments, a merge mode may be used as a sub-mode of inter picture prediction, in which a motion vector is derived from one or more motion vector predictors without benefit of coded motion vector components outside the predictors. In some other exemplary embodiments, there may be motion vector components applicable to the current block. Thus, the video encoder 703 may include components not explicitly shown in FIG. 7, such as a mode decision module, to determine the perduction mode of the processing block.

[0075] In the example of FIG. 7, the video encoder 703 includes an inter-encoder 730, an intra-encoder 722, a residual calculator 723, a switch 726, a residual encoder 724, an overall controller 721, and an entropy encoder 725 coupled to each other as shown in the exemplary configuration of FIG. 7.

[0076] The inter encoder 730 is configured to receive a sample of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., blocks in previous and subsequent pictures in display order), generate inter prediction information (e.g., a description of redundant information due to inter coding techniques, motion vectors, merge mode information), and calculate an inter prediction result (e.g., a prediction block) based on the inter prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture that has been decoded based on the encoded video information using a decoding unit 633 incorporated in the example encoder 620 of FIG. 6 (depicted as a residual decoder 728 of FIG. 7, as described in more detail below).

[0077] The intra encoder 722 is also configured to receive samples of a current block (e.g., a processing block), compare the block with blocks already coded in the same picture, generate quantized coefficients after transformation, and possibly generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). The intra encoder 722 can calculate an intra prediction result (e.g., a prediction block) based on the intra prediction information and a reference block in the same picture.

[0078] The overall controller 721 may be configured to determine overall control data and control other components of the video encoder 703 based on the overall control data. In one example, the overall controller 721 determines a prediction mode of the block and provides a control signal to the switch 726 based on the prediction mode. For example, if the prediction mode is an intra mode, the overall controller 721 controls the switch 726 to select an intra mode result to be used by the residual calculator 723, and controls the entropy encoder 725 to select intra prediction information and include the intra prediction information in the bitstream. Also, if the prediction mode of the block is an inter mode, the overall controller 721 controls the switch 726 to select an inter prediction result to be used by the residual calculator 723, and controls the entropy encoder 725 to select inter prediction information and include the inter prediction information in the bitstream.

[0079] The residual calculator 723 may be configured to calculate a difference (residual data) between the received block and a prediction result of the block selected from the intra encoder 722 or the inter encoder 730. The residual encoder 724 may be configured to encode the residual data to generate transform coefficients. For example, the residual encoder 724 may be configured to transform the residual data from the spatial domain to the frequency domain to generate transform coefficients. The transform coefficients then undergo a quantization process to obtain quantized transform coefficients. In various exemplary embodiments, the video encoder 703 also includes a residual decoder 728. The residual decoder 728 is configured to perform an inverse transform and generate decoded residual data. The decoded residual data may be used appropriately by the intra encoder 722 and the inter encoder 730. For example, the inter encoder 730 may generate a decoded block based on the decoded residual data and the inter prediction information, and the intra encoder 722 may generate a decoded block based on the decoded residual data and the intra prediction information. The decoded blocks are appropriately processed to generate a decoded picture, which may be buffered in a memory circuit (not shown) and used as a reference picture.

[0080] The entropy encoder 725 may be configured to format a bitstream to include the encoded block and perform entropy coding. The entropy encoder 725 is configured to include various information in the bitstream. For example, the entropy encoder 725 may be configured to include overall control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other suitable information in the bitstream. When coding a block in a merged sub-mode of either an inter mode or a bi-prediction mode, the residual information may not be present.

[0081] 8 shows a diagram of an example video decoder 810 according to another embodiment of the present disclosure. The video decoder 810 is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder 810 may be used in place of the video decoder 410 of the example of FIG. 4.

[0082] In the example of FIG. 8, the video decoder 810 includes an entropy decoder 871, an inter decoder 880, a residual decoder 873, a reconstruction module 874, and an intra decoder 872 coupled together as shown in the exemplary configuration of FIG.

[0083] The entropy decoder 871 may be configured to reconstruct from the coded picture certain symbols representing syntax elements that make up the coded picture. Such symbols may include, for example, prediction information (e.g., intra- or inter-prediction information) that may identify the mode in which the block is coded (e.g., intra- or bi-prediction mode, merged submode or another submode), certain samples or metadata used for prediction by the intra decoder 872 or the inter decoder 880, residual information, for example in the form of quantized transform coefficients, etc. In one example, when the prediction mode is an inter- or bi-prediction mode, the inter-prediction information is provided to the inter decoder 880, and when the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra decoder 872. The residual information may be dequantized and provided to the residual decoder 873.

[0084] The inter decoder 880 may be configured to receive the inter prediction information and generate inter prediction results based on the inter prediction information.

[0085] The intra decoder 872 may be configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0086] The residual decoder 873 may be configured to perform inverse quantization to extract inverse quantized transform coefficients and process the inverse quantized transform coefficients to transform the residual from the frequency domain to the spatial domain. The residual decoder 873 may also utilize certain control information (to include quantizer parameters (QP)) that may be provided by the entropy decoder 871 (a data path is not depicted since this is likely to be low data volume control information).

[0087] The reconstruction module 874 may be configured to combine, in the spatial domain, the residual output by the residual decoder 873 and a prediction result (possibly output by an inter-prediction module or an intra-prediction module) to form a reconstructed block that forms part of the reconstructed picture as part of the reconstructed video. It should be noted that other suitable operations, such as a deblocking operation, may also be performed to improve visual quality.

[0088] It should be noted that the video encoders 403, 603, and 703 and the video decoders 410, 510, and 810 may be implemented using any suitable technology. In some exemplary embodiments, the video encoders 403, 603, and 703 and the video decoders 410, 510, and 810 may be implemented using one or more integrated circuits. In another embodiment, the video encoders 403, 603, and 603 and the video decoders 410, 510, and 810 may be implemented using one or more processors executing software instructions.

[0089] Focusing on the block partitioning used for encoding and decoding, a typical partitioning may start from a base block and follow a predefined set of rules, a specific pattern, a partitioning tree, or any partitioning structure or scheme. The partitioning may be hierarchical and recursive. After isolating or dividing the base block according to any of the exemplary partitioning procedures described below or other procedures, or combinations thereof, a final set of partitions or coding blocks may be obtained. Each of these partitions may be at one of various partitioning levels in the partitioning hierarchy and may be partitions of various shapes. Each of the partitions may be referred to as a coding block (CB). In various exemplary partitioning implementations described further below, each resulting CB may be a CB of any of the allowed sizes and partitioning levels. Such partitions are referred to as coding blocks because they may form a unit for which some basic coding / decoding decisions may be made and coding / decoding parameters may be optimized, determined, and signaled in the encoded video bitstream. The highest or deepest level in the final partition represents the depth of the coding block partitioning structure of the tree. The coding block may be a luma coding block or a chroma coding block. The CB tree structure for each color is sometimes called the coding block tree (CBT).

[0090] The coding blocks of all color channels may be collectively referred to as a coding unit (CU). The hierarchical structure of all color channels may be collectively referred to as a coding tree unit (CTU). The partitioning pattern or structure of various color channels within a CTU may or may not be the same.

[0091] In some implementations, the split tree scheme or structure used for the luma channel and the chroma channel may not need to be the same. In other words, the luma channel and the chroma channel may have separate coding tree structures or patterns. Furthermore, whether the luma channel and the chroma channel use the same coding partition tree structure or different coding partition tree structures, and the actual coding partition tree structure to be used may depend on whether the slice being coded is a P slice, a B slice, or an I slice. For example, for an I slice, the chroma channel and the luma channel may have separate coding partition tree structures or coding partition tree structure modes, while for a P slice or a B slice, the luma channel and the chroma channel may share the same coding partition tree scheme. When separate coding partition tree structures or modes are applied, the luma channel may be split into CBs by one coding partition tree structure, and the chroma channel may be split into chroma CBs by another coding partition tree structure.

[0092] In some exemplary implementations, a predefined partitioning pattern may be applied to the base block. As shown in FIG. 9, an exemplary four-way partitioning tree may start at a first predefined level (e.g., 64×64 block level or other size as the base block size), and the base block may be partitioned hierarchically down to a predefined lowest level (e.g., 4×4 level). For example, the base block may follow four predefined partitioning options or patterns shown by 902, 904, 906, and 908, and the partition designated as R allows for recursive partitioning in that the same partitioning options shown in FIG. 9 may be repeated at lower scales down to the lowest level (e.g., 4×4 level). In some implementations, additional restrictions may be added to the partitioning scheme of FIG. 9. In the implementation of FIG. 9, rectangular partitions (e.g., 1:2 / 2:1 rectangular partitions) may be allowed but not recursive, and square partitions are allowed to be recursive. Subsequent partitioning of FIG. 9 by recursion, if necessary, generates a final set of coding blocks. A coding tree depth may be further defined to indicate the division depth from a root node or root block. For example, the coding tree depth for a root node or root block, e.g., a 64x64 block, may be set to 0, and the coding tree depth increases by 1 after the root block is further divided one time according to FIG. 9. The maximum or deepest level from the 64x64 base block to the 4x4 minimum partition is 4 (starting from level 0) in the above scheme. Such a division scheme may be applied to one or more of the color channels. Each color channel may be independently divided according to the scheme of FIG. 9 (e.g., for each of the color channels at each hierarchical level, a division pattern or option in the predefined pattern may be independently determined). Alternatively, two or more of the color channels may share the same hierarchical pattern tree of FIG. 9 (e.g., for two or more of the color channels at each hierarchical level, the same division pattern or option in the predefined pattern may be selected).

[0093] FIG. 10 illustrates another exemplary predefined partitioning pattern that allows recursive partitioning to form a partitioning tree. As illustrated in FIG. 10, an exemplary 10-way partitioning structure or pattern may be predefined. A root block may start from a predefined level (e.g., from a base block at a 128×128 level or a 64×64 level). The exemplary partitioning structure of FIG. 10 includes various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. A partition type with three subpartitions, illustrated at 1002, 1004, 1006, and 1008 in the second column of FIG. 10, may be referred to as a “T-type” partition. The “T-type” partitions 1002, 1004, 1006, and 1008 may be referred to as a left T-type, an upper T-type, a right T-type, and a lower T-type. In some exemplary implementations, none of the rectangular partitions in FIG. 10 may be further subdivided. A coding tree depth may be further defined to indicate a partitioning depth from a root node or root block. For example, the coding tree depth for a root node or root block, e.g., a 128x128 block, may be set to 0, and the coding tree depth increases by 1 after the root block is further split according to FIG. 10 one more time. In some implementations, only the square partitions of 1010 may allow recursive splitting to the next level of the split tree following the pattern of FIG. 10. In other words, recursive splitting may not be allowed for the square partitions in the T-shaped patterns 1002, 1004, 1006, and 1008. If necessary, the splitting procedure following FIG. 10 by recursion generates a final set of coding blocks. Such a scheme may be applied to one or more of the color channels. In some implementations, more flexibility may be added to the use of partitions less than 8x8 levels. For example, in some cases, 2x2 chroma inter prediction may be used.

[0094] In some other example implementations of coding block partitioning, a quadtree structure may be used to partition a base block or an intermediate block into quadtree partitions. Such quadtree partitioning may be applied hierarchically and recursively to any square partition. Whether a base block or an intermediate block or partition is further quadtree partitioned may be adapted to various local characteristics of the base block or intermediate block / partition. The quadtree partitioning at the picture boundary may be further adapted. For example, an implicit quadtree partitioning may be performed at the picture boundary such that a block continues to be quadtree partitioned until its size fits the picture boundary.

[0095] In some other example implementations, a hierarchical bisection from the base block may be used. For such a scheme, the base block or mid-level block may be divided into two partitions. The bisection may be either horizontal or vertical. For example, a horizontal bisection may divide the base block or mid-block into equal left and right partitions. Similarly, a vertical bisection may divide the base block or mid-block into equal top and bottom partitions. Such bisection may be hierarchical and recursive. It may be determined at each of the base block or mid-block whether the bisection scheme should continue and, if the scheme continues further, whether a horizontal or vertical bisection should be used. In some implementations, further division may stop at a predefined minimum partition size (in one or both dimensions). Alternatively, further division may stop when a predefined division level or depth from the base block is reached. In some implementations, the aspect ratio of the partitions may be limited. For example, the aspect ratio of the partitions may not be smaller than 1:4 (or larger than 4:1). Therefore, a vertical strip partition, having a vertical to horizontal aspect ratio of 4:1, can only be further divided vertically into an upper and lower partition, each having a vertical to horizontal aspect ratio of 2:1.

[0096] In yet some other examples, a ternary division scheme may be used to divide the base block or any intermediate blocks, as shown in FIG. 13. The ternary pattern may be implemented vertically, as shown at 1302 in FIG. 13, or horizontally, as shown at 1304 in FIG. 13. The exemplary division ratio in FIG. 13 is shown as 1:2:1, either vertically or horizontally, but other ratios may be predefined. In some implementations, two or more different ratios may be predefined. Such ternary division schemes may be used to complement quadtree or binary tree partitioning structures, in that such ternary tree divisions can capture objects located at block centers in one contiguous partition, whereas quadtrees and binary trees always divide along block centers, thus dividing objects into separate partitions. In some implementations, the width and height of the partitions of the exemplary ternary tree are always powers of two to avoid further transformations.

[0097] The above partitioning schemes may be combined in any manner at different partitioning levels. As an example, the above-mentioned quadtree and bipartitioning schemes may be combined to partition the base block into a quadtree-bipartition (QTBT) structure. In such a scheme, the base block or intermediate blocks / partitions may be either quadtree-partitioned or bipartitioned, if specified, according to a set of predefined conditions. A specific example is shown in FIG. 14. In the example of FIG. 14, the base block is first quadtree-partitioned into four partitions, as shown by 1402, 1404, 1406, and 1408. Each of the resulting partitions is then either quadtree-partitioned into four further partitions (such as 1408) or bipartitioned into two further partitions at the next level (e.g., either horizontally or vertically, such as 1402 or 1406, both of which are symmetric), or not partitioned (such as 1404). Bisection or quadtree partitioning may be recursively enabled for square partitions, as shown by the overall exemplary partition pattern in 1410 and the corresponding tree structure / representation in 1420, where solid lines represent quadtree partitioning and dashed lines represent bisection. A flag may be used for each bisection node (non-leaf binary partition) to indicate whether the bisection is horizontal or vertical. For example, flag "0" may represent horizontal bisection and flag "1" may represent vertical bisection, as shown in 1420, which matches the partition structure in 1410. In the case of quadtree partitioning, there is no need to indicate the partition type, since quadtree partitioning always splits a block or partition both horizontally and vertically to generate four sub-blocks / partitions of equal size. In some implementations, flag "1" may represent horizontal bisection and flag "0" may represent vertical bisection.

[0098] In some example implementations of QTBT, the quadtree and bisection rule sets may be represented by the following predefined parameters and their associated corresponding functions: -CTU size: quadtree root node size (base block size) -MinQTSize: The minimum allowable quadtree leaf node size. -MaxBTSize: Maximum allowable binary tree root node size -MaxBTDepth: Maximum allowed binary tree depth -MinBTSize: The minimum allowable binary tree leaf node size. In some exemplary implementations of the QTBT partitioning structure, the CTU size may be set as 128×128 luma samples with two corresponding 64×64 blocks of chroma samples (if exemplary chroma subsampling is considered and used), MinQTSize may be set as 16×16, MaxBTSize may be set as 64×64, MinBTSize (for both width and height) may be set as 4×4, and MaxBTDepth may be set as 4. Quad-tree partitioning may be applied to the CTU first to generate quad-tree leaf nodes. The quad-tree leaf nodes may have a size from its minimum allowed size of 16×16 (i.e., MinQTSize) to 128×128 (i.e., the CTU size). If a node is 128×128, it will not be split first by the binary tree because the size exceeds MaxBTSize (i.e., 64×64). Otherwise, nodes that do not exceed MaxBTSize may be split by the binary tree. In the example of FIG. 14, the base block is 128×128. The base block can only be quadtree split according to a predefined set of rules. The base block has a split depth of 0. Each of the resulting four partitions is 64×64, not exceeding MaxBTSize, and may be further quadtree or binary tree split at level 1. The process continues. When the binary tree depth reaches MaxBTDepth (i.e., 4), no further splits may be considered. When the width of a binary tree node is equal to MinBTSize (i.e., 4), no further horizontal splits may be considered. Similarly, when the height of a binary tree node is equal to MinBTSize, no further vertical splits may be considered.

[0099] In some example implementations, the above QTBT scheme may be configured to support flexibility for luma and chroma to have the same QTBT structure or separate QTBT structures. For example, for P slices and B slices, the luma CTB and chroma CTB in one CTU may share the same QTBT structure. However, for I slices, the luma CTB may be divided into CBs by a QTBT structure, and the chroma CTB may be divided into chroma CBs by another QTBT structure. This means that CUs may be used to refer to different color channels in an I slice, for example, an I slice may consist of a coding block of a luma component or a coding block of two chroma components, and a CU in a P slice or B slice may consist of coding blocks of all three color components.

[0100] In some other implementations, the QTBT scheme may be complemented with the ternary scheme described above. Such implementations may be referred to as multi-type tree (MTT) structures. For example, in addition to the bisection of the node, one of the trisection patterns of FIG. 13 may be selected. In some implementations, only square nodes may undergo trisection. Yet another flag may be used to indicate whether the trisection is horizontal or vertical.

[0101] The design of two-level or multi-level trees, such as the QTBT implementation and the QTBT complemented by trisection implementation, may be motivated primarily by reducing complexity. In theory, the complexity of traversing a tree is T D where T denotes the number of split types and D is the depth of the tree. Trade-offs may be made by using multitypes (T) while reducing the depth (D).

[0102] In some implementations, the CB may be further divided. For example, the CB may be further divided into multiple prediction blocks (PBs) for the purpose of intra-frame or inter-frame prediction during the encoding and decoding process. In other words, the CB may be further divided into different sub-partitions, where individual prediction decisions / configurations may be made. In parallel, the CB may be further divided into multiple transform blocks (TBs) for the purpose of describing the level at which the transformation or inverse transformation of the video data is performed. The division scheme of the CB into PBs and TBs may be the same or different. For example, each division scheme may be performed using its own procedure, for example, based on various characteristics of the video data. The division scheme of the PBs and TBs may be independent in some exemplary implementations. The division schemes and boundaries of the PBs and TBs may be correlated in some other exemplary implementations. In some implementations, for example, the TBs may be divided after the PB division, and in particular, each PB may be further divided into one or more TBs after being determined following the division of the coding block. For example, in some implementations, the PB may be divided into one, two, four, or other number of TBs.

[0103] In some implementations, the luma and chroma channels may be processed differently to split the base blocks into coding blocks and further into predictive and / or transform blocks. For example, in some implementations, splitting of coding blocks into predictive and / or transform blocks may be allowed for the luma channel, but such splitting of coding blocks into predictive and / or transform blocks may not be allowed for the chroma channels. Thus, in such implementations, the transform and / or prediction of luma blocks may be performed only at the coding block level. In another example, the minimum transform block size of the luma and chroma channels may be different, e.g., the coding blocks of the luma channel may be allowed to be split into smaller transform and / or predictive blocks than the chroma channels. In yet another example, the maximum depth of the splitting of coding blocks into transform and / or predictive blocks may be different between the luma and chroma channels, e.g., the coding blocks of the luma channel may be allowed to be split into deeper transform and / or predictive blocks than the chroma channels. As a specific example, a luma coding block may be divided into transform blocks of multiple sizes that can be represented by a recursive division down by up to two levels, and transform block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4, as well as transform block sizes from 4×4 to 64×64 may be allowed. However, for chroma blocks, only the largest possible transform block designated for the luma block may be allowed.

[0104] In some example implementations for partitioning a coding block into PBs, the depth, shape, and / or other characteristics of the PB partition may depend on whether the PB is intra-coded or inter-coded.

[0105] The division of the coding block (or prediction block) into transform blocks may be performed in various exemplary manners, including but not limited to quadtree division and predefined pattern division, recursively or non-recursively, further considering transform blocks at the boundaries of the coding block or prediction block. In general, the resulting transform blocks may be at different division levels, may not be the same size, and may not need to be square in shape (e.g., they may be rectangular with some allowed size and aspect ratio). Further examples are described in more detail below with respect to Figures 15, 16, and 17.

[0106] However, in some other implementations, the CB obtained via any of the above partitioning schemes may be used as a basic or smallest coding block for prediction and / or transformation. In other words, no further partitioning is performed for the purpose of performing inter / intra prediction and / or transformation. For example, the CB obtained from the above QTBT scheme may be used as it is as a unit for performing prediction. Specifically, such a QTBT structure removes the concept of multiple partition types, i.e., removes the separation of CU, PU, ​​and TU, and supports more flexibility for CU / CB partition shapes as described above. In such a QTBT block structure, the CU / CB can have either a square or a rectangle. The leaf nodes of such a QTBT are used as units for prediction and transformation processing without further partitioning. This means that the CU, PU, ​​and TU have the same block size in such an exemplary QTBT coding block structure.

[0107] The various CB division schemes described above, as well as further division of the CB into PB and / or TB (including no PB / TB division), may be combined in any manner. The following specific implementations are provided as non-limiting examples.

[0108] Specific exemplary implementations of the division of coding blocks and transform blocks are described below. In such exemplary implementations, the base block may be divided into coding blocks using a recursive quadtree division or a predefined division pattern described above (such as the division patterns of Figures 9 and 10). At each level, whether to continue further quadtree division of a particular partition may be determined by local video data characteristics. The resulting CBs may be at various quadtree division levels and of various sizes. The decision of whether to code a picture area using inter-picture (temporal) prediction or intra-picture (spatial) prediction may be made at the CB level (or at the CU level for all three color channels). Each CB may be further divided into one, two, four, or other number of PBs according to a predefined PB division type. Within one PB, the same prediction process may be applied, and related information may be transmitted to the decoder on a PB basis. After obtaining the residual block by applying the prediction process based on the PB partition type, the CB may be divided into TBs according to another quadtree structure similar to the coding tree for CBs. In this particular implementation, the CB or TB need not be limited to a square. Moreover, in this particular example, the PB may be a square or rectangular shape for inter prediction, and may only be a square for intra prediction. A coding block may be divided into, for example, four square TBs. Each TB may be further recursively divided (using quadtree division) into smaller TBs, which are referred to as residual quadtrees (RQTs).

[0109] Another exemplary implementation for splitting a base block into CB, PB, and / or TB is further described below. For example, rather than using multiple partition unit types such as those shown in FIG. 9 or FIG. 10, a quadtree with nested multi-type trees using bipartition and tripartition segmentation structures (e.g., QTBT or QTBT with tripartition as described above) may be used. Separation of CB, PB, and TB (i.e., splitting CB into PB and / or TB, and splitting PB into TB) may be abandoned except when necessary for CBs with sizes too large for the maximum transform length, when such CBs require further splitting. This exemplary splitting scheme may be designed to support further flexibility on CB partition shapes, such that both prediction and transformation may be performed at the CB level without further splitting. In such coding tree structures, CBs may have either square or rectangular shapes. Specifically, a coding tree block (CTB) may first be split by a quadtree structure. The quadtree leaf nodes may then be further split by a nested multi-type tree structure. An example of a nested multi-type tree structure using bisection or trisection is shown in Figure 11. Specifically, the example multi-type tree structure in Figure 11 includes four split types called vertical bisection (SPLIT_BT_VER) 1102, horizontal bisection (SPLIT_BT_HOR) 1104, vertical trisection (SPLIT_TT_VER) 1106, and horizontal trisection (SPLIT_TT_HOR) 1108. Then, CB corresponds to a leaf of the multi-type tree. In this example implementation, as long as CB is not too large for the maximum transform length, this segmentation is used for both prediction and transform processes without further splitting. This means that in most cases, CB, PB, and TB have the same block size in a quadtree with a nested multi-type tree coding block structure. An exception occurs when the maximum supported transform length is smaller than the width or height of the color components of CB.In some implementations, in addition to bisection or trisection, the nested pattern of FIG. 11 may further include a quadtree division.

[0110] One particular example of quadtrees (including quadtree options, bisection options and trisection options) with nested multi-type tree coding block structures of block division for one base block is shown in FIG. 12. More specifically, FIG. 12 shows that a base block 1200 is quadtree divided into four square partitions 1202, 1204, 1206 and 1208. A decision is made for each of the quadtree divided partitions to further use the multi-type tree structure and quadtrees of FIG. 11 for further division. In the example of FIG. 12, partition 1204 is not further divided. Partition 1202 and partition 1208 each adopt another quadtree division. For partition 1202, the second level quadtree partitioned top-left, top-right, bottom-left, and bottom-right partitions adopt third level partitioning of quadtree, horizontal bisection 1104 of FIG. 11, unsplit, and horizontal trisection 1108 of FIG. 11, respectively. Partition 1208 adopts another quadtree partitioning, and the second level quadtree partitioned top-left, top-right, bottom-left, and bottom-right partitions adopt third level partitioning of vertical trisection 1106 of FIG. 11, unsplit, unsplit, and horizontal bisection 1104 of FIG. 11, respectively. Two of the subpartitions of the third level top-left partition of 1208 are further partitioned according to horizontal bisection 1104 and horizontal trisection 1108 of FIG. 11, respectively. Partition 1206 employs a second level splitting pattern into two partitions following vertical bisection 1102 of Figure 11, which are further split at a third level according to horizontal bisection 1108 and vertical bisection 1102 of Figure 11. A fourth level split is further applied to one of these according to horizontal bisection 1104 of Figure 11.

[0111] In the above specific example, the maximum luma transform size may be 64×64, and the maximum supported chroma transform size may be different from the luma, e.g., 32×32. Even if the above example CB of FIG. 12 is not generally further divided into smaller PBs and / or TBs, when the width or height of a luma coding block or a chroma coding block is larger than the maximum transform width or maximum transform height, the luma coding block or the chroma coding block may be automatically divided in the horizontal and / or vertical directions to satisfy the transform size restrictions in that direction.

[0112] In the specific example of splitting the above base blocks into CBs, as described above, the coding tree scheme can support the ability of luma and chroma to have separate block tree structures. For example, for P slices and B slices, the luma CTB and chroma CTB in one CTU can share the same coding tree structure. For I slices, for example, luma and chroma can have separate coding block tree structures. When separate block tree structures are applied, the luma CTB may be split into luma CBs by one coding tree structure, and the chroma CTB is split into chroma CBs by another coding tree structure. This means that a CU in an I slice may be composed of a coding block of a luma component or a coding block of two chroma components, and a CU in a P slice or B slice is always composed of coding blocks of all three color components, unless the video is monochrome.

[0113] When a coding block is further divided into multiple transform blocks, the transform blocks therein may be ordered in the bitstream according to various orders or scanning schemes. Exemplary implementations for dividing a coding block or a predictive block into transform blocks and the coding order of the transform blocks are described in further detail below. In some exemplary implementations, as described above, the transform division can support transform blocks of multiple shapes, e.g., 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, with transform block sizes ranging from 4×4 to 64×64. In some implementations, when the coding block is smaller than or equal to 64×64, the transform block division may be applied only to the luma component, such that for chroma blocks, the transform block size is identical to the coding block size. Otherwise, if the width or height of the coding block is greater than 64, both the luma coding block and the chroma coding block may be implicitly partitioned into a multiple of min(W,64)×min(H,64) transform blocks and min(W,32)×min(H,32) transform blocks, respectively.

[0114] In some example implementations of transform block partitioning, for both intra-coded and inter-coded blocks, the coding block may be further partitioned into multiple transform blocks with a partition depth up to a predefined number of levels (e.g., two levels). The transform block partition depth and size may be related. For some example implementations, the mapping from the transform size of the current depth to the transform size of the next depth is shown in Table 1 below. [Table 1]

[0115] Based on the example mapping in Table 1, for a 1:1 square block, the next level transform partition can create four 1:1 square sub-transform blocks. The transform partition may stop at, for example, 4×4. Thus, the transform size of the current depth of 4×4 corresponds to the same size of 4×4 of the next depth. In the example of Table 1, for a 1:2 / 2:1 non-square block, the next level transform partition can create two 1:1 square sub-transform blocks, while for a 1:4 / 4:1 non-square block, the next level transform partition can create two 1:2 / 2:1 sub-transform blocks.

[0116] In some example implementations, further restrictions may be applied on transform block partitioning for the luma components of intra-coded blocks. For example, for each level of transform partitioning, all sub-transform blocks may be restricted to be of equal size. For example, for a 32×16 coding block, the level 1 transform partitioning creates two 16×16 sub-transform blocks, and the level 2 transform partitioning creates eight 8×8 sub-transform blocks. In other words, to keep the transform units of equal size, the second level partitioning must be applied to all first level sub-blocks. An example of transform block partitioning of an intra-coded square block according to Table 1 is shown in FIG. 15 with the coding order indicated by the arrows. Specifically, 1502 shows a square coding block. The first level partitioning according to Table 1 into four equal-sized transform blocks is shown in 1504 with the coding order indicated by the arrows. The second level partitioning of all first level equal-sized blocks according to Table 1 into 16 equal-sized transform blocks is shown in 1506 with the coding order indicated by the arrows.

[0117] In some example implementations, the above restrictions on intra-coding may not apply to the luma components of an inter-coded block. For example, after the first level of transform partitioning, any one of the sub-transform blocks may be further partitioned independently at another level. Thus, the resulting transform blocks may or may not be blocks of the same size. An example partitioning of an inter-coded block into transform blocks according to their coding order is shown in FIG. 16. In the example of FIG. 16, an inter-coded block 1602 is partitioned into transform blocks at two levels according to Table 1. At the first level, the inter-coded block is partitioned into four transform blocks of equal size. Then, only one of the four transform blocks (but not all of them) is further partitioned into four sub-transform blocks, resulting in a total of seven transform blocks with two different sizes, as shown by 1604. An example coding order of these seven transform blocks is shown by arrows at 1604 in FIG. 16.

[0118] In some example implementations, for chroma components, some additional restrictions on transform blocks may be applied: for example, for chroma components, the transform block size may be as large as the coding block size, but cannot be smaller than a predefined size, e.g., 8×8.

[0119] In some other example implementations, for coding blocks with either width (W) or height (H) greater than 64, both luma coding blocks and chroma coding blocks may be implicitly divided into multiples of min(W,64)×min(H,64) and min(W,32)×min(H,32) transform units, respectively, where in this disclosure, “min(a,b)” can return the smaller value between a and b.

[0120] FIG. 17 further illustrates another alternative exemplary scheme for partitioning a coding block or a predictive block into transform blocks. As illustrated in FIG. 17, instead of using a recursive transform partition, a set of predetermined partition types may be applied to a coding block according to the transform type of the coding block. In the particular example illustrated in FIG. 17, one of six exemplary partition types may be applied to partition the coding block into various numbers of transform blocks. Such a scheme for generating transform block partitions may be applied to either a coding block or a predictive block.

[0121] More specifically, the partitioning scheme of FIG. 17 provides up to six exemplary partition types for any given transform type (transform type refers to the type of first-order transform, such as ADST, for example). In this scheme, every coding block or predictive block may be assigned a transform partition type, for example, based on a rate-distortion cost. In one example, the transform partition type assigned to a coding block or predictive block may be determined based on the transform type of the coding block or predictive block. As shown by the six transform partition types illustrated in FIG. 17, a particular transform partition type may correspond to a transform block partition size and pattern. The correspondence between various transform types and various transform partition types may be predefined. An example is shown below, where the capitalized labels indicate transform partition types that may be assigned to a coding block or predictive block based on a rate-distortion cost. ·PARTITION_NONE: Allocate the transformation size equal to the block size. ·PARTITION_SPLIT: Allocates a height of 1 / 2 the block size and a height transformation size of 1 / 2 the block size. ·PARTITION_HORZ: Allocates a transformation size with width equal to the block size and height equal to 1 / 2 the block size. ·PARTITION_VERT: Allocates a transformation size with a width half the block size and a height equal to the block size. ·PARTITION_HORZ4: Allocates a transformation size with width equal to the block size and height 1 / 4 of the block size. ·PARTITION_VERT4: Allocates a transformation size with a width of 1 / 4 of the block size and a height equal to the block size.

[0122] In the above example, all transform partition types as shown in Figure 17 include uniform transform sizes for the partitioned transform blocks. This is merely an example and not a limitation. In some other implementations, mixed transform block sizes may be used for the partitioned transform blocks in a particular partition type (or pattern).

[0123] A video block (PB or CB, also referred to as PB when not further divided into multiple predictive blocks) may be predicted in various ways rather than being directly encoded, thereby exploiting various correlations and redundancies in the video data to improve compression efficiency. Correspondingly, such prediction may be performed in various modes. For example, a video block may be predicted via intra prediction or inter prediction. In particular, in an inter prediction mode, a video block may be predicted by one or more other reference blocks or inter predictor blocks from one or more other frames via either single reference inter prediction or mixed reference inter prediction. To perform inter prediction, a reference block may be specified by its frame identifier (the temporal location of the reference block) and a motion vector indicating a spatial offset between a current block being coded or decoded and the reference block (the spatial location of the reference block). The reference frame identification and the motion vector may be signaled in the bitstream. The motion vector as a spatial block offset may be directly signaled or may itself be predicted by another reference motion vector or a predictor motion vector. For example, the current motion vector may be predicted directly by a reference motion vector (e.g., of a candidate neighboring block) or by a combination of the reference motion vector and a motion vector differential (MVD) between the current motion vector and the reference motion vector. The latter is sometimes called merge mode with motion vector differential (MMVD). The reference motion vector may be identified in the bitstream, for example, as a pointer to a spatially neighboring block of the current block or a temporally neighboring but spatially co-located block.

[0124] Returning to the intra prediction process, samples in a block (e.g., luma or chroma prediction block, or coding block if not further divided into prediction blocks) are predicted by samples of neighboring lines, then neighboring lines, or one or more other lines, or combinations thereof, to generate a prediction block. The residual between the actual block being coded and the prediction block may then be processed through a transform followed by quantization. Various intra prediction modes may be made available, and parameters related to intra mode selection and other parameters may be signaled in the bitstream. Various intra prediction modes may relate, for example, to one or more line positions for predicting samples, the direction in which the prediction samples are selected from predicting one or more lines, and other special intra prediction modes.

[0125] For example, the set of intra-prediction modes (interchangeably referred to as "intra modes") may include a predefined number of directional intra-prediction modes. As described above with respect to the example implementation of FIG. 1, these intra-prediction modes may correspond to a predefined number of directions to follow when selecting a sample outside a block as a destination for a sample being predicted within a particular block. In another particular example implementation, eight main directional modes may be supported and predefined, corresponding to angles from 45 degrees to 207 degrees relative to the horizontal axis.

[0126] In some other implementations of intra prediction, the directional intra modes may be further extended to an angle set with finer granularity to further exploit the more diverse spatial redundancy in the directional texture. For example, the above eight angle implementation may be configured to provide eight nominal angles called V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED as shown in FIG. 19, and a predefined number (e.g., seven) of finer angles may be added to each nominal angle. Such an extension results in a larger total number of directional angles (e.g., 56 in this example), which may be available for intra prediction corresponding to the same number of predefined directional intra modes. The prediction angle may be represented by the nominal intra angle + angle delta. In the above specific example with seven finer angle directions for each nominal angle, the angle delta may be -3 to 3 times the step size of 3 degrees. Several angle schemes may be used, as shown in FIG. 18 with 65 different prediction angles.

[0127] In some implementations, instead of or in addition to the above directional intra modes, a predefined number of non-directional intra prediction modes may also be predefined and made available. For example, five non-directional intra modes called smooth intra prediction modes may be specified. These non-directional intra mode prediction modes may be specifically called DC, PAETH, SMOOTH, SMOOTH_V, and SMOOTH_H intra modes. Prediction of samples of a particular block under these exemplary non-directional modes is illustrated in FIG. 20. As an example, FIG. 20 shows that a 4×4 block 2002 is predicted by samples from an upper neighboring line and / or a left neighboring line. A particular sample 2010 in the block 2002 may correspond to a sample 2004 immediately above the sample 2010 of the upper neighboring line of the block 2002, a sample 2006 above and to the left of the sample 2010 as the intersection of the upper neighboring line and the left neighboring line, and a sample 2008 immediately to the left of the sample 2010 of the left neighboring line of the block 2002. In the exemplary DC intra prediction mode, the average of the left and top neighboring samples 2008 and 2004 may be used as a predictor for sample 2010. For the exemplary PAETH intra prediction mode, the top, left, and top left reference samples 2004, 2008, and 2006 may be fetched, and then the closest of these three reference samples to (top+left-top left) may be set as the predictor for sample 2010. For the exemplary SMOOTH_V intra prediction mode, sample 2010 may be predicted by quadratic interpolation in the vertical direction of the top left neighboring sample 2006 and the left neighboring sample 2008. For the exemplary SMOOTH_H intra prediction mode, sample 2010 may be predicted by quadratic interpolation in the horizontal direction of the top left neighboring sample 2006 and the top neighboring sample 2004. For the exemplary SMOOTH intra prediction mode, sample 2010 may be predicted by an average of quadratic interpolation in the vertical and horizontal directions. The above omni-directional intra-mode implementations are provided only as non-limiting examples.Other neighboring lines, and other non-directional selections of samples, and methods of combining predicted samples to predict a particular sample within a prediction block are also contemplated.

[0128] The selection of a particular intra-prediction mode by the encoder from the above directional or non-directional modes at various coding levels (picture, slice, block, unit, etc.) may be signaled in the bitstream. In some exemplary implementations, first, the eight exemplary nominal directional modes may be signaled along with the five non-angle smooth modes (a total of 13 options). Then, if the signaled mode is one of the eight nominal angle intra-modes, an index indicating the selected angle delta relative to the corresponding signaled nominal angle is further signaled. In some other exemplary implementations, all intra-prediction modes may be indexed together for signaling (e.g., 56 directional modes plus 5 non-directional modes to generate 61 intra-prediction modes).

[0129] In some example implementations, the example 56 or other number of directional intra-prediction modes may be implemented with a unified directional predictor that projects each sample of a block to a reference subsample position and interpolates the reference samples by a 2-tap bilinear filter.

[0130] In some implementations, additional filter modes, called FILTER INTRA modes, may be designed to capture the decaying spatial correlation with the references on the edges. In these modes, in addition to samples outside the block, samples predicted within the block may be used as intra prediction reference samples for some patches within the block. These modes may, for example, be predefined and made available for intra prediction of at least the luma block (or only the luma block). A predefined number (e.g., 5) of filter intra modes may be predesigned, each of which is represented by a set of n-tap filters (e.g., 7-tap filters) that reflect the correlation between a sample in a 4×2 patch and its adjacent n neighbors. In other words, the weighting coefficients of the n-tap filters may depend on the position. Taking an 8×8 block, a 4×2 patch, and 7-tap filtering as examples, the 8×8 block 2002 may be divided into eight 4×2 patches, as shown in FIG. 21. These patches are denoted by B0, B1, B1, B3, B4, B5, B6, and B7 in FIG. 21. For each patch, its seven neighbors, denoted by R0-R6 in FIG. 21, may be used to predict samples in the current patch. For patch B0, all neighbors may already be reconstructed. However, for other patches, some of the neighbors may not be reconstructed because they are in the current block, in which case the predicted values ​​of the immediate neighbors are used as reference. For example, all neighbors of patch B7 as shown in FIG. 21 have not been reconstructed, so the predicted samples of the neighbors are used instead.

[0131] In some implementations of intra prediction, one color component may be predicted using one or more other color components. A color component may be any one of the components of the YCrCb color space, the RGB color space, the XYZ color space, etc.

[0132] One type of intra prediction that predicts one color component using one or more other color components is luma-to-chroma (CfL) prediction. In CfL prediction, a chroma component is predicted based on a luma component. The predicted chroma component may include a chroma block, which may include a sample or a chroma sample. The predicted sample is referred to as a prediction sample. Also, the predicted chroma block may correspond to a luma block. In this specification, unless otherwise specified, the correspondence between a luma block and a chroma block refers to the chroma block being co-located with the luma block.

[0133] Also, as used herein and described in further detail below, the luma components used to predict the chroma components may include luma samples. The luma samples may include luma samples of the corresponding or co-located chroma block itself and / or may include neighboring luma samples that are luma samples of one or more neighboring luma blocks in the vicinity or adjacent to the co-located luma block corresponding to the predicted chroma block. Additionally, in at least some implementations, the luma samples used in the CfL prediction process are reconstructed luma samples, which may be copies of the original luma samples derived or reconstructed from a compressed version of the original luma samples using a decoding process.

[0134] In some implementations, an encoder (e.g., any of the encoders 403, 603, 703) and / or a decoder (e.g., any of the decoders 410, 510, 810) may be configured to perform CfL prediction via a CfL prediction process. Also, in at least some of these implementations, the encoder and / or the decoder may be configured in a CfL prediction mode to perform the CfL prediction process. As described in more detail below, the encoder and / or the decoder may be operable in at least one of a number of different CfL prediction modes. In the different CfL prediction modes, the encoder and / or the decoder may perform different respective CfL processes to generate chroma prediction samples.

[0135] 22A-22C, the encoder and / or decoder may include a CfL prediction unit 2202 configured to perform a CfL prediction process. In various implementations, the CfL prediction unit 2202 may be a standalone unit or a component or subunit of another unit of the encoder or decoder. For example, in any of various implementations, the CfL prediction unit 2202 may be a component or subunit of the intra prediction unit 552, the intra encoder 722, or the intra decoder 872. The CfL prediction unit 2202 may also be configured to perform various or multiple operations or functions to perform or implement the CfL process. For simplicity, the CfL prediction unit 2202 is described as being a component of the encoder and / or decoder that performs each of these operations or functions. However, in any of various implementations, the CfL prediction unit 2202 may be further configured or organized into multiple subunits each configured to perform one or more of the operations or functions of the CfL prediction process, or one or more units other than the CfL prediction unit 2202 may be configured to perform one or more operations of the CfL prediction process. Also, in any of various implementations, the CfL prediction unit 2202 may be implemented in hardware or a combination of hardware or software to perform and / or execute the operations or functions of the CfL process. For example, the CfL prediction unit may be implemented as an integrated circuit, or a processor configured to execute software or firmware stored in a memory, or a combination thereof. Also, in any of various implementations, a non-transitory computer-readable storage medium may store computer instructions executable by a processor to perform the functions or operations of the CfL prediction unit 2202.

[0136] For CfL prediction, the encoder and / or decoder may determine to apply a CfL prediction mode to a luma block of a received coded bitstream, such as via the CfL prediction unit 2202. The CfL prediction unit 2202 may then perform CfL prediction on the luma block according to the CfL prediction mode determined to be applied. Correspondingly, the encoder and / or decoder may reconstruct, at least in part, a chroma block that corresponds to or is co-located with the luma block through application of the CfL prediction mode, such as via the CfL prediction unit 2202.

[0137] As described above, the CfL prediction unit 2202 may operate in at least one of a plurality of CfL prediction modes. With reference to FIG. 22A, in a first CfL prediction mode (or a first set of one or more CfL prediction modes), the CfL prediction unit 2202 may be configured to generate a plurality of prediction samples of a chroma block corresponding to a luma block based on a luma sample of the luma block. As described above, the chroma block corresponding to the luma block is a chroma block co-located with the luma block. Correspondingly, in FIG. 22A, in the first CfL prediction mode, the CfL prediction unit uses a luma sample of a luma block co-located with the chroma block for which the prediction sample is to be generated.

[0138] FIG. 23 illustrates a flow diagram of an example method 2300 of a CfL prediction process that the CfL prediction unit 2202 may perform in the first CfL prediction mode. In some implementations, a CfL prediction process such as that illustrated in FIG. 23 generates a plurality of chroma prediction samples based on an alternating current (AC) contribution of a luma sample and a direct current (DC) contribution of a chroma sample. Each of the AC and DC contributions may be a prediction of a chroma component, and thus also referred to as an AC and DC contribution prediction. In certain of these implementations, the chroma prediction samples are modeled as a linear function of the luma sample, such as according to the following mathematical expression: CfL(α)=α×LAC +DC (1) Here, L AC where α represents the AC contribution of the luma component (luma sample), α represents a scaling parameter of the linear model, and DC represents the DC contribution of the chroma components. Also, in at least some implementations, the AC contributions are obtained for each of the samples of the block, while the DC contribution is obtained for the entire block.

[0139] In certain of these implementations shown in FIG. 23, in block 2302, multiple luma samples of a luma block may be subdivided (or downsampled) to a chroma resolution (e.g., 4:2:0, 4:2:2, or 4:4:4). In block 2304, the subsampled luma samples may be averaged to generate a luma average. In block 2306, the luma average may be subtracted from the luma sample to generate an AC contribution of the luma component. In block 2308, the AC contribution of the luma component may be multiplied with a scaling parameter α to generate a scaled AC contribution of the luma component. The scaled AC contribution of the luma component may also be an AC contribution prediction of the chroma component. In block 2310, a DC contribution prediction of the chroma component may be added to the chroma prediction sample to generate a chroma prediction sample according to a linear model. In at least some implementations, the scaling parameter α may be based on the original chroma sample and may be signaled in the bitstream. This may reduce decoder complexity and result in more accurate prediction. Additionally or alternatively, the DC contributions of the chroma components may be calculated using intra-DC modes within the chroma components in some example implementations.

[0140] Additionally, in some implementations of method 2300 or the first CfL mode, when some luma samples of a co-located luma block are outside a picture boundary, these luma samples may be padded, and the padded luma samples may be used to calculate a luma average, such as block 2304. Figure 24 shows a schematic diagram of luma samples inside and outside a picture defined by a picture boundary. In at least some implementations, luma samples outside the picture may be padded by working out the value of the closest available sample in the current block.

[0141] Additionally or alternatively, in some implementations, when performing CfL prediction, the subsampling performed in block 2302 may be combined with the averaging performed in block 2304 and / or the subtraction performed in block 2306. This can simplify the linear modeling equation while eliminating the subsampling division and rounding errors. The following equation (2) corresponds to the combination of both steps and simplifies to equation (3). Both equations (2) and (3) use integer division. Also, MxN is a matrix of pixels in the luma plane.

number

number

[0142] Based on chroma subsampling, S x ×S y ∈{1,2,4}. Also, M and N may both be powers of 2, and thus M×N is also a power of 2. For example, in the context of 4:2:0 chroma subsampling, instead of providing a box filter, a sum of four reconstructed luma pixels that match a chroma pixel may be used. Correspondingly, the CfL prediction may be scaled by 2.

[0143] In addition, as shown above for equation (1), the CfL prediction process of FIG. 23 can use only one linear model between luma samples and chroma samples in the entire coded block. However, only one linear model may not be optimal for the relationship between luma samples and chroma samples in the entire coded block. Additionally or alternatively, in the CfL prediction method 2300, the luma samples of the corresponding luma block are used to calculate the average and therefore the AC contribution. However, in at least some embodiments, the DC contribution may be determined or calculated by averaging neighboring luma samples. A non-alignment between the luma samples used for the AC and DC contributions, respectively, and the neighboring luma samples may result in an inaccurate or suboptimal prediction. Additionally or alternatively, signaling the scaling value α in the bitstream may consume some bits unnecessarily. Additionally or alternatively, in the CfL prediction method 2300 of FIG. 23, the chroma blocks are predicted from the co-located luma blocks, but the neighboring luma samples of the co-located luma blocks are not used.

[0144] Referring to FIG. 22B, in addition to or instead of operating in the first CfL prediction mode, in some implementations, the CfL prediction unit 2202 may operate in a second CfL prediction mode (or a second set of one or more CfL prediction modes) to generate a plurality of prediction samples of a chroma block corresponding to a luma block based on neighboring luma samples of the luma block. That is, the CfL prediction unit 2202 may generate a chroma prediction sample using neighboring luma samples of the luma block without using the luma samples of the luma block. Referring to FIG. 22C, in addition to or instead of operating in the first CfL prediction mode and / or the second CfL prediction mode, in some implementations, the CfL prediction unit 2202 may operate in a third CfL prediction mode (or a third set of one or more CfL prediction modes) to generate a plurality of prediction samples of a chroma block corresponding to a luma block based on the luma samples of the luma block and neighboring luma samples of the luma block.

[0145] FIG. 25 illustrates a schematic diagram of an example luma block 2502 and neighboring luma samples of the luma block 2502. In general, neighboring luma samples of a given luma block are luma samples of or within adjacent or neighboring luma blocks that are adjacent to or in the vicinity of the given luma block. Each neighboring luma sample can be or have multiple types of neighboring luma samples of a particular type. Each type may correspond to a relative spatial relationship with the given luma block. Similarly, adjacent or neighboring luma blocks can have a particular type that matches the particular type of neighboring luma samples contained therein. In at least some implementations, the multiple types of neighboring luma samples and / or blocks can include left, top left, top, top right, bottom right, bottom, and bottom left. Figure 25 shows that neighboring luma samples may be spatially arranged with respect to a given luma block 2502, including a left-neighboring luma sample 2504 in a left-neighboring luma block, an upper-left neighboring luma sample 2506 in an upper-left neighboring luma block, an upper-neighboring luma sample 2508 in an upper-neighboring luma block, an upper-right neighboring luma sample 2510 in an upper-right neighboring luma block, a right-neighboring luma sample 2512 in a right-neighboring luma block, a lower-right neighboring luma sample 2514 in a lower-right neighboring luma block, a lower-neighboring luma sample 2516 in a lower-neighboring luma block, and a lower-left neighboring luma sample 2518 in a lower-left neighboring luma block. Additionally, the upper-left, upper-right, lower-left, and lower-right neighboring luma samples and blocks may be generally and / or collectively referred to as corner neighboring luma samples and blocks, respectively. In any of various implementations in the second CfL mode and / or the third CfL mode, the CfL prediction unit 2202 may use all types, or at least one type, but not all, of neighboring luma samples when performing CfL prediction.

[0146] FIG. 26 shows a flowchart of an example method 2600 of a CfL prediction process that the CfL prediction unit 2202 may perform when operating in the second CfL prediction mode (FIG. 22B) and / or the third CfL prediction mode (FIG. 22C). In block 2602, the CfL prediction unit 2202 may determine multiple neighboring luma samples of a luma block. In block 2604, the CfL prediction unit 2202 may generate multiple predicted samples of a chroma block corresponding to the luma block based on the multiple neighboring luma samples.

[0147] FIG. 27 illustrates a flowchart of another example method 2700 of a CfL prediction process that the CfL prediction unit 2202 may perform when operating in the second CfL prediction mode (FIG. 22B) and / or the third CfL prediction mode (FIG. 22C). In block 2702, the CfL prediction unit 2202 may determine a plurality of neighboring luma samples of a luma block. In block 2704, the CfL prediction unit 2202 may generate AC and DC contributions of a plurality of predictive samples of a chroma block corresponding to the luma block. At least one of the AC or DC contributions is generated in block 2704 based on a set of luma samples including the plurality of neighboring luma samples determined in block 2702. In block 2706, the CfL prediction unit 2202 may generate a plurality of predictive samples of the chroma block based on the AC and DC contributions determined in block 2704. In at least some implementations, the CfL prediction unit 2202 may determine multiple chroma prediction samples in block 2706 using AC and DC contributions according to the linear model of equation (1) above.

[0148] In some implementations, the example methods 2600 and 2700 may be combined. For example, block 2602 may include block 2702, and block 2604 may include block 2704 and / or block 2706.

[0149] Figure 28 shows a flow diagram of another example method 2800 of a CfL prediction process that the CfL prediction unit 2202 may perform when operating in the third CfL prediction mode (Figure 22C). The CfL prediction process 2800 may be similar to the CfL prediction process 2300 of Figure 23, except that instead of averaging luma samples of a luma block, the CfL prediction process 2800 may average neighboring luma samples to generate a neighboring luma average. Thus, the AC contribution is based on both the luma samples of the luma block and the neighboring luma samples.

[0150] More specifically, in block 2802, multiple luma samples of a luma block may be subsampled to a chroma resolution. In block 2804, multiple neighboring luma samples of a luma block may be subsampled to a chroma resolution. In addition, in at least some implementations, the same subsampling (or downsampling) method may be used to subsample the luma samples and the neighboring luma samples (i.e., the same subsampling method is applied to both blocks 2802 and 2804). For example, if subsampling is performed according to a 4:2:0 format, two rows of the top neighboring area (e.g., area 2508 in FIG. 25), two rows of the left neighboring area (e.g., area 2504 in FIG. 25), and / or four pixels of the top-left area (e.g., area 2506 in FIG. 25) are subsampled (or downsampled). Correspondingly, when the CfL prediction unit 2202 determines the AC contribution (e.g., blocks 2806 and 2808 below), neighboring luma samples may be averaged and subtracted from the reconstructed luma sample value, as shown in FIG. 28 and further described below.

[0151] In more detail, in block 2806, the subsampled neighboring luma samples may be averaged to generate a neighboring luma average. In block 2808, the neighboring luma average may be subtracted from the subsampled luma sample to generate an AC contribution of the luma component. In block 2808, the AC contribution of the luma component may be multiplied with a scaling parameter α to generate a scaled AC contribution of the luma component. The scaled AC contribution of the luma component may also be an AC contribution prediction of the chroma component. In block 2812, a DC contribution prediction of the chroma component may be added to the chroma prediction sample to generate a chroma prediction sample, such as according to the linear model shown in equation (1) above. In at least some implementations, the scaling parameter α may be based on the original chroma sample and may be signaled in the bitstream. This may reduce decoder complexity and result in a more accurate prediction. Additionally or alternatively, the DC contribution of the chroma component may be calculated using an intra DC mode within the chroma component in some example implementations.

[0152] In other implementations of the example method 2800, the neighboring luma samples may not be subsampled before being used to determine the neighboring luma average, i.e., in these other implementations, block 2804 may be skipped or otherwise not performed.

[0153] Also, in some implementations, all or some of the blocks of method 2800 may be combined with method 2600 and / or method 2700. For example, after the neighboring luma samples are determined in blocks 2602 and / or 2702, they may be subsampled in block 2804 and / or averaged in block 2806. Additionally or alternatively, generating a chroma prediction sample based on the neighboring luma samples in block 2604 may include one or more of: subsampling the neighboring luma samples in block 2804, averaging the (subsampled) neighboring luma samples in block 2806, generating a luma AC contribution in block 2808, and / or scaling the luma AC contribution in block 2810. Additionally or alternatively, generating the AC contribution at block 2704 of method 2700 may include one or more of: subsampling neighboring luma samples at block 2804, averaging (subsampled) neighboring luma samples at block 2806, generating the luma AC contribution at block 2808, and / or scaling the luma AC contribution at block 2810. Additionally or alternatively, generating the chroma prediction sample based on the AC contribution and the DC contribution at block 2706 of method 2700 may include adding the DC contribution to the AC contribution at block 2812. Other ways to combine methods 2600, 2700, and / or 2800 may be possible.

[0154] FIG. 29 shows a flowchart of another example method 2900 of a CfL prediction process that the CfL prediction unit 2202 may perform when operating in the third CfL prediction mode (FIG. 22C). In block 2902, the CfL prediction unit 2202 may map multiple luma samples of a luma block to multiple neighboring chroma samples. For example, the CfL prediction unit 2202 may map each i-th luma pixel PL ij The kth neighboring chroma pixel PC kIn at least some implementations, in block 2902, the CfL prediction unit 2202 may also determine that a chroma block should be predicted in a CfL mode, where the chroma block corresponds to and / or is co-located with the luma block. The CfL prediction unit 2202 may then map the luma samples of the luma block to neighboring chroma samples in at least one neighboring chroma block in the vicinity of the chroma block to be predicted. In block 2904, the CfL prediction unit 2202 may perform CfL prediction of the chroma block using the neighboring chroma samples as prediction samples of the chroma block. For example, the CfL prediction unit 2202 may set the neighboring chroma samples as prediction samples of the chroma block corresponding to the luma block. That is, the CfL prediction unit 2202 may determine that each kth chroma pixel PC k The corresponding i-th to j-th chroma sample PC ij The CfL prediction unit 2202 may copy each k-th chroma pixel PC k for each of the i-th to j-th chroma samples PC ij The same relationship or mapping between the (i,j) position and the neighborhood index k determined in block 2902 may be used to set i,j=k k ...

[0155] In at least some implementations, the CfL prediction unit 2202 may map multiple luma samples of a luma block to multiple neighboring chroma samples in two stages in block 2902. First, the CfL prediction unit 2202 may map multiple luma samples to multiple neighboring luma samples. For example, the CfL prediction unit 2202 may map multiple luma samples of each of the i-th to j-th pixels PL ij For each k-th neighboring luma pixel PL kFor illustrative purposes, FIG. 30 illustrates the k-th neighboring luma sample PL k A specific chroma pixel PL mapped to (112) ij (111). The CfL prediction unit 2202 may then map the nearby luma samples to corresponding nearby chroma samples or otherwise determine nearby chroma samples that correspond to or are co-located with the nearby luma samples. For example, FIG. 30 illustrates, via arrow 3004, the neighboring chroma pixels PC k The kth neighboring luma pixel PL k (112). Through a two-stage process, the luma samples may be mapped or correspond to the neighboring chroma samples. The CfL prediction unit 2202 may then use the neighboring chroma samples as prediction samples of the corresponding chroma block for CfL prediction of the chroma block, as performed in block 2904. For example, FIG. 30 illustrates, via arrow 3006, the i-th to j-th luma samples PL ij The i-th to j-th chroma sample PCs ij The kth neighboring chroma sample PC is set as or copied to k Shows.

[0156] In addition, in at least some implementations of the two-stage process, when mapping luma samples to neighboring luma samples, the i-th to j-th luma samples PL ij For each luma sample PL ij In at least some of these implementations, the CfL prediction unit 2202 may determine a neighboring luma sample having a minimum difference from the i-th to j-th luma samples PL ij If the CfL prediction unit 2202 determines multiple neighboring luma samples having the smallest difference with the i-th to j-th luma samples PL ij The k-th neighboring luma sample of the target among the multiple minimum difference neighboring luma samples having the shortest distance to kThe CfL prediction unit 2202 may select the kth neighboring luma sample of the target in the minimum difference luma sample used to determine the chroma prediction sample.

[0157] Additionally or alternatively, in some implementations, when determining the minimum difference between a luma sample and a neighboring luma sample, the CfL prediction unit 2202 may determine the minimum difference according to a sum of absolute differences (SAD) measurement and / or a sum of squared errors (SSE) measurement.

[0158] Additionally or alternatively, in some implementations, the CfL prediction unit 2202 may map multiple luma samples together to multiple neighboring luma samples. The multiple luma samples that are mapped together may be referred to as a patch. For example, the CfL prediction unit 2202 may identify a given patch of luma samples and find a patch of neighboring luma samples that has a minimum difference with the given patch of luma samples. In at least some of these implementations, the CfL prediction unit 2202 may use a center sample or pixel of the patch to determine the neighboring luma patch that has the minimum difference with the given luma path. Upon identifying the neighboring luma patch with the minimum difference, the CfL prediction unit 2202 may map each luma sample of the luma patch to each neighboring luma sample of the minimum difference neighboring luma patch. In at least some of these implementations, the CfL prediction unit 2202 may use SSE to determine the minimum difference. Also, in some implementations, the sub-block sizes may be different. Additionally or alternatively, in some implementations, only a subset of all neighboring luma samples is analyzed to determine the minimum difference neighboring luma sample.

[0159] FIG. 31 illustrates an example of mapping luma samples to neighboring luma samples on a patch basis, which may be used as part of mapping multiple luma samples to multiple neighboring chroma samples in block 2902. In this example, three luma samples 11, 100, 21 may form a luma patch. The CfL prediction unit 2202 may analyze a neighboring luma patch having three neighboring luma samples. For each candidate neighboring luma patch, the CfL prediction unit 2202 may compare a central luma sample to a central neighboring luma sample of the candidate neighboring luma patch to determine a difference of the neighboring luma patch. In the example of FIG. 31, the CfL prediction unit 2202 determines that the neighboring luma sample having the smallest difference with the central luma sample 100 is the neighboring luma sample 101. Then, the CfL prediction unit 2202 may map the luma sample 11 with the neighboring luma sample 10, the luma sample 100 with the neighboring luma sample 101, and the luma sample 21 with the neighboring luma sample 20. As shown in Figure 31, after the luma samples of a luma block are mapped to neighboring luma samples on a patch basis, neighboring chroma samples corresponding to the neighboring luma samples may be determined, which may then be set as or copied to the chroma samples of the corresponding chroma block, as described above with respect to Figures 29 and 30.

[0160] Also, in some implementations, all or a portion of the method 2900 may be combined with the method 2600. For example, determining a neighboring luma sample in block 2602 may include mapping the luma sample to the neighboring luma sample performed in block 2902. Alternatively, the CfL prediction unit 2202 may determine a plurality of neighboring luma samples and then use these neighboring luma samples to map the luma sample to at least some of the neighboring luma samples in block 2602. Additionally or alternatively, in various implementations, generating a chroma prediction sample in block 2604 may include mapping the luma sample to the neighboring chroma sample in block 2902 and / or setting the neighboring chroma samples as a plurality of chroma prediction samples in block 2904.

[0161] Also, in any of the example methods 2600, 2700, 2800, 2900, the neighboring luma samples that the CfL prediction unit 2202 determines, subsamples, averages, and / or otherwise uses to generate the multiple chroma prediction samples may include all, or at least one, or less than all types of neighboring luma samples, including at least one of left, top left, top, top right, right, bottom right, bottom, or bottom left. For example, in some implementations, the at least one type may include at least one of left, top, or top left. In some other implementations, the at least one type may include at least one of left, top, right, bottom, top left, top right, bottom left, or bottom right. In some other implementations, the at least one type may include at least one of top right or bottom left. Also, in any of various implementations, including implementations of methods 2600, 2700, and / or 2800, when the CfL prediction unit 2202 determines to use a certain type of neighboring luma sample for the CfL prediction process, the CfL prediction unit 2202 may use one or more neighboring luma samples having this determined type.

[0162] Additionally or alternatively, in some implementations of example methods 2600, 2700, 2800, and / or 2900, signaling may be used to explicitly indicate or instruct the CfL prediction unit 2202 as to which type of neighboring luma samples to use for the CfL prediction process. For example, the CfL prediction unit 2202 may receive a signal generated internally, such as by another unit or component (e.g., an encoder or decoder) of the electronic device in which the CfL prediction unit 2202 is implemented, or a signal generated externally or remotely, such as by another unit or component (e.g., an encoder or decoder) of a different or separate electronic device in which the CfL prediction unit 2202 is implemented. In some other implementations, code information and / or characteristics of the luma block and / or predicted chroma block may be used to explicitly indicate which type of neighboring luma samples to use for the CfL prediction process, non-limiting examples of which include intra-prediction mode, block shape, block size, or block aspect ratio of the luma block and / or chroma block.

[0163] Additionally or alternatively, in some implementations of the example methods 2600, 2700, 2800, and / or 2900, only available types of neighboring luma samples may be used for the CfL prediction process. In some implementations, neighboring luma samples are available when not at a picture boundary or superblock boundary. For example, the CfL prediction unit 2202 may use the top neighboring luma sample for CfL prediction when only the top neighboring luma sample is available (e.g., determine the top neighboring luma sample in block 2602 and / or generate the chroma prediction sample based on the top neighboring luma sample in block 2604). As another example, the CfL prediction unit may use the left neighboring luma sample for CfL prediction when only the left neighboring luma sample is available (e.g., determine the left neighboring luma sample in block 2602 and / or generate the chroma prediction sample based on the left neighboring luma sample in block 2604).

[0164] Additionally or alternatively, in some implementations of example methods 2600, 2700, 2800, and / or 2900, when it is determined that a certain type of neighboring luma sample is used in the CfL prediction process and all of the certain type of neighboring luma samples are not available, the CfL prediction unit 2202 may pad the certain type of neighboring luma sample and use the padded neighboring luma sample in the CfL prediction process. For example, when a left neighboring luma sample is not available, the padding may include copying a luma sample of a left column of the current luma block as a left neighboring luma sample. As another example, when a top neighboring luma sample is not available, the padding may include copying a luma sample of a top row of the current luma block as a top neighboring luma sample. In at least some of these implementations, the CfL prediction unit may derange the neighboring luma samples according to the same padding method used in the intra-angle prediction mode.

[0165] Additionally or alternatively, in some implementations of example methods 2600, 2700, 2800, and / or 2900, when it is determined that upper neighboring luma samples are used for the CfL prediction process, the CfL prediction unit 2202 may use only those upper neighboring luma samples in the nearest upper reference line in the CfL prediction process. In certain of these implementations, the CfL prediction unit 2202 may use the upper neighboring luma samples in the nearest upper reference line as the neighboring luma samples for the CfL prediction process when the co-located luma block is located on a superblock boundary. In general, the AoMediaVideo model (AVM) uses coding blocks of various sizes, the largest of which is called a superblock. In some implementations, the largest block, i.e., the superblock, is 128×128 pixels or 64×64 pixels. The superblock size is signaled in the sequence header and has a default size of 128×128 pixels. The minimum coding block size is 4×4.

[0166] Additionally or alternatively, in some implementations of the example methods 2600, 2700, 2800, and / or 2900, the neighboring luma samples used in the CfL prediction process may include, and in some implementations may only include, neighboring luma samples in one or more nearest neighbor reference lines. At least some of these implementations may be used in combination with one or more of the other above-mentioned conditions or aspects. For example, when the CfL prediction unit 2202 determines to use a certain type of neighboring luma sample, the CfL prediction unit 2202 may use the certain type of neighboring luma sample that is in the certain type of nearest neighbor reference line. Figure 32 shows a diagram illustrating the top and left reference lines of co-located chroma blocks. In this example, reference line 0 may be the closest top neighbor reference in that it is closer than the other top reference lines 1, 2, and 3.

[0167] Additionally or alternatively, in some implementations of example methods 2600, 2700, 2800, and / or 2900, the CfL prediction unit 2202 may use one or more boundary luma samples of a luma block to perform the CfL prediction process. The boundary luma samples may be luma samples that define an outer boundary of a chroma block. For example, a left boundary luma sample of a given luma block defines, at least in part, a left boundary of the given chroma block. As another example, a top boundary defines, at least in part, an upper or upper boundary of the given luma block. The CfL prediction unit 2202 may use one or more boundary luma samples in combination with neighboring luma samples. For example, in one implementation of method 2800, the CfL prediction unit 2202 may determine a neighboring luma average by averaging a neighboring luma sample in combination with one or more boundary luma samples of a co-located chroma block. In various ones of these implementations, the boundary luma samples used for averaging may be obtained from the subsampled luma samples before subsampling, such as before the subsampling in block 2802, or after subsampling, such as after the subsampling in block 2802. In certain ones of these implementations, the one or more boundary luma samples may include one or more top boundary luma samples and / or one or more left boundary luma samples.

[0168] 33 shows a flowchart of another example method 3300 of a CfL prediction process that the CfL prediction unit 2202 may perform. In block 3302, the CfL prediction unit 2202 may determine a type of a CfL prediction process from a plurality of types of CfL prediction processes. The plurality of types of CfL prediction processes may correspond to at least two of the CfL prediction processes 2300, 2600, 2700, 2800, or 2900. In block 3304, the CfL prediction unit 2202 may perform a CfL prediction process according to the type of the CfL prediction process determined in block 3302.

[0169] In some implementations, the CfL prediction unit 2202 selects from only two of the CfL prediction processes 2300, 2600, 2700, 2800, 2900. The original two processes selected may be any two of the processes 2300, 2600, 2700, 2800, 2900 in any of various implementations. Additionally or alternatively, in at least some implementations, in block 3302, the CfL prediction unit 2202 may receive, identify, and / or use a flag to determine which type of CfL prediction process to use. In some of these implementations, the flag is bypass coded. In other implementations, the flag is context coded. In various embodiments, N contexts are used, where N is 1 or greater. In some other implementations, different CfL prediction processes are associated with different index values ​​of the chroma intra prediction mode syntax that the CfL prediction unit 2202 may receive, identify, and / or use. The chroma intra prediction mode syntax may indicate to the CfL prediction unit 2202 whether to perform a CfL prediction process, and if so, what type of CfL prediction process to perform. In any of various implementations, the index value may be one of two, three, or more possible different values ​​to indicate one of two, three, or more possible different CfL prediction processes.

[0170] The embodiments of the present disclosure may be used separately or combined in any order. Furthermore, each of the methods (or embodiments), the encoder, and the decoder may be implemented by a processing circuit (e.g., one or more processors, or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium. The embodiments of the present disclosure may be applied to a luma block or a chroma block.

[0171] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, FIG. 34 illustrates a computer system 3400 suitable for implementing certain embodiments of the disclosed subject matter.

[0172] The computer software may be coded using any suitable machine code or computer language that may undergo assembly, compilation, linking, or similar mechanisms to generate code including instructions that may be executed by one or more computer central processing units (CPUs) and / or graphics processing units (GPUs), either directly, or through interpretation and execution of microcode, etc.

[0173] The instructions may be executed on various types of computers or computer components including, for example, personal computers, tablet computers, servers, smart phones, gaming consoles, Internet of Things devices, and the like.

[0174] 34 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system 3400.

[0175] The computer system 3400 may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, via tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), olfactory input (not shown). The human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (voice, music, ambient sounds, etc.), images (scanned images, photographic images obtained from still image cameras, etc.), video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0176] The input human interface devices may include one or more of a keyboard 3401, a mouse 3402, a trackpad 3403, a touch screen 3410, a data glove (not shown), a joystick 3405, a microphone 3406, a scanner 3407, and a camera 3408 (only one of each is depicted).

[0177] The computer system 3400 may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, via haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen 3410, data gloves (not shown), or joystick 3405, although there may also be haptic feedback devices that do not function as input devices), audio output devices (such as speakers 3409, headphones (not depicted)), visual output devices (such as screens 3410 including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output or three-dimensional or higher output via means such as stereographic output, virtual reality glasses (not depicted), holographic displays, and smoke tanks (not depicted)), and printers (not depicted).

[0178] The computer system 3400 may also include human accessible storage devices and media associated with the storage devices such as optical media including CD / DVD ROM / RW 3420 having media 3421 such as CD / DVDs, thumb drives 3422, removable hard drives or solid state drives 3423, legacy magnetic media such as tapes and floppy disks (not shown), dedicated ROM / ASIC / PLD based devices such as security dongles (not shown), and the like.

[0179] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not encompass transmission media, carrier waves, or other transitory signals.

[0180] The computer system 3400 may also include an interface 3454 to one or more communication networks 3455. The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, cellular networks including wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., cable or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, vehicular and industrial including CAN bus, etc. Certain networks generally require an external network interface adapter connected to a particular general-purpose data port or peripheral bus 3449 (e.g., a USB port of the computer system 3400, etc.), while others are generally integrated into the core of the computer system 3400 by connection to a system bus as described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system 3400 can communicate with other entities. Such communications may be one-way, receive-only (e.g., television broadcast), one-way transmit-only (e.g., from a CANbus to a particular CANbus device), or two-way, e.g., to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks may be used in each of these networks and network interfaces, as described above.

[0181] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to the core 3440 of the computer system 3400.

[0182] The core 3440 may include one or more central processing units (CPUs) 3441, graphics processing units (GPUs) 3442, dedicated programmable processing units in the form of field programmable gate areas (FPGAs) 3443, hardware accelerators for specific tasks 3444, graphics adapters 3450, etc. These devices may be connected via a system bus 3448 along with read only memory (ROM) 3445, random access memory 3446, internal mass storage 3447 such as internal hard drives, SSDs, etc. that are not accessible to the user. In some computer systems, the system bus 3448 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 3448 or via a peripheral bus 3449. In one example, a screen 3410 may be connected to the graphics adapter 3450. Architectures for peripheral buses include PCI, USB, etc.

[0183] The CPU 3441, GPU 3442, FPGA 3443, and accelerator 3444 may execute certain instructions that may combine to constitute the aforementioned computer code. This computer code may be stored in ROM 3445 or RAM 3446. Transient data may also be stored in RAM 3446, while permanent data may be stored, for example, in internal mass storage 3447. Rapid storage and retrieval in any of the memory devices may be enabled by the use of cache memories that may be closely associated with one or more of the CPU 3441, GPU 3442, mass storage 3447, ROM 3445, RAM 3446, etc.

[0184] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the available kind well known to those skilled in the computer software arts.

[0185] As a non-limiting example, the architecture 3400, and specifically a computer system having the core 3440, can provide functionality as a result of the processor (including CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage as described above, as well as media associated with specific storage of the core 3440 that is non-transitory in nature, such as the core internal mass storage 3447 or ROM 3445. Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core 3440. The computer-readable media can include one or more memory devices or chips, depending on the particular needs. The software can cause the core 3440, and specifically the processor therein (including CPU, GPU, FPGA, etc.) to perform certain processes or certain parts of certain processes described herein, including defining data structures stored in RAM 3446 and modifying such data structures according to the software-defined processes. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 3444) that may operate in place of or together with software to perform certain processes or certain portions of certain processes described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that store software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0186] The subject matter of the present disclosure can also relate to or include, among other aspects, the following aspects:

[0187] In a first aspect, a method for video processing includes a step of determining that a chroma block should be predicted in luma-to-chroma (CfL) mode, where the chroma block corresponds to a luma block; a step of mapping a plurality of luma samples of the luma block to a plurality of neighboring chroma samples in at least one neighboring chroma block in the vicinity of the chroma block; and a step of performing CfL prediction of the chroma block using the plurality of neighboring chroma samples as a plurality of prediction samples of the chroma block.

[0188] A second aspect includes the first aspect and further includes that the mapping step includes a step of mapping a plurality of reconstructed luma samples to a plurality of reconstructed neighboring luma samples, and a step of mapping the plurality of reconstructed neighboring luma samples to a plurality of neighboring chroma samples.

[0189] A third aspect includes any of the first or second aspects, and further includes determining a plurality of minimum difference reconstructed neighboring luma samples corresponding to the plurality of reconstructed luma samples, and determining a plurality of reconstructed neighboring luma samples to be mapped to the plurality of reconstructed luma samples by setting the plurality of minimum difference reconstructed neighboring luma samples as the plurality of reconstructed neighboring luma samples.

[0190] A fourth aspect includes the third aspect and further includes the step of determining the plurality of minimum difference reconstructed neighboring luma samples includes the steps of identifying a minimum difference reconstructed neighboring luma sample for a luma sample of the plurality of reconstructed luma samples; selecting a target minimum difference reconstructed neighboring luma sample from the plurality of minimum difference reconstructed neighboring luma samples in response to a target minimum difference reconstructed neighboring luma sample having a shortest distance to the luma sample among the plurality of minimum difference reconstructed neighboring luma samples; and including the target minimum difference reconstructed neighboring luma sample in the plurality of minimum difference reconstructed neighboring luma samples.

[0191] A fifth aspect includes any of the third or fourth aspects, and further includes that determining the plurality of least-difference reconstructed neighboring luma samples includes determining the plurality of least-difference reconstructed neighboring luma samples according to at least one of a sum of absolute differences (SAD) measurement or a sum of squared errors (SSE) measurement.

[0192] A sixth aspect includes any of the first to fifth aspects, and further includes that mapping the plurality of reconstructed luma samples to the plurality of neighboring chroma samples includes mapping a patch of reconstructed luma samples to a neighboring patch of reconstructed luma samples.

[0193] A seventh aspect includes the sixth aspect, further including: the neighboring patches of reconstructed luma samples include a minimum difference patch that corresponds to a path of reconstructed luma samples.

[0194] An eighth aspect includes the seventh aspect, further including determining a closest distance patch based on a central luma sample of the patch of reconstructed luma samples and a central luma sample of the minimum difference patch.

[0195] A ninth aspect includes any of the second to eighth aspects, further including that the plurality of reconstructed neighboring luma samples includes less than all types of reconstructed neighboring luma samples.

[0196] A tenth aspect includes the ninth aspect, and further includes that the reconstructed luma samples of less than all types include at least one of a left reconstructed neighboring luma sample, an above reconstructed neighboring luma sample, or an above-left reconstructed neighboring luma sample.

[0197] An eleventh aspect includes any of the second to tenth aspects, further including that the plurality of reconstructed neighboring luma samples are within nearest neighbor reference lines.

[0198] A twelfth aspect includes a method for video processing including determining a type of luma to chroma (CfL) prediction process from a plurality of different types of CfL prediction processes, and performing the CfL prediction process according to the type of CfL prediction process.

[0199] A thirteenth aspect includes the twelfth aspect, further including that the plurality of different types of CfL prediction processes includes only two types of CfL prediction processes.

[0200] A fourteenth aspect includes the twelfth aspect, further including that the plurality of different types of CfL prediction processes includes only three types of CfL prediction processes.

[0201] A fifteenth aspect includes any of the twelfth to fourteenth aspects, and further includes: the first type CfL prediction process uses one or more neighboring luma samples to determine the plurality of chroma prediction samples; and the second type CfL prediction process does not use neighboring luma samples to determine the plurality of chroma prediction samples.

[0202] A sixteenth aspect includes any of the twelfth to fifteenth aspects, wherein the one type of CfL prediction process further includes mapping a plurality of luma samples to a plurality of neighboring chroma samples.

[0203] A seventeenth aspect includes the sixteenth aspect, further including that the one type of CfL prediction process does not use a scaling factor to determine alternating current (AC) contributions of the multiple chroma prediction samples.

[0204] An 18th aspect includes any of the 12th to 17th aspects, and further includes that the step of determining a type of CfL prediction process includes a step of identifying a flag indicating the type of CfL prediction process, the flag being bypass coded or context coded.

[0205] A 19th aspect includes any of the 12th to 18th aspects, further including that determining the type of CfL prediction process includes identifying a chroma intra prediction mode syntax that identifies the type of CfL prediction process.

[0206] A twentieth aspect includes a video processing device including a memory storing a set of instructions and a processor configured to execute the set of instructions, the video processing device being configured to perform any of the first to nineteenth aspects upon execution of the set of instructions.

[0207] A twenty-first aspect includes a non-transitory computer-readable storage medium storing a plurality of processor-executable instructions that, when executed by the processor, cause the processor to implement any of the first to nineteenth aspects.

[0208] In addition to the features mentioned in each of the independent aspects listed above, some examples may exhibit, alone or in combination, any of the features mentioned in the dependent aspects and / or disclosed in the above description and illustrated in the figures.

[0209] While this disclosure has described several exemplary embodiments, there are alterations, substitutions, and various substitute equivalents that fall within the scope of the disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the disclosure and are therefore within the spirit and scope of the disclosure.

[0210] Appendix A: Acronyms JEM: Joint Exploration Model VVC: Versatile Video Coding BMS: Benchmark Set MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Supplemental Extended Information VUI: Video Usability Information GOP: Group of Pictures TU: conversion unit PU: Prediction Unit CTU: Coding Tree Unit CTB: coding tree block PB: Predicted block HRD: Hypothetical Reference Decoder SNR: Signal to Noise Ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT: cathode ray tube LCD: Liquid crystal display OLED: Organic Light Emitting Diode CD:Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile Communications LTE: Long Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Area SSD: Solid State Drive IC: Integrated Circuit HDR: High Dynamic Range SDR: Standard Dynamic Range JVET: Joint Video Exploration Team MPM: Most Probable Mode WAIP: Wide-angle Intra Prediction CU: coding unit PU: Prediction Unit TU: conversion unit CTU: Coding Tree Unit PDPC: Position-dependent prediction combination ISP: Intra Subpartition SPS: Sequence Parameter Settings PPS: Picture Parameter Set APS: Adaptive Parameter Set VPS: Video Parameter Set DPS: Decoding Parameter Set ALF: Adaptive Loop Filter SAO: Sample Adaptive Offset CC-ALF: Cross-component adaptive loop filter CDEF: Constrained Directional Enhancement Filter CCSO: Cross component sample offset LSO: Local Sample Offset LR: Loop recovery filter AV1:AOMedia Video 1 AV2:AOMedia Video 2 LFNST: Low Frequency Nonseparable Transform IST: Intra-quadratic transformation [Explanation of symbols]

[0211] 101,2010 Sample, 102,103,3002,3004,3006 Arrow, 104 Square block, 180 Schematic diagram, 201-206,2002,2004,2006,2008 Block, 300,400 Communication system, 310,320,330,340 Terminal device, 350 Network, 401,601 Video source, 402 Stream, 403,603,703 Video encoder, 404 Video data, 405 Streaming server, 406,408 Client subsystem, 407,409 Copy, 410,510,810 Video decoder, 411 Video picture, 412,512 Display, 413 Video capture subsystem, 420,430,530,620 Electronic device, 515 Buffer memory, 520 entropy decoder / parser, 521 symbols, 531 receiver, 551 scaler / inverse transform unit, 552 intra picture prediction unit, 553 motion compensation prediction unit, 555 aggregator, 556 loop filter unit, 557, 634 reference picture memory, 558 picture buffer, 630 source coder, 632 coding engine, 633 local decoder, 635 predictor, 640 transmitter, 643 video sequence, 645 entropy coder, 650 controller, 660 communication channel, 721 general controller, 722 intra encoder, 723 residual calculator, 724 residual encoder, 725 entropy encoder, 726 switch, 728 residual decoder, 730 inter encoder, 871 entropy decoder, 872 intra decoder, 873 residual decoder, 874 Reconstruction module,880 Inter-decoder,902,904,906,908,1200,1502,1602 Base block,1002,1004,1006,1008,1010,1102,1104,1106,1108,1202,1204,1206,1208,1302,1304,1402,1404,406,1408,1420 Partition,1504,1506,1604 coding order, 2202 CfL prediction unit, 2300, 2600, 2700, 2800, 2900, 3300 method, 2302, 2304, 2306, 2308, 2310 block, 2502 luma block, 2504 left neighbor luma sample, 2506 top left neighbor luma sample, 2508 top neighbor luma sample, 2510 top right neighbor luma sample, 2512 right neighbor luma sample, 2514 bottom right neighbor luma sample, 2516 bottom neighbor luma sample, 2518 bottom left neighbor luma sample, 2602, 2604 block, 2702, 2704, 2706 block, 2802, 2804, 2806, 2808, 2810, 2812 block, 2902, 2904 block, 3302, 3304 block, 3400 computer system, 3401 keyboard, 3402 mouse, 3403 track pad, 3405 joystick, 3406 microphone, 3407 scanner, 3408 camera, 3409 speaker, 3410 touch screen, 3421 media, 3422 thumb drive, 3423 removable hard drive or solid state drive, 3440 core, 3441 central processing unit (CPU), 3442 graphics processing unit (GPU), 3443 field programmable gate area (FPGA), 3444 hardware accelerator, 3445 read only memory (ROM), 3446 random access memory, 3447 internal mass storage, 3448 system bus, 3449 peripheral bus, 3450 graphics adapter, 3454 Interfaces, 3455 Communication Networks,

Claims

1. A method for video processing executed by one or more processors, said method comprising: determining that a chroma block should be predicted in luma-to-chroma (CfL) mode, the chroma block corresponding to a luma block; mapping luma samples of the luma block to neighboring chroma samples in at least one neighboring chroma block in a neighborhood of the chroma block; performing CfL prediction for the chroma block using the plurality of neighboring chroma samples as a plurality of prediction samples for the chroma block; A method comprising:

2. said step of mapping mapping a plurality of reconstructed luma samples to a plurality of reconstructed neighboring luma samples; mapping the plurality of reconstructed nearby luma samples to the plurality of nearby chroma samples; 2. The method of claim 1, comprising: determining a plurality of least-difference reconstructed neighboring luma samples corresponding to a plurality of reconstructed luma samples; and setting the plurality of least-difference reconstructed neighboring luma samples as the plurality of reconstructed neighboring luma samples; determining the plurality of reconstructed neighboring luma samples to be mapped to the plurality of reconstructed luma samples by 3. The method of claim 1 or 2, further comprising:

4. determining the plurality of least-difference reconstructed neighboring luma samples comprises: identifying a least-different reconstructed neighboring luma sample for a luma sample of the plurality of reconstructed luma samples; selecting a target minimum difference reconstructed neighboring luma sample from the plurality of minimum difference reconstructed neighboring luma samples in response to a target minimum difference reconstructed neighboring luma sample having a shortest distance to the luma sample among the plurality of minimum difference reconstructed neighboring luma samples; including the target minimum difference reconstructed neighboring luma sample in the plurality of minimum difference reconstructed neighboring luma samples; 4. The method of claim 3, comprising:

5. 4. The method of claim 3, wherein determining the plurality of least-different reconstructed neighboring luma samples comprises determining the plurality of least-different reconstructed neighboring luma samples according to at least one of a sum of absolute differences (SAD) measurement or a sum of squared errors (SSE) measurement.

6. A method as described in claim 1 or 2, wherein the step of mapping a plurality of reconstructed luma samples to the plurality of neighboring chroma samples includes a step of mapping a patch of reconstructed luma samples to a patch of neighboring reconstructed luma samples.

7. The method of claim 6 , wherein the patches of neighboring reconstructed luma samples comprise minimum difference patches corresponding to paths of reconstructed luma samples.

8. The method of claim 7 , further comprising determining a minimum distance patch based on a central luma sample of the patch of reconstructed luma samples and a central luma sample of the minimum difference patch.

9. The method of claim 2 , wherein the plurality of reconstructed neighboring luma samples includes reconstructed neighboring luma samples of less than all types.

10. 10. The method of claim 9, wherein the reconstructed luma samples of less than all types include at least one of a left reconstructed neighboring luma sample, an above reconstructed neighboring luma sample, or an above-left reconstructed neighboring luma sample.

11. The method of claim 2 , wherein the plurality of reconstructed neighboring luma samples are within a nearest neighbor baseline.

12. A method for video processing executed by one or more processors, said method comprising: determining a type of luma to chroma (CfL) prediction process from a plurality of different types of CfL prediction processes; performing a CfL prediction process according to the CfL prediction process type; A method comprising:

13. The method of claim 12 , wherein the plurality of different types of CfL prediction processes includes only two types of CfL prediction processes.

14. The method of claim 12 , wherein the plurality of different types of CfL prediction processes includes only three types of CfL prediction processes.

15. 15. The method of claim 12, wherein a first type of CfL prediction process uses one or more neighboring luma samples to determine a plurality of chroma prediction samples, and a second type of CfL prediction process does not use neighboring luma samples to determine the plurality of chroma prediction samples.

16. 15. The method of claim 12, wherein one type of CfL prediction process maps multiple luma samples to multiple neighboring chroma samples.

17. 17. The method of claim 16, wherein the one type of CfL prediction process does not use a scaling factor to determine alternating current (AC) contributions of multiple chroma prediction samples.

18. determining a type of CfL prediction process, identifying a flag indicating the type of CfL prediction process, said flag being bypass coded or context coded; 15. The method of any one of claims 12 to 14, comprising:

19. determining a type of CfL prediction process, identifying a chroma intra prediction mode syntax that identifies a type of the CfL prediction process.

15. The method of any one of claims 12 to 14, comprising:

20. 1. A video processing device, comprising: a memory for storing a set of instructions; configured to execute the set of instructions, wherein upon execution of the set of instructions: mapping a plurality of reconstructed luma samples of a luma block to a plurality of neighboring chroma samples of a chroma block corresponding to the luma block; Setting the neighboring chrominance samples as prediction samples of the chrominance block. The processor and A video processing device comprising:

21. A computer program product for causing one or more processors to carry out the method of claim 1.

Images