Selecting a downsampling filter for chroma from luma prediction

Abstract

The present disclosure relates to video processing that includes iteratively predicting a chroma block in a chroma-from-luma (CfL) prediction mode based on downsampled luma samples downsampled from a plurality of downsampling filters, and selecting a target downsampling filter from a plurality of downsampling filters that corresponds to an error score determined for the iterative prediction.

Description

[Technical Field]

[0001] Incorporation by Reference

[0001] This application is based on and claims the benefit of priority to U.S. Non-Provisional Application No. 17 / 992,282, entitled "SELECTING DOWNSAMPLING FILTERS FOR CHROMA FROM LUMA PREDICTION," filed November 22, 2022, and U.S. Provisional Application No. 63 / 403,635, entitled "SELECTING DOWNSAMPLING FILTERS FOR CHROMA FROM LUMA INTRA PREDICTION MODE," filed September 2, 2022, each of which is incorporated by reference in its entirety into this specification.

[0002] This disclosure describes a set of advanced video coding techniques. More specifically, the disclosed techniques involve downsampling filter selection for chroma-from-luma prediction. [Background technology]

[0003]

[0003] This background discussion provided herein is intended to generally present the context for the present disclosure. Work of the inventors currently identified to the extent that their work is described in this background section, and aspects of the discussion that may not otherwise be considered prior art at the time of filing this application, are not expressly or implicitly admitted to be prior art to the present disclosure.

[0004] Video encoding and decoding can be performed using inter-picture prediction with motion compensation. Uncompressed digital video can include a series of pictures, each having spatial dimensions of, for example, 1920 x 1080 luma samples and associated full or subsampled chroma samples. The series of pictures can have a fixed or variable picture rate (alternatively called a frame rate), for example, 60 pictures per second, or 60 frames per second. Uncompressed video has inherent bitrate requirements for streaming or data processing. For example, a video with a pixel resolution of 1920 x 1080, a frame rate of 60 frames / second, and chroma subsampling of 4:2:0 with 8 bits per pixel per color channel requires a bandwidth of nearly 1.5 Gbit / s. One hour of such video requires more than 600 GByte of storage space.

[0005]

[0005] One goal of video encoding and decoding can be the reduction of redundancy in an uncompressed input video signal through compression. Compression can, in some cases, help reduce the aforementioned bandwidth and / or storage space requirements by more than two orders of magnitude. Both lossless and lossy compression, as well as combinations thereof, can be employed. Lossless compression refers to a technique in which an exact copy of the original signal can be reconstructed from the compressed original signal through a decoding process. Lossy compression refers to an encoding / decoding process in which the original video information is not fully preserved during encoding and therefore cannot be fully restored during decoding. When using lossy compression, the reconstructed signal may not be identical to the original signal, but even with some information loss, the distortion between the original and reconstructed signal is small enough to make the reconstructed signal useful for the intended application. In the case of video, lossy compression is widely adopted in many applications. The amount of tolerable distortion depends on the application. For example, users of certain consumer video streaming applications can tolerate greater distortion than users of movie or television broadcasting applications. The compression ratio achievable by a particular encoding algorithm can be selected or adjusted to reflect different distortion tolerances; generally, the more distortion that can be tolerated, the more lossy and the greater the compression ratio that is possible with an encoding algorithm.

[0006] Video encoders and decoders can utilize techniques from several broad categories and processes, including, for example, motion compensation, Fourier transform, quantization, and entropy coding.

[0007]

[0007] Video codec technology can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. When all blocks of samples are coded in intra mode, the picture can be called an intra-picture. Intra-pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and can therefore be used as the first picture in a coded video bitstream and video session, or as a still image. The samples of the block after intra-prediction can then be transformed into the frequency domain, and the transform coefficients thus generated can be quantized before entropy coding. Intra-prediction refers to a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value and the smaller the AC coefficients after the transformation, the fewer bits are required to represent the block after entropy coding at a given quantization step size.

[0008]

[0008] Traditional intra-coding, such as that known from the MPEG-2 generation of encoding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to encode / decode blocks based on surrounding sample data and / or metadata obtained during the encoding and / or decoding of spatially neighboring blocks and that precede the intra-coded or decoded data block in decoding order. Such techniques are hereinafter referred to as "intra-prediction" techniques. It should be noted that, at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed, and does not use reference data from other reference pictures.

[0009]

[0009] There can be many different forms of intra prediction. When two or more such techniques are available within a given video coding technique, the technique in use can be referred to as an intra prediction mode. In a particular codec, one or more intra prediction modes may be provided. In certain cases, a mode can have sub-modes and / or can be associated with various parameters, and the mode / sub-mode information and the intra coding parameters of a video block can be coded separately or collectively included in a mode codeword. Which codeword to use for a given mode, sub-mode, and / or parameter combination can also affect coding efficiency gains through intra prediction, and therefore, entropy coding techniques can be used to turn the codeword into a bitstream.

[0010]

[0010] Certain intra-prediction modes were introduced in H.264, improved in H.265, and further refined in newer coding techniques, such as the joint exploration model (JEM), versatile video coding (VVC), and benchmark set (BMS). In general, for intra-prediction, a predictor block can be formed using available neighboring sample values. For example, available values ​​of a specific set of neighboring samples along a specific direction and / or line may be copied into the predictor block. A reference to the direction in use can be coded in the bitstream or may itself be predicted.

[0011] 1A, a subset of nine predictor directions specified by the 33 possible intra predictor directions of H.265 (corresponding to the 33 angle modes of the 35 intra modes specified in H.265) is depicted in the lower right. The point 101 where the arrows converge represents the sample to be predicted. The arrows represent the direction in which neighboring samples are used to predict the sample at 101. For example, arrow 102 indicates that sample 101 is predicted from one or more neighboring samples located to the upper right and at an angle of 45 degrees from horizontal. Similarly, arrow 103 indicates that sample 101 is predicted from one or more neighboring samples located to the lower left of sample 101 and at an angle of 22.5 degrees from horizontal.

[0012]

[0012] With further reference to FIG. 1A, a square block 104 of 4x4 samples (indicated by a thick dotted line) is depicted in the upper left. The square block 104 includes 16 samples, each labeled with "S," the block's position in the Y dimension (e.g., row index), and the block's position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample of block 104 in both the Y and X dimensions. Because the block is 4x4 samples in size, S44 is located in the lower right. An example reference sample is also shown, following a similar numbering scheme. The reference sample is labeled R, the sample's Y position (e.g., row index) and X position (column index) relative to block 104. In both H.264 and H.265, prediction samples that are adjacent to the block being reconstructed are used.

[0013]

[0013] Intra-picture prediction of block 104 may begin by copying reference sample values ​​from neighboring samples according to the signaled prediction direction. For example, assume that the coded video bitstream includes signaling indicating the prediction direction of arrow 102 for this block 104—that is, the sample is predicted from one or more prediction samples located to the upper right and at a 45-degree angle from horizontal. In such a case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Thus, sample S44 is predicted from reference sample R08.

[0014] In certain cases, especially when the direction is not evenly divisible by 45 degrees, the values ​​of multiple reference samples may be combined, for example through interpolation, to calculate the reference sample.

[0015] The number of possible directions has increased as video coding technology continues to develop. In H.264 (2003), for example, nine different directions are available for intra prediction. This increased to 33 in H.265 (2013), and JEM / VVC / BMS can support up to 65 directions as of the time of this disclosure. Empirical studies have been conducted to help identify optimal intra prediction directions, and specific techniques in entropy coding may be used to encode these optimal directions with a small number of bits, accepting a specific bit penalty for the direction. Furthermore, the direction itself is sometimes predictable from neighboring directions used in intra prediction of decoded neighboring blocks.

[0016] FIG. 1B shows a diagram 180 depicting 65 intra prediction directions according to JEM to illustrate the increasing number of prediction directions in various encoding techniques developed over time.

[0017]

[0017] The manner in which bits representing intra-prediction directions are mapped to prediction directions in the encoded video bitstream may vary from one video coding technique to another, and may range, for example, from a simple direct mapping of prediction directions to codewords, to intra-prediction modes, to complex adaptive schemes involving most probable modes, and similar techniques. However, in all cases, there may be certain directions for intra-prediction that are statistically less likely to occur in the video content than certain other directions. Because the goal of video compression is to reduce redundancy, in a well-designed video coding technique, these less likely directions may be represented with a greater number of bits than more likely directions.

[0018]

[0018] Inter-picture prediction or inter-prediction may be based on motion compensation. In motion compensation, sample data (reference picture) from a previously reconstructed picture or part thereof may be used for predicting a newly reconstructed picture or part of a picture (e.g., a block) after being spatially shifted in a direction indicated by a motion vector (hereinafter, MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, with the third dimension indicating the reference picture in use (similar to a temporal dimension).

[0019]

[0019] In some video compression techniques, the current MV applicable to a particular area of ​​sample data can be predicted from other MVs, such as other MVs related to other areas of sample data that are spatially adjacent to the area being reconstructed and precede the current MV in decoding order. Doing so can substantially reduce the total amount of data required to encode the MV by relying on the elimination of redundancy in correlated MVs, thereby improving compression efficiency. For example, when encoding an input video signal derived from a camera (known as raw video), there is a statistical possibility that an area larger than the area to which a single MV is applicable moves in a similar direction in the video sequence. Therefore, in some cases, MV prediction can work effectively because it can be predicted using similar motion vectors derived from MVs of neighboring areas. This makes the actual MV of a given area similar or identical to the MV predicted from surrounding MVs. Then, after entropy encoding, such an MV may be represented with fewer bits than would be used if the MV were encoded directly rather than predicted from neighboring MVs. In some cases, the MV prediction may be an example of lossless compression of a signal (i.e., the MV) derived from the original signal (i.e., the sample stream). In other cases, the MV prediction itself may be lossy, for example, due to rounding errors when computing the predictor from several surrounding MVs.

[0020]

[0020] H.265 / HEVC (ITU-T Recommendation H.265, "High Efficiency Video Coding", December 2016) describes various MV prediction mechanisms. Among the many MV prediction mechanisms specified by H.265, a technique hereinafter referred to as "spatial merging" will be described below.

[0021]

[0021] Specifically, referring to Figure 2, a current block (201) contains samples that the encoder found during the motion search process to be predictable from a spatially shifted previous block of the same size. Instead of directly encoding this MV, the MV can be derived from metadata associated with one or more reference pictures, such as the most recent reference picture (in decoding order), using the MV associated with any one of five surrounding samples denoted A0, A1, and B0, B1, B2 (202 to 206, respectively). In H.265, MV prediction can use predictors from the same reference picture used by neighboring blocks. Summary of the Invention [Means for solving the problem]

[0022] Aspects of the present disclosure provide methods and apparatus for downsampling filter selection for chroma from luma (CfL) prediction.

[0023]

[0023] In some implementations, a method for video processing includes receiving an input chroma block from a video sequence, determining that the input chroma block should be predicted in a chroma-from-luma (CfL) prediction mode, applying a plurality of downsampling filters respectively to obtain a plurality of sets of downsampled luma samples corresponding to the input chroma block, iteratively predicting the input chroma block in the CfL prediction mode based on each of the plurality of sets of downsampled luma samples, calculating a plurality of error scores of the iterative prediction, each of the plurality of error scores corresponding to a respective one of the plurality of downsampling filters, selecting a target downsampling filter from the plurality of downsampling filters based on the plurality of error scores, and encoding the input chroma block in the CfL prediction mode by applying the selected target downsampling filter.

[0024]

[0024] In some other implementations, a method for video processing includes performing first pass encoding on a video sequence using a plurality of downsampling filters, determining a target downsampling filter from among the plurality of downsampling filters based on the first pass encoding, and performing second pass encoding on the video sequence after performing the first pass encoding using the target downsampling filter.

[0025]

[0025] In some other implementations, a device for processing video information is disclosed. The device may include circuitry configured to perform any one of the implementations of the above methods.

[0026]

[0026] An aspect of the present disclosure also provides a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding and / or encoding, cause the computer to perform a method for video decoding and / or encoding, such as any of the implementations of the methods described above.

[0027]

[0027] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Brief explanation of the drawings]

[0028] [Figure 1A] FIG. 10 is a schematic diagram of an exemplary subset of intra-prediction direction modes. [Figure 1B]

[0029] FIG. 1 is a diagram of exemplary intra-prediction directions. [Figure 2]

[0030] FIG. 1 is a schematic diagram of a current block and its surrounding spatial merge candidates for motion vector prediction in one example. [Figure 3]

[0031] 3 is a simplified block diagram schematic of a communication system 300 in accordance with an example embodiment. [Figure 4]

[0032] 4 is a schematic diagram of a simplified block diagram of a communication system 400 in accordance with an example embodiment. [Figure 5]

[0033] FIG. 2 is a schematic diagram of a simplified block diagram of a video decoder in accordance with an example embodiment; [Figure 6]

[0034] FIG. 2 is a schematic diagram of a simplified block diagram of a video encoder in accordance with an example embodiment; [Figure 7]

[0035] FIG. 10 is a block diagram of a video encoder according to another example embodiment. [Figure 8]

[0036] FIG. 10 is a block diagram of a video decoder according to another example embodiment. [Figure 9]

[0037] FIG. 1 is a diagram of a coding block partitioning scheme according to an example embodiment of the present disclosure. [Figure 10]

[0038] FIG. 10 is a diagram of another coding block partitioning scheme according to an example embodiment of the present disclosure. [Figure 11]

[0039] FIG. 10 is a diagram of another coding block partitioning scheme according to an example embodiment of the present disclosure. [Figure 12]

[0040] FIG. 10 is a diagram of an example partitioning of a base block into coding blocks according to an example partitioning scheme. [Figure 13]

[0041] FIG. 1 is a diagram of an example ternary partitioning scheme. [Figure 14]

[0042] FIG. 1 is a diagram of an example quadtree / binary tree coding block partitioning scheme. [Figure 15]

[0043] 1 is a diagram of a scheme for partitioning a coding block into multiple transform blocks and a coding order of the transform blocks according to an example embodiment of this disclosure. [Figure 16]

[0044] FIG. 10 is a diagram of another scheme for partitioning a coding block into multiple transform blocks and the coding order of the transform blocks according to an example embodiment of this disclosure. [Figure 17]

[0045] FIG. 10 is a diagram of another scheme for partitioning a coding block into multiple transform blocks according to an example embodiment of this disclosure. [Figure 18]

[0046] FIG. 10 is an example fine angle diagram for directional intra prediction. [Figure 19]

[0047] FIG. 10 is a diagram of nominal angles in directional intra prediction. [Figure 20]

[0048] 10 is a diagram of the top, left, and top-leftmost positions for the PAETH mode of a block. [Figure 21]

[0049] FIG. 10 is a diagram of an example recursive intra-filtering mode. [Figure 22]

[0050] FIG. 10 is a block diagram of a chroma-from-luma (CfL) prediction unit configured to generate prediction samples of chroma blocks based on input luma samples. [Figure 23A]

[0051] 1 is a flow diagram of an example CfL prediction process. [Figure 23B]

[0052] 10 is a flow diagram of another example CfL prediction process. [Figure 24]

[0053] FIG. 1 is a block diagram of luma samples inside and outside the picture boundary. [Figure 25]

[0054] 1 is a diagram of different chroma downsampling formats. [Figure 26]

[0055] A diagram of an AVI CfL downsampling filter. [Figure 27]

[0056] FIG. 1 is a diagram of an example 6-tap filter. [Figure 28]

[0057] FIG. 1 is a diagram of an example 4-tap filter. [Figure 29]

[0058] 1 is a flowchart of an example method of video processing. [Figure 30]

[0059] 10 is a flowchart of another example method of video processing. [Figure 31]

[0060] FIG. 10 is a schematic diagram of luma samples next to a luma block. [Figure 32]

[0061] 10 is a flowchart of another example method of video processing. [Figure 33]

[0062] FIG. 1 is a schematic diagram of a computer system according to an example embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0029]

[0063] FIG. 3 illustrates a simplified block diagram of a communication system 300 according to an embodiment of the present disclosure. The communication system 300 includes multiple terminal devices capable of communicating with each other, e.g., via a network 350. For example, the communication system 300 includes a first pair of terminal devices 310 and 320 interconnected via the network 350. In the example of FIG. 3, the first pair of terminal devices 310 and 320 may implement unidirectional transmission of data. For example, the terminal device 310 may encode video data (e.g., a stream of video pictures captured by the terminal device 310) for transmission to another terminal device 320 via the network 350. The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device 320 may receive the coded video data from the network 350, decode the coded video data to reconstruct the video pictures, and display the video pictures according to the reconstructed video data. The unidirectional data transmission may be implemented in a media serving application, etc.

[0030]

[0064] In another example, communication system 300 includes a second pair of terminal devices 330 and 340 that implements bidirectional transmission of encoded video data, such as may be implemented during a video conferencing application. For bidirectional transmission of data, in one example, each terminal device of terminal devices 330 and 340 may encode video data (e.g., of a stream of video pictures captured by the terminal device) for transmission to the other terminal device of terminal devices 330 and 340 over network 350. Each terminal device of terminal devices 330 and 340 may also receive encoded video data transmitted by the other terminal device of terminal devices 330 and 340, decode the encoded video data to recover the video pictures, and display the video pictures on an accessible display device in response to the recovered video data.

[0031]

[0065] In the example of FIG. 3 , terminal devices 310, 320, 330, and 340 may be implemented as a server, a personal computer, and a smartphone, although the applicability of the underlying principles of the present disclosure may not be so limited. Embodiments of the present disclosure may also be implemented in desktop computers, laptop computers, tablet computers, media players, wearable computers, dedicated videoconferencing equipment, and the like. Network 350 represents any number or type of network that conveys encoded video data between terminal devices 310, 320, 330, and 340, including, for example, wireline and / or wireless communication networks. Communication network 350 may exchange data over circuit-switched, packet-switched, and / or other types of channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of network 350 may not be important to the operation of the present disclosure unless explicitly described herein.

[0032]

[0066] 4 illustrates the arrangement of a video encoder and a video decoder in a video streaming environment as an example of application of the disclosed subject matter. The disclosed subject matter may be equally applicable to other video applications including, for example, video conferencing, digital TV broadcasting, gaming, virtual reality, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0033]

[0067] The video streaming system may include a video capture subsystem 413, which may include a video source 401, such as a digital camera, for producing a stream of uncompressed video pictures or images 402. In one example, the stream of video pictures 402 includes samples recorded by the digital camera of the video source 401. The stream of video pictures 402, depicted as a thick line to emphasize its larger data volume when compared to the encoded video data 404 (or encoded video bitstream), can be processed by an electronic device 420 including a video encoder 403 coupled to the video source 401. The video encoder 403 can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter as described in more detail below. The encoded video data 404 (or encoded video bitstream 404), depicted as a thin line to emphasize its smaller data volume when compared to the stream of uncompressed video pictures 402, can be stored on the streaming server 405 for future use or directly on a downstream video device (not shown). One or more streaming client subsystems, such as client subsystems 406 and 408 of FIG. 4, can access streaming server 405 to retrieve copies 407 and 409 of encoded video data 404. Client subsystem 406 may include a video decoder 410, for example within electronic device 430. Video decoder 410 decodes incoming copy of encoded video data 407 and produces an outgoing stream of video pictures 411 that is uncompressed and can be rendered to a display 412 (e.g., a display screen) or other rendering device (not shown). Video decoder 410 may be configured to perform some or all of the various functions described in this disclosure.In some streaming systems, the encoded video data 404, 407, and 409 (e.g., video bitstreams) may be encoded according to a particular video encoding / compression standard. Examples of these standards include ITU-T Recommendation H.265. In one example, a developing video encoding standard is informally known as Versatile Video Coding (VVC). The disclosed subject matter may also be used in the context of VVC, as well as other video encoding standards.

[0034]

[0068] It should be noted that electronic devices 420 and 430 may include other components (not shown). For example, electronic device 420 may include a video decoder (not shown), and electronic device 430 may similarly include a video encoder (not shown).

[0035]

[0069] 5 shows a block diagram of a video decoder 510 according to any of the embodiments of the present disclosure described below. The video decoder 510 may be included in an electronic device 530. The electronic device 530 may include a receiver 531 (e.g., receiving circuitry). The video decoder 510 may be used in place of the video decoder 410 in the example of FIG. 4.

[0036]

[0070] The receiver 531 may receive one or more encoded video sequences to be decoded by the video decoder 510. In the same or another embodiment, one encoded video sequence may be decoded at a time, with the decoding of each encoded video sequence being independent of other encoded video sequences. Each video sequence may be associated with multiple video frames or images. The encoded video sequences may be received from a channel 501, which may be a hardware / software link to a storage device that stored the encoded video data or a streaming source that transmitted the encoded video data. The receiver 531 may receive the encoded video data with other data, such as encoded audio data and / or ancillary data streams, and the encoded video data may be forwarded to its respective processing circuitry (not shown). The receiver 531 may separate the encoded video sequences from other data. To prevent network jitter, a buffer memory 515 may be disposed between the receiver 531 and the entropy decoder / parser 520 (hereinafter “parser 520”). In certain applications, the buffer memory 515 may be implemented as part of the video decoder 510. In other applications, the buffer memory 515 can be external to and separate from the video decoder 510 (not shown). In still other applications, there may be a buffer memory (not shown) external to the video decoder 510, e.g., to prevent network jitter, and another additional buffer memory 515 internal to the video decoder 510, e.g., to handle playback timing. When the receiver 531 is receiving data from a store / forward device with sufficient bandwidth and controllability or from an isosynchronous network, the buffer memory 515 may not be needed or may be small. For use with best-effort packet networks such as the Internet, a sufficiently sized buffer memory 515 may be required, and its size may be quite large.Such buffer memory may be provided in an adaptable size and may be implemented at least in part in an operating system or similar element (not shown) external to the video decoder 510.

[0037]

[0071] The video decoder 510 may include a parser 520 for reconstructing symbols 521 from the encoded video sequence. These symbol categories include information used to manage the operation of the video decoder 510 and possibly information for controlling a rendering device such as a display 512 (e.g., a display screen), which may or may not be an integral part of the electronic device 530 but may be coupled to the electronic device 530 as shown in FIG. 5. The control information for the rendering device may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser 520 may parse / entropy decode the encoded video sequence received by the parser 520. The entropy coding of the encoded video sequence may be in accordance with a video coding technique or standard and may follow various principles, including variable length coding, Huffman coding, context-independent or non-context-independent arithmetic coding, etc. The parser 520 may extract a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder from the coded video sequence based on at least one parameter corresponding to the subgroup. The subgroup may include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. Additionally, the parser 520 may extract coded video sequence information such as transform coefficients (e.g., Fourier transform coefficients), quantization parameter values, motion vectors, etc.

[0038]

[0072] Parser 520 may perform entropy decoding / parsing operations on the video sequence received from buffer memory 515 to produce symbols 521 .

[0039]

[0073] The reconstruction of symbols 521 may require several different processing or functional units, depending on the type of coded video picture or portion thereof (inter and intra pictures, inter and intra blocks, etc.), as well as other factors. The units required, and how they are required, may be controlled by subgroup control information parsed from the coded video sequence by parser 520. The flow of such subgroup control information between parser 520 and several of the processing or functional units described below is not depicted for the sake of simplicity.

[0040]

[0074] In addition to the functional blocks already mentioned, the video decoder 510 can be conceptually subdivided into several functional units as described below. In a practical implementation operating under commercial constraints, many of these functional units may interact closely with each other and be at least partially integrated with each other. However, to clearly explain the various functions of the disclosed subject matter, a conceptual subdivision into functional units is adopted in the following disclosure.

[0041]

[0075] The first unit may include a scalar / inverse transform unit 551. The scalar / inverse transform unit 551 may receive quantized transform coefficients and control information from the parser 520, including information indicating which type of inverse transform to use, block size, quantization factors / parameters, quantization scaling matrices, and lies as symbols 521. The scalar / inverse transform unit 551 may output blocks having sample values ​​that can be input to an aggregator 555.

[0042]

[0076] In some cases, the output samples of the scaler / inverse transform unit 551 may be for intra-coded blocks, i.e., blocks that can use prediction information from a previously reconstructed portion of the current picture rather than using prediction information from a previously reconstructed picture. Such prediction information can be provided by the intra-picture prediction unit 552. In some cases, the intra-picture prediction unit 552 may generate blocks of the same size and shape as the block being reconstructed using surrounding block information already reconstructed and stored in the current picture buffer 558. The current picture buffer 558 may, for example, buffer a partially reconstructed current picture and / or a fully reconstructed current picture. In some implementations, the aggregator 555 may add the prediction information generated by the intra-prediction unit 552 to the output sample information as provided by the scaler / inverse transform unit 551, on a sample-by-sample basis.

[0043]

[0077] In other cases, the output samples of the scalar / inverse transform unit 551 may relate to an inter-coded and potentially motion-compensated block. In such cases, the motion-compensated prediction unit 553 may access the reference picture memory 557 to fetch samples used for inter-picture prediction. After motion-compensating the fetched samples according to the symbols 521 for the block, these samples may be added to the output of the scalar / inverse transform unit 551 by the aggregator 555 to generate the output sample information (the output of unit 551 may be referred to as residual samples or residual signals). The addresses in the reference picture memory 557 from which the motion-compensated prediction unit 553 fetches the prediction samples may be controlled by a motion vector, available to the motion-compensated prediction unit 553 in the form of the symbols 521, which may have, for example, X and Y components (shift) and a reference picture component (time). Motion compensation may further include interpolation of sample values ​​as fetched from reference picture memory 557 when sub-sample accurate motion vectors are in use, may be further associated with a motion vector prediction mechanism, etc.

[0044]

[0078] The output samples of aggregator 555 may be subjected to various loop filtering techniques in loop filter unit 556. Video compression techniques can include in-loop filter techniques, controlled by parameters contained in the coded video sequence (also called coded video bitstream) and made available to loop filter unit 556 as symbols 521 from parser 520, but can also include those that respond to meta-information obtained during decoding of a coded picture or previous portion of the coded video sequence (in decoding order), and those that respond to previously reconstructed, loop-filtered sample values. As described in more detail below, several types of loop filters may be included as part of loop filter unit 556, in various orders.

[0045]

[0079] The output of the loop filter unit 556 may be a sample stream that may be output to the rendering device 512 or stored in the reference picture memory 557 for use during future inter-picture prediction.

[0046]

[0080] A particular coded picture can be used as a reference picture for future inter-picture prediction as soon as it is fully reconstructed. For example, once the coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by parser 520), the current picture buffer 558 can become part of reference picture memory 557, and an unused current picture buffer can be reallocated before starting reconstruction of the next coded picture.

[0047]

[0081] The video decoder 510 may perform decoding operations according to a predetermined video compression technique adopted in a standard such as ITU-T Recommendation H.265. An encoded video sequence may conform to the syntax specified by the video compression technique or standard used in the sense that the encoded video sequence conforms to both the syntax of the video compression technique or standard and the profile as documented in the video compression technique or standard. Specifically, a profile may select a particular tool from all tools available in the video compression technique or standard as the only tool available for use under this profile. To be standard-compliant, the complexity of the encoded video sequence may be within a range as defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstructed sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. In some cases, the limits set by the levels can be further restricted through the Hypothetical Reference Decoder (HRD) specification and metadata for HRD buffer management signaled in the coded video sequence.

[0048]

[0082] In some example embodiments, the receiver 531 may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence. The additional data may be used by the video decoder 510 to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0049]

[0083] 6 shows a block diagram of a video encoder 603 according to an example embodiment of the present disclosure. The video encoder 603 may be included in an electronic device 620. The electronic device 620 may further include a transmitter 640 (e.g., transmission circuitry). The video encoder 603 can be used in place of the video encoder 403 in the example of FIG.

[0050]

[0084] Video encoder 603 may receive video samples from video source 601 (not part of electronic device 620 in the example of FIG. 6) that may capture video images to be encoded by video encoder 603. In another example, video source 601 may be provided as part of electronic device 620.

[0051]

[0085] The video source 601 may provide a source video sequence to be encoded by the video encoder 603 in the form of a digital video sample stream, which can be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, etc.), any color space (e.g., BT.601 YCrCb, RGB, XYZ, etc.), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source 601 may be a storage device capable of storing previously prepared video. In a video conferencing system, the video source 601 may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures or images that convey motion when viewed in sequence. The picture itself may be organized as a spatial array of pixels, and each pixel may have one or more samples, depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following description focuses on samples.

[0052]

[0086] According to some example embodiments, the video encoder 603 may encode and compress pictures of a source video sequence into an encoded video sequence 643 in real time or under any other time constraints, as required by the application. Enforcing an appropriate encoding speed constitutes one function of the controller 650. In some embodiments, the controller 650 may be functionally coupled to and control other functional units, such as those described below. The coupling is not depicted for the sake of simplicity. Parameters set by the controller 650 may include rate control-related parameters (picture skip, quantizer, lambda value for rate-distortion optimization techniques, etc.), picture size, group-of-picture (GOP) layout, maximum motion vector search range, etc. The controller 650 can be configured to have other appropriate functions for the video encoder 603 optimized for a particular system design.

[0053]

[0087] In some example embodiments, the video encoder 603 may be configured to operate within an encoding loop. As an overly simplified explanation, in one example, the encoding loop may include a source coder 630 (e.g., responsible for producing symbols, such as a symbol stream, based on an input picture to be encoded and a reference picture) and a (local) decoder 633 embedded in the video encoder 603. The decoder 633 reconstructs the symbols to produce sample data in a similar manner as a (remote) decoder would have produced it even if the embedded decoder 633 had processed the video stream encoded by the source coder 630 without entropy encoding (since any compression between the symbols and the encoded video bitstream in entropy encoding may be lossless in the video compression techniques considered in the disclosed subject matter). The reconstructed sample stream (sample data) is input to a reference picture memory 634. Because decoding the symbol stream results in bit-exact results independent of the decoder location (local or remote), the contents of the reference picture memory 634 are also bit-exact between the local and remote encoders. In other words, the prediction portion of the encoder "sees" the reference picture sampling exactly the same sample values ​​that the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, e.g., due to channel errors) is used to improve coding quality.

[0054]

[0088] The operation of the "local" decoder 633 may be the same as a "remote" decoder, such as the video decoder 510 already described in detail above in conjunction with Figure 5. Furthermore, and briefly referring to Figure 5, however, the entropy decoding portion of the video decoder 510, including the buffer memory 515 and the parser 520, may not be entirely implemented in the local decoder 633 within the encoder, because symbols are available and the encoding / decoding of the symbols into an encoded video sequence by the entropy coder 645 and the parser 520 may be lossless.

[0055]

[0089] An observation that can be made at this point is that any decoder technology other than parsing / entropy decoding, which may only be present in the decoder, may not necessarily be present in substantially identical functional form in the corresponding encoder. For this reason, the disclosed subject matter may sometimes focus on decoder operations in cooperation with the decoding portion of the encoder. Therefore, a description of the encoder technology may be omitted, as it is the reverse of the decoder technology, which is described generically. Only in certain areas or aspects is a more detailed description of the encoder provided below.

[0056]

[0090] During operation in some example implementations, the source coder 630 may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine 632 codes color channel differences (or residuals) between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as prediction references for the input picture. The terms “residual” and its adjective form “residual” are sometimes used interchangeably.

[0057]

[0091] The local video decoder 633 may decode the encoded video data of pictures that may be designated as reference pictures based on the symbols produced by the source coder 630. The operation of the encoding engine 632 may advantageously be a lossy process. When the encoded video data may be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may be a copy of the source video sequence, typically containing some errors. The local video decoder 633 may replicate the decoding process that may be performed on reference pictures by the video decoder and store the reconstructed reference pictures in the reference picture cache 634. In this way, the video encoder 603 may locally store copies of reconstructed reference pictures that have content in common with reconstructed reference pictures that will be retrieved by a far-end (remote) video decoder (without transmission errors).

[0058]

[0092] The predictor 635 may perform a prediction search for the coding engine 632. That is, to encode a new picture, the predictor 635 may search the reference picture memory 634 for sample data (as candidate reference pixel blocks) or for specific metadata, such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new picture. The predictor 635 may operate on a sample block-by-pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor 635, the input picture may have prediction references created from multiple reference pictures stored in the reference picture memory 634.

[0059]

[0093] Controller 650 may manage the encoding operations of source coder 630, including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0060]

[0094] The output of all the aforementioned functional units may undergo entropy coding in entropy coder 645. Entropy coder 645 turns the symbols as produced by the various functional units into an encoded video sequence by lossless compression of the symbols through techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0061]

[0095] The transmitter 640 may buffer the encoded video sequence as produced by the entropy coder 645 in preparation for transmission over a communication channel 660, which may be a hardware / software link to a storage device that is to store the encoded video data. The transmitter 640 may merge the encoded video data from the video coder 603 with other data to be transmitted, such as, for example, encoded audio data and / or auxiliary data streams (sources not shown).

[0062]

[0096] The controller 650 may manage the operation of the video encoder 603. During encoding, the controller 650 may assign a particular coded picture type to each coded picture, which may affect the coding technique that may be applied to the respective picture. For example, pictures may often be assigned as one of the following picture types:

[0063]

[0097] An intra-picture (I-picture) may be coded and decoded without using any other pictures in a sequence as a source of prediction. Some video codecs allow different types of intra-pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0064]

[0098] A predicted picture (P picture) may be coded and decoded using intra- or inter-prediction, using at most one motion vector and reference index to predict the sample values ​​of each block.

[0065]

[0099] Bidirectionally predicted pictures (B-pictures) may be coded and decoded using intra- or inter-prediction, using at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures can use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0066]

[0100] A source picture may generally be spatially subdivided into multiple sample coding blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and coded block by block. Blocks may be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the block's respective picture. For example, blocks of an I-picture may be non-predictively coded or predictively coded with reference to previously coded blocks of the same picture (spatial prediction or intra-prediction). Pixel blocks of a P-picture may be predictively coded via spatial prediction or temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture may be predictively coded via spatial prediction or temporal prediction with reference to one or two previously coded reference pictures. Source pictures or intermediate processed pictures may also be subdivided into other types of blocks for other purposes. The division of coding blocks and other types of blocks may or may not follow the same manner as described in more detail below.

[0067]

[0101] The video encoder 603 may perform encoding operations according to a predetermined video encoding technique or standard, such as ITU-T Recommendation H.265. During its operation, the video encoder 603 may perform various compression operations, including predictive encoding operations that exploit temporal and spatial redundancies in the input video sequence. Thus, the encoded video data may conform to a syntax specified by the video encoding technique or standard used.

[0068]

[0102] In some example embodiments, the transmitter 640 may transmit additional data along with the encoded video. The source coder 630 may include data such as portions of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0069]

[0103] Video may be captured as a time sequence of multiple source pictures (video pictures). Intra-picture prediction (often abbreviated to intra-prediction) exploits spatial correlations within a given picture, while inter-picture prediction exploits temporal or other correlations between pictures. For example, a particular picture being encoded / decoded, called the current picture, may be partitioned into blocks. When a block in the current picture resembles a reference block in a previously coded and still buffered reference picture in the video, it may be coded by a vector, called a motion vector. A motion vector points to a reference block within a reference picture and may have a third dimension that identifies the reference picture in cases where multiple reference pictures are in use.

[0070]

[0104] In some example embodiments, bi-prediction techniques can be used for inter-picture prediction. Such bi-prediction techniques use two reference pictures, such as a first reference picture and a second reference picture, that both advance the current picture in the video in decoding order (but may be in the past or future, respectively, in display order). A block in the current picture can be coded by a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block can be jointly predicted by combining the first reference block and the second reference block.

[0071]

[0105] Furthermore, to improve coding efficiency, merge mode techniques may be used during inter-picture prediction.

[0072]

[0106] According to some example embodiments of the present disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed in units of blocks. For example, pictures in a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs within a picture may have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU may include three parallel coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU may be recursively quadtree split into one or more coding units (CUs). For example, a 64×64 pixel CTU may be split into one CU of 64×64 pixels or four CUs of 32×32 pixels. One or more of the 32×32 blocks may each be further split into four CUs of 16×16 pixels. In some example embodiments, each CU may be analyzed during encoding to determine a prediction type for the CU from various prediction types, such as an inter-prediction type or an intra-prediction type. The CU may be divided into one or more prediction units (PUs) according to temporal and / or spatial predictability. Generally, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations during coding (encoding / decoding) are performed in units of prediction blocks. The division of a CU into PUs (or PBs of different color channels) may be performed in various spatial patterns. For example, a luma or chroma PB may include a matrix of sample values ​​(e.g., luma values), such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 samples, etc.

[0073]

[0107] 7 shows a diagram of a video encoder 703 according to another example embodiment of this disclosure. The video encoder 703 is configured to receive a processed block (e.g., a predictive block) of sample values ​​in a current video picture in a sequence of video pictures and encode the processed block into an encoded picture that is part of an encoded video sequence. The example video encoder 703 may be used in place of the example video encoder (403) of FIG. 4.

[0074]

[0108] For example, the video encoder 703 receives a matrix of sample values ​​for a processing block, such as a predicted block of 8x8 samples. The video encoder 703 then determines, for example, using rate-distortion optimization (RDO), whether the processing block is best coded using intra mode, inter mode, or bi-predictive mode. When it is determined that the processing block is coded in intra mode, the video encoder 703 may encode the processing block into a coded picture using intra prediction techniques, and when it is determined that the processing block is coded in inter mode or bi-predictive mode, the video encoder 703 may encode the processing block into a coded picture using inter prediction or bi-predictive techniques, respectively. In some example embodiments, a merge mode may be used as a sub-mode of inter-picture prediction, in which a motion vector is derived from one or more motion vector predictors without the benefit of coded motion vector components outside the predictors. In some other example embodiments, there may be motion vector components applicable to the subject block. Thus, the video encoder 703 may include components not explicitly shown in FIG. 7, such as a mode decision module, to determine the prediction mode of a processing block.

[0075]

[0109] In the example of FIG. 7, the video encoder 703 includes an inter-encoder 730, an intra-encoder 722, a residual calculator 723, a switch 726, a residual encoder 724, a general controller 721, and an entropy encoder 725, coupled together as shown in the example arrangement of FIG. 7.

[0076]

[0110] The inter-encoder 730 is configured to receive a sample of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., blocks in previous and subsequent pictures in display order), generate inter-prediction information (e.g., a description of redundant information due to inter-coding techniques, motion vectors, merge mode information), and calculate an inter-prediction result (e.g., a predicted block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture decoded based on the encoded video information using a decoding unit 633 incorporated in the example encoder 620 of FIG. 6 (such as shown as residual decoder 728 of FIG. 7, as described in more detail below).

[0077]

[0111] The intra encoder 722 is configured to receive samples of a current block (e.g., a processing block), compare the block with previously coded blocks in the same picture, generate transformed quantized coefficients, and in some cases, further generate intra prediction information (e.g., intra prediction direction information according to one or more intra encoding techniques). The intra encoder 722 may calculate an intra prediction result (e.g., a predicted block) based on the intra prediction information and a reference block in the same picture.

[0078]

[0112] The general controller 721 may be configured to determine general control data and control other components of the video encoder 703 based on the general control data. In one example, the general controller 721 determines a prediction mode of the block and sends a control signal to the switch 726 based on the prediction mode. For example, when the prediction mode is intra-mode, the general controller 721 controls the switch 726 to select the intra-mode result for use by the residual calculator 723 and controls the entropy encoder 725 to select the intra-prediction information and include it in the bitstream; when the predicate mode of the block is inter-mode, the general controller 721 controls the switch 726 to select the inter-prediction result for use by the residual calculator 723 and controls the entropy encoder 725 to select the inter-prediction information and include it in the bitstream.

[0079]

[0113] The residual calculator 723 may be configured to calculate a difference (residual data) between the received block and a prediction result of the block selected from the intra encoder 722 or the inter encoder 730. The residual encoder 724 may be configured to encode the residual data to generate transform coefficients. For example, the residual encoder 724 may be configured to convert the residual data from the spatial domain to the frequency domain to generate the transform coefficients. The transform coefficients then undergo a quantization process to obtain quantized transform coefficients. In various illustrative embodiments, the video encoder 703 further includes a residual decoder 728. The residual decoder 728 is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be appropriately used by the intra encoder 722 and the inter encoder 730. For example, the inter encoder 730 can generate a decoded block based on the decoded residual data and the inter prediction information, and the intra encoder 722 can generate a decoded block based on the decoded residual data and the intra prediction information. The decoded blocks are suitably processed to generate a decoded picture, which is buffered in a memory circuit (not shown) and can be used as a reference picture.

[0080]

[0114] The entropy encoder 725 may be configured to format a bitstream to include the encoded block and perform entropy encoding. The entropy encoder 725 may be configured to include various information in the bitstream. For example, the entropy encoder 725 may be configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information in the bitstream. When encoding a block in an inter mode or a merged sub-mode of a bi-prediction mode, the residual information may be absent.

[0081]

[0115] 8 shows a diagram of an example video decoder 810 according to another embodiment of the present disclosure. The video decoder 810 is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder 810 may be used in place of the video decoder 410 in the example of FIG. 4.

[0082]

[0116] In the example of FIG. 8, the video decoder 810 includes an entropy decoder 871, an inter decoder 880, a residual decoder 873, a reconstruction module 874, and an intra decoder 872 linked together as shown in the example arrangement of FIG.

[0083]

[0117] The entropy decoder 871 can be configured to reconstruct, from the coded picture, certain symbols that represent syntax elements that constitute the coded picture. Such symbols can include, for example, prediction information (e.g., intra-mode, inter-mode, bi-prediction mode, merged sub-mode, or another sub-mode) that can identify the mode in which the block is coded, certain samples or metadata used for prediction by the intra decoder 872 or the inter decoder 880 (e.g., intra-prediction information or inter-prediction information), residual information in the form of, for example, quantized transform coefficients, etc. In one example, when the prediction mode is an inter- or bi-prediction mode, the inter-prediction information is provided to the inter decoder 880, and when the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra decoder 872. The residual information can undergo inverse quantization and is provided to the residual decoder 873.

[0084]

[0118] The inter decoder 880 may be configured to receive inter prediction information and generate inter prediction results based on the inter prediction information.

[0085]

[0119] The intra decoder 872 may be configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0086]

[0120] The residual decoder 873 may be configured to perform inverse quantization to extract dequantized transform coefficients, and process the dequantized transform coefficients to convert the residual from the frequency domain to the spatial domain. Additionally, the residual decoder 873 may utilize certain control information (to include quantization parameters (QPs)) that may be provided by the entropy decoder 871 (this may be only small data volume control information, so the data path is not depicted).

[0087]

[0121] The reconstruction module 874 may be configured to combine, in the spatial domain, the residual as output by the residual decoder 873 and the prediction result (as output by the inter- or intra-prediction module, as the case may be), to form a reconstructed block that forms part of the reconstructed picture as part of the reconstructed video. It should be noted that other appropriate operations, such as deblocking operations, may also be performed to improve visual quality.

[0088]

[0122] It should be noted that the video encoders 403, 603, and 703 and the video decoders 410, 510, and 810 may be implemented using any suitable technique. In some example embodiments, the video encoders 403, 603, and 703 and the video decoders 410, 510, and 810 may be implemented using one or more integrated circuits. In another embodiment, the video encoders 403, 603, and 603 and the video decoders 410, 510, and 810 may be implemented using one or more processors executing software instructions.

[0089]

[0123] Turning to block partitioning for encoding and decoding, general partitioning may start with a base block and may follow a predefined set of rules, a specific pattern, a partition tree, or any partition structure or scheme. The partitioning may be hierarchical and recursive. After dividing or partitioning the base block according to any or a combination of the example partitioning procedures described below or other procedures, a final set of partitions or coding blocks may be obtained. Each of these partitions may be at one of various partitioning levels in the partitioning hierarchy and may be of various shapes. Each of the partitions may be referred to as a coding block (CB). For various example partitioning implementations described further below, each resulting CB may be of any allowed size and partitioning level. Such partitions are referred to as coding blocks because they may form the unit at which some basic encoding / decoding decisions may be made, and at which encoding / decoding parameters may be optimized, determined, and signaled in the encoded video bitstream. The highest or deepest level in the final partition represents the depth of the coding block partitioning structure of the tree. The coding blocks may be luma coding blocks or chroma coding blocks. The CB tree structure for each color is sometimes called the coding block tree (CBT).

[0090]

[0124] The coding blocks of all color channels may be collectively referred to as a coding unit (CU). The hierarchical structure of all color channels may be collectively referred to as a coding tree unit (CTU). The partitioning pattern or structure of the various color channels in a CTU may or may not be the same.

[0091]

[0125] In some implementations, the partition tree schemes or structures used for the luma and chroma channels may not need to be the same. In other words, the luma and chroma channels may have separate coding tree structures or patterns. Furthermore, whether the luma and chroma channels use the same or different coding partition tree structures, and whether the actual coding partition tree structure should be used, may depend on whether the coded slice is a P slice, a B slice, or an I slice. For example, for an I slice, the chroma and luma channels may have separate coding partition tree structures or coding partition tree structure modes, while for a P or B slice, the luma and chroma channels may share the same coding partition tree scheme. When separate coding partition tree structures or modes are applied, the luma channel may be partitioned into CBs by one coding partition tree structure, and the chroma channel may be partitioned into chroma CBs by another coding partition tree structure.

[0092]

[0126] In some example implementations, a predetermined partitioning pattern may be applied to the base block. As shown in FIG. 9 , an example four-way partition tree may start at a first predefined level (e.g., a 64×64 block level or other size as the base block size), and the base block may be hierarchically partitioned down to a predefined lowest level (e.g., a 4×4 level). For example, the base block may follow four predefined partitioning options or patterns indicated by 902, 904, 906, and 908, and the partition designated as R may have the same partitioning option indicated in FIG. 9 repeated at a lower scale down to the lowest level (e.g., a 4×4 level), thereby allowing recursive partitioning. In some implementations, additional restrictions may be applied to the partitioning scheme of FIG. 9 . In the implementation of FIG. 9 , rectangular partitions (e.g., 1:2 / 2:1 rectangular partitions) may be allowed but are not allowed to be recursive, while square partitions are allowed to be recursive. Partitioning according to FIG. 9 with recursion generates a final set of coding blocks, if necessary. A coding tree depth may be further defined to indicate the division depth from the root node or root block. For example, the coding tree depth of the root node or root block, e.g., a 64×64 block, may be set to 0, and after the root block is further divided once according to FIG. 9, the coding tree depth increases by 1. The maximum or deepest level from the 64×64 base block to the smallest 4×4 partition should be 4 (starting from level 0) for the above scheme. Such a partitioning scheme may be applied to one or more of the color channels. Each color channel may be independently partitioned according to the scheme of FIG. 9 (e.g., a partitioning pattern or option from among predefined patterns may be independently determined for each color channel at each hierarchical level).Alternatively, two or more of the color channels may share the same hierarchical pattern tree of FIG. 9 (e.g., the same partitioning pattern or option from among predefined patterns may be chosen for two or more color channels per hierarchical level).

[0093]

[0127] FIG. 10 illustrates another example predefined partitioning pattern that allows recursive partitioning to form a partitioning tree. As shown in FIG. 10, an example 10-way partitioning structure or pattern may be predefined. The root block may begin at a predefined level (e.g., from a base block at a 128x128 level or a 64x64 level). The example partitioning structure of FIG. 10 includes various 2:1 / 1:2 and 4:1 / 1:4 rectangular partitions. The partition type with three sub-partitions, designated 1002, 1004, 1006, and 1008 in the second row of FIG. 10, may be referred to as a "T-type" partition. The "T-type" partitions 1002, 1004, 1006, and 1008 may be referred to as a left T-type, upper T-type, right T-type, and lower T-type. In some example implementations, none of the rectangular partitions of FIG. 10 are allowed to be further subdivided. A coding tree depth may be further defined to indicate the division depth from the root node or root block. For example, the coding tree depth of a root node or root block, such as a 128x128 block, may be set to 0, and after the root block is further divided once according to FIG. 10, the coding tree depth increases by 1. In some implementations, only the all-square partitions of 1010 may be allowed to recursively partition to the next level of the partitioning tree according to the pattern of FIG. 10. In other words, recursive partitioning may not be allowed for the square partitions in the T-type patterns 1002, 1004, 1006, and 1008. The partitioning procedure according to FIG. 10 with recursion generates a final set of coding blocks, if necessary. Such a scheme may be applied to one or more of the color channels. In some implementations, more flexibility may be added to the use of partitions below the 8x8 level. For example, 2x2 chroma inter prediction may be used in certain cases.

[0094]

[0128] In some other example implementations of coding block partitioning, a quadtree structure may be used to split a base block or an intermediate block into quadtree partitions. Such quadtree splitting may be applied hierarchically and recursively to any square-shaped partition. Whether a base block or an intermediate block or partition is further quadtree split may be adapted to various local characteristics of the base block or intermediate block / partition. The quadtree partitioning at picture boundaries may be further adapted. For example, implicit quadtree splitting may be implemented at picture boundaries so that a block continues to be quadtree split until its size fits the picture boundary.

[0095]

[0129] In some other example implementations, hierarchical binary partitioning from a base block may be used. In such a scheme, a base block or a mid-level block may be partitioned into two partitions. The partitioning may be horizontal or vertical. For example, horizontal partitioning may split a base block or a mid-level block into equal right and left partitions. Similarly, vertical partitioning may split a base block or a mid-level block into equal top and bottom partitions. Such partitioning may be hierarchical and recursive. For each base block or mid-level block, a determination may be made as to whether the binary partitioning scheme should continue and, if the scheme continues further, whether horizontal or vertical partitioning should be used. In some implementations, further partitioning may stop at a predefined minimum partition size (in one or both dimensions). Alternatively, further partitioning may stop as soon as a predefined partitioning level or depth from the base block is reached. In some implementations, the aspect ratio of the partitions may be limited. For example, the aspect ratio of a section may be no less than 1:4 (or no more than 4:1). Thus, a vertically elongated section with a vertical to horizontal aspect ratio of 4:1 may simply be further divided vertically into an upper section and a lower section, each with a vertical to horizontal aspect ratio of 2:1.

[0096]

[0130] In some other examples, a three-partitioning scheme may be used to partition the base block or any intermediate blocks, as shown in FIG. 13 . A three-partitioning scheme may be implemented vertically, as shown at 1302 in FIG. 13 , or horizontally, as shown at 1304 in FIG. 13 . The split ratio in the example in FIG. 13 is shown as 1:2:1 vertically or horizontally, but other ratios may be predefined. In some implementations, two or more different ratios may be predefined. Such a three-partitioning scheme may be used to complement a quadtree or two-partitioning structure, since such a ternary tree partitioning has the ability to capture an object at a block center within one contiguous partition, while quadtrees and two-partitions always split along the block center and therefore should split the object into separate partitions. In some implementations, the width and height of the partitions in the example ternary tree are always powers of two to avoid further transformations.

[0097]

[0131] The above partitioning schemes may be combined in any manner at different partitioning levels. As one example, the above-mentioned quadtree and binary partitioning schemes may be combined to partition a base block into a quadtree-binary-tree (QTBT) structure. In such a scheme, the base block or intermediate blocks / partitions may be quadtree-split or binary-split, if specified, subject to a set of predefined conditions. A specific example is illustrated in FIG. 14. In the example of FIG. 14, the base block is first quadtree-split into four partitions, as shown by 1402, 1404, 1406, and 1408. Each of the resulting partitions is then quadtree-partitioned into four further partitions (such as 1408), or binary-split into two further partitions at the next level (e.g., both symmetric, such as 1402 or 1406), or not split (such as 1404). Binary or quadtree splits may be recursively allowed for square-shaped partitions, as shown by the partition pattern of the overall example in 1410 and the corresponding tree structure / representation in 1420, where solid lines represent quadtree splits and dashed lines represent binary splits. A flag may be used at each binary split node (a non-leaf partition consisting of two) to indicate whether the binary split is horizontal or vertical. For example, as shown in 1420 in agreement with the partitioning structure in 1410, a flag "0" may represent a horizontal binary split and a flag "1" may represent a vertical binary split. In the case of quadtree split partitions, there is no need to indicate the split type, since a quadtree split always splits a block or partition both horizontally and vertically to produce four equal-sized sub-blocks / partitions. In some implementations, a flag "1" may represent a horizontal binary split and a flag "0" may represent a vertical binary split.

[0098]

[0132] In some example implementations of QTBT, the quadtree and binary splitting rule sets may be represented by the following predefined parameters and corresponding functions associated therewith: - CTU size: Root node size of the quadtree (size of the base block) - MinQTSize: Minimum allowable quadtree leaf node size - MaxBTSize: Maximum allowed binary tree root node size - MaxBTDepth: Maximum allowed binary tree depth - MinBTSize: The minimum allowable binary tree leaf node size In some example implementations of the QTBT partitioning structure, the CTU size may be set as 128x128 luma samples with two corresponding 64x64 blocks of chroma samples (when example chroma subsampling is considered and used), MinQTSize may be set as 16x16, MaxBTSize may be set as 64x64, MinBTSize (both width and height) may be set as 4x4, and MaxBTDepth may be set as 4. Quadtree partitioning may first be applied to the CTU to generate quadtree leaf nodes. The quadtree leaf nodes may have sizes from its minimum allowed size of 16x16 (i.e., MinQTSize) to 128x128 (i.e., CTU size). If the node is 128x128, it will not be initially split by the binary tree because its size exceeds MaxBTSize (i.e., 64x64). Otherwise, nodes that do not exceed MaxBTSize may be partitioned by a binary tree. In the example of FIG. 14, the base block is 128x128. According to a predefined rule set, the basic block can only be quadtree split. The base block has a partitioning depth of 0. Each of the resulting four partitions is 64x64, does not exceed MaxBTSize, and may undergo further quadtree or binary tree splits at level 1. The process continues. When the binary tree depth reaches MaxBTDepth (i.e., 4), further splits may not be considered. When a binary tree node has a width equal to MinBTSize (i.e., 4), further horizontal splits may not be considered. Similarly, when a binary tree node has a height equal to MinBTSize, further vertical splits are not considered.

[0099]

[0133] In some example implementations, the above QTBT scheme may be configured to support flexibility for luma and chroma to have the same or separate QTBT structures. For example, for P and B slices, the luma and chroma CTBs in one CTU may share the same QTBT structure. However, for an I slice, the luma CTB may be partitioned into CBs by a QTBT structure, and the chroma CTB may be partitioned into chroma CBs by a different QTBT structure. This means that CUs may be used to refer to different color channels in an I slice; for example, an I slice may consist of a coded block of a luma component or a coded block of two chroma components, while a CU in a P or B slice may consist of coded blocks of all three color components.

[0100]

[0134] In some other implementations, the QTBT scheme may be supplemented with the ternary scheme described above. Such implementations are sometimes referred to as multi-type tree (MTT) structures. For example, in addition to the binary division of nodes, one of the ternary partitioning patterns of FIG. 13 may be chosen. In some implementations, only square nodes may undergo ternary division. An additional flag may be used to indicate whether the ternary partitioning is horizontal or vertical.

[0101]

[0135] Two-level or multi-level tree designs, such as QTBT implementations, and QTBT implementations supplemented by three-partitions, may be primarily motivated by complexity reduction. Logically, the complexity of traversing a tree is T D where T represents the number of split types and D is the depth of the tree. Trade-offs may be made by using multiple types (T) while reducing the depth (D).

[0102]

[0136] In some implementations, the CB may be further partitioned. For example, the CB may be further partitioned into multiple prediction blocks (PBs) for intra- or inter-frame prediction during the encoding and decoding process. In other words, the CB may be further divided into different sub-partitions, and individual prediction decisions / configurations may be made. At the same time, the CB may be further partitioned into multiple transform blocks (TBs) to accurately represent the level at which a transform or inverse transform of video data is performed. The partitioning scheme of the CB into PBs and TBs may be the same or different. For example, each partitioning scheme may be implemented using a unique procedure based on, for example, various characteristics of the video data. The PB and TB partitioning schemes may be unrelated in some implementations. The PB and TB partitioning schemes and boundaries may be correlated in other implementations. In some implementations, for example, the TBs may be partitioned after the PB partitioning, and in particular, each PB may be determined following the partitioning of the coding block and then further partitioned into one or more TBs. For example, in some implementations, a PB may be split into one, two, four, or some other number of TBs.

[0103]

[0137] In some implementations, for the partitioning of base blocks into coding blocks and further into prediction blocks and / or transform blocks, the luma channel and the chroma channel may be treated separately. For example, in some implementations, partitioning of coding blocks into prediction blocks and / or transform blocks may be allowed for the luma channel, while such partitioning of coding blocks into prediction blocks and / or transform blocks may not be allowed for the chroma channel. In such implementations, transform and / or prediction of luma blocks may therefore only be performed at the coding block level. As another example, the minimum transform block size of the luma channel and the chroma channel may be different, e.g., coding blocks of the luma channel may be allowed to be partitioned into smaller transform and / or predictive blocks than the chroma channel. As yet another example, the maximum depth of partitioning of coding blocks into transform and / or predictive blocks may be different between the luma channel and the chroma channel, e.g., coding blocks of the luma channel may be allowed to be partitioned into deeper transform and / or predictive blocks than the chroma channel. As a specific example, a luma coding block may be partitioned into transform blocks of multiple sizes, which can be represented by recursive partitions down by up to two levels, and transform block shapes such as square, 2:1 / 1:2, and 4:1 / 1:4, as well as transform block sizes from 4x4 to 64x64 may be allowed. However, for chroma blocks, only the largest possible transform block specified for the luma block may be allowed.

[0104]

[0138] In some example implementations of the partitioning of coding blocks into PBs, the depth, shape, and / or other characteristics of the PB partitioning may depend on whether the PB is intra-coded or inter-coded.

[0105]

[0139] The partitioning of coding blocks (or prediction blocks) into transform blocks may be performed recursively or non-recursively in various illustrative schemes, including, but not limited to, quadtree division and predefined pattern division, and with further consideration of transform blocks at boundaries of coding blocks or prediction blocks. In general, the resulting transform blocks may be of different division levels, may not be the same size, and may not need to be square in shape (e.g., the resulting transform blocks can be rectangular with several allowed sizes and aspect ratios). Further examples are described in more detail below in connection with Figures 15, 16, and 17.

[0106]

[0140] However, in some other implementations, a CB obtained through any of the above partitioning schemes may be used as a basic or minimum coding block for prediction and / or transformation. In other words, no further division is performed for performing inter-prediction / intra-prediction and / or transformation. For example, a CB obtained from the above QTBT scheme may be directly used as a unit for performing prediction. Specifically, such a QTBT structure eliminates the concept of multiple partition types, i.e., such a QTBT structure eliminates the distinction between CUs, PUs, and TUs and supports more flexibility for the CU / CB partition shapes described above. In such a QTBT block structure, CUs / CBs can have square or rectangular shapes. Leaf nodes of such a QTBT are used as units for prediction and transformation processing without any further partitioning. This means that in such an example QTBT coding block structure, CUs, PUs, and TUs have the same block size.

[0107]

[0141] The various CB compartmentalization schemes described above, as well as further compartmentalization of the CB into PB and / or TB (not including PB / TB compartmentalization), may be combined in any manner. The following specific implementations are provided as non-limiting examples.

[0108]

[0142] Specific example implementations of coding block and transform block partitioning are described below. In such example implementations, a base block may be divided into coding blocks using recursive quadtree division or the predefined division patterns described above (such as those in FIGS. 9 and 10). For each level, local video data characteristics may determine whether further quadtree division of a particular partition should continue. The resulting CBs may be at various quadtree division levels and of various sizes. A decision as to whether a picture area should be coded using inter-picture (temporal) prediction or intra-picture (spatial) prediction may be made at the CB level (or at the CU level, for a total of three color channels). Each CB may be further divided into one, two, four, or other number of PBs according to a predefined PB division type. Within one PB, the same prediction process may be applied, and related information may be transmitted to the decoder for each PB. After obtaining the residual block by applying a prediction process based on the PB division type, the CB can be partitioned into TBs according to another quadtree structure similar to the coding tree of the CB. In this particular implementation, the CB or TB may be, but need not be, limited to a square shape. Furthermore, in this particular example, the PB may be square or rectangular for inter prediction, or only square for intra prediction. The coding block may be divided, for example, into four square-shaped TBs. Each TB may be further recursively divided (using quadtree division) into smaller TBs called residual quadtrees (RQTs).

[0109]

[0143] Another example implementation for partitioning a base block into CBs, PBs, and / or TBs is further described below. For example, rather than using multiple partition unit types such as those shown in FIG. 9 or FIG. 10, a quadtree with nested multitype trees using two- and three-split segmentation structures (e.g., QTBT or QTBT with three-split as described above) may be used. The separation of CBs, PBs, and TBs (i.e., partitioning CBs into PBs and / or TBs, and partitioning PBs into TBs) may be abandoned except when necessary for CBs with sizes too large for the maximum transform length; such CBs may require further partitioning. This example partitioning scheme may be designed to support more flexibility for CB partition shapes, so that both prediction and transformation can be performed on the CB level without further partitioning. In such a coding tree structure, the CBs may have a square or rectangular shape. Specifically, the coding tree block (CTB) may first be partitioned by a quadtree structure. The quadtree leaf nodes may then be further partitioned by a nested multitype tree structure. An example of a nested multitype tree structure using two or three splits is shown in FIG. 11. Specifically, the example multitype tree structure of FIG. 11 includes four split types, called vertical two-split (SPLIT_BT_VER) 1102, horizontal two-split (SPLIT_BT_HOR) 1104, vertical three-split (SPLIT_TT_VER) 1106, and horizontal three-split (SPLIT_TT_HOR) 1108. Therefore, CB corresponds to the leaf of the multitype tree. In this example implementation, as long as CB is not too large as the maximum transform length, this segmentation is used for both prediction and transform processing without any further partitioning. This means that in most cases, in a quadtree with a nested multitype tree coding block structure, CB, PB, and TB have the same block size. An exception occurs when the maximum supported transform length is smaller than the width or height of the color component of CB. In some implementations, in addition to the 2 or 3 splits, the nested pattern of FIG. 11 may further include a quadtree split.

[0110]

[0144] One specific example of a quadtree with a nested multitype tree coding block structure of block partitions (including quadtree, two-, and three-split options) for one base block is shown in FIG. 12. More specifically, FIG. 12 shows a base block 1200 quadtree split into four square partitions 1202, 1204, 1206, and 1208. The multitype tree structure of FIG. 11 and the decision to further use a quadtree for further splitting are made for each quadtree split partition. In the example of FIG. 12, partition 1204 is not further split. Partitions 1202 and 1208 each adopt another quadtree split. For partition 1202, the second-level quadtree-split top-left, top-right, bottom-left, and bottom-right partitions adopt a third-level split of the quadtree: horizontal two-split 1104 of FIG. 11, no split, and horizontal three-split 1108 of FIG. 11, respectively. Partition 1208 employs another quadtree splitting, with the second-level quadtree-split top-left, top-right, bottom-left, and bottom-right partitions employing a third-level split of vertical three-way split 1106 in FIG. 11 , no split, no split, and horizontal two-way split 1104 in FIG. 11 , respectively. Two of the subpartitions of the third-level top-left partition of 1208 are further split according to horizontal two-way split 1104 and horizontal three-way split 1108 in FIG. 11 , respectively. Partition 1206 employs a second-level splitting pattern according to vertical two-way split 1102 in FIG. 11 into two partitions that are further split according to horizontal three-way split 1108 and vertical two-way split 1102 in FIG. 11 at the third level. A fourth-level split is further applied to one of these according to horizontal two-way split 1104 in FIG. 11 .

[0111]

[0145] As a specific example of the above, the maximum luma transform size may be 64 × 64, and the maximum supported chroma transform size may be different from the luma, for example, at 32 × 32. Even if the CB in the above example of Figure 12 is not entirely further split into smaller PBs and / or TBs, when the width or height of the luma coding block or chroma coding block is greater than the maximum transform width or height, the luma coding block or chroma coding block may be automatically split in the horizontal and / or vertical directions to meet the transform size constraint in that direction.

[0112]

[0146] In the specific example of partitioning base blocks into CBs described above, the coding tree scheme may support the ability for luma and chroma to have separate block tree structures. For example, in the case of P and B slices, the luma and chroma CTBs in one CTU may share the same coding tree structure. In the case of an I slice, for example, the luma and chroma CTBs may have separate coding block tree structures. When separate block tree structures are applied, the luma CTB may be partitioned into luma CBs by one coding tree structure, and the chroma CTB is partitioned into chroma CBs by another coding tree structure. This means that a CU in an I slice may consist of a coding block of the luma component or a coding block of two chroma components, and that a CU in a P or B slice always consists of coding blocks of all three color components, unless the video is monochrome.

[0113]

[0147] When a coding block is further partitioned into multiple transform blocks, the transform blocks may be arranged in the bitstream according to various orders or scan patterns. Example implementations of partitioning coding or prediction blocks into transform blocks and the coding order of the transform blocks are described in further detail below. In some example implementations, as described above, the transform partitioning may support multiple shapes of transform blocks, such as 1:1 (square), 1:2 / 2:1, and 1:4 / 4:1, with transform block sizes ranging from 4×4 to 64×64, for example. In some implementations, when the coding block is smaller than or equal to 64×64, the transform block partitioning may be applied only to the luma component, and therefore, for chroma blocks, the transform block size is the same as the coding block size. Otherwise, if the coding block width or height is greater than 64, both the luma and chroma coding blocks may be implicitly split into multiples of min(W,64)×min(H,64) and min(W,32)×min(H,32), respectively.

[0114]

[0148] In some example implementations of transform block partitioning, for both intra- and inter-coded blocks, the coding blocks may be further partitioned into multiple transform blocks with a partition depth up to a predefined number of levels (e.g., two levels). The transform block partition depth and size may be related. For some example implementations, the mapping from the transform size of the current depth to the transform size of the next depth is shown below in Table 1.

[0115] [Table 1]

[0116]

[0149] Based on the example mapping in Table 1, for a 1:1 square block, the next level transform division may produce four 1:1 square sub-transform blocks. The transform partition may stop at, for example, 4x4. Thus, a transform size of 4x4 at the current depth corresponds to the same size of 4x4 at the next depth. In the example of Table 1, for a 1:2 / 2:1 non-square block, the next level transform division may produce two 1:1 square sub-transform blocks, while for a 1:4 / 4:1 non-square block, the next level transform division may produce two 1:2 / 2:1 sub-transform blocks.

[0117]

[0150] In some example implementations, additional restrictions on transform block partitioning may be applied to the luma component of an intra-coded block. For example, all sub-transform blocks at each level of transform partitioning may be restricted to be of equal size. For example, for a 32x16 coded block, level 1 transform partitioning creates two 16x16 sub-transform blocks, and level 2 transform partitioning creates eight 8x8 sub-transform blocks. In other words, second-level partitioning must be applied to all first-level sub-blocks to keep the transform units of equal size. An example of transform block partitioning for an intra-coded square block according to Table 1 is shown in Figure 15, with the coding order indicated by the arrows. Specifically, 1502 shows a square coded block. The first-level partitioning into four equal-sized transform blocks according to Table 1 is shown in 1504, with the coding order indicated by the arrows. All second-level partitioning of the first-level equal-sized block into 16 equal-sized transform blocks according to Table 1 is shown in 1506, with the coding order indicated by the arrows.

[0118]

[0151] In some example implementations, the above restrictions on intra-coding may not apply for the luma component of an inter-coded block. For example, after the first level of transform splitting, any one of the sub-transform blocks may be further split independently at another level. Thus, the resulting transform blocks may or may not be the same size. An example splitting of an inter-coded block into transform blocks in its coding order is shown in Figure 16. In the example of Figure 16, an inter-coded block 1602 is split into transform blocks at two levels according to Table 1. At the first level, the inter-coded block is split into four transform blocks of equal size. Then, only one of the four transform blocks (but not all of the transform blocks) is further split into four sub-transform blocks, resulting in a total of seven transform blocks with two different sizes, as indicated by 1604. The coding order of these seven transform blocks is indicated by the arrows at 1604 in Figure 16.

[0119]

[0152] In some example implementations, for chroma components, some additional restrictions on the transform blocks may apply: for example, for chroma components, the transform block size can be as large as the coding block size, but not smaller than a predefined size, such as 8x8.

[0120]

[0153] In some other example implementations, for coding blocks with width (W) or height (H) greater than 64, both luma and chroma coding blocks may be implicitly split into multiples of min(W,64) x min(H,64) and min(W,32) x min(H,32) transform units, respectively, where in this disclosure, "min(a,b)" may return the smaller value of a and b.

[0121]

[0154] Figure 17 further illustrates another alternative example scheme for partitioning a coding block or a prediction block into transform blocks. As shown in Figure 17, instead of using recursive transform partitioning, a predefined set of partitioning types may be applied to a coding block depending on the transform type of the coding block. In the particular example shown in Figure 17, one of six example partitioning types may be applied to split the coding block into various numbers of transform blocks. Such a scheme for generating transform block partitioning may be applied to a coding block or a prediction block.

[0122]

[0155] More specifically, the partitioning scheme of FIG. 17 provides up to six example partition types for any given transform type (transform type refers to the type of primary transform, such as ADST, for example). In this scheme, a transform partition type may be assigned for each coding block or predictive block, for example, based on rate-distortion cost. In an example, the transform partition type assigned to a coding block or predictive block may be determined based on the transform type of the coding block or predictive block. A particular transform partition type may correspond to a transform block division size and pattern, as shown by the six transform partition types illustrated in FIG. 17. The correspondence between various transform types and various transform partition types may be predefined. An example is shown below, where capitalized labels indicate transform partition types that may be assigned to a coding block or predictive block based on rate-distortion cost. ● PARTITION_NONE: Allocate a transformation size equal to the block size. ● PARTITION_SPLIT: Allocate a transformation size that is 1 / 2 the width of the block size and 1 / 2 the height of the block size. ● PARTITION_HORZ: Allocate a transform size that has the same width as the block size and 1 / 2 the height of the block size. ● PARTITION_VERT: Allocate a transformation size that has a width of 1 / 2 the block size and a height equal to the block size. ● PARTITION_HORZ4: Allocate a transformation size that has the same width as the block size and 1 / 4 of the height of the block size. ● PARTITION_VERT4: Allocates a transformation size that has a width of 1 / 4 of the block size and a height equal to the block size.

[0123]

[0156] In the above example, all transform partition types as shown in Figure 17 include uniform transform sizes for the partitioned transform blocks. This is by way of example only and not limitation. In some other implementations, mixed transform block sizes for the partitioned transform blocks of a particular partition type (or pattern) may be used.

[0124]

[0157] A video block (also referred to as a PB when not further partitioned into multiple predictive blocks, PB or CB) may be predicted in various manners rather than being directly encoded, thereby exploiting various correlations and redundancies in the video data to improve compression efficiency. Correspondingly, such prediction may be performed in various modes. For example, a video block may be predicted via intra prediction or inter prediction. In particular, in an inter prediction mode, a video block may be predicted by one or more other reference blocks or inter predictor blocks from one or more other frames via single-reference or mixed-reference inter prediction. For an inter prediction implementation, a reference block may be specified by its frame identifier (the temporal location of the reference block) and a motion vector that indicates the spatial offset between the current block being encoded or decoded and the reference block (the spatial location of the reference block). The reference frame identification and motion vector may be signaled in the bitstream. A motion vector, such as a spatial block offset, may be signaled directly or may itself be predicted by another reference motion vector or predictor motion vector. For example, the current motion vector may be predicted directly by a reference motion vector (e.g., of a candidate neighboring block) or by a combination of the reference motion vector and the motion vector difference (MVD) between the current and reference motion vectors. The latter is sometimes called merge mode with motion vector difference (MMVD). The reference motion vector may be identified in the bitstream, for example, as a pointer to a spatial neighboring block of the current block or a temporally neighboring but spatially aligned block.

[0125]

[0158] Returning to the intra-prediction process, samples within a block (e.g., a luma or chroma prediction block, or a coding block if not further divided into prediction blocks) are predicted by samples from neighboring, next-neighboring, or one or more other lines, or a combination thereof, to generate a prediction block. The residual between the actual block to be coded and the prediction block may then be processed through a transform before quantization. Various intra-prediction modes may be available, and parameters related to intra-mode selection and other parameters may be signaled in the bitstream. For example, the various intra-prediction modes may relate to one or more line positions for predicting samples, the direction in which prediction samples are selected from one or more prediction lines, and other special intra-prediction modes.

[0126]

[0159] For example, a set of intra-prediction modes (interchangeably referred to as "intra modes") may include a predefined number of directional intra-prediction modes. As described above with respect to the example implementation of FIG. 1, these intra-prediction modes may correspond to a predefined number of directions in which out-of-block samples are selected as predictions for samples predicted within a particular block. In another particular example implementation, eight (8) primary directional modes may be supported and predefined, corresponding to angles between 45 degrees and 207 degrees relative to the horizontal axis.

[0127]

[0160] In some other implementations of intra prediction, directional intra modes may be further expanded to a set of angles with finer granularity to further exploit a wider variety of spatial redundancies in directional textures. For example, the eight-angle implementation described above may be configured to provide eight nominal angles, designated V_PRED, H_PRED, D45_PRED, D135_PRED, D113_PRED, D157_PRED, D203_PRED, and D67_PRED, as illustrated in FIG. 19 , with a predefined number (e.g., seven) of finer angles added for each nominal angle. With such expansion, a larger total number (e.g., 56 in this example) of directional angles may be available for intra prediction corresponding to the same number of predefined directional intra modes. A prediction angle may be represented by a nominal intra angle and an angle delta. For the specific example described above with seven finer angle directions per nominal angle, the angle delta may be -3 to 3 multiplied by a step size of 3 degrees. Several as angle schemes may be used, such as that shown in FIG. 18 with 65 different prediction angles.

[0128]

[0161] In some implementations, a predefined number of non-directional intra-prediction modes may be predefined and made available as an alternative or addition to the above-described directional intra-modes. For example, five non-directional intra-modes called smooth intra-prediction modes may be specified. These non-directional intra-mode prediction modes may be specifically referred to as DC, PAETH, SMOOTH, SMOOTH_V, and SMOOTH_H intra-modes. Prediction of samples of a particular block under these example non-directional modes is illustrated in Figure 20. As an example, Figure 20 shows a 4x4 block 2002 predicted by samples from the top neighboring line and / or the left neighboring line. A particular sample 2010 in block 2002 may correspond to the immediately topmost sample 2004 of sample 2010 in the top-neighboring line of block 2002, the top-leftmost sample 2006 of sample 2010 as the intersection of the top- and left-neighboring lines, and the immediately leftmost sample 2008 of sample 2010 in the left-neighboring line of block 2002. For an example DC intra-prediction mode, the average of the left- and top-neighboring samples 2008 and 2004 may be used as a predictor for sample 2010. For an example PAETH intra-prediction mode, the top-most, left-most, and top-leftmost reference samples 2004, 2008, and 2006 may be fetched, and then any value among the nearest three reference samples (top-most + left-most - top-leftmost) may be set as a predictor for sample 2010. For the example SMOOTH_V intra-prediction mode, sample 2010 may be predicted by quadratic vertical interpolation of top-left most neighbor sample 2006 and left neighbor sample 2008. For the example SMOOTH_H intra-prediction mode, sample 2010 may be predicted by quadratic horizontal interpolation of top-left most neighbor sample 2006 and top-most neighbor sample 2004. For the example SMOOTH intra-prediction mode, sample 2010 may be predicted by an average of quadratic vertical and horizontal interpolations. The above non-directional intra-mode implementations are illustrated only as non-limiting examples.Other adjacent lines, and other non-directional selections of samples, and ways of combining prediction samples to predict a particular sample within a prediction block are also contemplated.

[0129]

[0162] The encoder's selection of a particular intra-prediction mode from the above directional or non-directional modes at various coding levels (picture, slice, block, unit, etc.) may be signaled in the bitstream. In some example implementations, eight exemplary nominal directional modes may be signaled first, along with five non-angle smooth modes (13 options in total). Then, if the signaled mode is one of the eight nominal angle intra-modes, an index is further signaled to indicate the selected angle delta relative to the corresponding signaled nominal angle. In some other example implementations, all intra-prediction modes may be indexed together for signaling (e.g., 56 directional modes and 5 non-directional modes to yield 61 intra-prediction modes).

[0130]

[0163] In some example implementations, the example 56 or other number of directional intra-prediction modes may be performed with an integrated directional predictor that projects each sample of a block to a reference sub-sample location, and interpolates the reference sample using a 2-tap bilinear filter.

[0131]

[0164] In some implementations, additional filter modes called FILTER INTRA modes may be designed to capture the attenuation of spatial correlation with references on edges. In these modes, predicted samples within a block, in addition to out-of-block samples, may be used as intra-prediction reference samples for some patches within the block. For example, these modes may be predefined and made available for intra-prediction for at least the luma block (or only the luma block). A predefined number (e.g., five) of filter intra modes may be predesigned, each represented by a set of n-tap filters (e.g., 7-tap filters) reflecting the correlation between samples in a 4x2 patch and its n neighbors. In other words, the weighting coefficients of the n-tap filters may be position-dependent. Taking an 8x8 block, a 4x2 patch, and 7-tap filtering as examples, the 8x8 block 2002 may be divided into eight 4x2 patches, as shown in Figure 21. These patches are designated in FIG. 21 as B0, B1, B1, B3, B4, B5, B6, and B7. For each patch, its seven neighbors, designated in FIG. 21 as R0 through R6, may be used to predict samples in the current patch. For patch B0, all of the neighbors may already be reconstructed. However, for other patches, some of the neighbors may be within the current block and therefore not reconstructed; therefore, the predicted values ​​of the immediate neighbors are used as a reference. For example, all of the neighbors of patch B7, as designated in FIG. 21, are not reconstructed, and therefore the predicted samples of the neighbors are used instead.

[0132]

[0165] In some implementations of intra prediction, one color component may be predicted using one or more other color components, which may be any one of the components in the YCrCb, RGB, XYZ color space, etc.

[0133]

[0166] One type of intra prediction, which predicts one color component using one or more other color components, is chroma-from-luma (CfL) prediction. In CfL prediction, a chroma component is predicted based on a luma component. The predicted chroma component may include a chroma block, which may include samples or chroma samples. The predicted samples are referred to as prediction samples. The predicted chroma block may also correspond to a luma block. In this specification, unless otherwise specified, a correspondence between a luma block and a chroma block refers to the chroma block being co-located with the luma block.

[0134]

[0167] Also, as used herein and as described in further detail below, the luma components used to predict the chroma components may include luma samples. The luma samples may include luma samples of the corresponding or co-located chroma block itself and / or may include neighboring luma samples, which are luma samples of one or more neighboring luma blocks next to or adjacent to the co-located luma block corresponding to the chroma block being predicted. Additionally, in at least some implementations, the luma samples used in the CfL prediction process may be reconstructed luma samples, which may be copies of the original luma samples derived or reconstructed from compressed versions of the original luma samples using a decoding process.

[0135]

[0168] In some implementations, an encoder (e.g., any of the encoders 403, 603, or 703) and / or a decoder (e.g., any of the decoders 410, 510, or 810) may be configured to perform CfL prediction via a CfL prediction process. Also, in at least some of these implementations, the encoder and / or decoder may be configured in a CfL prediction mode to perform the CfL prediction process. As described in further detail below, the encoder and / or decoder may be operable in at least one of a plurality of different CfL prediction modes. In different CfL prediction modes, the encoder and / or decoder may perform different respective CfL processes to generate chroma prediction samples.

[0136]

[0169] Referring to FIG. 22 , the encoder and / or decoder may include a CfL prediction unit 2202 configured to perform a CfL prediction process. In various implementations, the CfL prediction unit 2202 may be a standalone unit or a component or subunit of another unit of the encoder or decoder. For example, in any of various implementations, the CfL prediction unit 2202 may be a component or subunit of the intra prediction unit 552, the intra encoder 722, or the intra decoder 872. The CfL prediction unit 2202 may also be configured to perform various or multiple operations or functions to implement or execute the CfL process. For simplicity, the CfL prediction unit 2202 is described as a component of the encoder and / or decoder that performs each of these operations or functions. However, in any of various implementations, the CfL prediction unit 2202 may be further configured or organized into multiple subunits, each configured to perform one or more of the operations or functions of the CfL prediction process, or one or more units separate from the CfL prediction unit 2202 may be configured to perform one or more operations of the CfL prediction process. Also, in any of various implementations, the CfL prediction unit 2202 may be implemented in hardware or a combination of hardware and software to implement and / or perform the operations or functions of the CfL process. For example, the CfL prediction unit may be implemented as an integrated circuit, a processor configured to execute software or firmware stored in memory, or a combination thereof. Also, in any of various implementations, a non-transitory computer-readable storage medium may store computer instructions executable by a processor to implement the functions or operations of the CfL prediction unit 2202.

[0137]

[0170] For CfL prediction, the encoder and / or decoder may determine, such as via the CfL prediction unit 2202, that a CfL prediction mode should be applied to at least one luma block in a received video sequence and / or bitstream. The CfL prediction unit 2202 may then perform CfL prediction on the at least one luma block according to the determined CfL prediction mode to be applied to generate the at least one predicted chroma block. Correspondingly, the encoder and / or decoder, such as via the CfL prediction unit 2202, may encode or reconstruct at least one chroma block that corresponds to or is co-located with the at least one luma block, at least in part through application of the CfL prediction mode.

[0138]

[0171] Further, in general, the CfL prediction unit 2202 may operate in a CfL prediction mode. Referring to FIG. 22, in the CfL prediction mode, the CfL prediction unit 2202 may be configured to generate multiple prediction samples of at least one original or input chroma block that corresponds to or is co-located with at least one original or input luma block. The CfL prediction unit 2202 may generate a predicted chroma block for the original / input chroma block based on the original / input luma block that is co-located with the original / input chroma block and / or based on neighboring luma samples that are next to the co-located original / input luma block. Also, chroma prediction samples that constitute or are part of a chroma block are referred to as predicted chroma blocks. Furthermore, as used herein, the term “original” as used in conjunction with color (e.g., luma and chroma) components, such as pixels, samples, and blocks, refers to components that are not predicted by the prediction process, etc., and / or components that may be used as input to the prediction process to generate prediction samples.

[0139]

[0172] Figure 31 shows a schematic diagram of an example luma block 3102 and neighboring luma samples of the luma block 3102. In general, neighboring luma samples of a given luma block are luma samples of or within adjacent or neighboring luma blocks adjacent and / or next to the given luma block. Each neighboring luma sample may be or have a particular type among multiple types of neighboring luma samples. Each type may correspond to a relative spatial relationship with the given luma block. Similarly, each adjacent or neighboring luma block may have a particular type that matches the particular type of the neighboring luma sample it contains. In at least some implementations, the multiple types of neighboring luma samples and / or blocks may include left, top-left, top, top-right, right, bottom-right, bottom, and bottom-left. Figure 31 illustrates how neighboring luma samples may be spatially positioned relative to a given luma block 3102, including a left-neighboring luma sample 3104 in the left-neighboring luma block, an upper-left-neighboring luma sample 3106 in the upper-left-neighboring luma block, an upper-neighboring luma sample 3108 in the upper-neighboring luma block, an upper-right-neighboring luma sample 3110 in the upper-right-neighboring luma block, a right-neighboring luma sample 3112 in the right-neighboring luma block, a bottom-right-neighboring luma sample 3114 in the bottom-right-neighboring luma block, a bottom-most luma sample 3116 in the bottom-most luma block, and a bottom-left-most neighboring luma sample 3118 in the bottom-left-neighboring luma block. The top-left, top-right, bottom-left, and bottom-right neighboring luma samples and blocks may also be collectively and / or collectively referred to as corner neighboring luma samples and blocks, respectively. In some implementations, the CfL prediction unit 2202 may use all types of neighboring luma samples, or at least one type but not all, when performing CfL prediction. Also, a chroma block may have neighboring chroma samples in neighboring chroma blocks, similar to the luma block and its neighboring luma samples in Figure 31.

[0140]

[0173] Figure 23A shows a flow diagram of an example method 2300 of a CfL prediction process that the CfL prediction unit 2202 may perform in a first CfL prediction mode. In some implementations, a CfL prediction process such as that shown in Figure 23 generates multiple chroma prediction samples based on alternating current (AC) contributions of luma samples and direct current (DC) contributions of chroma samples. Each of the AC and DC contributions may be predictions of chroma components and are also referred to as AC and DC contribution predictions. In details of these implementations, the chroma prediction samples are modeled as a linear function of the luma samples, such as by the following mathematical formula: CfL(α)=α×L AC +DC (1) Here, L AC represents the AC contribution of the luma component (luma sample), α represents a scaling parameter of the linear model, and DC represents the DC contribution of the chroma component. In at least some implementations, the AC contribution is obtained for each sample of the block, while the DC contribution is obtained for the entire block. Furthermore, the scaling parameter α and the luma component L AC may be considered to be the AC contribution of the chroma prediction.

[0141]

[0174] In details of these implementations as shown in Figure 23A, in block 2302A, multiple original luma samples of a luma block co-located with a chroma block to be predicted may be downsampled (or subsampled) to a chroma resolution (e.g., 4:2:0, 4:2:2, or 4:4:4). As described in further detail below, the luma samples may be downsampled using or with a downsampling filter. In block 2304A, the downsampled luma samples may be averaged to generate a luma mean. In block 2306A, the luma mean may be subtracted from the luma samples of the original luma block to generate an AC contribution for the luma component. As shown in Figure 23A, the luma mean may be subtracted from the downsampled luma samples. In block 2308A, the AC contribution of the luma component may be multiplied by a scaling parameter α to generate a scaled AC contribution for the luma component. The scaled AC contribution of the luma component may also be a chroma AC contribution prediction for the chroma component. In block 2310A, the DC contribution prediction of the chroma component may be added to the AC contribution prediction of the chroma component to generate a chroma prediction sample for the predicted chroma block according to a linear model. In at least some implementations, the scaling parameter α may be based on the original chroma sample and signaled in the bitstream. This may reduce decoder complexity and result in a more accurate prediction. Additionally or alternatively, the DC contribution of the chroma component may be calculated using an intra-DC mode within the chroma component in some example implementations.

[0142]

[0175] Additionally, in some implementations of method 2300A or the first CfL mode, when some luma samples of a co-located luma block are outside a picture boundary, these luma samples may be padded, and the padded luma samples may be used to calculate a luma average, such as in block 2304A. Figure 24 shows a schematic diagram of luma samples inside and outside a picture defined by a picture boundary. In at least some implementations, outside picture luma samples may be padded by copying the value of the nearest available sample in the current block.

[0143]

[0176] Additionally or alternatively, in some implementations, when performing CfL prediction, the downsampling performed in block 2302 may be combined with the averaging performed in block 2304A and / or the subtraction performed in block 2306. This can then simplify the linear modeling equation while eliminating downsampling division and rounding errors. Equation (2) below corresponds to the combination of both steps and simplifies to equation (3). Both equations (2) and (3) use integer division. Also, M×N is a matrix of pixels in the luma plane.

[0144]

number

[0145]

[0177] Based on chroma subsampling, S x ×S y ∈{1,2,4}. Also, both M and N are powers of 2, and similarly M×N is a power of 2. For example, in the context of 4:2:0 chroma subsampling, instead of applying a box filter, the sum of four reconstructed luma pixels that match the chroma pixels may be used. As an example, a 4-tap {1 / 4, 1 / 4, 1 / 4, 1 / 4} filter is used to downsample the co-located luma samples to align the chroma resolution. Correspondingly, the CfL prediction may be scaled by a factor of 2.

[0146]

[0178] Figure 23B shows a flowchart of another example method 2300B of a CfL prediction process that the CfL prediction unit 2202 may perform in a CfL prediction mode. The CfL prediction process 2300B may be similar to the CfL prediction process 2300A of Figure 23A, except that instead of averaging luma samples of a luma block co-located with a chroma block to be predicted, the CfL prediction process 2300B may average neighboring luma samples of the co-located luma block to generate a neighboring luma average. Thus, the AC contributions, and similarly, the predicted chroma samples, are based on both the original luma samples of the co-located luma block and the neighboring luma samples next to the co-located luma block.

[0147]

[0179] More specifically, in block 2302B, multiple luma samples of an original co-located luma block that is co-located with the original chroma block to be predicted may be downsampled (or subsampled) to the chroma resolution. In block 2304B, multiple neighboring luma samples that are neighbors of the original co-located luma block may be downsampled to the chroma resolution. Additionally, in at least some implementations, the same downsampling (or subsampling) method or filter is used to subsample the original luma samples and neighboring luma samples of the co-located luma block (i.e., the same downsampling method or filter is applied in both blocks 2302B and 2304B). For example, if subsampling is performed according to the 4:2:0 format, two rows in the neighboring area above (e.g., area 3108 in FIG. 31), two columns in the neighboring area to the left (e.g., area 3104 in FIG. 31), and / or four pixels in the top-left area (e.g., area 3106 in FIG. 31) are downsampled (or subsampled). Correspondingly, when the CfL prediction unit 2202 determines the AC contribution (e.g., blocks 2306B and 2308B below), the neighboring luma samples may be averaged and subtracted from the downsampled luma sample value, as shown in FIG. 23B and further described below.

[0148]

[0180] More specifically, in block 2306B, the downsampled neighboring luma samples may be averaged to generate a neighboring luma average. In block 2808B, the neighboring luma average may be subtracted from the downsampled luma sample to generate an AC contribution for the luma component. In block 2810B, the AC contribution for the luma component may be multiplied by a scaling parameter α to generate a scaled AC contribution for the luma component. The scaled AC contribution for the luma component may also be an AC contribution prediction for the chroma component or an AC contribution for the predicted chroma sample. In block 2812B, the DC contribution prediction for the chroma component may be added to the AC contribution prediction for the chroma component to generate a chroma prediction sample, such as according to the linear model depicted above in equation (1). In at least some implementations, the scaling parameter α may be based on the original chroma sample and signaled in the bitstream. This can reduce decoder complexity and result in more accurate predictions. Additionally or alternatively, the DC contribution of the chroma component may be calculated using an internal DC mode within the chroma component in some example implementations.

[0149]

[0181] Furthermore, there may be different YUV formats according to different chroma downsampling phases. For example, Figure 25 shows different chroma downsampling formats. Different chroma formats may define different downsampling grids (phases) for different color components. For the 4:2:0 format, there may be two different downsampling formats, including 4:2:0 MPEG1 or 4:2:0 MPEG2, as shown in Figure 25.

[0150]

[0182] In some implementations, for the luma downsampling filter in AV1, equation (4) below is applied to derive the luma reconstructed samples.

[0151]

number

[0152]

[0183] The downsampling filter in AV1 may assume that the chroma downsampling format corresponds to the 4:2:0 MPEG1 downsampling format, as shown below in Figure 26. In some implementations, multiple downsampling filters may be supported. In at least some of these implementations, the filter type may be signaled, such as in a high-level syntax. Additionally or alternatively, the multiple filters may include one or more 4-tap filters, one or more 6-tap filters, and another 4-tap filter in AVI. Figures 27 and 28 show an example 6-tap filter and an example 4-tap filter, respectively.

[0153]

[0184] Equation (5) below is the mathematical equation used to determine the downsampled values ​​of a 6-tap filter, such as the 6-tap filter shown in FIG.

[0154]

number

[0155]

[0185] Equation (6) below is the mathematical equation used to determine the downsampled values ​​of a 4-tap filter, such as the 4-tap filter shown in FIG.

[0156]

number

[0157]

[0186] Additionally, in some implementations, an implicit CfL prediction method may be implemented. The difference between implicit CfL prediction and AV1 is that the scaling factor α is not explicitly signaled, for example, in the bitstream or sequence, but is instead implicitly derived. The neighboring reconstructed chroma sample and its corresponding (downsampled) luma sample are used to derive the scaling factor α. A least square error of Sum(RecC-α·RecY)2 may be applied. Furthermore, the division operations involved in the derivation process may be replaced with multiplication operations and lookup table operations, and the same lookup table as in the motion distortion process is used.

[0158]

[0187] Also, in some implementations, when CfL prediction is applied to a block, an additional flag may be signaled to indicate which CfL mode should be used. In one CfL mode, the DC in equation (1) is calculated based on the neighboring Y (downsampled) reconstructed luma samples to match the chroma sample locations for the DC calculation. This differs from the second CfL mode, in which the average of the current C (downsampled) reconstructed luma block is assigned to the DC.

[0159]

[0188] Furthermore, in some implementations with CfL prediction, only one downsampling filter is supported. However, for some content or different chroma downsampling formats, this only downsampling filter may not be the optimal filter. Also, even if multiple downsampling filters are supported, the encoder or decoder may not have the ability to choose the best or optimal downsampling filter for a given video sequence. The following provides a scheme for video processing involving determining a target downsampling filter from multiple downsampling filters.

[0160]

[0189] 29 is a flowchart of an example method 2900 of video processing involving determining a target downsampling filter. In various implementations, the actions performed to determine the downsampling filter may generally be referred to as a detection process or algorithm that detects or determines a target downsampling filter from among multiple downsampling filters. At least some implementations involve the target downsampling filter being a downsampling filter available to or supported by an encoder or decoder that results in the smallest or best error between the original color components (e.g., original chroma components) and the corresponding predicted color components (e.g., predicted chroma components). Additionally, although method 2900 is described as being performed by the CfL prediction unit 2200, other components or subcomponents, such as those of an encoder and / or decoder, may be used to perform one or more of the functions and / or actions of method 2900.

[0161]

[0190] In block 2902, the CfL prediction unit 2202 may receive at least one input chroma block from a video sequence. In block 2904, the CfL prediction unit 2202 may determine that the at least one input chroma block should be predicted in a CfL prediction mode. In block 2906, the CfL prediction unit 2202 may apply multiple downsampling filters to obtain multiple sets of downsampled luma samples corresponding to the at least one input chroma block, respectively. In block 2908, the CfL prediction unit 2202 may iteratively predict the input chroma block in the CfL prediction mode based on the multiple sets of downsampled luma samples. In block 2910, the CfL prediction unit 2202 may calculate multiple error scores or costs of the iterative prediction, each error score corresponding to a respective one of the multiple downsampling filters. In block 2912, the CfL prediction unit 2202 may select a target downsampling filter from among the multiple downsampling filters based on the multiple error scores. In block 2914, the CfL prediction unit 2202 may encode at least one original input chroma block in a CfL prediction mode by applying the selected target downsampling filter.

[0162]

[0191] 30 is a flowchart of an example method 3000 of video processing including selecting a target downsampling filter. Similar to method 2900, method 3000 is described as being performed by the CfL prediction unit 2200, although in other implementations, some or all of the functions or actions of method 3000 may be performed by another component or subcomponent of the encoder and / or decoder. Also, in some implementations, some or all of the actions or functions of method 3000 may be performed in combination with method 2900. For example, one or more of the actions performed in method 3000 may be performed as part of or for selecting the target downsampling filter in block 2904 of method 2900.

[0163]

[0192] More specifically, in block 3002, the CfL prediction unit 2202 may split or organize multiple original or input luma samples of a video sequence into multiple input luma blocks. In at least some implementations, the input luma samples may be within the same frame of the video sequence as each other, and correspondingly, the input luma blocks into which the original luma samples are split may be part of the same frame of the video sequence. In block 3004, the CfL prediction unit 2202 may downsample the multiple input luma blocks with multiple downsampling filters, respectively, to generate multiple sets of downsampled luma blocks, each set of the multiple sets of downsampled luma blocks corresponding to a respective one of the multiple downsampling filters. For example, the CfL prediction unit 2202 may downsample the multiple input luma blocks with a first downsampling filter to generate a first set of downsampled luma blocks, and may downsample the multiple input luma blocks with an Nth downsampling filter to generate an Nth set of downsampled luma blocks. In at least some implementations, the downsampling in block 3004 may be performed as part of or consistent with the CfL prediction process, as described in Figures 23A and 23B. For example, the downsampling performed in block 3004 may be part of the downsampling performed in blocks 2302A of Figure 23A and / or 2302B of Figure 23B.

[0164]

[0193] In block 3006, the CfL prediction unit 2202 may generate multiple sets of predicted chroma blocks according to the CfL prediction mode and based on the multiple sets of downsampled luma blocks, where each set of predicted chroma blocks corresponds to a respective one of the multiple downsampling filters. For example, the CfL prediction unit 2202 may generate the first set of predicted chroma blocks using the first set of downsampled luma blocks generated with the first downsampling filter, and may generate the Nth set of predicted chroma blocks using the Nth set of downsampled luma blocks generated with the Nth downsampling filter. Also, in at least some implementations, the CfL prediction unit may generate the multiple sets of predicted chroma blocks according to the CfL prediction process of Figure 23A and / or Figure 23B. For example, in block 3006, for each generated predicted chroma block, the CfL prediction unit 2202 may subtract a neighboring luma average (generated from a co-located luma sample, such as in block 2304A of Figure 23A, or from a neighboring luma sample, such as in block 2306B of Figure 23B) from the downsampled luma sample as performed in block 2306A or block 2308B to generate a luma AC contribution, multiply the luma AC contribution by a scaling factor α to generate the AC contribution of the predicted chroma block (as performed in block 2308A of Figure 23A or block 2310B of Figure 23B), and add the AC contribution of the predicted chroma block to the DC contribution of the predicted chroma block (as performed in block 2310A of Figure 23A or block 2312B of Figure 23B) to generate the predicted chroma block.

[0165]

[0194] Additionally or alternatively, in some implementations, the CfL prediction unit 2202 may perform iterative prediction in a CfL prediction mode at block 2908 by performing some or all of the actions at block 3006. For example, the iterative prediction performed by the CfL prediction unit 2202 at block 2908 may include generating a plurality of predicted chroma blocks corresponding to an input chroma block to be predicted in a CfL prediction mode, each of the plurality of predicted chroma blocks corresponding to a respective one of a plurality of downsampling filters. In some of these implementations, the CfL prediction unit 2202 may generate each of the plurality of predicted chroma blocks by adding together AC and DC contributions of the predicted chroma block, such as in accordance with the CfL prediction of FIG. 23A and / or 23B . To generate the AC contribution, the CfL prediction unit 2202 may subtract a neighboring luma mean from a downsampled input luma block that was downsampled from one of multiple downsampling filters to generate the luma AC contribution, and multiply the luma AC contribution by a scaling factor α, as previously described with reference to Figures 23A and / or 23B. The CfL prediction unit 2202 may iterate through this process for each of multiple downsampling filters to generate multiple predicted chroma blocks.

[0166]

[0195] At block 3008, the CfL prediction unit 2202 may determine or calculate multiple error scores or costs, each error score or cost corresponding to a respective set of predicted chroma blocks and / or a respective one of multiple downsampling filters. In at least some implementations, the CfL prediction unit 2202 may calculate the multiple error scores at block 2910 of method 2900 of FIG. 29 in response to the actions performed at block 3008 of method 3000 of FIG. 23. Additionally or alternatively, in at least some implementations, the CfL prediction unit 2202 may determine or calculate the multiple error scores or costs based on or in response to at least one of a sum of absolute differences (SAD) algorithm or mathematical formula, or a sum of squared differences (SSD) algorithm or mathematical formula. Equations (1) and (2) below are example SAD and SSD algorithms, respectively, that may be used to determine multiple error scores or costs in block 3008 and / or block 2910. Cost = Σabs(C p -C o )÷(width × height) (7) Cost = Σabs(C p -C o )×abs(C p -C o ))÷(Width × Height) (8) In at least some implementations, such as those using equation (7) and / or equation (8), an error score or cost may be determined for each predicted chroma block and corresponding original / input chroma block. For example, in equations (7) and (8), Cp represents a predicted sample of the predicted chroma block, and C0 represents a corresponding original / input sample of the corresponding original / input chroma block. An absolute difference (abs(Cp-C0)) or absolute squared difference (abs(Cp-C0) x abs(Cp-C0)) for the SAD is determined for each chroma sample, and the absolute difference or absolute squared difference of the chroma samples of the predicted and original chroma blocks are summed together and then normalized using the (width and height) of the chroma block to determine or generate a cost or error score specifically for or specific to the predicted chroma block. Thus, each of the multiple error scores generated in block 3008 may be an overall cost or error score for a particular downsampling filter based on the specific cost or error scores of the predicted chroma blocks generated from the particular downsampling filter. One or more mathematical operations may be used to generate an overall cost based on the multiple specific cost or error scores. For example, the specific costs or error scores may be added together to generate the overall cost or error score, although other mathematical operations may be possible.

[0167]

[0196] For illustrative purposes, assume that the original / input luma samples are divided into multiple input luma blocks that are downsampled using a first downsampling filter to generate a first set of downsampled luma blocks and then a first set of predicted chroma blocks according to a CfL prediction process, such as in Figures 23A and / or 23B. Multiple specific costs or error scores are determined for each of the predicted chroma blocks in the first set, such as using Equation (7), Equation (8), or another SAD or SSD algorithm. An overall error or cost of the first downsampling filter may then be determined based on the multiple specific costs or error scores. For example, the multiple specific costs or error scores may be summed together to generate an overall cost or error score for the first downsampling filter. This process may be repeated for each downsampling filter in block 3008 to generate multiple overall costs or error scores, each corresponding to a respective one of the downsampling filters.

[0168]

[0197] Also, in some implementations, an error score or cost of the downsampling filter may be determined for at least one of the blue chroma saturation component Cb or the red chroma saturation component Cr. In particular implementations, the error score or cost is determined for both Cb and Cr. For example, each set of the multiple sets of predicted chroma blocks may include a set of blue chroma predicted chroma blocks and a set of red chroma predicted chroma blocks. Correspondingly, a first cost or error score may be determined for the Cb chroma blocks, and a second cost or error score may be determined for the corresponding Cr chroma blocks. Based on a combination of the first cost / error score and the second cost / error score, a combined cost of the corresponding Cb and Cr chroma blocks may be determined. For example, one or more mathematical operations, such as, but not limited to, a summation operation or an average operation, may be performed on the first and second cost / error scores. Correspondingly, multiple combined cost / error scores for the multiple sets of Cb and Cr predicted chroma blocks are determined. An overall cost / error score for a particular downsampling filter may then be determined based on the multiple combined cost / error scores, e.g., the multiple combined cost / error scores may be added together to generate an overall cost or error score for the particular downsampling filter at block 3008, as a non-limiting example.

[0169]

[0198] In block 3010, the CfL prediction unit 2202 may identify an error score from the multiple error scores determined in block 3008. In at least some implementations, the error score identified by the CfL prediction unit 2202 in block 3010 is a best error score or a value indicating the minimum error between one or more input chroma blocks and one or more corresponding predicted chroma blocks. In block 3012, the CfL prediction unit 2202 may select a target downsampling filter from the multiple downsampling filters. The selected target downsampling filter corresponds to the error score identified in block 3010. That is, the target downsampling filter is the downsampling filter that provides the minimum error or cost between the original / input chroma block and the predicted chroma block.

[0170]

[0199] In some implementations, the CfL prediction unit 2202 may determine a target downsampling filter using only a single block size. For example, in block 3002, the CfL prediction unit 2202 may split the original luma samples into multiple luma blocks, each having the same predetermined block size, such as NxN, where non-limiting values ​​of N may include 8, 16, 32, and 64. In such implementations, the CfL prediction unit 2202 may determine an optimal downsampling filter for this single block size. In other implementations, the CfL prediction unit 2202 may vary the block size of the luma blocks over several iterations and determine an error score / cost for each different block size. For example, a first plurality of error scores may be determined for one luma block size, a second plurality of error scores may be determined for a second luma block size, and so on. In doing so, the CfL prediction unit 2202 can determine both a target or optimal downsampling filter from among multiple downsampling filters and an optimal block size for the luma block that results in the smallest or lowest cost or error between the original chroma block and the predicted chroma block for the video sequence.

[0171]

[0200] Additionally or alternatively, when generating the predicted chroma block, the scaling factor α used to multiply with the luma AC contribution may be determined in any of the various manners previously described with reference to Figures 23A and 23B, or using the implicit CfL method described above.

[0172]

[0201] Additionally or alternatively, when generating a predicted chroma block, the predicted DC contribution of a given predicted chroma block corresponding to the original / input chroma block may be, in any of various implementations, the average of the samples of the original / input chroma block, the average of the samples of neighboring chroma blocks, or a combination thereof.

[0173]

[0202] Additionally or alternatively, in some implementations, the CfL prediction unit 2202 may be configured to favor one or more downsampling filters over other downsampling filters among multiple downsampling filters available to the CfL prediction unit 2202. For example, the CfL prediction unit 2202 may be configured to bias one or more of the costs or error scores for one or more of the downsampling filters to increase the likelihood that the CfL prediction unit 2202 will select one of these one or more downsampling filters as the target downsampling filter. For example, the CfL prediction unit 2202 may multiply the error or cost by a scaling factor that is less than 1. As another example, the CfL prediction unit 2202 may subtract a value, such as a positive value, from the cost. In still other implementations, biasing may not be performed.

[0174]

[0203] Additionally or alternatively, a target downsampling filter may be selected based on CfL prediction for blocks of at least one frame of the video sequence. For example, the CfL prediction unit 2202 may determine the target downsampling filter using a first frame of the video sequence. As another example, the at least one frame may not include the first frame, or may include one or more frames other than or in addition to the first frame. For example, the CfL prediction unit 2202 may select one or more frames from the video sequence for which the target downsampling filter is to be selected. Additionally or alternatively, the CfL prediction unit 2202 may be configured to choose one or more particular key or intra-only picture frames of the video sequence for which the target downsampling filter is to be determined. In some implementations, the CfL prediction unit 2202 may select a target downsampling filter from multiple downsampling filters for each frame of the video sequence. Additionally or alternatively, in some implementations, the CfL prediction unit 2202 may select a target downsampling filter from among multiple downsampling filters for each intra-frame of a video sequence. Also, in at least some implementations, when the CfL prediction unit 2202 determines a target downsampling filter, the CfL prediction unit 2202 may use the target downsampling filter for chroma prediction in subsequent frames in the bitstream until it identifies a next frame (e.g., a next keyframe) for which it will determine a target downsampling filter.

[0175]

[0204] Additionally, although the above implementations of the detection algorithm are described with respect to CfL prediction, other implementations may implement the detection algorithm with one or more other prediction modes other than or in addition to the CfL prediction mode, non-limiting examples of which include a DC mode and a smooth mode. For example, an encoder and / or decoder may include a prediction unit that receives original samples of one or more color components and generates predicted samples of the one or more color components based on the original / input samples. The prediction unit may use a downsampling filter as part of the prediction process. Correspondingly, the prediction unit may perform a detection process to determine a target downsampling filter from among multiple downsampling filters. As part of the detection process, the prediction unit may use an algorithm such as a SAD algorithm (e.g., according to Equation (7)) and / or an SSD algorithm (e.g., according to Equation (8)) to determine a cost or error score between the original sample and the predicted sample, determine multiple cost or error scores, each corresponding to a respective downsampling filter, and then determine a target downsampling filter corresponding to the best or lowest cost or error.

[0176]

[0205] FIG. 32 is a flowchart of an example method 3200 of video processing including a multi-pass encoding method. In block 3202, an encoder may perform first-pass encoding on a video sequence. In block 3204, the encoder may perform second-pass encoding on a video sequence after the first-pass encoding has been performed on the video sequence. In at least some implementations, in block 3202, the encoder may apply first-pass encoding to a chroma plane using multiple downsampling filters, such as downsampling filters available to or that may be supported by the encoder. Additionally or alternatively, intra-only encoding may be used on the original / input luma samples to perform prediction (e.g., CfL prediction). Additionally or alternatively, only a limited number of intra-prediction modes, such as CfL mode, DC mode, or smooth mode, may be used during the first-pass encoding. The target downsampling filter determined during the first pass in block 3202 may be used as a downsampling filter for second-pass encoding applied to the video sequence after the first-pass encoding. Additionally, in any of various implementations, the first pass encoding and / or second pass encoding may include one or more of partitioning, intra and / or inter prediction, quantization, transform, or entropy coding estimation.

[0177]

[0206] The embodiments of the present disclosure may be used separately or in combination in any order. Furthermore, each of the method (or embodiment), encoder, and decoder may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium. The embodiments of the present disclosure may be applied to luma blocks or chroma blocks.

[0178]

[0207] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 33 illustrates a computer system 3300 suitable for implementing certain embodiments of the disclosed subject matter.

[0179]

[0208] Computer software can be coded using any suitable machine or computer language, which may be subjected to mechanisms such as assembling, compiling, linking, etc. to produce code comprising instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., either directly or through translation, microcode execution, etc.

[0180]

[0209] The instructions may be executable by various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, and the like.

[0181]

[0210] 33 for computer system 3300 are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system 3300.

[0182]

[0211] The computer system 3300 may include certain human interface input devices that can respond to input by one or more human users, for example, through tactile input (e.g., keystrokes, swipes, data glove movements), audio input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human interface devices can also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from a still image camera), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic video).

[0183]

[0212] The input human interface devices may include one or more (only one of each depicted) of a keyboard 3301, a mouse 3302, a trackpad 3303, a touchscreen 3310, a data glove (not shown), a joystick 3305, a microphone 3306, a scanner 3307, and a camera 3308.

[0184]

[0213] The computer system 3300 may further include certain human interface output devices. Such human interface output devices may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touchscreen 3310, data gloves (not shown), or joystick 3305, although haptic feedback devices that do not function as input devices are also possible), audio output devices (such as speakers 3309, headphones (not shown)), visual output devices (such as screens 3310, to include CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touchscreen input capability, and each with or without haptic feedback capability—some of which may be capable of outputting two-dimensional visual output or three- or more-dimensional output through means such as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0185]

[0214] The computer system 3300 may also include human-accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW 3320, including media 3321 such as CD / DVDs, thumb drives 3322, removable hard drives or solid state drives 3323, older magnetic media such as tape and floppy disks (not shown), and specialized ROM / ASIC / PLD-based devices (not shown) such as security dongles.

[0186]

[0215] Furthermore, those skilled in the art will understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transient signals.

[0187]

[0216] The computer system 3300 may also include an interface 3354 to one or more communication networks 3355. For example, the network may be wireless, wireline, optical. Furthermore, the network may be local, wide area, metropolitan, vehicular and industrial, real-time, delay-tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks to include GSM, 3G, 4G, 5G, LTE, etc., TV wireline or wireless wide area digital networks to include cable TV, satellite TV, and terrestrial television broadcast, vehicular and industrial to include CAN bus, etc. Particular networks generally require an external network interface adapter attached to a particular general-purpose data port or peripheral bus 3349 (e.g., a USB port on the computer system 3300), while others are generally integrated into the core of the computer system 3300 by attachment to a system bus as described below (e.g., an Ethernet interface on a PC computer system, or a cellular network interface on a smartphone computer system). Using any of these networks, the computer system 3300 may communicate with other entities. Such communications can be one-way receive-only (e.g., television broadcast), one-way transmit-only (e.g., CANbus to a specific CANbus device), or two-way, e.g., to other computer systems using local or wide-area digital networks. Specific protocols and protocol stacks are available for use with each of these networks and network interfaces, as described above.

[0188]

[0217] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to the core 3340 of the computer system 3300 .

[0189]

[0218] The core 3340 may include one or more central processing units (CPUs) 3341, graphics processing units (GPUs) 3342, specialized programmable processing units 3343 in the form of field programmable gate areas (FPGAs), hardware accelerators 3344 for specific tasks, graphics adapters 3350, etc. These devices may be connected through a system bus 3348, along with read-only memory (ROM) 3345, random access memory 3346, internal mass storage 3347 such as an internal non-user-accessible hard drive, SSD, etc. In some computer systems, the system bus 3348 may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus 3348 or through a peripheral bus 3349. In one example, a screen 3310 may be connected to the graphics adapter 3350. Peripheral bus architectures include PCI, USB, etc.

[0190]

[0219] The CPU 3341, GPU 3342, FPGA 3343, and accelerator 3344 can execute specific instructions that, in combination, can make up the aforementioned computer code. This computer code can be stored in ROM 3345 or RAM 3346. Temporary data can also be stored in RAM 3346, while permanent data can be stored, for example, in internal mass storage 3347. Fast storage and retrieval to and from any of the memory devices can be enabled through the use of cache memory, which can be closely associated with one or more of the CPU 3341, GPU 3342, mass storage 3347, ROM 3345, RAM 3346, etc.

[0191]

[0220] The computer-readable medium can bear computer code for performing various computer-executed operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those skilled in the computer software arts.

[0192]

[0221] As a non-limiting example, computer system 3300 having an architecture, and specifically core 3340, can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be user-accessible mass storage as introduced above, as well as media associated with specific storage of core 3340 that is non-transitory in nature, such as core internal mass storage 3347 or ROM 3345. Software that executes various embodiments of the present disclosure can be stored on such devices and executed by core 3340. Computer-readable media can include one or more memory devices or chips, depending on particular needs. Software can cause core 3340, and specifically the core's processor (including a CPU, GPU, FPGA, etc.), to execute specific processes, or specific portions of specific processes, described herein, including defining data structures stored in RAM 3346 and modifying such data structures in accordance with software-defined processes. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator 3344) that can operate in place of or together with software to perform particular processes, or particular portions of particular processes, described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass, where appropriate, circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry that embodies logic for execution, or both. The present disclosure encompasses any appropriate combination of hardware and software.

[0193]

[0222] The subject matter of the present disclosure may further specifically relate to or include the following aspects:

[0194]

[0223] In a first aspect, a method for video processing includes receiving an input chroma block from a video sequence; determining, for the video sequence, that the input chroma block should be predicted in a chroma-from-luma (CfL) prediction mode; applying a plurality of downsampling filters, respectively, to obtain a plurality of sets of downsampled luma samples corresponding to the input chroma block; iteratively predicting the input chroma block in the CfL prediction mode based on each of the plurality of sets of downsampled luma samples; calculating a plurality of error scores of the iterative prediction, each of the plurality of error scores corresponding to a respective one of the plurality of downsampling filters; selecting a target downsampling filter from the plurality of downsampling filters based on the plurality of error scores; and encoding the input chroma block in the CfL prediction mode by applying the selected target downsampling filter.

[0195]

[0224] A second aspect includes the first aspect, further including identifying a best error score from among the plurality of error scores.

[0196]

[0225] A third aspect includes the second aspect, further including that selecting the target downsampling filter includes selecting one downsampling filter from among the plurality of downsampling filters corresponding to the best error score as the target downsampling filter.

[0197]

[0226] A fourth aspect includes any of the first to third aspects, and further includes splitting a plurality of input luma samples of a frame of a video sequence into a plurality of input luma blocks; downsampling the plurality of luma blocks with each of a plurality of downsampling filters to generate a plurality of sets of downsampled luma blocks, where each set of downsampled luma blocks corresponds to a respective one of the plurality of downsampling filters; and generating a plurality of sets of predicted chroma blocks according to a CfL prediction mode and based on the plurality of sets of downsampled luma blocks, where each set of predicted chroma blocks corresponds to a respective one of the plurality of downsampling filters; and calculating a plurality of error scores includes calculating a plurality of error scores based on the plurality of sets of predicted chroma blocks and the plurality of input chroma blocks.

[0198]

[0227] A fifth aspect includes the fourth aspect, further including varying block sizes of the plurality of input luma blocks to calculate the plurality of error scores.

[0199]

[0228] A sixth aspect includes the fifth aspect, further including determining a best block size for the plurality of input luma blocks based on varying the block sizes.

[0200]

[0229] A seventh aspect includes any of the fourth through sixth aspects, and further includes: each set of the plurality of sets of predicted chroma blocks includes a set of blue chroma predicted chroma blocks and a set of red chroma predicted chroma blocks; and each error score of the plurality of error scores is based on a first error corresponding to the respective set of blue chroma predicted chroma blocks and a second error corresponding to the respective set of red chroma predicted chroma blocks.

[0201]

[0230] An eighth aspect includes any of the first to seventh aspects, and further includes calculating the plurality of errors includes calculating the plurality of errors based on a sum of absolute differences (SAD) algorithm or a sum of squared differences (SSD) algorithm.

[0202]

[0231] A ninth aspect includes any of the first through eighth aspects, further including biasing one of the plurality of error scores to increase the likelihood that a downsampling filter corresponding to one of the plurality of error scores is selected.

[0203]

[0232] A tenth aspect includes any of the first to ninth aspects, and further includes: the input chroma block is part of a first picture frame of the video sequence.

[0204]

[0233] An eleventh aspect includes any of the first through tenth aspects, and further includes the input chroma block being part of a selected picture frame of the video sequence.

[0205]

[0234] A twelfth aspect includes any of the first to eleventh aspects, further including selecting one or more predetermined keyframes of the video sequence that will result in selecting a corresponding target downsampling filter from among the plurality of downsampling filters.

[0206]

[0235] A thirteenth aspect includes any of the first to twelfth aspects, and further includes, for each frame of the video sequence, selecting a corresponding target downsampling filter from among the plurality of downsampling filters.

[0207]

[0236] A fourteenth aspect includes any of the first to thirteenth aspects, further including selecting a corresponding target downsampling filter for each intra-frame of the video sequence.

[0208]

[0237] In a fifteenth aspect, a method for video processing includes performing first pass encoding on a video sequence using a plurality of downsampling filters, determining a target downsampling filter from among the plurality of downsampling filters based on the first pass encoding, and performing second pass encoding on the video sequence after performing the first pass encoding using the target downsampling filter.

[0209]

[0238] A sixteenth aspect includes the fifteenth aspect, further including determining a best error score from among a plurality of error scores corresponding to a plurality of downsampling filters, wherein determining the target downsampling filter includes determining the target downsampling filter corresponding to the best error score.

[0210]

[0239] A seventeenth aspect includes any of the fifteenth or sixteenth aspects, and further includes that performing the first pass encoding includes performing CfL prediction on the video sequence using a plurality of downsampling filters.

[0211]

[0240] An eighteenth aspect includes an apparatus comprising a memory storing a plurality of instructions; and a processor configured to execute the plurality of instructions, the processor configured, upon execution of the plurality of instructions, to perform any of the first through seventeenth aspects.

[0212]

[0241] A nineteenth aspect includes a non-transitory computer-readable storage medium having stored thereon a plurality of instructions executable by a processor, the plurality of instructions being configured, when executed by the processor, to cause the processor to perform any of the first through seventeenth aspects.

[0213]

[0242] In addition to the features mentioned in each of the independent aspects listed above, some examples may exhibit optional features, alone or in combination, mentioned in the dependent aspects and / or as disclosed in the above description and illustrated in the figures.

[0214]

[0243] While this disclosure has described several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that are within the scope of this disclosure. It will thus be recognized that those skilled in the art will be capable of devising numerous systems and methods that, while not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure.

[0215] Appendix A: Acronyms JEM: Joint Exploration Model VVC: Versatile Video Coding BMS: Benchmark Set MV: Motion Vector HEVC: High Efficiency Video Coding SEI:Supplementary Enhancement Information VUI: Video Usability Information GOP: Group of Pictures TU: Conversion unit PU: Prediction Unit CTU: Coding Tree Unit CTB: coding tree block PB: Predicted Block HRD: Hypothetical Reference Decoder SNR: Signal to Noise Ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT: cathode ray tube LCD: Liquid crystal display OLED: Organic Light Emitting Diode CD: Compact Disc DVD: Digital Video Disc ROM: Read-only memory RAM: Random Access Memory ASIC: Application Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile Communications LTE: Long Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Area SSD: Solid State Drive IC: Integrated Circuit HDR: High Dynamic Range SDR: Standard Dynamic Range JVET: Joint Video Exploration Team MPM: Most Probable Mode WAIP: Wide-Angle Intra Prediction CU: Coding Unit PU: Prediction Unit TU: Conversion unit CTU: Coding Tree Unit PDPC: Position Dependent Prediction Combination ISP: Intra Sub-Partition SPS: Sequence Parameter Setting PPS: Picture Parameter Set APS: Calibration parameter set VPS: Video Parameter Set DPS: Decoding Parameter Set ALF: Adaptive Loop Filter SAO: Sample Adaptive Offset CC-ALF: Cross-Component Adaptive Loop Filter CDEF: Constrained Directional Enhancement Filter CCSO: Cross-Component Sample Offset LSO: Local Sample Offset LR: Loop Restoration Filter AV1:AOMedia Video 1 AV2:AOMedia Video 2 LFNST: Low Frequency Non-Separable Transform IST: Intra-quadratic transformation [Explanation of symbols]

[0216] 101 points, sample 102 Arrow 103 Arrow 104 Square Blocks 180 Schematic 201 Current Block 300 Communication Systems 350 Network 310 Terminal Devices 320 terminal devices 330 Terminal Devices 340 Terminal Devices 400 Communication Systems 401 Video Source 402 Video picture or image stream, video picture stream 403 Video Encoder 404 Encoded Video Data, Encoded Video Bitstream 405 Streaming Server 406 Client Subsystem 407 Copy of encoded video data, encoded video data 408 Client Subsystem 409 Copy of encoded video data, encoded video data 410 Video Decoder 411 Video Picture Stream 412 Display 413 Video Capture Subsystem 420 Electronic Devices 430 Electronic Devices 501 Channel 510 Video Decoder 512 displays, rendering devices 515 buffer memory 520 Entropy Decoder / Parser, Parser 521 Symbol 530 Electronic Devices 531 Receiver 551 Scaler / Inverse Transformation Unit, Scaler / Inverse Transformation, Unit 552 Intra-picture prediction unit, intra-prediction unit 553 Motion Compensation Prediction Unit 555 Aggregator 556 Loop Filter Unit 557 Reference Picture Memory 558 Current Picture Buffer 601 Video Sources 603 Video Encoder 620 Electronic Devices 630 Source Coder 632 encoding engine 633 decoder, local video decoder, (local) decoder, decoding unit 634 Reference Picture Memory, Reference Picture Cache 635 Predictor 640 Transmitter 643 coded video sequence 645 Entropy Coder 650 Controller 660 Communication Channels 703 Video Encoder 721 General Controller 722 Intra Encoder 723 Residual Calculator 724 Residual Encoder 725 Entropy Encoder 726 Switch 728 Residual Decoder 730 Interencoder 810 Video Decoder 871 Entropy Decoder 872 Intra Decoder 873 Residual Decoder 874 Reconstruction Module 880 Interdecoder 1002 "T-type" section, T-type pattern 1004 "T-type" section, T-type pattern 1006 "T-type" section, T-type pattern 1008 "T-type" section, T-type pattern 1010 All square plots 1102 Vertical binary split, SPLIT_BT_VER 1104 Horizontal binary split, SPLIT_BT_HOR 1106 Vertical split into three, SPLIT_TT_VER 1108 Horizontal split into three, SPLIT_TT_HOR 1202 Square Plot, Plot 1204 Square Plot, Plot 1206 Square Plot, Plot 1208 Square Plot, Plot 1602 Inter-coded blocks 3300 Computer Systems 3301 Keyboard 3302 Mouse 3303 Trackpad 3305 Joystick 3306 Microphone 3307 Scanner 3308 Camera 3309 Speaker 3310 Touchscreen, Screen 3320 CD / DVD ROM / RW 3321 CD / DVD and other media 3322 thumb drive 3323 Removable Hard Drive or Solid State Drive 3340 cores 3341 Central Processing Unit (CPU) 3342 Graphics Processing Unit (GPU) 3343 FPGA, Specialized Programmable Processing Unit 3344 Hardware Accelerator, Accelerator 3345 Read-Only Memory (ROM) 3346 Random Access Memory, RAM 3347 Internal Mass Storage, Mass Storage, Core Internal Mass Storage 3349 Peripheral bus 3350 graphics adapter 3354 Interface 3355 Communication Networks

Claims

1. 1. A method for video processing, comprising: receiving an input chroma block from a video sequence; determining that the input chroma blocks for the video sequence should be predicted in a chroma-from-luma (CfL) prediction mode; applying a plurality of downsampling filters, respectively, to obtain a plurality of sets of downsampled luma samples corresponding to the input chroma block; iteratively predicting the input chroma block based on each of the plurality of sets of downsampled luma samples in the CfL prediction mode; calculating a plurality of error scores of the iterative prediction, each of the plurality of error scores corresponding to a respective one of the plurality of downsampling filters; selecting a target downsampling filter from the plurality of downsampling filters based on the plurality of error scores; encoding the input chroma block in the CfL prediction mode by applying the selected target downsampling filter; A method comprising:

2. The method of claim 1 , further comprising identifying a best error score from among the plurality of error scores.

3. 3. The method of claim 2, wherein selecting the target downsampling filter comprises selecting one downsampling filter from the plurality of downsampling filters corresponding to the best error score as the target downsampling filter.

4. splitting a plurality of input luma samples of a frame of the sequence frames into a plurality of input luma blocks; downsampling the plurality of luma blocks with each of the plurality of downsampling filters to generate a plurality of sets of downsampled luma blocks, each set of downsampled luma blocks corresponding to a respective one of the plurality of downsampling filters; generating a plurality of sets of predicted chroma blocks according to the CfL prediction mode and based on the plurality of sets of downsampled luma blocks, each set of predicted chroma blocks corresponding to a respective one of the plurality of downsampling filters; further comprising calculating the plurality of error scores includes calculating the plurality of error scores based on the plurality of sets of predicted saturation blocks and the plurality of input saturation blocks; The method according to any one of claims 1 to 3.

5. The method of claim 4 , further comprising varying block sizes of the input luma blocks to calculate the error scores.

6. The method of claim 5 , further comprising determining a best block size for the plurality of input luma blocks based on the varying block size step.

7. 7. The method of claim 4, wherein each set of the plurality of sets of predicted chroma blocks comprises a set of blue chroma predicted chroma blocks and a set of red chroma predicted chroma blocks, and wherein each error score of the plurality of error scores is based on a first error corresponding to the respective set of blue chroma predicted chroma blocks and a second error corresponding to the respective set of red chroma predicted chroma blocks.

8. 8. The method according to claim 1, wherein the step of calculating the plurality of errors comprises calculating the plurality of errors based on a sum of absolute differences (SAD) algorithm or a sum of squared differences (SSD) algorithm.

9. 9. The method of claim 1, further comprising biasing one of the plurality of error scores to increase the likelihood that a downsampling filter corresponding to said one of the plurality of error scores will be selected.

10. The method according to any of claims 1 to 9, wherein the input chroma block is part of a first picture frame of the video sequence.

11. The method according to any of claims 1 to 10, wherein the input chroma block is part of a selected picture frame of the video sequence.

12. The method of any of claims 1 to 11, further comprising selecting one or more predetermined keyframes of the video sequence for which a corresponding target downsampling filter will be selected from among the plurality of downsampling filters.

13. The method according to any one of claims 1 to 12, further comprising the step of selecting, for each frame of the video sequence, a corresponding target downsampling filter from among the plurality of downsampling filters.

14. The method according to any of claims 1 to 13, further comprising the step of selecting, for each intra-frame of the video sequence, a corresponding target down-sampling filter from among the plurality of filters.

15. 1. A method for video processing, comprising: performing a first pass encoding on the video sequence using a plurality of downsampling filters; determining a target downsampling filter from among the plurality of downsampling filters based on the first pass encoding; performing a second pass encoding on the video sequence after performing the first pass encoding using the target downsampling filter; A method comprising:

16. 16. The method of claim 15, further comprising determining a best error score from among a plurality of error scores corresponding to the plurality of downsampling filters, wherein determining the target downsampling filter comprises determining the target downsampling filter that corresponds to the best error score.

17. The method of claim 15 or 16, wherein performing the first pass encoding comprises performing CfL prediction on the video sequence using the plurality of downsampling filters.

18. a memory storing a plurality of instructions; configured to execute the plurality of instructions, wherein upon execution of the plurality of instructions: receiving an input chroma block from a video bitstream; determining that the input chroma block should be predicted in a chroma-from-luma (CfL) prediction mode; applying a plurality of downsampling filters respectively to obtain a plurality of sets of downsampled luma samples corresponding to the input chroma block; iteratively predicting the input chroma block based on each of the plurality of sets of downsampled luma samples in the CfL prediction mode; calculating a plurality of error scores of the iterative prediction, each of the plurality of error scores corresponding to a respective one of the plurality of downsampling filters; selecting a target downsampling filter from among the plurality of downsampling filters based on the plurality of error scores; and encoding the input chroma block in the CfL prediction mode by applying the target downsampling filter; a processor configured to: An apparatus comprising:

19. 20. The apparatus of claim 18, wherein the processor, upon execution of the plurality of instructions, is further configured to identify a best error score from among the plurality of error scores.

20. 20. The apparatus of claim 19, wherein to select the target downsampling filter, the processor is configured, upon execution of the instructions, to select as the downsampling filter one downsampling filter from among the plurality of downsampling filters corresponding to the best error score.

Images