Mixed Model Cross-Component Prediction Mode

Abstract

A feature value is determined based on at least one of (i) reconstructed chroma samples in a neighborhood of a current chroma block, and (ii) reconstructed luma samples in a neighborhood of a luma block co-located with the current chroma block. Based on a threshold value of the feature value, the chroma samples of the current chroma block and the luma samples of the luma block co-located with the current chroma block are grouped into a plurality of groups. Each of the plurality of groups includes a respective chroma sample and a respective luma sample. A respective cross-component prediction mode for each of the plurality of groups is determined by comparing the respective chroma sample and the respective luma sample of each of the respective groups with the determined feature value. The current chroma block is reconstructed based on the determined cross-component prediction modes of the plurality of groups.

Description

[Technical Field]

[0001]

[0001] Incorporation by Reference This application claims priority to U.S. Patent Application No. 17 / 982,189, entitled "Mixed Model Cross-Component Prediction Mode," filed November 7, 2022, which in turn claims priority to U.S. Provisional Application No. 63 / 388,117, entitled "Mixed Model Cross-Component Prediction Mode," filed July 11, 2022. The disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002]

[0002] Technical Field This disclosure describes embodiments generally related to video coding. [Background technology]

[0003] The background discussion provided herein is intended to generally present the context of the present disclosure. Work under the names of the current inventors is not admitted, expressly or impliedly, as prior art to the present disclosure to the extent that that work is described in this background section or in a descriptive manner that may not otherwise qualify as prior art as of the filing date.

[0004]

[0004] Uncompressed digital images and / or video can include a series of pictures, each with spatial dimensions of, for example, 1920 x 1080 luminance samples and associated chrominance samples. The series of pictures may have a fixed or variable picture rate (informally known as the frame rate), for example, 60 pictures per second or 60 Hz. Uncompressed images and / or video have significant bitrate requirements. For example, 1080p60 4:2:0 video (1920 x 1080 luminance sample resolution at a 60 Hz frame rate) with 8 bits per sample requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 Gbytes of storage space.

[0005] One of the goals of image and / or video encoding and decoding is the reduction of redundancy in the input image and / or video signal through compression. Compression can help reduce the aforementioned bandwidth and / or storage space requirements, in some cases by more than two orders of magnitude. While the description herein uses video encoding / decoding as an illustrative example, the same techniques can be applied to image encoding / decoding in a similar manner without departing from the spirit of this disclosure. Both lossless and non-lossless compression, as well as combinations thereof, can be used. Lossless compression (reversible compression) refers to a technique in which an exact copy of the original signal can be reconstructed from a compressed version of the original signal. When non-lossless compression is used, the reconstructed signal may not be identical to the original signal, but the distortion between the original and reconstructed signal is small enough that the reconstructed signal is useful for the intended application. In the case of video, non-lossless compression is widely used. The amount of acceptable distortion depends on the application; for example, users of a particular consumer streaming application may be able to tolerate higher distortion than users of a television distribution application. The achievable compression ratio may reflect that a higher tolerable / tolerable distortion may result in a higher compression ratio.

[0006] Video encoders and decoders may utilize techniques in several broad categories, including, for example, motion compensation, transform processing, quantization, and entropy coding.

[0007] Video codec technology can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, pictures are spatially divided into blocks of samples. If all blocks of samples are coded in intra mode, the picture can be an intra-picture. Intra-pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and therefore can be used as the first picture in a coded video bitstream and video session, or as still images. Samples in intra-blocks are subjected to a transform, and the transform coefficients can be quantized before entropy coding. Intra-prediction can be a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after the transform and the smaller the AC coefficients, the fewer bits required for a given quantization step size to represent the block after entropy coding.

[0008]

[0008] Traditional intra-coding, such as that used in MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to perform prediction based on surrounding sample data and / or metadata obtained during the encoding and / or decoding of a block of data. Such techniques are hereinafter referred to as "intra-prediction" techniques. Note that, at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed, and does not use reference data from a reference picture.

[0009]

[0009] Many different forms of intra-prediction may exist. When more than one such technique may be used in a given video coding technique, the particular technique used may be coded as a particular intra-prediction mode that uses that particular technique. In certain cases, an intra-prediction mode may have submodes and / or parameters, which may be coded separately or included in a mode codeword, which defines the prediction mode used. The codeword used for a given mode, submode, and / or parameter combination may affect the coding efficiency gain from intra-prediction and may also affect the entropy coding technique used to convert the codeword into a bitstream.

[0010]

[0010] A specific mode of intra prediction was introduced in H.264, improved in H.265, and further refined in newer coding techniques such as JEM (joint exploration model), VVC (versatile video coding), and MBS (benchmark set). A prediction block can be formed using neighboring sample values ​​of already available samples. The sample values ​​of the neighboring samples are copied to the predictor block according to a certain direction. A reference to the direction to use can be coded in the bitstream or can be predicted itself.

[0011]

[0011] Referring to FIG. 1A, depicted at the bottom right is a subset of nine predictor directions known from the 33 possible predictor directions defined in H.265 (corresponding to the 33 angular modes of the 35 intra modes). The point where the arrows converge (101) represents the predicted sample. The arrows represent the direction from which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from a sample pointing to the upper right at a 45-degree angle from horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from a sample pointing to the lower left of sample (101) at a 22.5-degree angle from horizontal.

[0012]

[0012] Still referring to FIG. 1A, a square block (104) of 4x4 samples is depicted at the top left (indicated by the dashed bold line). The square block (104) contains 16 samples, each labeled with "S," its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in block (104) in both the Y and X dimensions. Because the block size is 4x4 samples, S44 is located at the bottom right. Also shown are reference samples that follow a similar numbering scheme. The reference samples are labeled with their Y position (e.g., row index) and X position (column index) relative to block (104) and R. In both H.264 and H.265, the prediction samples are in the neighborhood of the block being reconstructed, so there is no need to use negative values.

[0013]

[0013] Intra-picture prediction can be performed by copying reference sample values ​​from nearby samples indicated by the signaled prediction direction. For example, assume that the coded video bitstream includes signaling indicating a prediction direction consistent with arrow (102) for this block, i.e., that samples are predicted from samples pointing upward and to the right at a 45-degree angle from horizontal. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Thus, sample S44 is predicted from reference sample R08.

[0014] In certain cases, the values ​​of multiple reference samples can be combined, for example by interpolation, to calculate the reference sample; especially when the direction is not evenly divisible by 45 degrees.

[0015]

[0015] As video coding technology evolves, the number of possible directions increases. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013). Currently, JEM / VVC / BMS can support up to 65 directions. Experiments have been conducted to identify the most likely directions, using specific techniques in entropy coding to represent these more likely directions with fewer bits while accepting a certain penalty for less likely directions. Furthermore, the directions themselves can sometimes be predicted from nearby directions used in nearby, already decoded blocks.

[0016] FIG. 1B shows a diagram (110) depicting the 65 intra prediction directions according to JEM, showing the increasing number of prediction directions over time.

[0017]

[0017] The mapping of intra-prediction direction bits to represent directions within a coded video bitstream can vary from one video coding technique to another. Such mappings can range, for example, from simple direct mappings to complex adaptive schemes involving codewords, most likely modes, and similar techniques. However, in most cases, there will likely be certain directions that are statistically less likely to occur in the video content than certain other directions. Because the goal of video compression is to reduce redundancy, these less likely directions are represented with more bits than more likely directions in well-performing video coding techniques.

[0018]

[0018] Image and / or video encoding and decoding may be performed using inter-picture prediction with motion compensation. Motion compensation may be a non-lossless compression technique and may refer to a technique used to predict a newly reconstructed picture or picture portion after blocks of sample data from a previously reconstructed picture or portion thereof (reference picture) are spatially shifted in a direction indicated by a motion vector (hereinafter referred to as MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions, X and Y, or three dimensions, the third of which is an index of the reference picture used (the latter may indirectly be a temporal dimension).

[0019] In some video compression techniques, the motion vector applicable to a given region of sample data can be predicted from other motion vectors, e.g., from a motion vector associated with another area of ​​sample data that is spatially adjacent to the area being reconstructed and precedes that motion vector in decoding order. Doing so can significantly reduce the amount of data required to code the motion vector, thereby eliminating redundancy and increasing compression. For example, when coding an input video signal derived from a camera (known as natural video), motion vector prediction can work effectively because there is a statistical likelihood that areas larger than the area to which a single motion vector is applicable will move in a similar direction, and therefore, in some cases, can be predicted using similar motion vectors derived from motion vectors in neighboring areas. This results in a motion vector for a given area that is found to be similar or identical to the motion vector predicted from surrounding motion vectors, which, after entropy coding, can be represented using fewer bits than would be used to code the motion vector directly. In some cases, motion vector prediction can be an example of lossless compression of a signal (i.e., a motion vector) derived from the original signal (i.e., a sample stream). In other cases, the MV prediction itself may be non-lossless due to rounding errors, for example when computing a predictor from several surrounding MVs.

[0020]

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms provided by H.265, one that will be described with reference to Figure 2 is a technique hereafter referred to as "spatial merging".

[0021]

[0021] Referring to Figure 2, the current block (201) contains samples that have been found by the encoder during the motion search process to be predictable from a previous block of the same size but spatially shifted. Instead of coding its MV directly, the MV can be derived from metadata associated with one or more reference pictures, e.g., from the most recent reference picture (in decoding order) using MVs associated with any of five surrounding samples (202-206), denoted as A0, A1, and B0, B1, B2, respectively. In H.265, MV prediction can use predictors from the same reference picture as neighboring blocks. Summary of the Invention

[0022] Aspects of the disclosure provide a method and apparatus for video encoding / decoding. In some examples, the video decoding apparatus includes a processing circuit.

[0023] According to an aspect of the disclosure, a video decoding method is provided that is executed in a video decoder. The method may receive coded information for a current chroma block and a luma block collocated with the current chroma block. A feature value may be determined based on at least one of (i) neighboring reconstructed chroma samples of the current chroma block and (ii) neighboring reconstructed luma samples of the luma block collocated with the current chroma block. The chroma samples of the current chroma block and the luma samples of the luma block collocated with the current chroma block may be grouped into a plurality of groups based on a feature value threshold. Each of the plurality of groups may include a respective chroma sample and a respective luma sample. A respective cross-component prediction mode may be determined for each of the plurality of groups by comparing the respective chroma samples and the respective luma samples of each of the respective groups with the determined feature values. The current chroma block may be reconstructed based on the determined cross-component prediction modes of the plurality of groups.

[0024]

[0024] In one example, the feature value can be determined as one of the average value of the reconstructed chroma samples in the vicinity of the current chroma block and the average value of the reconstructed luma samples in the vicinity of the luma block at the same position.

[0025]

[0025] In one example, the feature value can be determined as one of the average gradient value of the reconstructed chroma samples in the vicinity of the current chroma block and the average gradient value of the reconstructed luma samples in the vicinity of the luma block at the same position.

[0026]

[0026] In one example, the feature value can be determined as the average value of the reconstructed chroma samples in the vicinity of the current chroma block and the reconstructed luma samples in the vicinity of the luma block at the same position.

[0027]

[0027] To group the chroma samples of the current chroma block and the luma samples of the co-located luma block, a characteristic value associated with each luma sample of the co-located luma block may be determined. It may be determined whether the characteristic value associated with each luma sample of the co-located luma block is greater than a threshold feature value.

[0028] In one example, based on the characteristic value associated with each luma sample being greater than a threshold feature value, (i) each luma sample among the luma samples of a co-located luma block and (ii) each chroma sample among the chroma samples of the current chroma block corresponding to each luma sample may be grouped into a first group.

[0029] In one example, (i) each luma sample among the luma samples of the co-located luma block and (ii) each chroma sample among the chroma samples of the current chroma block corresponding to each luma sample can be grouped into a second group based on the characteristic value associated with each luma sample being less than a threshold feature value.

[0030]

[0028] In some embodiments, the characteristic value of each luma sample may include one of (i) the luma sample value of each luma sample, and (ii) the average value of reconstructed luma samples in the vicinity of the luma block at the same position.

[0031]

[0029] In some embodiments, the respective cross-component prediction mode may include one of a cross-component linear model (CCLM), chroma from luma (CfL), a convolutional cross-component model (CCCM), a multiple filter linear model (MFLM), a gradient linear model (GLM), a combination of CCLM, CfL, and CCCM, and an angular intra prediction mode.

[0032]

[0030] In some embodiments, the respective cross-component prediction mode may be determined based on a corresponding flag included in the coded information.

[0033]

[0031] In an embodiment, whether each of the multiple groups shares a cross-component prediction mode may be determined based on a flag included in the coded information.

[0034] In one example, a cross-component prediction mode for each of the plurality of groups may be determined based on a flag indicating that each of the plurality of groups shares a cross-component prediction mode.

[0035] In one example, in response to the flag indicating that each of the multiple groups does not share a cross-component prediction mode, a respective cross-component prediction mode for each of the multiple groups can be determined based on a corresponding syntax element.

[0036] In an embodiment, a first cross-component prediction mode for a first group of the plurality of groups may be determined based on a first index included in the coded information. A second cross-component prediction mode for a second group of the plurality of groups may be determined based on a second index. The second index may be determined based on a sum of a delta index included in the coded information and the first index.

[0037] To reconstruct the current chroma block, in one example, prediction samples for the chroma samples in each of the multiple groups can be generated based on the respective cross-component prediction modes, and a filter can then be applied to the prediction samples for the chroma samples in each of the multiple groups.

[0038] In one example, a prediction sample may be generated for the chroma samples in each of the multiple groups based on the respective cross-component prediction modes, and the prediction sample for the current chroma block may be determined as a weighted combination of the prediction samples for the chroma samples in each of the multiple groups.

[0039] According to another aspect of the present disclosure, there is provided an apparatus, the apparatus including a processing circuit, the processing circuit being configurable to perform any method for video encoding / decoding.

[0040]

[0036] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer for video coding, cause the computer to perform any method for video encoding / decoding. [Brief explanation of the drawings]

[0041]

[0037] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and accompanying drawings. [Figure 1A] FIG. 1A is a schematic diagram of an example subset of intra-prediction modes. [Figure 1B] FIG. 1B is a diagram of an exemplary intra-prediction direction. [Figure 2]

[0040] Figure 2 is a schematic diagram of a current block and its surrounding spatial merge candidates in one example. [Figure 3] FIG. 3 is a simplified block diagram schematic of a communication system (300) according to an embodiment. [Figure 4]

[0042] FIG. 4 is a simplified block diagram schematic of a communication system (400) according to an embodiment. [Figure 5]

[0043] FIG. 5 is a schematic illustration of a simplified block diagram of a decoder according to an embodiment. [Figure 6]

[0044] FIG. 6 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment. [Figure 7]

[0045] FIG. 7 shows a block diagram of an encoder according to another embodiment. [Figure 8]

[0046] FIG. 8 shows a block diagram of a decoder according to another embodiment. [Figure 9]

[0047] FIG. 9 illustrates exemplary locations of spatial merge candidates according to an embodiment of the present disclosure. [Figure 10]

[0048] FIG. 10 illustrates an exemplary redundancy check of spatial merge candidates. [Figure 11]

[0049] FIG. 11 shows a schematic diagram of motion vector scaling for temporal merge candidates according to an embodiment. [Figure 12]

[0050] FIG. 12 illustrates exemplary candidate positions for temporal merge candidates according to an embodiment. [Figure 13]

[0051] FIG. 13 illustrates an example template and reference samples of the example template in a reference picture according to an embodiment. [Figure 14]

[0052] FIG. 14 illustrates an exemplary derivation of sub-block-based merging candidates according to an embodiment. [Figure 15]

[0053] FIG. 15 shows a schematic diagram of the merge with motion vector difference (MMVD) search process. [Figure 16A]

[0054] Figure 16A shows a schematic diagram of the search points in the MMVD process. [Figure 16B]

[0055] Figure 16B shows a schematic diagram of the search points of the MMVD process. [Figure 17]

[0056] FIG. 17 shows a schematic diagram of the additional refinement positions of the MMVD process. [Figure 18]

[0057] FIG. 18 illustrates an exemplary cross-component linear mode (CCLM) prediction mode. [Figure 19]

[0058] FIG. 19 shows an exemplary chroma from luma (CfL) process. [Figure 20]

[0059] FIG. 20 illustrates an exemplary CfL prediction process. [Figure 21A]

[0060] FIG. 21A illustrates an exemplary model of a CCLM according to an embodiment. [Figure 21B]

[0061] FIG. 21B illustrates an exemplary adjusted model of CCLM according to an embodiment. [Figure 22]

[0062] FIG. 22 illustrates an exemplary convolution filter according to an embodiment. [Figure 23]

[0063] FIG. 23 shows a schematic diagram of the calculation of filter coefficients according to an embodiment. [Figure 24]

[0064] Figure 24 shows a schematic diagram of grouping luma and chroma samples for a block. [Figure 25]

[0065] FIG. 25 shows a flowchart outlining an exemplary decoding process according to some embodiments of the present disclosure. [Figure 26]

[0066] FIG. 26 shows a flowchart outlining an exemplary encoding process according to some embodiments of the present disclosure. [Figure 27]

[0067] FIG. 27 is a schematic diagram of a computer system according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0042]

[0068] Detailed Description of the Embodiments Figure 3 shows an exemplary block diagram of a communication system (300). The communication system (300) includes multiple terminal devices capable of communicating with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of Figure 3, the first pair of terminal devices (310) and (320) engage in unidirectional transmission of data. For example, the terminal device (310) may code video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to another terminal device (320) via the network (350). The coded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (320) can receive the coded video data from the network (350), decode the coded video data to reconstruct the video picture, and display the video picture according to the reconstructed video data. Unidirectional data transmission may be common in media serving applications, etc.

[0043]

[0069] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) for bidirectional transmission of coded video data, e.g., during a video conference. For bidirectional data transmission, for example, each of the terminal devices (330) and (340) can code video data (e.g., a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) over the network (350). Each of the terminal devices (330) and (340) can also receive coded video data transmitted by the other of the terminal devices (330) and (340), decode the coded video data to reconstruct the video pictures, and display the video pictures on an accessible display device in accordance with the reconstructed video data.

[0044]

[0070] In the example of FIG. 3 , terminal devices 310, 320, 330, and 340 are shown as a server, a personal computer, and a smartphone, respectively, although the principles of the present disclosure need not be so limited. Embodiments of the present disclosure may be used with laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 350 represents any number of networks that carry coded video data between terminal devices 310, 320, 330, and 340, including, for example, wired and / or wireless communication networks. Communication network 350 may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of the present disclosure, the architecture and topology of network 350 may not be important to the operation of the present disclosure, unless otherwise described below.

[0045]

[0071] Figure 4 illustrates a video encoder and video decoder in a streaming environment as an example application of the disclosed subject matter, which is equally applicable to other video-enabled applications including, for example, video conferencing, digital TV, streaming services, storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.), etc.

[0046]

[0072] The streaming system may include a video source (401), such as a digital camera, and may also include a capture subsystem (413) capable of generating a stream of uncompressed video pictures (402). In one example, the video picture stream (402) includes samples captured by the digital camera. The video picture stream (402), depicted as a thick line to emphasize its large amount of data when compared to the encoded video data (404) (or coded video bitstream), may be processed by an electronic device (420) including a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in detail below. The encoded video data (404) (or encoded video bitstream), depicted as a thin line to emphasize its smaller amount of data compared to the stream of video pictures (402), can be stored on the streaming server (405) for future use. One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include a video decoder (410), for example, within the electronic device (430). The video decoder (410) decodes the incoming copy of the encoded video data (407) and generates an output stream of video pictures (411) that can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., video bitstream) may be encoded according to a particular video coding / compression standard.Examples of these standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0047]

[0073] It should be noted that the electronic devices 420 and 430 may include other components (not shown). For example, the electronic device 420 may include a video decoder (not shown), and the electronic device 430 may include a video encoder (not shown).

[0048]

[0074] 5 shows an example block diagram of a video decoder (510). The video decoder (510) can be included in an electronic device (530). The electronic device (530) can include a receiver (531) (e.g., receiving circuitry). The video decoder (510) can be used in place of the video decoder (410) in the example of FIG. 4.

[0049]

[0075] The receiver (531) can receive one or more coded video sequences to be decoded by the video decoder (510). In some embodiments, the receiver can receive one coded video sequence at a time, where the decoding of each coded video sequence is independent of the decoding of other coded video sequences. The coded video sequences can be received from a channel (501), which can be a hardware or software link to a storage device that stores the coded video data. The receiver (531) can receive the coded video data along with other data, such as coded audio data and / or auxiliary data streams, which can be transferred using respective entities (not shown). The receiver (531) can separate the coded video sequences from other data. To address network jitter, a buffer memory (515) can be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter referred to as the "parser (520)"). In certain applications, the buffer memory (515) is part of the video decoder (510). In other cases, it may be external to the video decoder (510) (not shown). In yet another example, there may be a buffer memory (not shown) external to the video decoder (510), for example, to deal with network jitter, and there may even be another buffer memory (515) internal to the video decoder (510), for example, to handle playback timing. If the receiver (531) is receiving data from a store-and-forward device with sufficient bandwidth and controllability or from a synchronous network, the buffer memory (515) may not be needed or may be small.For use in a best effort packet network such as the Internet, a buffer memory (515) may be required, which may be relatively large and may advantageously be adaptively sized, and may be implemented at least in part in an operating system or similar element (not shown) outside the video decoder (510).

[0050]

[0076] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the coded video sequence. These symbol categories include information used to manage the operation of the video decoder (510) and, potentially, information for controlling a rendering device, such as a rendering device (512) (e.g., a display screen) that is not an integral part of the electronic device (530) but may be coupled to the electronic device (530), as shown in FIG. 5. The rendering device control information may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a variety of video coding techniques or standards, including variable length coding, Huffman coding, arithmetic coding with or without context effects, and so forth. The parser (520) can extract from the coded video sequence a set of subgroup parameters for at least one subgroup of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroup can include a group of pictures (GOP), a picture, a tile, a slice, a macroblock, a coding unit (CU), a block, a transform unit (TU), a prediction unit (PU), etc. The parser (520) can also extract from the coded video sequence information such as transform coefficients, quantization parameter values, motion vectors, etc.

[0051]

[0077] The parser (520) is capable of performing an entropy decoding / parsing process on the video sequence received from the buffer memory (515) to generate symbols (521).

[0052]

[0078] The reconstruction of the symbols (521) may include several different units depending on the type of coded video picture or part thereof (e.g., inter and intra pictures, inter and intra blocks) and other factors. Which units are included and how can be controlled by subgroup control information parsed by the parser (520) from the coded video sequence. The flow of such subgroup control information between the parser (520) and subsequent units is not depicted for clarity.

[0053]

[0079] Beyond the functional blocks already described, the video decoder (510) may be conceptually subdivided into a number of functional units, as described below. In a practical implementation operating within commercial constraints, many of these units will interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate.

[0054]

[0080] The first unit is a scalar / inverse transform unit (551), which receives quantized transform coefficients as well as control information (including the transform to use, block size, quantization factor, quantization scaling matrix, etc.) from the parser (520) as symbols (521). The scalar / inverse transform unit (551) can output blocks containing sample values ​​that can be input to an aggregator (555).

[0055]

[0081] In some cases, the output samples of the scalar / inverse transform unit (551) may relate to intra-coded blocks. Intra-coded blocks are blocks that do not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates blocks of the same size and shape as the block being reconstructed using already reconstructed surrounding information retrieved from the current picture buffer (558). The current picture buffer (558), for example, buffers the partially reconstructed and / or fully reconstructed current picture. The aggregator (555) optionally adds, on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (552) to the output sample information as provided by the scalar / inverse transform unit (551).

[0056]

[0082] In other cases, the output samples of the scalar / inverse transform unit (551) may relate to a block that may be inter-coded and motion-compensated. In such cases, the motion-compensated prediction unit (553) may access the reference picture memory (557) to retrieve samples used for prediction. After motion-compensating the retrieved samples according to the symbols (521) associated with the block, these samples are added by the aggregator (555) to the output of the scalar / inverse transform unit (551) (in this case, referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (557) from which the motion-compensated prediction unit (553) retrieves prediction samples may be controlled by motion vectors available to the motion-compensated prediction unit (553), for example, in the form of symbols (521), which may have X, Y, and reference picture components. Motion compensation can also include interpolation of sample values ​​taken from a reference picture memory (557), motion vector prediction mechanisms, etc., where sub-sample accurate motion vectors are used.

[0057]

[0083] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). Video compression techniques may include in-loop filtering techniques controlled by parameters contained in the coded video sequence (also called the coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520). Video compression may also depend on meta-information obtained during decoding of previous portions (in decoding order) of the coded picture or coded video sequence, as well as on previously reconstructed loop-filtered sample values.

[0058]

[0084] The output of the loop filter unit (556) can be a stream of samples that can be output to a rendering device (512) or stored in a reference picture memory (557) for use in future inter-picture prediction.

[0059]

[0085] Once a given coded picture is fully reconstructed, it can be used as a reference picture for future prediction. For example, once the coded picture corresponding to the current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and a fresh current picture buffer can be reallocated before starting the reconstruction of a subsequent coded picture.

[0060]

[0086] The video decoder (510) may perform decoding operations according to a standard or predetermined video compression technology, such as ITU-T Rec. H.265. A coded video sequence may conform to the syntax specified by the video compression technology or standard in use, in the sense that the coded video sequence conforms to both the syntax of the video compression technology or standard and the profile, as documented in the video compression technology or standard. Specifically, a profile may select certain tools from all tools available in the video compression technology or standard as the only tools that can be used under that profile. Compliance also requires that the complexity of the coded video sequence fall within a range defined by the level of the video compression technology or standard. In some cases, the level may limit the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the levels may in some cases be further constrained by Hypothetical Reference Decoder (HRD) specifications and metadata for HRD buffer management signaled in the coded video sequence.

[0061]

[0087] In an embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0062]

[0088] 6 shows an example block diagram of a video encoder (603). The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmitting circuit). The video encoder (603) can be used in place of the video encoder (403) in the example of FIG. 4.

[0063]

[0089] The video encoder (603) can receive video samples from a video source (601) (which in the example of FIG. 6 is not part of the electronic device (620)) that can capture video images to be coded by the video encoder (603). In another example, the video source (601) is part of the electronic device (620).

[0064]

[0090] The video source (601) may provide a source video sequence to be coded by the video encoder (603) in the form of a digital video sample stream, which may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCB, RGB, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (601) may be a storage device that stores pre-prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures that convey motion when viewed sequentially. The picture itself may be organized as a spatial array of pixels, each of which may contain one or more samples, depending on the sampling structure, color space, etc., in use. Those skilled in the art will readily understand the relationship between pixels and samples. The following discussion focuses on examples.

[0065]

[0091] According to an embodiment, the video encoder (603) is capable of coding and compressing pictures of a source video sequence into a coded video sequence (643) in real time or under any other required time constraints. Enforcing an appropriate coding rate is one function of the controller (650). In some embodiments, the controller (650) controls and is functionally coupled to other functional units as described below, the coupling of which is not depicted for clarity. Parameters set by the controller (650) may include rate control-related parameters (picture skip, quantizer, lambda value for rate-distortion optimization techniques, etc.), picture size, group-of-picture (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured with other appropriate functions associated with the video encoder (603) optimized for a particular system design.

[0066]

[0092] In some embodiments, the video encoder (603) is configured to operate in a coding loop. As a simplified explanation, in one example, the coding loop may include a source coder (630) (e.g., responsible for generating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to generate sample data in a manner similar to that generated by the (remote) decoder. The reconstructed sample stream (sample data) is input to a reference picture memory (634). Because decoding of the symbol stream produces bit-exact results independent of the location (local or remote) of the decoder, the contents of the reference picture memory (634) are also bit-exact between the local and remote encoders. In other words, the predictor in the encoder "sees" as reference picture samples exactly the same sample values ​​that the decoder would "see" if it were using prediction during decoding. This basic principle of reference picture synchronization (and the resulting drift if synchronization cannot be maintained, e.g., due to channel errors) is used in several related technologies as well.

[0067]

[0093] The operation of the "local" decoder (633) may be the same as that of a "remote" decoder, such as the video decoder (510) already described in detail above in connection with Figure 5. However, briefly referring also to Figure 5, because symbols are available and the encoding / decoding of the symbols into a coded video sequence by the entropy coder (645) and parser (520) may be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and parser (520), may not be fully implemented in the local decoder (633).

[0068]

[0094] In embodiments, decoder techniques other than analysis / entropy decoding present in a decoder are also present in the corresponding encoder, ideally or in substantially the same functional form. Therefore, the disclosed subject matter focuses on the operation of the decoder. A description of the encoder techniques can be omitted, as they are the reverse of the decoder techniques described generically. In certain areas, more detailed descriptions are provided below.

[0069]

[0095] During operation, in some examples, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (632) codes differences between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as predictive references for the input picture.

[0070]

[0096] The local video decoder (633) can decode coded video data of pictures that can be designated as reference pictures based on symbols generated by the source coder (630). The operation of the coding engine (632) can advantageously be a non-lossless process. When the coded video data can be decoded by a video decoder (not shown in FIG. 6), the reconstructed video sequence can typically be a replica of the source video sequence with some errors. The local video decoder (633) can repeat the decoding process that can be performed by the video decoder on the reference pictures, causing the reconstructed reference pictures to be stored in the reference picture cache (634). In this way, the video encoder (603) can locally store copies of reconstructed reference pictures that have common content with reconstructed reference pictures obtained by a far-end video decoder (assuming there are no transmission errors).

[0071]

[0097] The predictor (635) can perform a prediction search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) can search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or predetermined metadata (reference picture motion vectors, block shapes, etc.), which may serve as suitable prediction references for the new picture. The predictor (635) can operate on a sample-block-pixel-block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).

[0072]

[0098] The controller (650) can manage the coding operations of the source coder (630), including, for example, setting parameters and subgroup parameters used to encode the video data.

[0073]

[0099] All outputs of the aforementioned functional units can be entropy coded in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0074]

[0100] The transmitter (640) can buffer the coded video sequence, as produced by the entropy coder (645), and prepare it for transmission over a communication channel (660), which may be a hardware / software link to a storage device that stores the coded video data. The transmitter (640) can merge the coded video data from the video coder (603) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0075]

[0101] The controller (650) can manage the operation of the video encoder (603). During coding, the controller (650) can assign a particular coded picture type to each coded picture, which can affect the coding technique that can be applied to each picture. For example, pictures may often be assigned as one of the following picture types:

[0102] An intra picture (I-picture) is one that can be coded and decoded without using any other picture in the sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0076]

[0103] A predicted picture (P-picture) can be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0077]

[0104] Bidirectionally predicted pictures (B-pictures) can be coded and decoded using intra- or inter-prediction, which uses at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures can use more than two reference pictures and associated metadata for the reconstruction of a block.

[0078]

[0105] A source picture is typically spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and can be coded block by block. Blocks can be predictively coded with reference to other (already coded) blocks, as determined by the coding assignment applied to the respective picture. For example, blocks of an I-picture can be non-predictively coded, or they can be predictively coded with reference to previously coded blocks of the same picture (spatial or intra prediction). Pixel blocks of a P-picture can be predictively coded with spatial or temporal prediction with reference to one previously coded reference picture. Blocks of a B-picture can be predictively coded with spatial or temporal prediction with reference to one or two previously coded reference pictures.

[0079]

[0106] The video encoder (603) may perform coding operations according to a predetermined video coding technique or standard, such as ITU-T Rec. H.266. In this operation, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancies in the input video sequence. The coded video data may therefore conform to a syntax specified by the video coding technique or standard being used.

[0080]

[0107] In some embodiments, the transmitter (640) can transmit additional data along with the coded video. The source coder (630) can include such data as part of the coded video sequence. The additional data can include temporal, spatial, and SNR enhancement layers, as well as other forms of redundant data (such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.).

[0081]

[0108] Video can be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) exploits spatial correlation within a given picture, while inter-picture prediction exploits correlation (temporal or otherwise) between pictures. In one example, a particular picture under encoding / decoding, called the current picture, is partitioned into blocks. If a block in the current picture is similar to a reference block in a reference picture that was previously coded and is still buffered in the video, the block in the current picture can be coded by a vector called a motion vector. The motion vector points to a reference block within the reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0082]

[0109] In some embodiments, bi-prediction techniques may be used for inter-picture prediction. Bi-prediction techniques use two reference pictures, such as a first reference picture and a second reference picture, that both precede the current picture in decoding order (but may be past and future in display order, respectively) in the video. A block in the current picture may be coded with a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block may be predicted by a combination of the first and second reference blocks.

[0083]

[0110] Furthermore, to improve coding efficiency, it is possible to use merge mode techniques for inter-picture prediction.

[0084]

[0111] According to some embodiments of the present disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed on a block-by-block basis. For example, according to the HEVC standard, pictures in a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs within a picture have the same size, such as 64x64 pixels, 32x32 pixels, or 16x16 pixels. Generally, a CTU includes three coding tree blocks (CTBs): one luma CTB and two chroma CTBs. Each CTU can be recursively quadtree partitioned into one or more coding units (CUs). For example, a 64x64 pixel CTU can be partitioned into one 64x64 pixel CU, four 32x32 pixel CUs, or sixteen 16x16 pixel CUs. In one example, each CU is analyzed to determine the CU's prediction type, such as an inter prediction type or an intra prediction type. A CU is divided into one or more prediction units (PUs) depending on temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values ​​(e.g., luma values) for pixels, such as 8x8 pixels, 16x16 pixels, 8x16 pixels, 16x8 pixels, etc.

[0085]

[0112] 7 shows an example diagram of a video encoder (703). The video encoder (703) is configured to receive a processed block of sample values ​​(e.g., a predicted block) in a current video picture in a sequence of video pictures and encode the processed block into a coded picture that is part of the coded video sequence. In one example, the video encoder (703) is used in place of the video encoder (403) in the example of FIG. 4.

[0086]

[0113] In an HEVC example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as an 8x8 sample prediction block. The video encoder (703) determines whether the processing block is best coded using intra-mode, inter-mode, or bi-prediction mode, e.g., using rate-distortion optimization. If the processing block is to be coded in intra-mode, the video encoder (703) may encode the processing block into a coded picture using intra-prediction techniques; if the processing block is to be coded in inter-mode or bi-prediction mode, the video encoder (703) may encode the processing block into a coded picture using inter-prediction techniques or bi-prediction techniques, respectively. In certain video coding techniques, the merge mode may be an inter-picture prediction submode, in which case the motion vector is derived from one or more motion vector predictors without the benefit of any motion vector components coded outside the predictor. In certain other video coding techniques, there may be motion vector components applicable to the current block. In one example, the video encoder (703) includes other components, such as a mode decision module (not shown), to determine the mode of the processing block.

[0087]

[0114] In the example of Figure 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together as shown in Figure 7.

[0088]

[0115] The inter-encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., a block in a previous picture and a block in a subsequent picture), generate inter-prediction information (e.g., a description of redundant information due to the inter-coding technique, motion vectors, merge mode information), and calculate an inter-prediction result (e.g., a predicted block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture that has been decoded based on coded video information.

[0089]

[0116] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), optionally compare the block to previously coded blocks in the same picture, generate transformed and quantized coefficients, and optionally generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). In one example, the intra encoder (722) also calculates an intra prediction result (e.g., a prediction block) based on the intra prediction information and reference blocks in the same picture.

[0090]

[0117] The general-purpose controller (721) is configured to determine general control data and control other components of the video encoder (703) based on the general control data. In one example, the general-purpose controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, if the mode is intra mode, the general-purpose controller (721) controls the switch (726) to select intra mode results for use by the residual calculator (723) and the entropy encoder (725) to select intra prediction information and include it in the bitstream; and if the mode is inter mode, the general-purpose controller (721) controls the switch (726) to select inter prediction results for use by the residual calculator (723) and the entropy encoder (725) to select inter prediction information and include it in the bitstream.

[0091]

[0118] The residual calculator (723) is configured to calculate the difference (residual data) between the received block and a prediction result selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) is configured to operate on the residual data to encode the residual data and generate transform coefficients. In one example, the residual encoder (724) is configured to transform the residual data from the spatial domain to the frequency domain to generate transform coefficients. The transform coefficients are then subjected to a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be used by the intra-encoder (722) and the inter-encoder (730), as appropriate. For example, the inter-encoder (730) may generate decoded blocks based on the decoded residual data and inter-prediction information, and the intra-encoder (722) may generate decoded blocks based on the decoded residual data and intra-prediction information. The decoded blocks are processed appropriately to generate decoded pictures, which may be buffered in memory circuitry (not shown) and, in some cases, used as reference pictures.

[0092]

[0119] The entropy encoder (725) is configured to format a bitstream to include the coded blocks. The entropy encoder (725) is configured to include various information in the bitstream in accordance with an appropriate standard, such as the HEVC standard. In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information in the bitstream. Note that, when coding blocks in a merged sub-mode of either an inter mode or a bi-prediction mode in accordance with the disclosed subject matter, the residual information is not present.

[0093]

[0120] 8 shows an example diagram of a video decoder (810). The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (810) is used in place of the video decoder (410) in the example of FIG. 4.

[0094]

[0121] In the example of Figure 8, the video decoder (810) includes an entropy decoder (871), an inter-decoder (880), a residual decoder (873), a reconstruction module (874), and an intra-decoder (872) coupled together as shown in Figure 8.

[0095]

[0122] The entropy decoder (871) can be configured to reconstruct, from a coded picture, specific symbols that represent syntax elements that make up the coded picture. Such symbols can include, for example, the mode in which the block is coded (e.g., intra mode, inter mode, bi-prediction mode, merged submode or another submode within the latter two), prediction information (e.g., intra-prediction information or inter-prediction information), and other information that can identify specific samples or metadata used for prediction by the intra-decoder (872) or inter-decoder (880), respectively. The symbols can also include residual information, for example, in the form of quantized transform coefficients. In one example, if the prediction mode is an inter- or bi-prediction mode, the inter-prediction information is provided to the inter-decoder (880); if the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra-decoder (872). The residual information can be inverse-quantized and provided to the residual decoder (873).

[0096]

[0123] The inter decoder (880) is configured to receive inter prediction information and generate inter prediction results based on the inter prediction information.

[0097]

[0124] The intra decoder (872) is configured to receive intra prediction information and generate a prediction result based on the intra prediction information.

[0098]

[0125] The residual decoder (873) is configured to perform inverse quantization to extract unquantized transform coefficients and process the unquantized transform coefficients to transform the residual information from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (including quantization parameters (QPs)), which may be provided by the entropy decoder (871) (this may be only a small amount of control information, so a data path is not depicted).

[0099]

[0126] The reconstruction module (874) is configured to combine, in the spatial domain, the residual information as output by the residual decoder (873) and the prediction result (possibly output by an inter- or intra-prediction module) to form a reconstructed block, which is part of a reconstructed picture, which may be part of a reconstructed video. It should be noted that other appropriate processes, such as deblocking processes, may be performed to improve visual quality.

[0100]

[0127] The video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technology. In some embodiments, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In other embodiments, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0101]

[0128] The present disclosure includes embodiments related to mixed model design for cross-component prediction modes.

[0102]

[0129] ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) published the H.265 / HEVC (High Efficiency Video Coding) standard in 2013 (Version 1), 2014 (Version 2), 2015 (Version 3), and 2016 (Version 4). In 2015, the two standardization organizations jointly formed the Joint Video Exploration Team (JVET) to explore the possibility of developing the next video coding standard beyond HEVC. In October 2017, the two standardization organizations issued a Joint Call for Proposals on Video Compression with Capability beyond HEVC (CfP). By February 15, 2018, 22 CfP responses had been submitted for standard dynamic range (SDR), 12 for high dynamic range (HDR), and 12 for the 360 ​​video category. In April 2018, all accepted CfP responses were evaluated at the 122 MPEG / 10th JVET meeting. As a result of this meeting, JVET officially launched the standardization process for next-generation video coding beyond HEVC. The new standard was named VVC (Versatile Video Coding), and JVET was renamed JVET (Joint Video Experts Team). In 2020, ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) published the VVC video coding standard (Version 1).

[0103]

[0130] In inter prediction, for an inter-predicted coding unit (CU), motion parameters are required, for example, for VVC coding features to be used in generating inter-predicted samples. The motion parameters may include motion vectors, reference picture indices, reference picture list usage indices, and / or additional information. The motion parameters may be signaled in an explicit or implicit manner. If a CU is coded in skip mode, the CU may be associated with one PU, and significant residual coefficients, coded motion vector deltas, and / or reference picture indices may not be required. When a CU is coded in merge mode, its motion parameters can be obtained from neighboring CUs. Neighboring CUs may include spatial and temporal candidates, as well as schedules (or additional candidates) as introduced in VVC. Merge mode may also be applied to any inter-predicted CUs, not just for skip mode. An alternative to merge mode is explicit transmission of motion parameters, in which case motion vectors, corresponding reference picture indices for each reference picture list, reference picture list usage flags, and / or other necessary information can be explicitly signaled for each CU.

[0104]

[0131] For VVC, the VVC Test model (VTM) reference software includes a number of new refined inter-predictive coding tools, including: (1) Extended merge prediction, (2) Merge motion vector difference (MMVD) mode, (3) adaptive motion vector prediction (AMVP) mode with symmetric MVD signaling; (4) Affine motion compensation prediction, (5) Subblock-based temporal motion vector prediction (SbTMVP), (6) adaptive motion vector resolution (AMVR) (7) Motion field storage (1 / 16 luma sample MV storage and 8x8 motion field compression), (8) Bi-prediction with CU-level weights (BCW), (9) Bi-directional optical flow (BDOF), PROF), (10) Decoder side motion vector refinement (DMVR), (11) Combined inter and intra prediction (CIIP), (12) Geometric Partitioning Mode (GPM) It may contain one or more of the following:

[0105]

[0132] As in VTM4, a merge candidate list can be constructed by including five types of candidates. The merge candidate list can be constructed in the following order: (1) Spatial MVP (motion vector predictor(s),MVP) from spatially neighboring CUs. (2) Temporal MVP from collocated CUs (collocated CU(s)), (3) History-based MVP(s) from FIFO tables (HMVP), (4) Pairwise average MVP, and (5) Zero MV(s).

[0106]

[0133] The size of the merge candidate list can be signaled in the slice header. The maximum allowed size of the merge candidate list can be 6, as in VTM4. For each CU coded in merge mode, the index of the best merge candidate can be coded using, for example, truncated unary binarization (TU). The first bin of the merge index can be coded using context, and bypass coding can be used for the other bins.

[0107]

[0134] In spatial candidate derivation, for example, in VVC, the derivation of spatial merge candidates may be the same as or similar to the derivation of spatial merge candidates in HEVC. For example, the maximum number of merge candidates (e.g., four merge candidates) may be selected from the candidates located at positions shown in FIG. 9. As shown in FIG. 9, the current block (901) may include neighboring blocks (902)-(906) located at positions A0, A1, B0, B1, and B2, respectively. The derivation order of spatial merge candidates may be B1, A1, B0, A0, B2. Position B2 may be considered only if any CUs (or blocks) at positions A0, B0, B1, and A1 are unavailable (e.g., because the CU belongs to another slice or tile) or are intra-coded. After the candidate (or block) at position A1 is added, the addition of the remaining candidates (or blocks) may be subject to a redundancy check.

[0108] The redundancy check can ensure that candidates with the same motion information are removed from the merge list, thereby improving coding efficiency. To reduce computational complexity, the redundancy check may not consider all possible candidate pairs. Instead, only the candidate pairs connected by the arrows in Figure 10 may be considered. For example, the redundancy check can be applied to five candidate pairs, such as candidate pair A1 and B1, candidate pair A1 and A0, and so on. A candidate may be added to the merge list only if the corresponding candidates used in the redundancy check do not contain the same motion information. For example, candidate B0 may be added to the merge list only if the corresponding candidate B1 does not contain the same motion information.

[0109]

[0135] In temporal candidate derivation, only one candidate may be added to the merge list. For example, as shown in FIG. 11, in deriving a temporal merge candidate for a current CU (1114), a scaled motion vector may be derived based on the co-located CU (1104) belonging to the co-located reference picture (1112). The reference picture list used to derive the co-located CU (1104) may be explicitly signaled in the slice header. The scaled motion vector for the temporal merge candidate may be obtained as shown by the dotted line in FIG. 11, which is scaled from the motion vector of the co-located CU (1104) using picture order count (POC) distances tb and td. tb may be defined as the POC difference between the current picture's reference picture (e.g., Curr_ref) (1106) and the current picture (e.g., Curr_pic) (1108). td may be defined as the POC difference between the co-located picture's reference picture (e.g., Col_ref) 1110 and the co-located picture (e.g., Col_pic) 1112. The reference picture index of the temporal merge candidate may be set to zero.

[0110]

[0136] The temporal candidate location can be selected between candidates C0 and C1, as shown in Figure 12. For example, location C1 can be used if the CU at location C0 is unavailable, intra-coded, or outside the current row of the CTU. Alternatively, location C0 can be used in the derivation of the temporal merge candidate.

[0111]

[0137] Adaptive reordering of merge candidates with template matching (ARMC-TM). Merge candidates can be adaptively reordered using template matching (TM). ARMC-TM can be applied to various prediction modes, such as regular merge mode, TM merge mode, and affine merge mode (except for SbTMVP candidates). For TM merge mode, merge candidates can be reordered before the refinement process.

[0112]

[0138] After the merge candidate list is constructed, the merge candidates can be divided into subgroups. The subgroup size can be set to 5 for regular merge mode and TM merge mode. For affine merge mode, the subgroup size can be set to 3. The merge candidates within each subgroup can be sorted in ascending order according to their cost values ​​based on template matching. For simplicity, merge candidates in the last subgroup but not the first subgroup may not be sorted.

[0113]

[0139] The template matching cost of a merge candidate can be measured by the sum of absolute differences (SAD) between the template's samples of the current block and the corresponding reference samples of the current block. The template can include a set of reconstructed samples adjacent to or near the current block. The template's reference samples can be located by the motion information of the merge candidate.

[0114]

[0140] If the merge candidate uses bi-directional prediction, the reference samples for the merge candidate template may also be generated using bi-directional prediction. In the example shown in Figure 13, a current block (1302) is included in a current picture (1304). The current block (1302) may have a template (1306) located adjacent to the top and left sides of the current block (1302). The template (1306) may include reconstructed samples from the surrounding area of ​​the current block (1302).

[0115] Based on template matching, a first reference block 1308 in a first reference picture 1310 of a first reference list 0 may be determined within the search range, in which case the minimum SAD between template 1312 of the first reference block 1308 and template 1306 may be determined. The first reference block (1308) may be specified by a MV (1314).

[0116] Also, based on template matching, a second reference block 1316 in a second reference picture 1318 of a second reference list 1 can be determined within the search range, and in this case, the minimum SAD between a template 1320 of the second reference block 1316 and the template 1306 can be determined. The second reference block (1316) can be specified by a MV (1322).

[0117] A predicted sample of the current block (1302) can be further determined based on the first reference block (1308) and the second reference block (1316).

[0118]

[0141] For a subblock-based merging candidate with a subblock size equal to W × H, the top template of the current block may include several subtemplates with a size of W × 1, and the left template of the current block may include several subtemplates with a size of 1 × H. As shown in Figure 14, the motion information of the subblocks in the first row and first column of the current block can be used to derive the reference samples for each subtemplate.

[0119]

[0142] FIG. 14 illustrates an exemplary derivation of sub-block-based merging candidates. As shown in FIG. 14, a current block (1402) may be included in a current picture (1404). The current block (1402) may include sub-block A-G in the first row and first column. The current block (1402) may include a template (1406) adjacent to the top and left of the current block (1402). The current block (1402) may include a block (1408) at a co-located position in a reference picture (1410). The co-located block (1408) may include sub-block A-G in the first row and first column corresponding to sub-block A-G in the current block (1402). The motion information (e.g., affine motion vector) of sub-block A-G in the first row and first column of the current block (1402) may be used to derive reference samples for the sub-template (or sub-reference template) of the equivalently positioned block (1408). For example, the motion information of sub-blocks A, E, F, and G of the current block (1402) may be applied to derive reference samples for the sub-templates located adjacent to the left of sub-blocks A, E, F, and G of the equivalently positioned block (1408). The sub-templates adjacent to the left of sub-blocks A, E, F, and G of the equivalently positioned block (1408) may form the left reference template of the equivalently positioned block (1408). The motion information of sub-blocks A, B, C, and D of the current block (1402) may be applied to derive reference samples for the sub-templates located adjacent to the top of sub-blocks A, B, C, and D of the equivalently positioned block (1408). The sub-templates adjacent to the top side of the sub-blocks A, B, C, and D of the equivalent position block (1408) can further form the upper reference template of the equivalent position block (1408).

[0120]

[0143] Merge with Motion Vector Difference (MMVD) can be used for certain prediction modes, such as skip mode and merge mode using a motion vector representation. MMVD can reuse merge candidates like VVC. It can select merge candidates from among merge candidates and further enhance (or refine) them using a motion vector representation. MMVD can provide new motion vector representations with simplified signaling. The motion vector representation can include the starting point, motion magnitude, and motion direction.

[0121]

[0144] MMVD can use a merge candidate list as in VVC. Candidates with the default merge type (e.g., MRG_TYPE_DEFAULT_N) can be considered for extending MMVD. In MMVD, a base candidate index may define a starting point. The base candidate index (IDX) can specify the best candidate from among the candidates in a list, for example, as shown in Table 1 below.

[0122] Table 1. Base candidate IDX

[0123] [Table 1]

[0145] If the number of base candidates is equal to 1, the base candidate IDX may not be signaled. The distance index may provide motion magnitude information. The distance index may indicate a predefined distance from the starting point. The predefined distance based on the distance index may be provided as shown in Table 2 below: Table 2. Distance IDX

[0124] [Table 2]

[0146] The direction index can represent the direction of MVD relative to the starting point. The direction index can represent four directions, as shown in Table 3. The MMVD flag can be signaled immediately if the skip flag and merge flag are sent. If the skip flag and merge flag are true, the MMVD flag can be parsed. If the MMVD flag is equal to 1, the MMVD syntax can be parsed. However, if the MMVD flag is not 1, the AFFINE flag can be parsed. If the AFFINE flag is equal to 1, the AFFINE mode can be applied. However, if the AFFINE flag is not 1, the skip / merge index can be parsed for the skip / merge mode.

[0125] Table 3. Direction IDX

[0126] [Table 3]

[0147] 15 shows an exemplary MMVD search process. As shown in FIG. 15, the starting MV can be indicated by (1511) (e.g., by the direction IDX and the base candidate IDX), the offset can be indicated by (1512) (e.g., by the distance IDX and the direction IDX), and the final MV predictor can be indicated by (1513). In another example, the starting MV can be indicated by (1521) (e.g., by the direction IDX and the base candidate IDX), the offset can be indicated by (1522) (e.g., by the distance IDX and the direction IDX), and the final MV predictor can be indicated in FIG. 15 by (1523).

[0127]

[0148] 16A and 16B show exemplary search points of the MMVD. As shown in FIG. 16A, the starting MV can be indicated in the first reference list L0 by (1611) (e.g., by the direction IDX and the base candidate IDX). In the example of FIG. 16A, four search directions such as +Y, -Y, +X, and -X are used, and the four search directions can be indexed by 0, 1, 2, and 3. The distance can be indexed by 0 (the distance to the starting MV is 0), 1 (1 s (or 1 sample) to the starting MV), 2 (2 s to the starting MV), 3 (3 s to the starting point), etc. Thus, if the direction IDX is 3 and the distance IDX is 2, the final MV predictor is indicated as (1615).

[0128]

[0149] In another example, the search direction and distance can be combined for indexing. For example, in the second reference list L1, the starting point MV is indicated by (1621) (e.g., by the direction IDX and the base candidate IDX). The search direction and distance are combined to be indexed from 0 to 12, as shown in Figure 16B.

[0129]

[0150] In template-matching-based candidate sorting for MMVD and affine MMVD, the MMVD offset can be expanded for both MMVD and affine MMVD modes. In one example, additional refinement positions along the k × π / 8 diagonal can be added first. Exemplary additional refinement positions can be shown in FIG. 17, where the number of directions can be increased from 4 to 16. Second, all possible MMVD refinement positions (e.g., 16 × 6) for each base candidate can be sorted based on the SAD cost between the template (e.g., one row above and one column to the left of the current block) and the template reference for each refinement position. Finally, the top 1 / 8 refinement positions resulting in the smallest template SAD cost can be retained as available positions and, therefore, can be retained for MMVD index coding. The MMVD index can be binarized by a code such as the Rice code with a parameter equal to 2.

[0130]

[0151] In another example, on top of the MMVD extension as described above, affine MMVD reordering or reordering can also be extended, where additional refinement positions along the k×π / 4 diagonal can be added. After reordering, the top half of refinement positions with the smallest template SAD cost can be kept.

[0131]

[0152] To reduce cross-component redundancy, it is possible to use the cross-component linear model (CCLM) prediction mode, as in VVC. In CCLM prediction mode, the chroma samples of the current CU can be predicted based on the reconstructed luma samples of the current CU by using a linear model such as the following equation (1):

[0132]

number

[0133]

[0153] The CCLM parameters (e.g., α and β) can be derived based on at most four nearby chroma samples and the corresponding down-sampled luma samples of the nearby chroma samples. If the current chroma block has dimensions W × H (where W is the width of the current chroma block and H is the height of the current chroma block), the width W' of the first reference region and the height H' of the second reference region can be defined as follows: When LM mode is applied, W'=W,H'=H; When LM-A mode is applied, W' = W + H; When the LM-L mode is applied, H' = H + W. The first reference area can be adjacent to the top of the current chroma block, and the second reference area can be adjacent to the left of the current chroma block.

[0134] In LM mode, neighboring chroma samples can be located in both the first and second reference regions.

[0135] In LM-A mode, nearby chroma samples can be located within the first reference area.

[0136] In LM-L mode, nearby chroma samples can be positioned within a second reference region.

[0137]

[0154] Therefore, the position of the top neighborhood (or the position of the first reference region) is S[0,-1]...S[W'-1,-1] and the left neighbor position (or the position of the second reference region) can be expressed as S[-1,0]...S[-1,H'-1] It can be expressed as: Therefore, the locations of the four nearby chroma samples can be chosen as follows: When LM mode is applied and both top and left nearby samples are available S[W' / 4,-1],S[3*W' / 4,-1],S[-1,H' / 4],S[-1,3*H' / 4]; When LM-A mode is applied or only samples near the top are available S[W' / 8,-1],S[3*W' / 8,-1],S[5*W' / 8,-1],S[7*W' / 8,-1]; When LM-L mode is applied or only the left nearest sample is available S[-1,H' / 8],S[-1,3*H' / 8],S[-1,5*H' / 8],S[-1,7*H' / 8]

[0155] The four luma samples corresponding to the four nearby chroma samples at the selected position are downsampled and compared four times to find the larger of the two values ​​x 0 A and x 1 A , and two smaller values ​​x 0 B and x 1 B The chroma sample values ​​of the four nearby chroma samples corresponding to the four luma samples are y 0 A ,y 1 A ,y 0B , and y 1 B It can be expressed as follows: Parameter X a ,X b ,Y a ,Y b can be derived from equations (2)-(5) as follows:

[0138]

number

[0156] Finally, the linear model parameters α and β can be calculated using equations (6) and (7), respectively.

[0139]

number

[0157] Figure 18 shows example locations of samples near the left and top of the current block, as well as samples of the current block involved in the CCLM prediction mode. As shown in Figure 18, the current chroma CU (1802) may have a size of NxN (e.g., 8x8). The corresponding luma CU (1804) of the current chroma CU (1802) may have a size of 2Nx2N (e.g., 16x16). The chroma samples α and β near the current chroma CU (1802) used to derive the linear model parameters may be located in a first reference region (or near-top position) (1806) and / or a second reference region (or near-left position) (1808). The luma samples corresponding to the nearby chroma samples (1806) and (1808) can be located in a first reference area (1810) and a second reference area (1812), respectively. As shown in Figure 18, the luma samples (1810) and (1812) can be downsampled to match the nearby chroma samples (1806) and (1808) one-to-one. The reconstructed values ​​Rec' of the luma samples (1810) and (1812) are L and the reconstructed values ​​of the nearby chroma samples (1806) and (1808) cand apply it to derive the linear model parameters α and β based on equations (2)-(7). Once the linear model parameters α and β are obtained, the chroma samples in the current chroma CU (1802) can be predicted based on the downsampled reconstructed luma samples in the corresponding luma CU (1804) using the CCLM prediction mode.

[0140]

[0158] The division operation to calculate the parameter α can be implemented using a look-up table. To reduce the memory required to store the table, the diff value (e.g., the difference between the maximum and minimum values) and the parameter α can be expressed in exponential notation. For example, the diff can be approximated with a 4-bit significant part and exponent. Thus, the table for 1 / diff can be reduced to 16 elements for 16 significant values, as shown in Equation (8) below.

[0141]

number

[0159] This provides the advantage of reducing the computational complexity as well as reducing the memory size required to store the required tables.

[0142]

[0160] As shown in Figure 18, a top template (e.g., (1806)) and a left template (e.g., (1808)) can be used together to calculate linear model coefficients (e.g., linear model parameters α and β). The top and left templates can also be used alternatively in two other LM modes, called LM_A (or LM_T) mode and LM_L mode.

[0143]

[0161] In LM_A mode, the linear model coefficients can be calculated using only the upper template. To obtain more samples, the upper template can be extended to (W+H) samples.

[0144] In LM_L mode, the linear model coefficients can be calculated using only the left template. To obtain more samples, the left template can be extended to (H+W) samples.

[0145]

[0162] In the LM_LT mode, both the left and top templates can be used to calculate the linear model coefficients.

[0146]

[0163] To match the chroma sample positions for 4:2:0 video sequences, two types of down-sampling filters can be applied to the luma samples to achieve a 2:1 down-sampling ratio in both the horizontal and vertical directions. The choice of down-sampling filter can be specified by the SPS level flag. The two down-sampling filters can be provided as shown in equations (9) and (10) below, corresponding to "type-0" content and "type-2" content, respectively.

[0147]

number

[0164] Note that if the upper reference line is at a CTU boundary, only one luma line is available to create the downsampled luma samples (e.g., a common line buffer in intra prediction).

[0148]

[0165] The calculation of the parameters (e.g., α and β) can be performed as part of the decoding process and not necessarily as part of the encoder search operation, so that no syntax can be used to communicate the α and β values ​​to the decoder.

[0149]

[0166] For chroma intra mode coding, a total of eight intra modes may be allowed for chroma intra mode coding. The nine intra modes may include five conventional intra modes and three cross-component linear model modes (e.g., CCLM, LM_A, LM_L). The chroma mode signaling and derivation process may be as shown in Table 4. The chroma mode coding of the current chroma block may directly depend on the intra prediction mode of the corresponding luma block. Because separate block partitioning structures for luma and chroma components may be enabled in an I slice, one chroma block may correspond to multiple luma blocks. Therefore, for chroma DM mode, the intra prediction mode of the corresponding luma block covering the center position of the current chroma block may be directly inherited.

[0150] Table 4. Derivation of Chroma Prediction Mode from Luma Mode when CCLM is Enabled

[0151] [Table 4]

[0167] In Table 5, a single binarization table can be used regardless of the value of the flag (eg, sps_cclm_enabled_flag) indicating whether CCLM is enabled or not.

[0152] Table 5. Unified Binarization for Chroma Prediction Modes

[0153] [Table 5]

[0168] As shown in Table 5, the first bin may indicate whether regular (0) mode or LM mode (1) is applied. If the first bin indicates LM mode, the next bin (or the second bin) may indicate whether LM_CHROMA (0) is applied. If the second bin indicates LM_CHROMA, the next bin (or the third bin) may indicate whether LM_L (0) or LM_A (1) is applied. If the CCLM flag (e.g., sps_cclm_enabled_flag) is 0, the first bin of the binarization table of the corresponding chroma intra mode (e.g., intra_chroma_pred_mode) may be discarded before entropy coding. Or, in other words, the first bin may be inferred as 0 and therefore not coded. The binarization table may be used when the CCLM flag (i.e., sps_cclm_enabled_flag) is 0 or 1. The first two bins of Table 5 can be table context coded using the corresponding context model, and the remaining bins can be bypass coded.

[0154]

[0169] Additionally, if a 64x64 luma coding tree node is partitioned using "Not Split" (and ISP is not used for 64x64 CUs) or QT to reduce luma-chroma latency in dual trees, the chroma CUs in a 32x32 / 32x16 chroma coding tree node can be licensed to use CCLM under the following conditions: (1) If a 32x32 chroma node is not split or partitioned by QT splitting, all chroma CUs within the 32x32 node can use CCLM.

[0155] (2) If a 32x32 chroma node is partitioned with horizontal BT and its 32x16 child nodes are not partitioned or use vertical BT partitioning, all chroma CUs within the 32x16 chroma node can use CCLM. In other luma and chroma coding tree splitting conditions, CCLM may not be acceptable for chroma CUs.

[0156]

[0170] In some embodiments, in addition to the 56 directional and 5 non-directional modes for the chroma components, Chroma from Luma (CfL) may be a chroma-only intra prediction mode, as in AV1. CfL can model chroma pixels as a linear function of coincident reconstructed luma pixels. CfL prediction can be expressed as Equation (11) below:

[0157]

number

[0158]

[0171] In CfL mode, if some samples in a co-located luma block are outside the picture boundary, they can be padded and used to calculate the average luma sample. As shown in Figure 20, samples (2002) in the gray area of ​​the current block are outside the picture, and these samples (2002) can be padded by copying the value of the nearest available sample in the current block. For example, sample (2002) can be padded by copying the value of sample (2004), which is the nearest available sample in the current block.

[0159]

[0172] In CfL mode, the luma subsampling step can be combined with the average subtraction step, which can be shown in Figure 19. Therefore, the mathematical expressions for the subsampling step and the average subtraction step can be simplified, and the subsampling division and the corresponding rounding error can be eliminated. The expression corresponding to the combination of both steps can be given in Equation (12), which can be further simplified to Equation (13). Note that both Equations (12) and (13) can use integer division. Let M × N be a matrix of pixels in the luma plane.

[0160]

number

[0173] Based on the supported chroma subsampling, x ×S y ∈{1,2,4} and M×N can also be said to be powers of two, since both M and N are powers of two.

[0161]

[0174] For example, in a 4:2:0 chroma subsampling situation, instead of applying a box filter, the CfL mode simply requires the sum of the four reconstructed luma pixels that match the chroma pixels. 4-tap {1 / 4,1 / 4,1 / 4,1 / 4} filter It is possible to downsample the colocated luma samples to match the chroma resolution using . Then, when a CfL mode scales the luma pixels to improve the accuracy of the prediction, only a scaling by 2 is required.

[0162]

[0175] CCLM may be extended by adding three multi-model LM (MMLM) modes, as in JVET-D0110. In each MMLM mode, reconstructed neighboring samples can be classified into two classes using a threshold that can be taken as the mean of the luma reconstructed neighboring samples. A linear model for each class can be derived using the least-mean-square (LMS) method. For CCLM mode, a linear model can also be derived using the LMS method. A slope adjustment can be applied to CCLM and multi-model LM predictions. This adjustment can tilt the linear function that maps luma values ​​to chroma values ​​about a center point determined by the mean luma value of the reference samples.

[0163]

[0176] CCLM is able to map luma values ​​to chroma values ​​using a two-parameter model. The slope parameter “a” and the bias parameter “b” define the mapping as shown in Equation (14): chromaVal=a*lumaVal+b Eq.(14) The adjustment "u" to the gradient parameter "a" can be signaled to form an updated model as in equation (15) below: chromaVal=a'*lumaVal+b' Eq.(15) where: a'=a+u Eq.(16) b'=bu*y r Eq.(17) Based on the adjustment “u”, the mapping function (e.g., Eq. (15)) calculates the luminance value y r The luminance value y can be tilted or rotated around a point having a center. ris determined based on the average of the reference luma samples used in the model and can result in meaningful modifications to the model. Figure 21A shows a CCLM model with two parameters, "a" and "b". Figure 21B shows an updated CCLM model with two parameters, a' and b', where a' = a + u and b' = bu * y. r is.

[0164]

[0177] The tilt adjustment parameter u may be provided as an integer between (-4) and (4) inclusive and signaled in the bitstream. The units of the tilt adjustment parameter u may be 1 / 8 of a chroma sample value per (or relative to) a luma sample value (e.g., for 10-bit content).

[0165]

[0178] Tilt adjustment can be implemented for CCLM models (e.g., "LM_CHROMA_IDX" and "MMLM_CHROMA_IDX") that use reference samples adjacent to the top and left of a block, but not for "single side" modes that use reference samples adjacent to only one side of a block. The decision to use tilt adjustment can be made based on a trade-off between coding efficiency and coding complexity.

[0166]

[0179] When gradient adjustment is applied to a multi-mode CCLM model, it is possible to adjust both models and thus signal up to two gradient updates for a single chroma block.

[0167]

[0180] The encoder may perform a SATD-based search for the best value of the gradient update for the Cr of the TU, and a similar SATD-based search for the Cb of the TU. If either of the two searches results in a non-zero gradient adjustment parameter, the combined gradient adjustment pair (e.g., a SATD-based update for Cr and a SATD-based update for Cb) may be included in the rate distortion (RD) check list for the TU.

[0168]

[0181] A convolutional cross-component model (CCCM) can be applied to predict chroma samples from the reconstructed luma samples in a manner similar to CCLM. As with CCLM, the reconstructed luma samples can be downsampled to match a lower resolution chroma grid when chroma subsampling is used.

[0169]

[0182] Also, similar to CCLM, single-model or multi-model variants of CCCM can be applied. The multi-model variant can use two models: one model can be derived for samples above the mean luma reference value, and another model can be derived for the remaining samples. The multi-model CCCM mode can be selected for PUs with at least 128 available reference samples.

[0170]

[0183] In CCCM, a convolutional 7-tap filter can be applied. The 7-tap filter may include a spatial 5-tap component with a plus sign shape, a nonlinear term, and a bias term. The input to the spatial 5-tap component of the 7-tap filter may include a center (C) luma sample (which may be co-located with the chroma sample for prediction), and / or a top / north (N), bottom / south (S), left / west (W), and right / east (E) neighboring sample, which may be co-located with the chroma sample for prediction. An exemplary spatial portion (e.g., spatial 5-tap component) (2200) of a conventional 7-tap filter may be that shown in FIG. 22.

[0171]

[0184] As shown in equation (18), the nonlinear term P can be expressed as a power of two of the central luma sample C and can be scaled to the content sample value range (e.g., 10 bits): P=(C*C+midVal) >> bitDepth Eq.(18) That is, for 10-bit content, the nonlinear term P can be calculated as follows: P=(C*C+512) >> 10 Eq.(19) The bias term B can represent a scalar offset between the input and output (similar to the offset term in CCLM) and can be set as an intermediate chroma value (e.g., 512 for 10-bit content). The output of the filter can be shown in Equation (20). The output of the filter is determined by the filter coefficient c i The chroma sample value is calculated based on a convolution between the input and output values, and can be clipped to the range of valid chroma samples.

[0172] predChromaVal = c0C+c1N+c2S+c3E+c4W+c5P+c6B Eq.(20)

[0185] filter coefficient c i can be calculated by minimizing the MSE between the predicted and reconstructed chroma samples in the reference area. Figure 23 shows a reference area (2302) that can include six adjacent lines of chroma samples on the top and left sides of a PU (2304). The reference area (2302) can be extended to the right by 1-PU width (2308) and downward by 1-PU height (2310) beyond the boundary of the PU (2304). The reference area can be adjusted to include only available samples. Extensions (2306) to the reference area (2302) can be included to support "side samples" of plus-shaped spatial filters (e.g., the spatial 5-tap component (2200)), padding out the unavailable areas.

[0173]

[0186] The MSE minimization can be performed by calculating the autocorrelation matrix of the luma input and the cross-correlation vector between the luma input and the chroma output. The autocorrelation matrix can be LDL decomposed, and the final filter coefficients can be calculated using back-substitution. The MSE minimization can follow the calculation of the ALF filter coefficients in the ECM. However, instead of Cholesky decomposition, LDL decomposition may be chosen to avoid using square root operations. Furthermore, only integer arithmetic may be used in the MSE minimization.

[0174]

[0187] In bitstream signaling, the use of CCCM mode can be signaled with a CABAC coded PU-level flag. A new CABAC context can be included to support the PU-level flag. CCCM can be considered a sub-mode of CCLM. That is, the CCCM flag can be signaled only if the intra prediction mode is LM_CHROMA_IDX (to enable single-mode CCCM) or MMLM_CHROMA_IDX (to enable multi-model CCCM).

[0175]

[0188] During encoder operation, the encoder can perform two new RD checks in the chroma prediction mode loop: one to check for single-model CCCM mode and one to check for multi-model CCCM mode.

[0176]

[0189] When a cross-component prediction mode using multiple models is applied, all models may be required to follow the same prediction mode (or cross-component prediction mode). For example, in the MMLM mode, two models are applied, and both models are required to apply the CCLM mode. Another example is the multi-model CCCM method, which requires both models to apply the CCCM mode. However, different cross-component prediction modes can be designed under different image model assumptions, and the restriction of the same prediction mode for multiple models may limit coding performance.

[0177]

[0190] In the present disclosure, a feature may be predetermined from reconstructed samples in a neighborhood of the current block. One of a plurality of cross-component prediction modes may be applied based on a comparison of the feature value to one or more samples of the current block, and in some cases, each sample. In one example, the samples of the current block may be grouped into a plurality of classes (or groups) based on the feature value. A respective cross-component prediction mode may be applied to each of the plurality of groups.

[0178]

[0191] In an embodiment, the feature may be derived from a predetermined function using reconstructed samples (e.g., luma samples) in the vicinity of a luma block that is positioned at the same position as the current chroma block, and / or reconstructed samples (e.g., chroma samples) in the vicinity of the current chroma block.

[0179]

[0192] In one example, the feature can be an average value of reconstructed samples, such as nearby reconstructed samples, which may include nearby reconstructed samples (e.g., luma samples) of a luma block at a comparable position to the current chroma block, or the average value of nearby reconstructed samples (e.g., chroma samples) of the current chroma block.

[0180]

[0193] In one example, the feature may be the average gradient value of nearby reconstructed samples (e.g., nearby reconstructed luma samples) of a luma block at an equivalent position of the current chroma block, or the average gradient value of nearby reconstructed samples (e.g., nearby reconstructed chroma samples) of the current chroma block.

[0181]

[0194] In one example, the feature may be the average value of a pair of nearby chroma samples of the current chroma block and nearby luma samples at the same position in the luma block at the same position as the current chroma block. Thus, the feature may be equal to the average value of nearby chroma samples and nearby luma samples at the same position.

[0182]

[0195] In an embodiment, the cross-component prediction modes may include, but are not limited to, one or more of CCLM, CfL, CCCM, multiple filter linear model (MFLM), GLM, and a combination of CCLM / CfL / CCCM with smooth and angular intra prediction mode.

[0183]

[0196] In embodiments, one or more thresholds of feature values ​​may be defined, such as a threshold T. For each sample (e.g., chroma sample) of the current chroma block, an equally-located luma sample of each chroma sample in an equally-located luma block of the current chroma block may be fetched. The feature value of each equally-located luma sample in the equally-located luma block, i.e., F, may be compared to the threshold T.

[0184] If F is less than (or equal to) T, the chroma samples and their co-located luma samples may be grouped into a set (or group), namely G0.

[0185] If F is greater than (or equal to) T, the chroma samples and their co-located luma samples may be grouped into another set, namely G1.

[0186]

[0197] In one example, the characteristic value of each equally-located luma sample may be the luma sample value of each equally-located luma sample.

[0187] In one example, the feature value of each equally-located luma sample may be the average value of the reconstructed luma samples in the vicinity of the equally-located luma block.

[0188] In one example, the feature value of each equally-located luma sample may be the average gradient value of the reconstructed luma samples in the vicinity of the equally-located luma block.

[0189] In one example, the feature value of each equally-located luma sample may be the average value of nearby reconstructed chroma samples of the current chroma block and nearby reconstructed luma samples of the equally-located luma block.

[0190]

[0198] For samples in G0 (e.g., chroma samples and co-located luma samples), one of multiple cross-component prediction modes can be selected to generate predicted samples. For samples in G1, one of multiple cross-component prediction modes can be selected to generate predicted samples.

[0191]

[0199] An example of grouping chroma samples and equivalently positioned luma samples may be shown in FIG. 24. As shown in FIG. 24, the vertical axis “Cb / Cr” may indicate the chroma sample values ​​of the chroma samples in the current chroma block. The horizontal axis “luma” may indicate the luma sample values ​​of the equivalently positioned luma samples in the equivalently positioned luma block. The feature value T (2402) may be determined as the average value of the nearby reconstructed luma samples. The feature value F of each equivalently positioned luma sample in the equivalently positioned luma block may be determined as the luma sample value of the respective equivalently positioned luma sample.

[0192] If F is less than T, each equally-positioned luma sample of the equally-positioned luma block and its corresponding chroma sample of the current chroma block may be grouped into a first group G0.

[0193] If F is greater than T, each equally-positioned luma sample of the equally-positioned luma block and its corresponding chroma sample of the current chroma block may be grouped into a second group G1.

[0194] Different cross-component prediction modes may also be applied to the first group G0 and the second group G1. In the example of Figure 24, a CCCM mode may be applied to G0 and a GGLM mode may be applied to G1.

[0195]

[0200] In an embodiment, for each group or samples within a selected group, a flag or other coding information may be signaled to indicate which of multiple cross-component prediction modes has been selected.

[0196]

[0201] In one example, when entropy coding is applied to a flag indicating which of multiple cross-component prediction modes is selected for each group, the context of the entropy coding may depend on the flag values ​​of other groups being signaled, e.g., the flag for group G1 may be based on the flag for group G0 being signaled.

[0197]

[0202] In an embodiment, a first flag may be signaled to indicate whether any or all groups (e.g., G0 and G1) share the same cross-component prediction mode. If the first flag is true, another syntax (or syntax element) may be signaled to indicate which cross-component prediction mode is applied. If the first flag is false, a respective syntax may be signaled for each group to indicate which of multiple cross-component prediction modes is applied to the respective group. For example, if the first flag is false, a first syntax may be signaled for G0 and a second syntax may be signaled for G1.

[0198]

[0203] In one example, when the first flag is false and the first cross-component prediction mode is applied to other groups except one or more specific groups, such as the last group, a second cross-component prediction mode for the last group may be implicitly derived or signaled. However, the signaled second cross-component prediction mode may differ from the first cross-component prediction mode. Thus, the second cross-component prediction mode for the last group may be determined excluding the first cross-component prediction mode selected for the other groups.

[0199]

[0204] In an embodiment, an index syntax may be signaled to indicate which cross-component mode from among a plurality of cross-component prediction modes is applied in the first group. i index and group i-1 The delta value, which is the difference between the index of the group i where i may be a positive value greater than 1. For example, an index syntax may be signaled for a first group G0. Additionally, a delta value may be signaled for a second group G1. The delta value may indicate a difference between an index of the first group G0 and an index of the second group G1. The index of the second group G1 may be determined as the sum of the index syntax and the delta value.

[0200]

[0205] In one example, the multiple cross-component prediction modes may include CCLM and CCCM.

[0201]

[0206] In an embodiment, if different filters (or cross-component prediction) are applied to multiple groups of samples (e.g., G0 and G1) within a coding block, a predicted sample can be generated for each group. If all predicted samples for multiple groups are available, a filter can be applied to all predicted samples of the coding block.

[0202]

[0207] In one example, the filter applied to the predicted samples of the coding block may include a smoothing filter, such as a Gaussian, maximum, average, median, minimum, non-local mean, percentile, and rank filter. In one example, the filter applied to the predicted samples of the coding block may be a sharpening filter, such as a Laplace filter.

[0203]

[0208] In an embodiment, multiple prediction blocks of a current block may be initially generated separately according to multiple cross-component modes, and a final prediction block may be determined based on a weighted average of the prediction blocks. For example, as shown in FIG. 24, a first prediction block may be generated based on a CCCM mode. The first prediction block may represent prediction samples of chroma samples in a first group G0. A second prediction block may be generated based on a CCLM mode. The second prediction block may represent prediction samples of chroma samples in a second group G1. The final prediction block may further be determined based on a weighted average of the first prediction block and the second prediction block. The final prediction block may represent prediction samples of chroma samples of the current chroma block (or chroma samples of the current block).

[0204]

[0209] In one example, a flag can be first signaled to indicate whether multiple prediction blocks of the current block are generated separately by multiple cross-component modes and the final prediction block is determined based on a weighted average of the prediction blocks. If the flag is true, a syntax element can be signaled to indicate which cross-component mode is used to generate the weighted average of the prediction blocks.

[0205]

[0210] In one example, the multiple cross-component modes may include CCLM and / or CCCM.

[0206]

[0211] FIG. 25 shows a flowchart outlining an exemplary decoding process (2500) according to some embodiments of the present disclosure. FIG. 26 shows a flowchart outlining an exemplary encoding process (2600) according to some embodiments of the present disclosure. The proposed processes may be used separately or combined in any order. Furthermore, each of the processes (or embodiments), encoders, and decoders may be implemented by processing circuitry (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0207]

[0212] The process operations (e.g., 2500 and 2600) may be combined or arranged in any quantity or order as desired. In embodiments, two or more of the process operations (e.g., 2500 and 2600) may be performed in parallel.

[0208]

[0213] The processes (e.g., 2500 and 2600) may be used in the reconstruction and / or encoding of a block to generate a prediction block for the block being reconstructed. In various embodiments, the processes (e.g., 2500 and 2600) are performed by processing circuitry, such as processing circuitry within the terminal devices (310), (320), (330), and (340), processing circuitry performing the functions of the video encoder (403), processing circuitry performing the functions of the video decoder (410), processing circuitry performing the functions of the video decoder (510), processing circuitry performing the functions of the video encoder (603), etc. In some embodiments, the processes (e.g., 2500 and 2600) are embodied in software instructions, such that the processing circuitry performs the processes (e.g., 2500 and 2600) when it executes the software instructions.

[0209]

[0214] As shown in Figure 25, the process 2500 can start at S2501 and proceed to S2510, where coded information for a current chroma block and a luma block co-located with the current chroma block is received.

[0210]

[0215] In (S2520), a feature value is determined based on at least one of (i) reconstructed chroma samples in the vicinity of the current chroma block, and (ii) reconstructed luma samples in the vicinity of a luma block at a position equivalent to the current chroma block.

[0211]

[0216] In (S2530), based on a threshold value of the feature value, the chroma samples of the current chroma block and the luma samples of the luma block at the same position as the current chroma block are grouped into a plurality of groups, each of which includes a respective chroma sample and a respective luma sample.

[0212]

[0217] At (S2540), a respective cross-component prediction mode for each of the plurality of groups is determined by comparing a respective chroma sample and a respective luma sample for each respective group with the determined feature value.

[0213]

[0218] At (S2550), the current chroma block is reconstructed based on the determined cross-component prediction modes of the plurality of groups.

[0214]

[0219] In one example, the feature value is determined as one of the average value of the reconstructed chroma samples in the vicinity of the current chroma block and the average value of the reconstructed luma samples in the vicinity of the co-located luma block.

[0215]

[0220] In one example, the feature value is determined as one of the average gradient value of the reconstructed chroma samples in the vicinity of the current chroma block and the average gradient value of the reconstructed luma samples in the vicinity of the luma block at the same position.

[0216]

[0221] In one example, the feature value is determined as the average value of the reconstructed chroma samples in the vicinity of the current chroma block and the reconstructed luma samples in the vicinity of the co-located luma block.

[0217]

[0222] To group the chroma samples of the current chroma block and the luma samples of the co-located luma block, the characteristic values ​​associated with each luma sample of the co-located luma block are determined. It is determined whether the characteristic values ​​associated with each luma sample of the co-located luma block are greater than a threshold feature value.

[0218] In one example, (i) each luma sample among the luma samples of a co-located luma block and (ii) each chroma sample among the chroma samples of the current chroma block corresponding to each luma sample are grouped into a first group based on the characteristic value associated with each luma sample being greater than a feature value threshold.

[0219] In one example, (i) each luma sample among the luma samples of the co-located luma block and (ii) each chroma sample among the chroma samples of the current chroma block corresponding to each luma sample are grouped into a second group based on the characteristic value associated with each luma sample being less than a threshold feature value.

[0220]

[0223] In some embodiments, the characteristic value of each luma sample includes one of (i) the luma sample value of the respective luma sample, and (ii) the average value of reconstructed luma samples in the vicinity of the co-located luma block.

[0221]

[0224] In some embodiments, the respective cross-component prediction mode includes one of a cross-component linear model (CCLM), chroma-from-luma (CfL), a convolutional cross-component model (CCCM), a multi-filter linear model (MFLM), a gradient linear model (GLM), a combination of CCLM, CfL, and CCCM, and an angular intra prediction mode.

[0222]

[0225] In some embodiments, the respective cross-component prediction mode is determined based on a corresponding flag included in the coded information.

[0223]

[0226] In an embodiment, it is determined whether each of the multiple groups share a cross-component prediction mode based on a flag included in the coded information.

[0224] In one example, a cross-component prediction mode for each of the plurality of groups is determined based on the flag indicating that each of the plurality of groups shares a cross-component prediction mode.

[0225] In one example, based on the flag indicating that each of the multiple groups does not share a cross-component prediction mode, a respective cross-component prediction mode for each of the multiple groups is determined based on a corresponding syntax element.

[0226]

[0227] In an embodiment, a first cross-component prediction mode is determined for a first group of the plurality of groups based on a first index included in the coded information, and a second cross-component prediction mode is determined for a second group of the plurality of groups based on a second index, the second index being determined based on a sum of a delta index and the first index included in the coded information.

[0227]

[0228] To reconstruct the current chroma block, in one example, prediction samples for the chroma samples in each of the multiple groups are generated based on the respective cross-component prediction modes, and a filter can then be applied to the prediction samples for the chroma samples in each of the multiple groups.

[0228]

[0229] In one example, a prediction sample for the chroma samples in each of the plurality of groups is generated based on the respective cross-component prediction mode, and a prediction sample for the current chroma block is determined as a weighted combination of the prediction samples for the chroma samples in each of the plurality of groups.

[0229]

[0230] After (S2550), the process proceeds to (S2599) and ends.

[0230]

[0231] The method 2500 may be adapted as appropriate. Steps in the process 2500 may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0231]

[0232] As shown in Figure 26, the process 2600 can start at S2601 and proceed to S2610, where a feature value is determined based on at least one of (i) reconstructed chroma samples in the vicinity of the current chroma block and (ii) reconstructed luma samples in the vicinity of a luma block co-located with the current chroma block.

[0232]

[0233] In (S2620), based on a threshold value of the feature value, the chroma samples of the current chroma block and the luma samples of the luma block at the same position as the current chroma block are grouped into a plurality of groups, where each of the plurality of groups includes a respective chroma sample and a respective luma sample.

[0233]

[0234] At (S2630), a respective cross-component prediction mode for each of the plurality of groups is determined by comparing a respective chroma sample and a respective luma sample for each respective group with the determined feature value.

[0234]

[0235] At (S2640), a predicted sample of the current chroma block is generated based on the determined cross-component prediction mode of the plurality of groups, and coding information indicating the respective cross-component prediction modes of the plurality of groups is also generated.

[0235]

[0236] The process then proceeds to (S2699) and ends.

[0236]

[0237] The method 2600 may be adapted as appropriate. Steps in the process 2600 may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0237]

[0238] The techniques described above may be implemented as computer software using computer-readable instructions and may be physically stored on one or more computer-readable media. For example, Figure 27 illustrates a computer system (2700) suitable for implementing certain embodiments of the disclosed subject matter.

[0238]

[0239] Computer software may be coded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms to create code that contains instructions that may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or that may go through interpretation, microcode execution, etc.

[0239]

[0240] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0240]

[0241] 27 for computer system 2700 are exemplary in nature and are not intended to suggest any limitation on the scope of functionality or application of the computer software implementing embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement related to any one or combination of components illustrated in the exemplary embodiment of computer system 2700.

[0241]

[0242] The computer system 2700 may include certain human interface input devices that may respond to input by one or more human users, for example, via tactile input (e.g., keystrokes, swipes, data glove movements), auditory input (e.g., voice, claps), visual input (e.g., gestures), or olfactory input (not shown). Human interface devices may also be used to capture certain media that do not necessarily involve direct human conscious input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still-image cameras), and video (e.g., two-dimensional video, three-dimensional video, including stereoscopic pictures).

[0242]

[0243] The input human interface devices may include one or more of (only one of each is depicted) a keyboard (2701), a mouse (2702), a trackpad (2703), a touch screen (2710), a data glove (not shown), a joystick (2705), a microphone (2706), a scanner (2707), and a camera (2708).

[0243]

[0244] The computer system 2700 may also include certain human interface output devices that may stimulate one or more of the human user's senses, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices can include haptic output devices (e.g., haptic feedback via touch screens (2710), data gloves (not shown), joysticks (2705), although there can be haptic feedback devices that do not serve as input devices), auditory output devices (e.g., speakers (2709), headphones (not shown)), visual output devices (e.g., screens (2710), including CRT screens, LCD screens, plasma screens, and OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output, three-dimensional or higher output by means such as stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0244]

[0245] The computer system (2700) may also include human-accessible storage devices and associated media, such as optical media including CD / DVD ROM / RW (2720) using media such as CD / DVD (2721), thumb drives (2722), removable hard drives or solid state drives (2723), legacy magnetic media (not shown) such as tape and floppy disks (not shown), and specialized ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0245]

[0246] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not encompass transmission media, carrier waves, or other transitional signals.

[0246]

[0247] The computer system (2700) may also include interfaces to one or more communications networks (2755). Networks may be, for example, wireless, wired, or optical. Networks may further be local, wide area, metropolitan, automotive, real-time, delay-tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), TV wired or wireless wide area digital networks (including cable TV, satellite TV, and terrestrial TV), automotive networks including CANBus, etc. Particular networks generally require an external network interface adapter attached to a particular general-purpose data port or peripheral bus (2749) (e.g., a USB port on the computer system (2700)); others are commonly integrated into the core of the computer system (2700) by attaching to a system bus, as described below (e.g., an Ethernet interface is integrated in a PC computer system, and a cellular network interface is integrated in a smartphone computer system). Using any of these networks, the computer system (2700) can communicate with other entities. Such communication can be one-way receive-only (e.g., broadcast TV), one-way transmit-only (e.g., CANbus to certain CANbus devices), or bidirectional, such as with other computer systems using local or wide-area digital networks. Specific protocols and protocol stacks, as described above, can be used with each of these networks and network interfaces.

[0247]

[0248] The aforementioned human interface devices, human accessible storage devices, and network interfaces can be attached to the core (2740) of the computer system (2700).

[0248]

[0249] The core (2740) may include one or more central processing units (CPUs) (2741), graphics processing units (GPUs) (2742), specialized programmable processing units in the form of field programmable gate arrays (FPGAs) (2743), task-specific hardware accelerators (2744), graphics adapters (2750), etc. These devices, along with read-only memory (ROM) (2745), random access memory (2746), and internal mass storage devices (e.g., internal non-user-accessible hard drives, SSDs, etc.) (2747), may be connected via a system bus (2748). In some computer systems, the system bus (2748) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (2748) or via a peripheral bus (2749). In one example, the screen 2710 can be connected to a graphics adapter 2750. Peripheral bus architectures include PCI, USB, etc.

[0249]

[0250] The CPU (2741), GPU (2742), FPGA (2743), and accelerator (2744) may combine to execute specific instructions that may constitute the aforementioned computer code. The computer code may be stored in ROM (2745) or RAM (2746). Temporary data may be stored in RAM (2746), while persistent data may be stored in, for example, internal mass storage (2747). Rapid storage and retrieval from any memory device may be enabled through the use of cache memory, which may be closely associated with one or more of the CPU (2741), GPU (2742), mass storage (2747), ROM (2745), RAM (2746), etc.

[0250]

[0251] The computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and computer code can be those specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those having skill in the computer software arts.

[0251]

[0252] By way of example, and not limitation, a computer system having the architecture (2700), and in particular the core (2740), can provide functionality as a result of operations by a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, as well as specific storage of the core (2740) that is non-transitory in nature, such as the core's internal mass storage (2747) or ROM (2745). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (2740). The computer-readable media can include one or more memory devices or chips, depending on particular needs. The software can cause the core (2740), and in particular the processor (including a CPU, GPU, FPGA, etc.) therein, to perform certain processes or portions of certain processes described herein, including defining data structures stored in RAM (2746) and modifying such data structures according to processes defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embedded in circuitry (e.g., accelerator (2744)), which may execute in place of or in conjunction with software to perform the particular processes or portions of particular processes described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry embodying logic for execution, or both, as appropriate. The present disclosure encompasses any appropriate combination of hardware and software.

[0252] Appendix A: Acronyms JEM:joint exploration model VVC:versatile video coding BMS:benchmark set MV: Motion Vector HEVC:High Efficiency Video Coding SEI:Supplementary Enhancement Information VUI:Video Usability Information GOPs:Groups of Pictures TUs:Transform Units PUs:Prediction Units CTUs:Coding Tree Units CTBs:Coding Tree Blocks PBs:Prediction Blocks HRD:Hypothetical Reference Decoder SNR:Signal Noise Ratio CPUs:Central Processing Units GPUs:Graphics Processing Units CRT:Cathode Ray Tube LCD:Liquid-Crystal Display OLED:Organic Light-Emitting Diode CD:Compact Disc DVD:Digital Video Disc ROM:Read-Only Memory RAM:Random Access Memory ASIC:Application-Specific Integrated Circuit PLD:Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile communications LTE: Long-Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI:Peripheral Component Interconnect FPGA:Field Programmable Gate Areas SSD: solid-state drive IC: Integrated Circuit CU: Coding Unit

[0253] While this disclosure describes a number of exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be understood that those skilled in the art will be able to devise many systems and methods that, while not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope.

[0253]

[0254] Additional notes (Appendix 1) 1. A video decoding method performed in a video decoder, comprising: receiving coded information of a current chroma block and a luma block co-located with the current chroma block; determining a feature value based on at least one of (i) reconstructed chroma samples in a vicinity of the current chroma block, and (ii) reconstructed luma samples in a vicinity of a luma block co-located with the current chroma block; grouping the chroma samples of the current chroma block and the luma samples of the luma block co-located with the current chroma block into a plurality of groups based on a threshold value of the feature value, each of the plurality of groups including a respective chroma sample and a respective luma sample; determining a respective cross-component prediction mode for each of the plurality of groups by comparing the respective chroma samples and the respective luma samples of each respective group to determined feature values; and reconstructing the current chroma block based on the determined cross-component prediction modes of the plurality of groups; A method comprising:

[0254] (Appendix 2) 10. The method of claim 1, wherein the step of determining the feature value further comprises: determining the feature value as one of an average value of reconstructed chroma samples in a neighborhood of the current chroma block and an average value of reconstructed luma samples in a neighborhood of the equally positioned luma block; A method comprising:

[0255] (Appendix 3) 10. The method of claim 1, wherein the step of determining the feature value further comprises: determining the feature value as one of an average gradient value of reconstructed chroma samples in a neighborhood of the current chroma block and an average gradient value of reconstructed luma samples in a neighborhood of the equally positioned luma block; A method comprising:

[0256] (Appendix 4) 10. The method of claim 1, wherein the step of determining the feature value further comprises: determining the feature value as an average value of reconstructed chroma samples in a neighborhood of the current chroma block and reconstructed luma samples in a neighborhood of the equally positioned luma block; A method comprising:

[0257] (Appendix 5) 10. The method of claim 1, wherein the grouping step further comprises: determining a characteristic value associated with each luma sample of the co-located luma block; determining whether a characteristic value associated with each luma sample of the co-located luma block is greater than the feature value threshold; grouping (i) each luma sample among the luma samples of the equivalently positioned luma block and (ii) each chroma sample among the chroma samples of the current chroma block corresponding to the each luma sample into a first group based on a characteristic value associated with the each luma sample being greater than the feature value threshold; and grouping (i) each luma sample among the luma samples of the equivalently positioned luma block and (ii) each chroma sample among the chroma samples of the current chroma block corresponding to each luma sample into a second group based on the characteristic value associated with the each luma sample being less than the feature value threshold; A method comprising:

[0258] (Appendix 6) 6. The method of claim 5, wherein the characteristic value of each luma sample comprises one of: (i) a luma sample value of the each luma sample; and (ii) an average value of reconstructed luma samples in the vicinity of the equivalently positioned luma block.

[0259] (Appendix 7) 10. The method of claim 1, wherein each cross-component prediction mode comprises one of a cross-component linear model (CCLM), chroma-from-luma (CfL), a convolutional cross-component model (CCCM), a multi-filter linear model (MFLM), a gradient linear model (GLM), a combination of CCLM, CfL, and CCCM, and an angular intra prediction mode.

[0260] (Appendix 8) 10. The method of claim 1, wherein determining a respective cross-component prediction mode for each of the plurality of groups further comprises: determining the respective cross-component prediction modes based on corresponding flags included in the coded information; A method comprising:

[0261] (Appendix 9) 10. The method of claim 1, wherein determining a respective cross-component prediction mode for each of the plurality of groups further comprises: determining whether each of the plurality of groups shares a cross-component prediction mode based on a flag included in the coded information; determining a cross-component prediction mode for each of the plurality of groups in response to the flag indicating that each of the plurality of groups shares a cross-component prediction mode; and determining a respective cross-component prediction mode for each of the plurality of groups based on a corresponding syntax element in response to the flag indicating that each of the plurality of groups does not share a cross-component prediction mode; A method comprising:

[0262] (Appendix 10) 10. The method of claim 1, wherein determining a respective cross-component prediction mode for each of the plurality of groups further comprises: determining a first cross-component prediction mode for a first group of the plurality of groups based on a first index included in the coded information; and determining a second cross-component prediction mode for a second group of the plurality of groups based on a second index, the second index being determined based on a sum of a delta index included in the coded information and the first index; A method comprising:

[0263] (Appendix 11) 2. The method of claim 1, wherein the step of reconstructing the current chroma block comprises: generating prediction samples for the chroma samples in each of the plurality of groups based on the respective cross-component prediction modes; and applying a filter to prediction samples for the chroma samples in each of the plurality of groups; A method comprising:

[0264] (Appendix 12) 2. The method of claim 1, wherein the step of reconstructing the current chroma block comprises: generating prediction samples for the chroma samples in each of the plurality of groups based on the respective cross-component prediction modes; and determining a prediction sample for the current chroma block as a weighted combination of prediction samples for the chroma samples in each of the plurality of groups; A method comprising:

[0265] (Appendix 13) 1. An apparatus including a processing circuit, the processing circuit comprising: receiving coded information of a current chroma block and a luma block co-located with the current chroma block; determining a feature value based on at least one of (i) reconstructed chroma samples in a vicinity of the current chroma block, and (ii) reconstructed luma samples in a vicinity of a luma block co-located with the current chroma block; grouping the chroma samples of the current chroma block and the luma samples of the luma block co-located with the current chroma block into a plurality of groups based on a threshold value of the feature value, each of the plurality of groups including a respective chroma sample and a respective luma sample; determining a respective cross-component prediction mode for each of the plurality of groups by comparing the respective chroma samples and the respective luma samples of each respective group to determined feature values; and reconstructing the current chroma block based on the determined cross-component prediction modes of the plurality of groups; 20. An apparatus configured to:

[0266] (Appendix 14) 14. The apparatus of claim 13, wherein the processing circuitry: determining the feature value as one of an average value of reconstructed chroma samples in a neighborhood of the current chroma block and an average value of reconstructed luma samples in a neighborhood of the equally positioned luma block; 20. An apparatus configured to:

[0267] (Appendix 15) 14. The apparatus of claim 13, wherein the processing circuitry: determining the feature value as one of an average gradient value of reconstructed chroma samples in a neighborhood of the current chroma block and an average gradient value of reconstructed luma samples in a neighborhood of the equally positioned luma block; 20. An apparatus configured to:

[0268] (Appendix 16) 14. The apparatus of claim 13, wherein the processing circuitry: determining the feature value as an average value of reconstructed chroma samples in a neighborhood of the current chroma block and reconstructed luma samples in a neighborhood of the equally positioned luma block; 20. An apparatus configured to:

[0269] (Appendix 17) 14. The apparatus of claim 13, wherein the processing circuitry: determining a characteristic value associated with each luma sample of the co-located luma block; determining whether a characteristic value associated with each luma sample of the co-located luma block is greater than the feature value threshold; grouping (i) each luma sample among the luma samples of the equivalently positioned luma block and (ii) each chroma sample among the chroma samples of the current chroma block corresponding to the each luma sample into a first group based on a characteristic value associated with the each luma sample being greater than the feature value threshold; and grouping (i) each luma sample among the luma samples of the equivalently positioned luma block and (ii) each chroma sample among the chroma samples of the current chroma block corresponding to each luma sample into a second group based on the characteristic value associated with the each luma sample being less than the feature value threshold; 20. An apparatus configured to:

[0270] (Appendix 18) 18. The apparatus of claim 17, wherein the characteristic value of each luma sample comprises one of: (i) a luma sample value of the each luma sample; and (ii) an average value of reconstructed luma samples in a neighborhood of the equivalently positioned luma block.

[0271] (Appendix 19) 14. The apparatus of claim 13, wherein each cross-component prediction mode comprises one of a cross-component linear model (CCLM), a chroma-from-luma (CfL), a convolutional cross-component model (CCCM), a multi-filter linear model (MFLM), a gradient linear model (GLM), a combination of CCLM, CfL, and CCCM, and an angular intra prediction mode.

[0272] (Appendix 20) 14. The apparatus of claim 13, wherein the processing circuitry: determining the respective cross-component prediction modes based on corresponding flags included in the coded information; 20. An apparatus configured to:

Claims

1. 1. A video decoding method performed in a video decoder, comprising: receiving coded information of a current chroma block and a luma block co-located with the current chroma block; determining a feature value based on at least one of (i) reconstructed chroma samples in a neighborhood of the current chroma block, and (ii) reconstructed luma samples in a neighborhood of a luma block co-located with the current chroma block; grouping the chroma samples of the current chroma block and the luma samples of a luma block co-located with the current chroma block into a plurality of groups based on a threshold value of the feature value, each of the plurality of groups including a respective chroma sample and a respective luma sample; determining a respective cross-component prediction mode for each of the plurality of groups by comparing the respective chroma samples and the respective luma samples of each respective group to determined feature values, wherein which of the plurality of cross-component prediction modes is selected is based on a signaled syntax element; and reconstructing the current chroma block based on the determined cross-component prediction modes of the plurality of groups; A method comprising:

2. 10. The method of claim 1, wherein the step of determining the feature value further comprises: determining the feature value as one of an average value of reconstructed chroma samples in a neighborhood of the current chroma block and an average value of reconstructed luma samples in a neighborhood of the equally positioned luma block; A method comprising:

3. 10. The method of claim 1, wherein the step of determining the feature value further comprises: determining the feature value as one of an average gradient value of reconstructed chroma samples in a neighborhood of the current chroma block and an average gradient value of reconstructed luma samples in a neighborhood of the equally positioned luma block; A method comprising:

4. 10. The method of claim 1, wherein the step of determining the feature value further comprises: determining the feature value as an average value of reconstructed chroma samples in the vicinity of the current chroma block and reconstructed luma samples in the vicinity of the equally positioned luma block; A method comprising:

5. 10. The method of claim 1, wherein the grouping step further comprises: determining a characteristic value associated with each luma sample of the co-located luma block; determining whether a characteristic value associated with each luma sample of the co-located luma block is greater than the feature value threshold; grouping (i) each luma sample among the luma samples of the co-located luma block and (ii) a chroma sample among the chroma samples of the current chroma block corresponding to the each luma sample into a first group based on a characteristic value associated with the each luma sample being greater than the feature value threshold; and grouping (i) each luma sample among the luma samples of the equivalently positioned luma block and (ii) each chroma sample among the chroma samples of the current chroma block corresponding to the each luma sample into a second group based on the characteristic value associated with the each luma sample being less than the feature value threshold; A method comprising:

6. 6. The method of claim 5, wherein the characteristic value of each luma sample comprises one of: (i) a luma sample value of the each luma sample; and (ii) an average value of reconstructed luma samples in a neighborhood of the equally positioned luma block.

7. 10. The method of claim 1, wherein each cross-component prediction mode comprises one of a cross-component linear model (CCLM), a chroma-from-luma (CfL), a convolutional cross-component model (CCCM), a multi-filter linear model (MFLM), a gradient linear model (GLM), a combination of CCLM, CfL, and CCCM, and an angular intra prediction mode.

8. 10. The method of claim 1, wherein determining a respective cross-component prediction mode for each of the plurality of groups further comprises: determining the respective cross-component prediction modes based on corresponding flags included in the coded information; A method comprising:

9. 10. The method of claim 1, wherein determining a respective cross-component prediction mode for each of the plurality of groups further comprises: determining whether each of the plurality of groups shares a cross-component prediction mode based on a flag included in the coded information; determining a cross-component prediction mode for each of the plurality of groups in response to the flag indicating that each of the plurality of groups shares a cross-component prediction mode; and determining a respective cross-component prediction mode for each of the plurality of groups based on a corresponding syntax element in response to the flag indicating that each of the plurality of groups does not share a cross-component prediction mode; A method comprising:

10. 10. The method of claim 1, wherein determining a respective cross-component prediction mode for each of the plurality of groups further comprises: determining a first cross-component prediction mode for a first group of the plurality of groups based on a first index included in the coded information; and determining a second cross-component prediction mode for a second group of the plurality of groups based on a second index, the second index being determined based on a sum of a delta index included in the coded information and the first index; A method comprising:

11. 10. The method of claim 1, wherein the step of reconstructing the current chroma block comprises: generating prediction samples for the chroma samples in each of the plurality of groups based on the respective cross-component prediction modes; and applying a filter to prediction samples for the chroma samples in each of the plurality of groups; A method comprising:

12. 10. The method of claim 1, wherein the step of reconstructing the current chroma block comprises: generating prediction samples for the chroma samples in each of the plurality of groups based on the respective cross-component prediction modes; and determining a prediction sample for the current chroma block as a weighted combination of prediction samples for the chroma samples in each of the plurality of groups; A method comprising:

13. 13. Apparatus comprising processing circuitry for carrying out the method of any one of claims 1-12.

14. 1. A video encoding method performed in a video encoder, comprising: creating an encoded bitstream including coded information of a current chroma block and a luma block co-located with the current chroma block; and transmitting the encoded bitstream to a video decoder, the video encoder comprising: based on a feature value threshold determined based on at least one of (i) reconstructed chroma samples in the vicinity of the current chroma block, and (ii) reconstructed luma samples in the vicinity of a luma block co-located with the current chroma block, the chroma samples of the current chroma block and the luma samples of the luma block co-located with the current chroma block are grouped into a plurality of groups, each of the plurality of groups including a respective chroma sample and a respective luma sample; a respective cross-component prediction mode for each of the plurality of groups is determined based on a comparison of the respective chroma sample and the respective luma sample of each respective group with the feature value, and which of the plurality of cross-component prediction modes is selected based on a signaled syntax element; and The method, wherein the current chroma block is reconstructed based on the determined cross-component prediction modes of the plurality of groups.

Images