Mixed Model Cross-Component Prediction Mode

Abstract

A feature value is determined based on at least one of (i) reconstructed chroma samples in a neighborhood of a current chroma block, and (ii) reconstructed luma samples in a neighborhood of a luma block co-located with the current chroma block. Based on a threshold value of the feature value, the chroma samples of the current chroma block and the luma samples of the luma block co-located with the current chroma block are grouped into a plurality of groups. Each of the plurality of groups includes a respective chroma sample and a respective luma sample. A respective cross-component prediction mode for each of the plurality of groups is determined by comparing the respective chroma sample and the respective luma sample of each of the respective groups with the determined feature value. The current chroma block is reconstructed based on the determined cross-component prediction modes of the plurality of groups.

Description

[Technical field]

[0001]

[0001] Incorporation by Reference This application claims priority to U.S. Patent Application No. 17 / 982,189, entitled "Mixed Model Cross-Component Prediction Mode," filed November 7, 2022, which claims priority to U.S. Provisional Application No. 63 / 388,117, entitled "Mixed Model Cross-Component Prediction Mode," filed July 11, 2022. The disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002]

[0002] Technical Field This disclosure describes embodiments generally relating to video coding. [Background technology]

[0003] The background statement provided in this case is intended to generally present the context of the present disclosure. Work under the names of the current inventors is not admitted, expressly or impliedly, as prior art to the present disclosure to the extent that such work is described in this background section or in a descriptive manner that may not otherwise qualify as prior art at the time of filing.

[0004]

[0004] Uncompressed digital images and / or videos may include a sequence of pictures, each having spatial dimensions of, for example, 1920x1080 luminance samples and associated chrominance samples. The sequence of pictures may have a fixed or variable picture rate (informally known as the frame rate), for example, 60 pictures / second, or 60 Hz. Uncompressed images and / or videos have significant bitrate requirements. For example, 1080p60 4:2:0 video (1920x1080 luminance sample resolution at a frame rate of 60 Hz) with 8 bits per sample requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 Gbytes of storage space.

[0005]

[0005] One of the goals of image and / or video encoding and decoding can be said to be the reduction of redundancy in the input image and / or video signal through compression. Compression can help reduce the aforementioned bandwidth and / or storage space requirements, in some cases by more than one order of magnitude. The description in this application uses video encoding / decoding as an illustrative example, but the same techniques can be applied to image encoding / decoding in a similar manner without departing from the spirit of the present disclosure. Both lossless and non-lossless compression, as well as combinations thereof, can be used. Lossless compression (reversible compression) refers to techniques whereby an exact copy of the original signal can be reconstructed from the compressed original signal. When using non-lossless compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original signal and the reconstructed signal is small enough that the reconstructed signal is useful for the intended application. For video, non-lossless compression is widely used. The amount of distortion that is tolerated depends on the application; for example, users of a particular consumer streaming application may be able to tolerate higher distortion than users of a television distribution application. The achievable compression ratio may reflect that a higher tolerable / tolerable distortion may result in a higher compression ratio.

[0006]

[0006] Video encoders and decoders may utilize techniques in several broad categories, including, for example, motion compensation, transform processing, quantization, and entropy coding.

[0007]

[0007] Video codec techniques may include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially divided into blocks of samples. If all blocks of samples are coded in intra mode, the picture may be an intra picture. Intra pictures and their derivatives, such as independent decoder refresh pictures, may be used to reset the decoder state and therefore may be used as the first picture in a coded video bitstream and video session or as still images. Samples of an intra block are subjected to a transform and the transform coefficients may be quantized before entropy coding. Intra prediction may be a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after the transform and the smaller the AC coefficients, the fewer bits are required for a given quantization step size to represent the block after entropy coding.

[0008]

[0008] Conventional intra-coding, for example as used in the MPEG-2 generation of coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to perform prediction based on surrounding sample data and / or metadata obtained during the encoding and / or decoding of a block of data. Such techniques are hereinafter referred to as "intra-prediction" techniques. It should be noted that, at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed, and does not use reference data from a reference picture.

[0009]

[0009] There may be many different forms of intra prediction. When more than one such technique may be used in a given video coding technique, the particular technique used may be coded as a particular intra prediction mode that uses that particular technique. In certain cases, an intra prediction mode may have submodes and / or parameters, in which case the submodes and / or parameters may be coded separately or included in a mode codeword, which defines the prediction mode used. Which codeword is used for a given mode, submode, and / or parameter combination may affect the coding efficiency gain from intra prediction, and may also affect the entropy coding technique used to convert the codeword into a bitstream.

[0010]

[0010] A specific mode of intra prediction was introduced in H.264, improved in H.265, and further refined in newer coding techniques such as JEM (joint exploration model), VVC (versatile video coding), and MBS (benchmark set). A prediction block can be formed using sample values ​​in the neighborhood of already available samples. The sample values ​​of the neighboring samples are copied to the predictor block according to a certain direction. The reference to the direction to use can be coded in the bitstream or can be predicted itself.

[0011]

[0011] Referring to FIG. 1A, depicted at the bottom right is a subset of 9 predictor directions known from the 33 possible predictor directions defined in H.265 (corresponding to the 33 angle modes of the 35 intra modes). The point where the arrows converge (101) represents the sample to be predicted. The arrows represent the direction from which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from a sample that is at an angle of 45 degrees from the horizontal and points to the upper right. Similarly, arrow (103) indicates that sample (101) is predicted from a sample that is at an angle of 22.5 degrees from the horizontal and points to the lower left of sample (101).

[0012]

[0012] With further reference to FIG. 1A, at the top left, a square block (104) of 4x4 samples is depicted (indicated by the dashed bold line). The square block (104) contains 16 samples, each labeled with "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in the block (104) in both the Y and X dimensions. Since the size of the block is 4x4 samples, S44 is located at the bottom right. Also shown are reference samples that follow a similar numbering scheme. The reference samples are labeled with their Y position (e.g., row index) and X position (column index) relative to the block (104) and R. In both H.264 and H.265, since the prediction samples are in the neighborhood of the block being reconstructed, there is no need to use negative values.

[0013]

[0013] Intra-picture prediction can be performed by copying reference sample values ​​from nearby samples indicated by the signaled prediction direction. For example, assume that the coded video bitstream contains signaling indicating for this block a prediction direction consistent with the arrow (102), i.e., the sample is predicted from a sample pointing to the upper right at an angle of 45 degrees from the horizontal. In this case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Thus, sample S44 is predicted from reference sample R08.

[0014] In certain cases, values ​​of multiple reference samples can be combined, for example by interpolation, to calculate a reference sample; especially when the orientation is not evenly divisible by 45 degrees.

[0015]

[0015] As video coding techniques develop, the number of possible directions is increasing. In H.264 (2003), nine different directions could be represented. This has increased to 33 in H.265 (2013). Currently, JEM / VVC / BMS can support up to 65 directions. Experiments have been carried out to identify the most likely directions, using certain techniques in entropy coding to represent these more likely directions with fewer bits, while accepting certain penalties for less likely directions. Furthermore, the directions themselves can sometimes be predicted from nearby directions used in nearby already decoded blocks.

[0016] FIG. 1B shows a diagram (110) illustrating the 65 intra prediction directions according to JEM, showing the increasing number of prediction directions over time.

[0017]

[0017] The mapping of intra-prediction direction bits to represent directions in the coded video bitstream may vary from one video coding technique to another. Such mappings may range from simple direct mappings to complex adaptation schemes involving codewords, most likely modes, and similar techniques. However, in most cases, there may be certain directions that are statistically less likely to occur in the video content than other certain directions. Since the goal of video compression is to reduce redundancy, these less likely directions are represented by a larger number of bits than the more likely directions in well-performing video coding techniques.

[0018]

[0018] Image and / or video encoding and decoding may be performed using inter-picture prediction with motion compensation. Motion compensation may be a non-lossless compression technique and may relate to a technique in which blocks of sample data from a previously reconstructed image or part thereof (reference image) are spatially shifted in a direction indicated by a motion vector (hereafter MV) and used for the prediction of a newly reconstructed picture or part of a picture. In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions X and Y, or three dimensions, the third being an index of the reference picture to be used (the latter may indirectly be a temporal dimension).

[0019]

[0019] In some video compression techniques, the MV applicable to a given region of sample data can be predicted from other MVs, for example from an MV associated with another area of ​​sample data that is spatially adjacent to the area being reconstructed and precedes that MV in decoding order. By doing so, the amount of data required to code the MV can be significantly reduced, thereby eliminating redundancy and increasing the degree of compression. For example, when coding an input video signal derived from a camera (known as natural video), MV prediction can work effectively because there is a statistical likelihood that areas larger than the area to which a single MV is applicable move in similar directions and therefore can in some cases be predicted using similar motion vectors derived from MVs of neighboring areas. This results in an MV that, for a given area, is found to be similar or identical to the MV predicted from the surrounding MVs, which, after entropy coding, can be represented with fewer bits than would be used to code the MV directly. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., an MV) derived from the original signal (i.e., a sample stream). In other cases, the MV prediction itself may be non-lossless due to rounding errors, for example when computing a predictor from several surrounding MVs.

[0020]

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms provided by H.265, one that will be described with reference to Fig. 2 is a technique hereafter referred to as "spatial merging".

[0021]

[0021] Referring to Fig. 2, the current block (201) contains samples that have been found by the encoder during the motion search process to be predictable from a previous block of the same size but spatially shifted. Instead of coding its MV directly, the MV can be derived from metadata associated with one or more reference pictures, e.g., from the most recent reference picture (in decoding order), using MVs associated with any of the five surrounding samples (202 to 206, respectively), denoted as A0, A1, and B0, B1, B2. In H.265, MV prediction can use predictors from the same reference picture as the nearby blocks use. Summary of the Invention

[0022]

[0022] Aspects of the disclosure provide a method and apparatus for video encoding / decoding. In some examples, the video decoding apparatus includes a processing circuit.

[0023]

[0023] According to an aspect of the disclosure, a video decoding method is provided that is performed in a video decoder. In the method, coded information of a current chroma block and a luma block collocated with the current chroma block may be received. A feature value may be determined based on at least one of (i) neighboring reconstructed chroma samples of the current chroma block and (ii) neighboring reconstructed luma samples of the luma block collocated with the current chroma block. The chroma samples of the current chroma block and the luma samples of the luma block collocated with the current chroma block may be grouped into a plurality of groups based on a feature value threshold. Each of the plurality of groups may include a respective chroma sample and a respective luma sample. A respective cross-component prediction mode may be determined for each of the multiple groups by comparing the respective chroma samples and the respective luma samples of each of the respective groups to the determined feature values. The current chroma block may be reconstructed based on the determined cross-component prediction modes of the multiple groups.

[0024]

[0024] In one example, the feature value can be determined as one of the average value of reconstructed chroma samples in the vicinity of the current chroma block and the average value of reconstructed luma samples in the vicinity of a luma block at an equivalent position.

[0025]

[0025] In one example, it is possible to determine a feature value as one of an average gradient value of reconstructed chroma samples in the vicinity of a current chroma block and an average gradient value of reconstructed luma samples in the vicinity of a luma block at an equivalent position.

[0026]

[0026] In one example, a feature value can be determined as an average value of reconstructed chroma samples in the vicinity of the current chroma block and reconstructed luma samples in the vicinity of a luma block at the same position.

[0027]

[0027] To group the chroma samples of the current chroma block and the luma samples of the co-located luma block, a characteristic value associated with each of the luma samples of the co-located luma block may be determined. It may be determined whether the characteristic value associated with each of the luma samples of the co-located luma block is greater than a threshold feature value.

[0028] In one example, (i) each luma sample among the luma samples of a co-located luma block and (ii) each chroma sample among the chroma samples of the current chroma block corresponding to each luma sample may be grouped into a first group based on a characteristic value associated with each luma sample being greater than a feature value threshold.

[0029] In one example, (i) each luma sample among the luma samples of the co-located luma block and (ii) each chroma sample among the chroma samples of the current chroma block corresponding to each luma sample may be grouped into a second group based on a characteristic value associated with each luma sample being less than a feature value threshold.

[0030]

[0028] In some embodiments, the characteristic value of each luma sample may include one of (i) the luma sample value of the respective luma sample, and (ii) the average value of reconstructed luma samples in the vicinity of the luma block at the equivalent position.

[0031]

[0029] In some embodiments, each cross-component prediction mode may include one of a cross-component linear model (CCLM), chroma from luma (CfL), a convolutional cross-component model (CCCM), a multiple filter linear model (MFLM), a gradient linear model (GLM), a combination of CCLM, CfL, and CCCM, and an angular intra prediction mode.

[0032]

[0030] In some embodiments, each cross-component prediction mode may be determined based on a corresponding flag included in the coded information.

[0033]

[0031] In an embodiment, whether each of the multiple groups shares a cross-component prediction mode may be determined based on a flag included in the coded information.

[0034] In one example, a cross-component prediction mode for each of the multiple groups can be determined based on a flag indicating that each of the multiple groups shares a cross-component prediction mode.

[0035] In one example, in response to the flag indicating that each of the multiple groups do not share a cross-component prediction mode, a respective cross-component prediction mode for each of the multiple groups can be determined based on a corresponding syntax element.

[0036]

[0032] In an embodiment, a first cross-component prediction mode for a first group of the plurality of groups can be determined based on a first index included in the coded information. A second cross-component prediction mode for a second group of the plurality of groups can be determined based on a second index. The second index can be determined based on a sum of a delta index included in the coded information and the first index.

[0037]

[0033] To reconstruct the current chroma block, in one example, a prediction sample for the chroma samples in each of the multiple groups can be generated based on a respective cross-component prediction mode. A filter can then be applied to the prediction sample for the chroma samples in each of the multiple groups.

[0038]

[0034] In one example, a prediction sample may be generated for the chroma samples in each of the multiple groups based on the respective cross-component prediction modes. A prediction sample for the current chroma block may be determined as a weighted combination of the prediction samples for the chroma samples in each of the multiple groups.

[0039]

[0035] According to another aspect of the present disclosure, an apparatus is provided. The apparatus includes a processing circuit. The processing circuit can be configured to perform any method for video encoding / decoding.

[0040]

[0036] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer for video coding, cause the computer to perform any method for video encoding / decoding. [Brief description of the drawings]

[0041] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Figure 1A] FIG. 1A is a schematic diagram of an example subset of intra-prediction modes. [Figure 1B] FIG. 1B is a diagram of an example intra-prediction direction. [Diagram 2] FIG. 2 is a schematic diagram of a current block and its surrounding spatial merge candidates in one example. [Diagram 3] FIG. 3 is a schematic diagram of a simplified block diagram of a communication system (300) according to an embodiment. [Figure 4]

[0042] FIG. 4 is a schematic diagram of a simplified block diagram of a communication system (400) according to an embodiment. [Diagram 5]

[0043] FIG. 5 is a schematic diagram of a simplified block diagram of a decoder according to an embodiment. [Figure 6]

[0044] FIG. 6 is a schematic diagram of a simplified block diagram of an encoder according to an embodiment. [Figure 7]

[0045] FIG. 7 shows a block diagram of an encoder according to another embodiment. [Figure 8]

[0046] FIG. 8 shows a block diagram of a decoder according to another embodiment. [Figure 9]

[0047] FIG. 9 illustrates example locations of spatial merging candidates according to an embodiment of the present disclosure. [Figure 10]

[0048] FIG. 10 illustrates an exemplary redundancy check of spatial merge candidates. [Figure 11]

[0049] FIG. 11 shows a schematic diagram of motion vector scaling for temporal merge candidates according to an embodiment. [Figure 12]

[0050] FIG. 12 illustrates example candidate positions for temporal merge candidates according to an embodiment. [Figure 13]

[0051] FIG. 13 illustrates an example template and a reference sample of the example template in a reference picture according to an embodiment. [Figure 14]

[0052] FIG. 14 illustrates an exemplary derivation of sub-block based merging candidates according to an embodiment. [Figure 15]

[0053] FIG. 15 shows a schematic diagram of the merge with motion vector difference (MMVD) search process. [Figure 16A]

[0054] Figure 16A shows a schematic diagram of the search points of the MMVD process. [Figure 16B]

[0055] Figure 16B shows a schematic diagram of the search points of the MMVD process. [Figure 17]

[0056] FIG. 17 shows a schematic diagram of additional refinement positions of the MMVD process. [Figure 18]

[0057] FIG. 18 illustrates an exemplary cross-component linear mode (CCLM) prediction mode. [Figure 19]

[0058] FIG. 19 illustrates an exemplary chroma from luma (CfL) process. [Figure 20]

[0059] FIG. 20 illustrates an exemplary CfL prediction process. [Figure 21A]

[0060] FIG. 21A illustrates an exemplary model of a CCLM according to an embodiment. [Figure 21B]

[0061] FIG. 21B illustrates an exemplary adjusted model of CCLM according to an embodiment. [Figure 22]

[0062] FIG. 22 illustrates an exemplary convolution filter according to an embodiment. [Diagram 23]

[0063] FIG. 23 shows a schematic diagram of the calculation of filter coefficients according to an embodiment. [Figure 24]

[0064] Figure 24 shows a schematic diagram of grouping luma and chroma samples for a block. [Diagram 25]

[0065] FIG. 25 shows a flow chart outlining an exemplary decoding process according to some embodiments of the present disclosure. [Figure 26]

[0066] FIG. 26 shows a flowchart outlining an exemplary encoding process according to some embodiments of the present disclosure. [Figure 27]

[0067] FIG. 27 is a schematic diagram of a computer system according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0042]

[0068] Detailed Description of the Embodiments FIG. 3 illustrates an example block diagram of a communication system (300). The communication system (300) includes a plurality of terminal devices capable of communicating with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) perform a unidirectional transmission of data. For example, the terminal device (310) may code video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to the other terminal device (320) via the network (350). The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (320) can receive the coded video data from the network (350), decode the coded video data to reconstruct the video picture, and display the video picture according to the reconstructed video data. The unidirectional data transmission may be common in media serving applications, etc.

[0043]

[0069] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) for bidirectional transmission of coded video data, for example during a video conference. With respect to the bidirectional transmission of data, for example, each of the terminal devices (330) and (340) can code video data (e.g., a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) over the network (350). Each of the terminal devices (330) and (340) can also receive coded video data transmitted by the other of the terminal devices (330) and (340), can decode the coded video data to reconstruct the video pictures, and can display the video pictures on an accessible display device according to the reconstructed video data.

[0044]

[0070] In the example of FIG. 3, terminal devices (310), (320), (330), and (340) are shown as a server, a personal computer, and a smartphone, respectively, although the principles of the present disclosure need not be so limited. Embodiments of the present disclosure may find application with laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network (350) represents any number of networks that carry coded video data between terminal devices (310), (320), (330), and (340), including, for example, wired (wired) and / or wireless communication networks. Communications network (350) may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of the present disclosure, the architecture and topology of network (350) may not be important to the operation of the present disclosure, unless otherwise described below.

[0045]

[0071] 4 illustrates a video encoder and video decoder in a streaming environment as an example application of the disclosed subject matter, which is equally applicable to other video-enabled applications including, for example, video conferencing, digital TV, streaming services, storage of compressed video on digital media (including CDs, DVDs, memory sticks, etc.), etc.

[0046]

[0072] The streaming system may include a video source (401), e.g., a digital camera, and may include a capture subsystem (413) capable of generating a stream of, e.g., uncompressed video pictures (402). In one example, the stream of video pictures (402) includes samples captured by a digital camera. The stream of video pictures (402), depicted as a thick line to emphasize the amount of data when compared to the encoded video data (404) (or coded video bitstream), may be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter as described in more detail below. The encoded video data (404) (or encoded video bitstream), depicted as thin lines to emphasize the smaller amount of data when compared to the stream of video pictures (402), can be stored in the streaming server (405) for future use. One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include a video decoder (410), for example within the electronic device (430). The video decoder (410) decodes the incoming copy of the encoded video data (407) and generates an output stream of video pictures (411) that can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., a video bitstream) may be encoded according to a particular video coding / compression standard.Examples of these standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as Versatile Video Coding (VVC). The disclosed subject matter may be used in the context of VVC.

[0047]

[0073] It should be noted that electronic devices (420) and (430) can include other components (not shown). For example, electronic device (420) can include a video decoder (not shown), and electronic device (430) can include a video encoder (not shown).

[0048]

[0074] 5 shows an example block diagram of a video decoder (510). The video decoder (510) may be included in an electronic device (530). The electronic device (530) may include a receiver (531) (e.g., receiving circuitry). The video decoder (510) may be used in place of the video decoder (410) in the example of FIG. 4.

[0049]

[0075] The receiver (531) can receive one or more coded video sequences to be decoded by the video decoder (510). In an embodiment, it can receive one coded video sequence at a time, where the decoding of each coded video sequence is independent of the decoding of the other coded video sequences. The coded video sequences can be received from a channel (501), which can be a hardware / software link to a storage device that stores the coded video data. The receiver (531) can receive the coded video data together with other data, such as coded audio data and / or auxiliary data streams, which can be transferred using respective entities (not shown). The receiver (531) can separate the coded video sequences from the other data. To address network jitter, a buffer memory (515) can be coupled between the receiver (531) and the entropy decoder / parser (520), hereafter referred to as the "parser (520)". In certain applications, the buffer memory (515) is part of the video decoder (510). In other cases, it may be outside the video decoder (510) (not shown). In yet another example, there may be a buffer memory (not shown) outside the video decoder (510), for example to deal with network jitter, and even another buffer memory (515) inside the video decoder (510), for example to handle playback timing. If the receiver (531) is receiving data from a store-and-forward device with sufficient bandwidth and controllability, or from a synchronous network, the buffer memory (515) may not be needed or may be small.For use in a best effort packet network such as the Internet, a buffer memory (515) may be required, which may be relatively large and may advantageously be adaptively sized and may be implemented at least in part in an operating system or similar element (not shown) outside the video decoder (510).

[0050]

[0076] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the coded video sequence. These symbol categories include information used to manage the operation of the video decoder (510) and potentially information for controlling a rendering device such as a rendering device (512) (e.g., a display screen) that is not an integral part of the electronic device (530) but may be coupled to the electronic device (530) as shown in FIG. 5. The rendering device control information may be in the form of a Supplemental Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles including variable length coding, Huffman coding, arithmetic coding with or without context effects, etc. The parser (520) can extract from the coded video sequence a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on the at least one parameter corresponding to the group. The subgroups can include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser (520) can also extract from the coded video sequence information such as transform coefficients, quantization parameter values, motion vectors, etc.

[0051]

[0077] The parser (520) is capable of performing an entropy decoding / parsing process on the video sequence received from the buffer memory (515) to generate symbols (521).

[0052]

[0078] The reconstruction of the symbols (521) may include different units depending on the type of coded video picture or part thereof (e.g.: inter and intra pictures, inter and intra blocks) and other factors. Which units are included and how can be controlled by subgroup control information parsed by the parser (520) from the coded video sequence. The flow of such subgroup control information between the parser (520) and subsequent units is not depicted for clarity.

[0053]

[0079] Beyond the functional blocks already mentioned, the video decoder (510) may be further conceptually subdivided into a number of functional units as described below. In a practical implementation operating within commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is adequate.

[0054]

[0080] The first unit is a scalar / inverse transform unit (551), which receives quantized transform coefficients as well as control information (including the transform to use, block size, quantization factor, quantization scaling matrix, etc.) from the parser (520) as symbols (521). The scalar / inverse transform unit (551) can output blocks containing sample values ​​that can be input to an aggregator (555).

[0055]

[0081] In some cases, the output samples of the scalar / inverse transform unit (551) may relate to intra-coded blocks. Intra-coded blocks are blocks that do not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed part of the current picture. Such prediction information may be provided by the intra picture prediction unit (552). In some cases, the intra picture prediction unit (552) generates a block of the same size and shape of the block being reconstructed using already reconstructed surrounding information taken from the current picture buffer (558). The current picture buffer (558) may, for example, buffer the partially reconstructed current picture and / or the fully reconstructed current picture. The aggregator (555) adds, possibly on a sample-by-sample basis, the prediction information generated by the intra prediction unit (552) to the output sample information as provided by the scalar / inverse transform unit (551).

[0056]

[0082] In other cases, the output samples of the scalar / inverse transform unit (551) may relate to a block that may be inter-coded and motion compensated. In such a case, the motion compensation prediction unit (553) may access the reference picture memory (557) to retrieve samples used for prediction. After motion compensating the retrieved samples according to the symbols (521) related to the block, these samples are added by the aggregator (555) to the output of the scalar / inverse transform unit (551) (in this case called residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (557) from which the motion compensation prediction unit (553) retrieves the prediction samples may be controlled by the motion vectors available to the motion compensation prediction unit (553), for example in the form of symbols (521), which may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​taken from a reference picture memory (557), motion vector prediction mechanisms, etc., where sub-sample accurate motion vectors are used.

[0057]

[0083] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). Video compression techniques may include in-loop filtering techniques controlled by parameters contained in the coded video sequence (also called coded video bitstream) and made available to the loop filter unit (556) as symbols (521) from the parser (520). Video compression may also depend on meta-information obtained during decoding of previous parts (in decoding order) of the coded picture or coded video sequence, as well as on previously reconstructed loop filtered sample values.

[0058]

[0084] The output of the loop filter unit (556) may be a sample stream that can be output to a rendering device (512) or stored in a reference picture memory (557) for use in future inter-picture prediction.

[0059]

[0085] Once a given coded picture has been fully reconstructed, it can be used as a reference picture for future predictions. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557) and a fresh current picture buffer can be reallocated before starting the reconstruction of a future coded picture.

[0060]

[0086] The video decoder (510) may perform decoding operations according to a standard or a given video compression technique, such as ITU-T Rec. H.265. The coded video sequence may comply with the syntax specified by the video compression technique or standard used in the sense that the coded video sequence conforms to both the syntax of the video compression technique or standard and the profile as documented in the video compression technique or standard. In particular, a profile may select a particular tool from all tools available in the video compression technique or standard as the only tool that may be used under that profile. Also, what is required for compliance is that the complexity of the coded video sequence falls within a range defined by the level of the video compression technique or standard. In some cases, that level limits the maximum picture size, the maximum frame rate, the maximum reconstruction sample rate (e.g., measured in megasamples per second), the maximum reference picture size, etc. The limits set by the levels may in some cases be further constrained by Hypothetical Reference Decoder (HRD) specifications and metadata for HRD buffer management signaled in the coded video sequence.

[0061]

[0087] In an embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0062]

[0088] 6 shows an example block diagram of a video encoder (603). The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used in place of the video encoder (403) in the example of FIG. 4.

[0063]

[0089] The video encoder (603) may receive video samples from a video source (601) (which in the example of FIG. 6 is not part of the electronic device (620)) capable of capturing video images to be coded by the video encoder (603). In another example, the video source (601) is part of the electronic device (620).

[0064]

[0090] The video source (601) may provide a source video sequence to be coded by the video encoder (603) in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 YCrCB, RGB, ...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media serving system, the video source (601) may be a storage device that stores pre-prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a number of individual pictures that convey motion when viewed in sequence. The picture itself may be organized as a spatial array of pixels, each of which may contain one or more samples depending on the sampling structure, color space, etc. in use. Those skilled in the art will readily appreciate the relationship between pixels and samples. The following discussion focuses on examples.

[0065]

[0091] According to an embodiment, the video encoder (603) is capable of coding and compressing pictures of a source video sequence into a coded video sequence (643) in real-time or under any other required time constraint. Enforcing an appropriate coding rate is one function of the controller (650). In some embodiments, the controller (650) controls and is operatively coupled to other functional units as described below, the couplings of which are not depicted for clarity. Parameters set by the controller (650) may include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other appropriate functions associated with the video encoder (603) optimized for a particular system design.

[0066]

[0092] In some embodiments, the video encoder (603) is configured to operate in a coding loop. As a simplified explanation, in one example, the coding loop may include a source coder (630) (e.g., responsible for generating symbols, such as a symbol stream, based on an input picture to be coded and reference pictures) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to generate sample data in a manner similar to that generated by the (remote) decoder. The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since the decoding of the symbol stream produces bit-exact results independent of the location of the decoder (local or remote), the contents in the reference picture memory (634) are also bit-exact between the local and remote encoders. In other words, the predictor of the encoder "sees" exactly the same sample values ​​as the reference picture samples that the decoder would "see" if it were to use prediction during decoding. This basic principle of reference picture synchronization (and the resulting drift if synchronization cannot be maintained, e.g. due to channel errors) is used in several related techniques as well.

[0067]

[0093] The operation of the "local" decoder (633) may be the same as that of a "remote" decoder, such as the video decoder (510) already described in detail above in connection with Figure 5. However, with brief reference to Figure 5 as well, because symbols are available and the encoding / decoding of the symbols into a coded video sequence by the entropy coder (645) and parser (520) may be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515) and parser (520), may not be fully implemented in the local decoder (633).

[0068]

[0094] In an embodiment, decoder techniques other than analysis / entropy decoding present in a decoder are also present in the corresponding encoder, ideally or in substantially the same functional form. Thus, the disclosed subject matter focuses on the operation of the decoder. A description of the encoder techniques can be omitted, since they are the inverse of the decoder techniques described generically. In certain areas, more detailed descriptions are provided below.

[0069]

[0095] During operation, in some examples, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (632) codes differences between pixel blocks of the input picture and pixel blocks of the reference pictures that may be selected as predictive references for the input picture.

[0070]

[0096] The local video decoder (633) can decode the coded video data of the pictures that can be designated as reference pictures based on the symbols generated by the source coder (630). The operation of the coding engine (632) can advantageously be a non-lossless process. If the coded video data can be decoded by a video decoder (not shown in FIG. 6), the reconstructed video sequence can be a replica of the source video sequence, typically with some errors. The local video decoder (633) can repeat the decoding process that can be performed by the video decoder on the reference pictures, causing the reconstructed reference pictures to be stored in the reference picture cache (634). In this way, the video encoder (603) can locally store copies of reconstructed reference pictures that have common content as reconstructed reference pictures obtained by the far-end video decoder (assuming there are no transmission errors).

[0071]

[0097] The predictor (635) may perform a prediction search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or predefined metadata (reference picture motion vectors, block shapes, etc.), which may serve as suitable prediction references for the new picture. The predictor (635) may operate on a sample block-pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).

[0072]

[0098] The controller (650) may manage the coding operations of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0073]

[0099] All outputs of the aforementioned functional units can be subjected to entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by applying a lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0074]

[0100] The transmitter (640) can buffer the coded video sequence as produced by the entropy coder (645) and prepare it for transmission over a communication channel (660), which may be a hardware / software link to a storage device that stores the coded video data. The transmitter (640) can merge the coded video data from the video coder (603) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0075]

[0101] A controller (650) may manage the operation of the video encoder (603). During coding, the controller (650) may assign a particular coded picture type to each of the coded pictures, which may affect the coding technique that may be applied to each picture. For example, pictures may often be assigned as one of the following picture types:

[0102] An Intra picture (I-picture) is one that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow different types of Intra pictures, including, for example, Independent Decoder Refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures and their respective uses and characteristics.

[0076]

[0103] A predictive picture (P-picture) is one that can be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict the sample values ​​of each block.

[0077]

[0104] Bidirectionally predicted pictures (B-pictures) are those that can be coded and decoded using intra- or inter-prediction, which uses at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, multiple predicted pictures can use more than two reference pictures and associated metadata for the reconstruction of a block.

[0078]

[0105] A source picture is usually spatially subdivided into several sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and can be coded block by block. Blocks can be predictively coded with reference to other (already coded) blocks as determined by the coding assignment applied to the respective picture. For example, blocks of I pictures can be non-predictively coded or they can be predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). Pixel blocks of P pictures can be predictively coded with spatial or temporal prediction with reference to one previously coded reference picture. Blocks of B pictures can be predictively coded with spatial or temporal prediction with reference to one or two previously coded reference pictures.

[0079]

[0106] The video encoder (603) may perform coding operations according to a given video coding technique or standard, such as ITU-T Rec. H.266, in which the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancies in the input video sequence. The coded video data may thus conform to a syntax specified by the video coding technique or standard being used.

[0080]

[0107] In an embodiment, the transmitter (640) can transmit additional data along with the encoded video. The source coder (630) can include such data as part of the coded video sequence. The additional data can include temporal / spatial / SNR enhancement layers, other forms of redundant data (redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.).

[0081]

[0108] A video can be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra prediction) exploits spatial correlation in a given picture, while inter-picture prediction exploits correlation (temporal or otherwise) between pictures. In one example, a particular picture under encoding / decoding, called the current picture, is partitioned into blocks. If a block in the current picture is similar to a reference block in a reference picture that was previously coded in the video and is still buffered, the block in the current picture can be coded by a vector called a motion vector. The motion vector points to a reference block in the reference picture and can have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0082]

[0109] In some embodiments, bi-prediction techniques may be used for inter-picture prediction. According to bi-prediction techniques, two reference pictures are used, such as a first reference picture and a second reference picture, both preceding the current picture in decoding order (but potentially past and future, respectively, in display order) in a video. A block in the current picture may be coded with a first motion vector that points to a first reference block in the first reference picture and a second motion vector that points to a second reference block in the second reference picture. A block may be predicted by a combination of the first and second reference blocks.

[0083]

[0110] Furthermore, to improve coding efficiency, it is possible to use merge mode techniques for inter-picture prediction.

[0084]

[0111] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed on a block-by-block basis. For example, according to the HEVC standard, pictures in a sequence of video pictures are partitioned into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. In general, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quad-tree partitioned into one or more coding units (CUs). For example, a 64×64 pixel CTU can be partitioned into one CU-1 of 64×64 pixels, four CU-4 of 32×32 pixels, or sixteen CU-16 of 16×16 pixels. In one example, each CU is analyzed to determine a prediction type of the CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more Prediction Units (PUs) depending on temporal and / or spatial predictability. In general, each PU includes a luma prediction block (PB) and two chroma PBs. In an embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Taking a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values ​​(e.g., luma values) for pixels, such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0085]

[0112] 7 shows an example diagram of a video encoder (703). The video encoder (703) is configured to receive a processed block of sample values ​​(e.g., a predicted block) in a current video picture in a sequence of video pictures and to encode the processed block into a coded picture that is part of a coded video sequence. In one example, the video encoder (703) is used in place of the video encoder (403) of the example of FIG. 4.

[0086]

[0113] In an HEVC example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a predicted block of 8x8 samples. The video encoder (703) determines whether the processing block is best coded using intra mode, inter mode, or bi-predictive mode, e.g., using rate-distortion optimization. If the processing block is to be coded in intra mode, the video encoder (703) may code the processing block into a coded picture using intra prediction techniques; if the processing block is to be coded in inter mode or bi-predictive mode, the video encoder (703) may code the processing block into a coded picture using inter prediction techniques or bi-predictive techniques, respectively. In certain video coding techniques, the merge mode may be an inter picture prediction sub-mode, in which case the motion vector is derived from one or more motion vector predictors without benefit of a coded motion vector component outside the predictor. In certain other video coding techniques, there may be motion vector components applicable to the current block. In one example, the video encoder (703) includes other components, such as a mode decision module (not shown) to determine the mode of the processing blocks.

[0087]

[0114] In the example of FIG. 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a generalized controller (721), and an entropy encoder (725), coupled together as shown in FIG. 7.

[0088]

[0115] The inter encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block with one or more reference blocks in a reference picture (e.g., a block of a previous picture and a block in a subsequent picture), generate inter prediction information (e.g., a description of redundant information due to an inter coding technique, motion vectors, merge mode information), and calculate an inter prediction result (e.g., a prediction block) based on the inter prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture that has been decoded based on the coded video information.

[0089]

[0116] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), optionally compare the block to previously coded blocks in the same picture, generate transformed and quantized coefficients, and optionally also generate intra prediction information (e.g., intra prediction direction information according to one or more intra coding techniques). In one example, the intra encoder (722) also calculates an intra prediction result (e.g., a prediction block) based on the intra prediction information and reference blocks in the same picture.

[0090]

[0117] The generic controller (721) is configured to determine generic control data and control other components of the video encoder (703) based on the generic control data. In one example, the generic controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, if the mode is intra mode, the generic controller (721) controls the switch (726) to select intra mode results for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and include the intra prediction information in the bitstream; and if the mode is inter mode, the generic controller (721) controls the switch (726) to select inter prediction results for use by the residual calculator (723) and controls the entropy encoder (725) to select inter prediction information and include the inter prediction information in the bitstream.

[0091]

[0118] The residual calculator (723) is configured to calculate a difference (residual data) between the received block and a prediction result selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) is configured to operate on the residual data to encode the residual data and generate transform coefficients. In one example, the residual encoder (724) is configured to transform the residual data from a spatial domain to a frequency domain to generate transform coefficients. The transform coefficients are then subjected to a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be used by the intra-encoder (722) and the inter-encoder (730) as appropriate. For example, the inter encoder (730) may generate decoded blocks based on the decoded residual data and the inter prediction information, and the intra encoder (722) may generate decoded blocks based on the decoded residual data and the intra prediction information. The decoded blocks are appropriately processed to generate decoded pictures, which may be buffered in a memory circuit (not shown) and, in some cases, used as reference pictures.

[0092]

[0119] The entropy encoder (725) is configured to format a bitstream to include the encoded blocks. The entropy encoder (725) is configured to include various information in the bitstream according to an appropriate standard, such as the HEVC standard. In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information in the bitstream. It should be noted that when coding blocks in a merged sub-mode of either the inter mode or the bi-prediction mode in accordance with the disclosed subject matter, the residual information is not present.

[0093]

[0120] 8 shows an example diagram of a video decoder (810). The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (810) is used in place of the video decoder (410) in the example of FIG. 4.

[0094]

[0121] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872), coupled together as shown in FIG. 8.

[0095]

[0122] The entropy decoder (871) can be configured to reconstruct from the coded picture certain symbols that represent syntax elements that make up the coded picture. Such symbols can include, for example, the mode in which the block is coded (e.g., intra mode, inter mode, bi-predictive mode, merged submode or another submode in the latter two), prediction information (e.g., intra prediction information or inter prediction information) that can identify certain samples or metadata used for prediction by the intra decoder (872) or the inter decoder (880), respectively. The symbols can also include residual information, for example, in the form of quantized transform coefficients. In one example, if the prediction mode is an inter or bi-predictive mode, the inter prediction information is provided to the inter decoder (880); if the prediction type is an intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information can be inverse quantized and provided to the residual decoder (873).

[0096]

[0123] The inter decoder (880) is configured to receive the inter prediction information and to generate inter prediction results based on the inter prediction information.

[0097]

[0124] The intra decoder (872) is configured to receive intra prediction information and to generate a prediction result based on the intra prediction information.

[0098]

[0125] The residual decoder (873) is configured to perform inverse quantization to extract unquantized transform coefficients, and to process the unquantized transform coefficients to transform the residual information from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (including a quantization parameter (QP)), which may be provided by the entropy decoder (871) (this may only be a small amount of control information, so a data path is not depicted).

[0099]

[0126] The reconstruction module (874) is configured to combine, in the spatial domain, the residual information as output by the residual decoder (873) and the prediction result (possibly output by an inter- or intra-prediction module) to form a reconstructed block, which is part of a reconstructed picture, which may be part of a reconstructed video. It should be noted that other suitable processes, such as a deblocking process, may be performed to improve visual quality.

[0100]

[0127] The video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technology. In an embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0101]

[0128] This disclosure includes embodiments related to mixed model design for cross-component prediction modes.

[0102]

[0129] ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) published the H.265 / HEVC (High Efficiency Video Coding) standard in 2013 (version 1), 2014 (version 2), 2015 (version 3), and 2016 (version 4). In 2015, the two standardization organizations jointly formed the Joint Video Exploration Team (JVET) to explore the possibility of developing the next video coding standard beyond HEVC. In October 2017, the two standardization organizations issued a Joint Call for Proposals on Video Compression with Capability beyond HEVC (CfP). By February 15, 2018, 22 CfP responses for standard dynamic range (SDR), 12 CfP responses for high dynamic range (HDR), and 12 CfP responses for 360 video category were submitted. In April 2018, all accepted CfP responses were evaluated at the 122 MPEG / 10th JVET meeting. As a result of this meeting, JVET formally started the standardization process for next-generation video coding beyond HEVC, the new standard was named VVC (Versatile Video Coding), and JVET was renamed JVET (Joint Video Experts Team). In 2020, ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC 29 / WG 11) published the VVC video coding standard (version 1).

[0103]

[0130] In inter prediction, for an inter predicted coding unit (CU), motion parameters are required for the coding features of VVC, for example, to be used for inter predicted sample generation. The motion parameters may include motion vectors, reference picture indices, reference picture list usage indices, and / or additional information. The motion parameters may be signaled in an explicit or implicit manner. If the CU is coded in skip mode, it may be associated with one PU, and significant residual coefficients, coded motion vector delta, and / or reference picture index may not be required. If a CU is coded in merge mode, the motion parameters of the CU can be obtained from neighboring CU(s), which may include spatial and temporal candidates, and schedule (or additional candidates) as introduced in VVC. The merge mode may be applied not only for skip mode, but also for any inter-predicted CUs. An alternative to the merge mode is explicit transmission of motion parameters, in which case motion vectors, corresponding reference picture indexes for each reference picture list, reference picture list usage flags, and / or other necessary information can be explicitly signaled per CU.

[0104]

[0131] In VVC, the VVC Test model (VTM) reference software includes a number of new refinement inter-prediction coding tools, including: (1) Enhanced merge prediction, (2) merge motion vector difference (MMVD) mode; (3) adaptive motion vector prediction (AMVP) mode with symmetric MVD signaling; (4) Affine motion compensation prediction, (5) Subblock-based temporal motion vector prediction (SbTMVP), (6) adaptive motion vector resolution (AMVR); (7) Motion field storage (1 / 16 luma sample MV storage, and 8x8 motion field compression), (8) Bi-prediction with CU-level weights (BCW), (9) Bi-directional optical flow (BDOF), PROF), (10) Decoder side motion vector refinement (DMVR); (11) Combined inter and intra prediction (CIIP), (12) Geometric partitioning mode (GPM) may include one or more of the following:

[0105]

[0132] As in VTM4, a merge candidate list can be constructed by including five types of candidates. The merge candidate list can be constructed in the following order: (1) Spatial MVP (motion vector predictor(s),MVP) from spatially neighboring CUs, (2) Temporal MVP from collocated CU(s), (3) History-based MVP(s) from FIFO tables (HMVP), (4) Pairwise average MVP, and (5) zero MV(s).

[0106]

[0133] The size of the merge candidate list can be signaled in the slice header. The maximum allowed size of the merge candidate list can be six, as in VTM4. For each CU coded in merge mode, the index of the best merge candidate can be coded, for example, using truncated unary binarization (TU). The first bin of the merge index can be coded with the context and bypass coding can be used for the other bins.

[0107]

[0134] In spatial candidate derivation, e.g., in VVC, the derivation of spatial merge candidates may be the same or similar to the derivation of spatial merge candidates in HEVC. For example, the maximum number of merge candidates (e.g., 4 merge candidates) may be selected from the candidates at the positions shown in FIG. 9. As shown in FIG. 9, a current block (901) may include neighboring blocks (902)-(906) at positions A0, A1, B0, B1, and B2, respectively. The derivation order of spatial merge candidates may be B1, A1, B0, A0, B2. Position B2 may be considered only if any CU (or block) at positions A0, B0, B1, and A1 is unavailable (e.g., due to the CU belonging to another slice or tile) or is intra-coded. After the candidate (or block) at position A1 is added, the addition of the remaining candidates (or blocks) may be subject to a redundancy check.

[0108] The redundancy check can ensure that candidates with the same motion information are removed from the merge list, which improves coding efficiency. To reduce computational complexity, the redundancy check may not consider all possible candidate pairs. Instead, only the candidate pairs connected by the arrows in FIG. 10 may be considered. For example, the redundancy check can be applied to five candidate pairs, such as candidate pair A1 and B1, candidate pair A1 and A0, and so on. A candidate may be added to the merge list only if the corresponding candidates used for the redundancy check do not contain the same motion information. For example, candidate B0 may be added to the merge list only if the corresponding candidate B1 does not contain the same motion information.

[0109]

[0135] In temporal candidate derivation, only one candidate may be added to the merge list. For example, as shown in FIG. 11, in derivation of a temporal merge candidate for a current CU (1114), a scaled motion vector may be derived based on the co-located CU (1104) belonging to the co-located reference picture (1112). The reference picture list used to derive the co-located CU (1104) may be explicitly signaled in the slice header. The scaled motion vector for the temporal merge candidate may be obtained as shown by the dotted line in FIG. 11, which is scaled from the motion vector of the co-located CU (1104) using picture order count (POC) distances tb and td. tb may be defined as the POC difference between the reference picture (eg, Curr_ref) (1106) of the current picture and the current picture (eg, Curr_pic) (1108). td may be defined as the POC difference between the co-located picture's reference picture (eg, Col_ref) (1110) and the co-located picture (eg, Col_pic) (1112). The reference picture index of the temporal merge candidate may be set to zero.

[0110]

[0136] The location of the temporal candidate can be selected between candidates C0 and C1, as shown in Figure 12. For example, location C1 can be used if the CU in location C0 is not available, is intra-coded, or is outside the current row of the CTU. Alternatively, location C0 can be used in the derivation of the temporal merge candidate.

[0111]

[0137] Adaptive reordering of merge candidates with template matching,ARMC-TMMerge candidates can be adaptively reordered using template matching (TM). ARMC-TM can be applied to various prediction modes, such as regular merge mode, TM merge mode, and affine merge mode (except for SbTMVP candidates). For TM merge mode, merge candidates can be reordered before the refinement process.

[0112]

[0138] After the merge candidate list is constructed, the merge candidates may be divided into subgroups. The subgroup size may be set to 5 for regular merge mode and TM merge mode. For affine merge mode, the subgroup size may be set to 3. The merge candidates in each subgroup may be sorted in ascending order according to their cost values ​​based on template matching. For simplicity, the merge candidates in the last subgroup but not the first subgroup may not be sorted.

[0113]

[0139] The template matching cost of a merge candidate may be measured by the sum of absolute differences (SAD) between the template's samples of the current block and the corresponding reference samples of the current block. The template may include a set of reconstructed samples adjacent to or near the current block. The template's reference samples may be located by the motion information of the merge candidate.

[0114]

[0140] If the merge candidate uses bi-directional prediction, the reference samples of the template of the merge candidate may also be generated by bi-directional prediction. In the example shown in FIG. 13, a current block (1302) is included in a current picture (1304). The current block (1302) may have a template (1306) located adjacent to the top side and left side of the current block (1302). The template (1306) may include reconstructed nearby samples of the current block (1302).

[0115] Based on the template matching, a first reference block 1308 in a first reference picture 1310 of a first reference list 0 may be determined within the search range, in which case a minimum SAD may be determined between the template 1312 of the first reference block 1308 and the template 1306. The first reference block (1308) may be specified by a MV (1314).

[0116] Also, based on template matching, a second reference block 1316 in a second reference picture 1318 of a second reference list 1 may be determined within the search range, in which case a minimum SAD may be determined between a template 1320 of the second reference block 1316 and the template 1306. The second reference block (1316) may be specified by a MV (1322).

[0117] A predicted sample of the current block (1302) may further be determined based on the first reference block (1308) and the second reference block (1316).

[0118]

[0141] For a subblock-based merging candidate with subblock size equal to W×H, the top template of the current block may include some sub-templates with size of W×1, and the left template of the current block may include some sub-templates with size of 1×H. As shown in Fig. 14, the motion information of the subblocks in the first row and first column of the current block can be used to derive the reference samples of each sub-template.

[0119]

[0142] FIG. 14 illustrates an example derivation of sub-block-based merging candidates. As shown in FIG. 14, a current block (1402) may be included in a current picture (1404). The current block (1402) may include a sub-block AG in the first row and the first column. The current block (1402) may include a template (1406) adjacent to the top and left of the current block (1402). The current block (1402) may include a block (1408) that is co-located in a reference picture (1410). The co-located block (1408) may include a sub-block AG in the first row and the first column that corresponds to the sub-block AG in the current block (1402). The motion information (e.g., affine motion vector) of the subblock A G in the first row and first column of the current block (1402) may be used to derive a reference sample of a sub-template (or sub-reference template) of the equivalently positioned block (1408). For example, the motion information of the subblocks A, E, F, and G of the current block (1402) may be applied to derive a reference sample of a sub-template adjacent to the left side of the subblocks A, E, F, and G of the equivalently positioned block (1408). The sub-template adjacent to the left side of the subblocks A, E, F, and G of the equivalently positioned block (1408) may form a left reference template of the equivalently positioned block (1408). The motion information of the subblocks A, B, C, and D of the current block (1402) may be applied to derive a reference sample of a sub-template adjacent to the top side of the subblocks A, B, C, and D of the equivalently positioned block (1408). The sub-templates adjacent to the top side of the sub-blocks A, B, C, and D of the equivalent position block (1408) may further form an upper reference template of the equivalent position block (1408).

[0120]

[0143] Merge with Motion Vector Difference (MMVD) can be used for certain prediction modes, such as skip mode and merge mode with motion vector representation. MMVD can reuse merge candidates like VVC. It can select merge candidates among them and further extend (or refine) them with motion vector representation. MMVD can provide new motion vector representation with simplified signaling. The motion vector representation can include the starting point, the motion magnitude, and the motion direction.

[0121]

[0144] MMVD can use a merge candidate list as in VVC. Candidates with default merge type (e.g., MRG_TYPE_DEFAULT_N) can be considered for the extension of MMVD. In MMVD, a base candidate index may define a starting point. The base candidate index (IDX) can specify the best candidate among the candidates in a list such as Table 1 below.

[0122] Table 1. Base candidate IDX

[0123] [Table 1]

[0145] If the number of base candidates is equal to 1, the base candidate IDX may not be signaled. The distance index may provide motion magnitude information. The distance index may indicate a predefined distance from the starting point. The predefined distance based on the distance index may be provided as in Table 2 below: Table 2. Distance IDX

[0124] [Table 2]

[0146] The direction index can represent the direction of the MVD relative to the start point. The direction index can represent four directions, as shown in Table 3. The MMVD flag can be signaled immediately if the skip and merge flags are sent. If the skip and merge flags are true, the MMVD flag can be parsed. If the MMVD flag is equal to 1, the MMVD syntax can be parsed. However, if the MMVD flag is not 1, the AFFINE flag can be parsed. If the AFFINE flag is equal to 1, the AFFINE mode can be applied. However, if the AFFINE flag is not 1, the skip / merge index can be parsed for the skip / merge mode.

[0125] Table 3. Direction IDX

[0126] [Table 3]

[0147] FIG. 15 illustrates an exemplary MMVD search process. As shown in FIG. 15, the starting MV can be indicated by (1511) (e.g., by direction IDX and base candidate IDX), the offset can be indicated by (1512) (e.g., by distance IDX and direction IDX), and the final MV predictor can be indicated by (1513). In another example, the starting MV can be indicated by (1521) (e.g., by direction IDX and base candidate IDX), the offset can be indicated by (1522) (e.g., by distance IDX and direction IDX), and the final MV predictor can be indicated in FIG. 15 by (1523).

[0127]

[0148] 16A and 16B show exemplary search points of MMVD. As shown in FIG. 16A, the starting point MV can be indicated in the first reference list L0 as (1611) (e.g., by the direction IDX and the base candidate IDX). In the example of FIG. 16A, four search directions such as +Y, -Y, +X, -X are used, and the four search directions can be indexed by 0, 1, 2, 3. The distance can be indexed by 0 (distance to the starting point MV is 0), 1 (1s (or 1 sample) to the starting point MV), 2 (2s to the starting point MV), 3 (3s to the starting point), etc. Thus, if the direction IDX is 3 and the distance IDX is 2, the final MV predictor is indicated as (1615).

[0128]

[0149] In another example, it is possible to index the search direction and distance in combination. For example, in the second reference list L1, the starting point MV is indicated by (1621) (e.g., by the direction IDX and the base candidate IDX). The search direction and distance are combined to be indexed by 0-12, as shown in FIG. 16B.

[0129]

[0150] In template matching based candidate reordering for MMVD and affine MMVD, the MMVD offset can be expanded for MMVD and affine MMVD modes. In one example, additional refinement positions along the k×π / 8 diagonal can be added first. Exemplary additional refinement positions can be shown in FIG. 17, where the number of directions can be increased from 4 to 16. Secondly, all possible MMVD refinement positions (e.g., 16×6) for each base candidate can be reordered based on the SAD cost between the template (e.g., one row up and one column left of the current block) and the template reference for each refinement position. Finally, the top 1 / 8 refinement positions that result in the smallest template SAD cost can be retained as available positions and, therefore, can be retained for MMVD index coding. The MMVD index can be binarized by a code such as the Rice code with a parameter equal to 2.

[0130]

[0151] In another example, on top of the MMVD extension as described above, affine MMVD reordering or reordering can also be extended, where additional refinement positions along the k×π / 4 diagonal can be added. After reordering, the top ½ refinement positions with the smallest template SAD cost can be kept.

[0131]

[0152] To reduce the cross-component redundancy, it is possible to use the Cross-Component Linear Model (CCLM) prediction mode, as in VVC. In the CCLM prediction mode, the chroma samples of the current CU can be predicted based on the reconstructed luma samples of the current CU by using a linear model as shown in the following equation (1):

[0132]

number

[0133]

[0153] The CCLM parameters (e.g., α and β) can be derived based on at most four nearby chroma samples and the corresponding down-sampled luma samples of the nearby chroma samples. If the current chroma block has dimensions W×H (where W is the width of the current chroma block and H is the height of the current chroma block), the width W' of the first reference region and the height H' of the second reference region can be defined as follows: When LM mode is applied, W'=W,H'=H; When LM-A mode is applied, W'=W+H; When LM-L mode is applied, H'=H+W. The first reference area can be adjacent to the top of the current chroma block. The second reference area can be adjacent to the left of the current chroma block.

[0134] In LM mode, neighboring chroma samples can be located in both the first and second reference regions.

[0135] In LM-A mode, nearby chroma samples can be located within the first reference area.

[0136] In LM-L mode, nearby chroma samples can be located within a second reference region.

[0137]

[0154] Therefore, the position of the top neighborhood (or the position of the first reference area) is S[0,-1]...S[W'-1,-1] and the left near position (or the position of the second reference region) can be expressed as: S[-1,0]...S[-1,H'-1] It can be expressed as: Thus, the positions of the four nearby chroma samples can be chosen as follows: When LM mode is applied and both top and left near samples are available S[W' / 4,-1],S[3*W' / 4,-1],S[-1,H' / 4],S[-1,3*H' / 4]; When LM-A mode is applied or only upper samples are available S[W' / 8,-1],S[3*W' / 8,-1],S[5*W' / 8,-1],S[7*W' / 8,-1]; When LM-L mode is applied or only left near samples are available S[-1,H' / 8],S[-1,3*H' / 8],S[-1,5*H' / 8],S[-1,7*H' / 8]

[0155] The four luma samples corresponding to the four nearby chroma samples at the selected position are downsampled and compared four times to find the larger of the two values ​​x 0 A and x 1 A , as well as the two smaller values ​​of x 0 B and x 1 B The chroma sample values ​​of the four nearby chroma samples corresponding to the four luma samples are y 0 A ,y 1 A ,y 0B , and y 1 B It can be expressed as follows: Parameter X a ,X b ,Y a ,Y b can be derived from equations (2)-(5) as follows:

[0138]

number

[0156] Finally, the linear model parameters α and β can be calculated using equations (6) and (7), respectively.

[0139]

number

[0157] FIG. 18 shows example locations of samples near the left and top of a current block, and samples of the current block for CCLM prediction mode. As shown in FIG. 18, a current chroma CU (1802) may have a size of NxN (e.g., 8x8). A corresponding luma CU (1804) of the current chroma CU (1802) may have a size of 2Nx2N (e.g., 16x16). The chroma samples α and β near the current chroma CU (1802) used to derive the linear model parameters may be located in a first reference region (or near top position) (1806) and / or a second reference region (or near left position) (1808). The luma samples corresponding to the nearby chroma samples (1806) and (1808) can be located in a first reference region (1810) and a second reference region (1812), respectively. As shown in FIG. 18, the luma samples (1810) and (1812) can be downsampled to match the nearby chroma samples (1806) and (1808) one-to-one. The reconstructed values ​​Rec' of the luma samples (1810) and (1812) can be L and the reconstructed values ​​of the nearby chroma samples (1806) and (1808) cand apply it to derive the linear model parameters α and β based on equations (2)-(7). Once the linear model parameters α and β are obtained, the chroma samples in the current chroma CU (1802) can be predicted based on the down-sampled reconstructed luma samples in the corresponding luma CU (1804) using the CCLM prediction mode.

[0140]

[0158] The division operation to calculate the parameter α can be implemented using a look-up table. To reduce the memory required to store the table, the diff value (e.g., the difference between the maximum and minimum values) and the parameter α can be expressed in exponential notation. For example, the diff can be approximated with a 4-bit significant part and exponent. Thus, the table for 1 / diff can be reduced to 16 elements for 16 significant values, as shown in Equation (8) below.

[0141]

number

[0159] Thus, the advantage is obtained of reducing the computational complexity as well as reducing the memory size required to store the required tables.

[0142]

[0160] As shown in Figure 18, the top template (e.g., (1806)) and the left template (e.g., (1808)) can be used together to calculate the linear model coefficients (e.g., the linear model parameters α and β). The top and left templates can also be used alternatively in two other LM modes, called the LM_A (or LM_T) mode and the LM_L mode.

[0143]

[0161] In LM_A mode, the linear model coefficients can be calculated using only the top template. To obtain more samples, the top template can be extended to (W+H) samples.

[0144] In LM_L mode, the linear model coefficients can be calculated using only the left template. To obtain more samples, the left template can be extended to (H+W) samples.

[0145]

[0162] In the LM_LT mode, it is possible to use both the left and top templates to calculate the linear model coefficients.

[0146]

[0163] To match the chroma sample positions for 4:2:0 video sequences, two types of down-sampling filters can be applied to the luma samples to achieve a 2:1 down-sampling ratio in both horizontal and vertical directions. The choice of down-sampling filter can be specified by the SPS level flag. The two down-sampling filters can be provided as shown in the following equations (9) and (10), which correspond to the “type-0” content and the “type-2” content, respectively.

[0147]

number

[0164] Note that if the upper reference line is on a CTU boundary, only one luma line is available to create the downsampled luma samples (e.g., a common line buffer in intra prediction).

[0148]

[0165] The computation of the parameters (e.g., α and β) can be performed as part of the decoding process and not necessarily as part of the encoder search operation, so that no syntax can be used to communicate the α and β values ​​to the decoder.

[0149]

[0166] For chroma intra mode coding, a total of eight intra modes may be allowed for chroma intra mode coding. The nine intra modes may include five conventional intra modes and three cross-component linear model modes (e.g., CCLM, LM_A, LM_L). The chroma mode signaling and derivation process may be as shown in Table 4. The chroma mode coding of the current chroma block may directly depend on the intra prediction mode of the corresponding luma block. Since separate block partitioning structures for luma and chroma components may be enabled in an I slice, one chroma block may correspond to multiple luma blocks. Therefore, for the chroma DM mode, the intra prediction mode of the corresponding luma block covering the center position of the current chroma block may be directly inherited.

[0150] Table 4. Derivation of Chroma Prediction Mode from Luma Mode when CCLM is enabled

[0151] [Table 4]

[0167] In Table 5, a single binarization table can be used regardless of the value of the flag (eg, sps_cclm_enabled_flag) indicating whether CCLM is enabled or not.

[0152] Table 5. Unified binarization for chroma prediction modes

[0153] [Table 5]

[0168] As shown in Table 5, the first bin may indicate whether the regular (0) mode or the LM mode (1) is applied. If the first bin indicates the LM mode, the next bin (or the second bin) may indicate whether the LM_CHROMA (0) is applied. If the second bin indicates the LM_CHROMA, the next one bin (or the third bin) may indicate whether the LM_L (0) or the LM_A (1) is applied. If the CCLM flag (e.g., sps_cclm_enabled_flag) is 0, the first bin of the binarization table of the corresponding chroma intra mode (e.g., intra_chroma_pred_mode) may be discarded before entropy coding. Or, in other words, the first bin may be inferred as 0 and thus not coded. The binarization table may be used if the CCLM flag (i.e., sps_cclm_enabled_flag) is 0 or 1. The first two bins of Table 5 can be table context coded using the corresponding context model, and the remaining bins can be bypass coded.

[0154]

[0169] Additionally, to reduce luma-chroma latency in dual trees, if a 64x64 luma coding tree node is partitioned using “Not Split” (and ISP is not used for 64x64 CUs) or QT, the chroma CUs in a 32x32 / 32x16 chroma coding tree node can be licensed for CCLM use under the following conditions: (1) If a 32x32 chroma node is not split or partitioned by QT splitting, all chroma CUs within the 32x32 node can use CCLM.

[0155] (2) If a 32x32 chroma node is partitioned with horizontal BT and a 32x16 child node is not partitioned or uses vertical BT partitioning, all chroma CUs within the 32x16 chroma node can use CCLM. In other luma and chroma coding tree splitting conditions, CCLM may not be permissible for a chroma CU.

[0156]

[0170] In some embodiments, in addition to the 56 directional and 5 non-directional modes for the chroma components, Chroma from Luma (CfL) may be a chroma-only intra prediction mode, as in AV1. CfL can model chroma pixels as a linear function of coincident reconstructed luma pixels. CfL prediction can be expressed as the following equation (11):

[0157]

number

[0158]

[0171] In CfL mode, if some samples in a co-located luma block are outside the picture boundary, they can be padded and used to calculate the average of the luma samples. As shown in Figure 20, samples (2002) in the gray area of ​​the current block are outside the picture, and these samples (2002) can be padded by copying the value of the nearest available sample in the current block. For example, sample (2002) can be padded by copying the value of sample (2004), which is the nearest available sample in the current block.

[0159]

[0172] In the CfL mode, the luma subsampling step can be combined with the average subtraction step, which can be shown in FIG. 19. Thus, the formulas of the subsampling step and the average subtraction step can be simplified, and the subsampling division and the corresponding rounding error can be eliminated. The formula corresponding to the combination of both steps can be given in equation (12), which can be further simplified to equation (13). Note that both equations (12) and (13) can use integer division. Let M×N be a matrix of pixels in the luma plane.

[0160]

number

[0173] Based on the supported chroma subsampling, x ×S y ∈{1,2,4} and M×N can also be said to be powers of two, since both M and N are powers of two.

[0161]

[0174] For example, in a 4:2:0 chroma subsampling situation, instead of applying a box filter, the CfL mode simply requires the sum of the four reconstructed luma pixels that match the chroma pixels. 4-tap {1 / 4,1 / 4,1 / 4,1 / 4} filter It is possible to downsample the comparably positioned luma samples to match the chroma resolution using . Then, when a CfL mode scales the luma pixels to improve the accuracy of the prediction, only a scaling by 2 is required.

[0162]

[0175] CCLM may be extended by adding three Multi-model LM (MMLM) modes, as in JVET-D0110. In each MMLM mode, the reconstructed neighboring samples can be classified into two classes using a threshold that can be taken as the average of the luma reconstructed neighboring samples. A linear model for each class can be derived using the Least-Mean-Square (LMS) method. For CCLM modes, it is also possible to derive the linear model using the LMS method. A slope adjustment can be applied to the CCLM and Multi-model LM predictions. This adjustment can tilt the linear function that maps luma values ​​to chroma values ​​with respect to a center point determined by the average luma value of the reference samples.

[0163]

[0176] CCLM is able to map luma values ​​to chroma values ​​using a model with two parameters. The tilt parameter “a” and the bias parameter “b” define the mapping as shown in the following equation (14): chromaVal=a*lumaVal+b Eq.(14) The adjustment "u" to the gradient parameter "a" can be signaled to form an updated model as in equation (15) below: chromaVal=a'*lumaVal+b' Eq.(15) Where: a'=a+u Eq.(16) b'=bu*y r Eq.(17) Based on the adjustment “u”, the mapping function (e.g., Eq. (15)) maps the luminance value y r The luminance value y can be tilted or rotated around a point having ris determined based on the average of the reference luma samples used in the model and can bring about meaningful modifications to the model. Figure 21A shows a CCLM model with two parameters “a” and “b”. Figure 21B shows an updated CCLM model with two parameters a’ and b’, where a’=a+u and b’=bu*y r It is.

[0164]

[0177] The tilt adjustment parameter u may be provided as an integer between (-4) and (4), inclusive, and signaled in the bitstream. The units of the tilt adjustment parameter u may be 1 / 8 of a chroma sample value per (or relative to) luma sample value (e.g., for 10-bit content).

[0165]

[0178] Tilt adjustment can be implemented for CCLM models (e.g., “LM_CHROMA_IDX” and “MMLM_CHROMA_IDX”) that use the top and left neighboring reference samples of the block, but not for the “single side” modes that use only the neighboring reference samples on one side of the block. The decision to use tilt adjustment can be made based on a trade-off consideration between coding efficiency and coding complexity.

[0166]

[0179] When gradient adjustment is applied to a multi-mode CCLM model, it is possible to adjust both models and thus signal up to two gradient updates for a single chroma block.

[0167]

[0180] The encoder may perform a SATD-based search for the best value of the slope update for the Cr of the TU, and a similar SATD-based search for the Cb of the TU. If either of the two searches results in a non-zero slope adjustment parameter, the combined slope adjustment pair (e.g., a SATD-based update for Cr and a SATD-based update for Cb) may be included in the list of rate distortion (RD) checks for the TU.

[0168]

[0181] A Convolutional Cross-Component Model (CCCM) can be applied to predict the chroma samples from the reconstructed luma samples in a similar manner to the CCLM mode. As with CCLM, the reconstructed luma samples can be downsampled to match a lower resolution chroma grid when chroma subsampling is used.

[0169]

[0182] Also similar to CCLM, single-model or multi-model variants of CCCM can be applied. The multi-model variant can use two models: one model can be derived for samples above the mean luma reference value and another model can be derived for the remaining samples. The multi-model CCCM mode can be selected for PUs with at least 128 available reference samples.

[0170]

[0183] In CCCM, a convolutional 7-tap filter may be applied. The 7-tap filter may include a spatial 5-tap component with a plus sign shape, a nonlinear term, and a bias term. The input to the spatial 5-tap component of the 7-tap filter may include a center (C) luma sample (which may be co-located with the chroma sample of the prediction) that may be co-located with the chroma sample for the prediction, and / or a top / north (N), bottom / south (S), left / west (W), and right / east (E) neighbor sample. An exemplary spatial portion (e.g., spatial 5-tap component) (2200) of a conventional 7-tap filter may be as shown in FIG. 22.

[0171]

[0184] As shown in equation (18), the nonlinear term P can be expressed as a power of two of the central luma sample C and scaled to the content sample value range (e.g., 10 bits): P=(C*C+midVal) >> bitDepth Eq.(18) That is, for 10-bit content, the nonlinear term P can be calculated as follows: P=(C*C+512) >> 10 Eq.(19) The bias term B can represent a scalar offset between the input and output (similar to the offset term in CCLM) and can be set as an intermediate chroma value (e.g., 512 for 10-bit content). The output of the filter can be shown in Equation (20). The output of the filter is expressed as the filter coefficient c i The chroma sample value is calculated based on a convolution between the input and the chroma sample values, and can be clipped to the range of valid chroma samples.

[0172] predChromaVal = c0C+c1N+c2S+c3E+c4W+c5P+c6B Eq.(20)

[0185] Filter coefficient c i can be calculated by minimizing the MSE between the predicted and reconstructed chroma samples in the reference area. FIG. 23 shows a reference area (2302) that can include 6 lines of chroma samples from the top and left neighbors of a PU (2304). The reference area (2302) can be extended to the right by 1-PU width (2308) and downward by 1-PU height (2310) beyond the boundary of the PU (2304). The reference area can be adjusted to include only available samples. Extensions (2306) to the reference area (2302) can be included to support "side samples" of a plus-shaped spatial filter (e.g., the spatial 5-tap component (2200)), padding out the unavailable areas.

[0173]

[0186] The MSE minimization can be performed by computing the autocorrelation matrix of the luma input and the cross-correlation vector between the luma input and the chroma output. The autocorrelation matrix can be LDL decomposed and the final filter coefficients can be calculated using back-substitution. The MSE minimization can follow the calculation of the ALF filter coefficients in the ECM. However, instead of Cholesky decomposition, LDL decomposition may be chosen to avoid using square root operations. Furthermore, only integer arithmetic may be used in the MSE minimization.

[0174]

[0187] In bitstream signaling, the use of CCCM mode can be signaled with a CABAC coded PU level flag. A new CABAC context can be included to support the PU level flag. CCCM can be considered as a sub-mode of CCLM, i.e., the CCCM flag can be signaled only if the intra prediction mode is LM_CHROMA_IDX (to enable single mode CCCM) or MMLM_CHROMA_IDX (to enable multi-model CCCM).

[0175]

[0188] During encoder operation, the encoder can implement two new RD checks in the chroma prediction mode loop: one to check for single-model CCCM mode and one to check for multi-model CCCM mode.

[0176]

[0189] When a cross-component prediction mode using multiple models is applied, all models may need to follow the same prediction mode (or cross-component prediction mode). For example, in the MMLM mode, two models are applied, and both models need to apply the CCLM mode. Another example is in the multi-model CCCM method, both models need to apply the CCCM mode. However, it is possible to design different cross-component prediction modes under different image model assumptions, and the constraint of the same prediction mode for multiple models may limit the coding performance.

[0177]

[0190] In the present disclosure, a feature may be predefined from reconstructed samples in a neighborhood of the current block. One of a plurality of cross-component prediction modes may be applied based on a comparison of one or more samples of the current block, and in some examples each sample, to the value of the feature. In one example, the samples of the current block may be grouped into a plurality of classes (or groups) based on the feature values. For each of the plurality of groups, a respective cross-component prediction mode may be applied.

[0178]

[0191] In an embodiment, the feature may be derived from a predefined function using reconstructed samples (e.g., luma samples) in the vicinity of a luma block that is positioned at a position equivalent to the current chroma block, and / or reconstructed samples (e.g., chroma samples) in the vicinity of the current chroma block.

[0179]

[0192] In one example, the feature can be an average value of reconstructed samples, such as nearby reconstructed samples, which may include nearby reconstructed samples (e.g., luma samples) of a luma block at a comparable position of the current chroma block, or an average value of nearby reconstructed samples (e.g., chroma samples) of the current chroma block.

[0180]

[0193] In one example, the feature may be an average gradient value of nearby reconstructed samples (e.g., nearby reconstructed luma samples) of a luma block at an equivalent position of the current chroma block, or an average gradient value of nearby reconstructed samples (e.g., nearby reconstructed chroma samples) of the current chroma block.

[0181]

[0194] In one example, the feature can be the average value of a pair of a nearby chroma sample of the current chroma block and a nearby luma sample at the same position of the luma block at the same position as the current chroma block. Thus, the feature can be equal to the average value of the nearby chroma sample and the nearby luma sample at the same position.

[0182]

[0195] In an embodiment, the cross-component prediction modes may include, but are not limited to, one or more of CCLM, CfL, CCCM, multiple filter linear model (MFLM), GLM, and a combination of CCLM / CfL / CCCM and smooth and angular intra prediction mode.

[0183]

[0196] In an embodiment, one or more thresholds of feature values ​​may be defined, such as a threshold T. For each sample (e.g., chroma sample) of the current chroma block, a comparably positioned luma sample of the respective chroma sample in the comparably positioned luma block of the current chroma block may be fetched. The feature value of each comparably positioned luma sample in the comparably positioned luma block, i.e., F, may be compared to the threshold T.

[0184] If F is less than (or equal to) T, the chroma samples and the co-located luma samples of the chroma samples may be grouped into a set (or group), namely G0.

[0185] If F is greater than (or equal to) T, the chroma samples and the co-located luma samples of the chroma samples may be grouped into another set, namely G1.

[0186]

[0197] In one example, the characteristic value of each equally-located luma sample may be the luma sample value of each equally-located luma sample.

[0187] In one example, the feature value of each co-located luma sample may be the average value of reconstructed luma samples in the vicinity of the co-located luma block.

[0188] In one example, the feature value of each co-located luma sample may be the average gradient value of reconstructed luma samples in the vicinity of the co-located luma block.

[0189] In one example, the feature value of each equally-located luma sample may be the average value of nearby reconstructed chroma samples of the current chroma block and nearby reconstructed luma samples of the equally-located luma block.

[0190]

[0198] For samples in G0 (e.g., chroma samples and co-located luma samples), one of multiple cross-component prediction modes can be selected to generate predicted samples. For samples in G1, one of multiple cross-component prediction modes can be selected to generate predicted samples.

[0191]

[0199] An example of grouping chroma samples and co-located luma samples may be shown in FIG. 24. As shown in FIG. 24, the vertical axis “Cb / Cr” may indicate the chroma sample values ​​of the chroma samples in the current chroma block. The horizontal axis “luma” may indicate the luma sample values ​​of the co-located luma samples in the co-located luma block. The feature value T (2402) may be determined as the average value of the nearby reconstructed luma samples. The feature value F of each co-located luma sample in the co-located luma block may be determined as the luma sample value of the respective co-located luma sample.

[0192] If F is less than T, each equally-located luma sample of the equally-located luma block and its corresponding chroma sample of the current chroma block may be grouped into a first group G0.

[0193] If F is greater than T, each equally-located luma sample of the equally-located luma block and its corresponding chroma sample of the current chroma block may be grouped into a second group G1.

[0194] Different cross-component prediction modes may further be applied to the first group G0 and the second group G1. In the example of Figure 24, a CCCM mode may be applied to G0 and a GGLM mode may be applied to G1.

[0195]

[0200] In an embodiment, for each group or samples within a selected group, a flag or other coding information may be signaled to indicate which of multiple cross-component prediction modes was selected.

[0196]

[0201] In one example, when entropy coding is applied to a flag indicating which of multiple cross-component prediction modes is selected for each group, the context of the entropy coding may depend on the flag values ​​of other groups being signaled, e.g., the flag for group G1 may be based on the flag for group G0 being signaled.

[0197]

[0202] In an embodiment, a first flag may be signaled to indicate whether any or all groups (e.g., G0 and G1) share the same cross-component prediction mode. If the first flag is true, another syntax (or syntax element) may be signaled to indicate which cross-component prediction mode applies. If the first flag is false, a respective syntax may be signaled for each group to indicate which of multiple cross-component prediction modes applies to the respective group. For example, if the first flag is false, a first syntax may be signaled for G0 and a second syntax may be signaled for G1.

[0198]

[0203] In one example, when the first flag is false and the first cross-component prediction mode is applied to other groups except one or more specific groups, such as the last group, a second cross-component prediction mode for the last group may be implicitly derived or signaled. However, the signaled second cross-component prediction mode may be different from the first cross-component prediction mode. Thus, the second cross-component prediction mode for the last group may be determined excluding the first cross-component prediction mode selected for the other groups.

[0199]

[0204] In an embodiment, an index syntax may be signaled to indicate which cross-component mode from among a plurality of cross-component prediction modes is applied in the first group. i Index and group i-1 The delta value, which is the difference between the index of i For example, an index syntax may be signaled for the first group G0. Furthermore, a delta value may be signaled for the second group G1. The delta value may indicate a difference between an index of the first group G0 and an index of the second group G1. The index of the second group G1 may be determined as a sum of the index syntax and the delta value.

[0200]

[0205] In one example, the multiple cross-component prediction modes may include CCLM and CCCM.

[0201]

[0206] In an embodiment, if a different filter (or cross-component prediction) is applied to multiple groups of samples (e.g., G0 and G1) in a coding block, a predicted sample may be generated for each group. If all predicted samples of multiple groups are obtained, a filter may be applied to all predicted samples of the coding block.

[0202]

[0207] In one example, the filter applied to the predicted samples of the coding block may include a smoothing filter, such as a Gaussian, maximum, average, median, minimum, non-local mean, percentile, and rank filter. In one example, the filter applied to the predicted samples of the coding block may be a sharpening filter, such as a Laplace filter.

[0203]

[0208] In an embodiment, a plurality of prediction blocks of a current block may be first generated separately according to a plurality of cross-component modes, and a final prediction block may be determined based on a weighted average of the prediction blocks. For example, as shown in FIG. 24, a first prediction block may be generated based on a CCCM mode. The first prediction block may indicate a prediction sample of a chroma sample in a first group G0. A second prediction block may be generated based on a CCLM mode. The second prediction block may indicate a prediction sample of a chroma sample in a second group G1. A final prediction block may be further determined based on a weighted average of the first prediction block and the second prediction block. The final prediction block may indicate a prediction sample of a chroma sample of a current chroma block (or a chroma sample of a current block).

[0204]

[0209] In one example, a flag can be signaled first to indicate whether multiple prediction blocks of the current block are generated separately by multiple cross-component modes and the final prediction block is determined based on a weighted average of the prediction blocks. If the flag is true, a syntax element can be signaled to indicate which cross-component mode is used to generate the weighted average of the prediction blocks.

[0205]

[0210] In one example, the multiple cross-component modes may include CCLM and / or CCCM.

[0206]

[0211] FIG. 25 shows a flow chart outlining an exemplary decoding process (2500) according to some embodiments of the present disclosure. FIG. 26 shows a flow chart outlining an exemplary encoding process (2600) according to some embodiments of the present disclosure. The proposed processes may be used separately or combined in any order. Furthermore, each of the processes (or embodiments), the encoder, and the decoder may be implemented by a processing circuit (e.g., one or more processors, or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0207]

[0212] The process operations (e.g., 2500 and 2600) may be combined or arranged in any quantity or order as desired. In an embodiment, two or more of the process operations (e.g., 2500 and 2600) may be performed in parallel.

[0208]

[0213] The processes (e.g., (2500) and (2600)) may be used in the reconstruction and / or encoding of a block to generate a prediction block for the block being reconstructed. In various embodiments, the processes (e.g., (2500) and (2600)) are performed by processing circuitry, such as processing circuitry in the terminal devices (310), (320), (330), (340), processing circuitry performing the functions of a video encoder (403), processing circuitry performing the functions of a video decoder (410), processing circuitry performing the functions of a video decoder (510), processing circuitry performing the functions of a video encoder (603), etc. In some embodiments, the processes (e.g., (2500) and (2600)) are embodied in software instructions, such that the processing circuitry performs the processes (e.g., (2500) and (2600)) when it executes the software instructions.

[0209]

[0214] As shown in Figure 25, the process (2500) may start at (S2501) and proceed to (S2510), where coded information of a current chroma block and a luma block co-located with the current chroma block is received.

[0210]

[0215] In (S2520), a feature value is determined based on at least one of (i) reconstructed chroma samples in a vicinity of the current chroma block, and (ii) reconstructed luma samples in a vicinity of a luma block at a position equivalent to the current chroma block.

[0211]

[0216] In (S2530), based on a feature value threshold, the chroma samples of the current chroma block and the luma samples of the luma blocks co-located with the current chroma block are grouped into a plurality of groups, each of the plurality of groups including a respective chroma sample and a respective luma sample.

[0212]

[0217] At (S2540), a respective cross-component prediction mode for each of the plurality of groups is determined by comparing a respective chroma sample and a respective luma sample for each respective group to the determined feature values.

[0213]

[0218] At (S2550), the current chroma block is reconstructed based on the determined cross-component prediction modes of the multiple groups.

[0214]

[0219] In one example, the feature value is determined as one of the average value of reconstructed chroma samples in the vicinity of the current chroma block and the average value of reconstructed luma samples in the vicinity of a luma block at a comparable position.

[0215]

[0220] In one example, the feature value is determined as one of the average gradient value of reconstructed chroma samples in the vicinity of the current chroma block and the average gradient value of reconstructed luma samples in the vicinity of a luma block at an equivalent position.

[0216]

[0221] In one example, the feature value is determined as the average value of the reconstructed chroma samples in the vicinity of the current chroma block and the reconstructed luma samples in the vicinity of the co-located luma block.

[0217]

[0222] To group the chroma samples of the current chroma block and the luma samples of the co-located luma block, a characteristic value associated with each luma sample of the co-located luma block is determined. It is determined whether the characteristic value associated with each luma sample of the co-located luma block is greater than a feature value threshold.

[0218] In one example, (i) each luma sample among the luma samples of a co-located luma block and (ii) each chroma sample among the chroma samples of the current chroma block corresponding to each luma sample are grouped into a first group based on a characteristic value associated with each luma sample being greater than a feature value threshold.

[0219] In one example, (i) each luma sample among the luma samples of the co-located luma block and (ii) each chroma sample among the chroma samples of the current chroma block corresponding to each luma sample are grouped into a second group based on a characteristic value associated with each luma sample being less than a feature value threshold.

[0220]

[0223] In some embodiments, the characteristic value of each luma sample includes one of: (i) the luma sample value of the each luma sample; and (ii) an average value of reconstructed luma samples in the vicinity of the co-located luma block.

[0221]

[0224] In some embodiments, the respective cross-component prediction mode includes one of a cross-component linear model (CCLM), chroma-from-luma (CfL), a convolutional cross-component model (CCCM), a multi-filter linear model (MFLM), a gradient linear model (GLM), a combination of CCLM, CfL, and CCCM, and an angular intra prediction mode.

[0222]

[0225] In some embodiments, the respective cross-component prediction mode is determined based on a corresponding flag included in the coded information.

[0223]

[0226] In an embodiment, it is determined whether each of the multiple groups share a cross-component prediction mode based on a flag included in the coded information.

[0224] In one example, a cross-component prediction mode for each of the multiple groups is determined based on the flag indicating that each of the multiple groups shares a cross-component prediction mode.

[0225] In one example, a respective cross-component prediction mode for each of the multiple groups is determined based on a corresponding syntax element based on the flag indicating that each of the multiple groups does not share a cross-component prediction mode.

[0226]

[0227] In an embodiment, a first cross-component prediction mode is determined for a first group of the plurality of groups based on a first index included in the coded information, and a second cross-component prediction mode is determined for a second group of the plurality of groups based on a second index, the second index being determined based on a sum of a delta index and the first index included in the coded information.

[0227]

[0228] To reconstruct the current chroma block, in one example, a prediction sample for the chroma samples in each of the multiple groups is generated based on a respective cross-component prediction mode, and a filter can then be applied to the prediction sample for the chroma samples in each of the multiple groups.

[0228]

[0229] In one example, a prediction sample for the chroma samples in each of the multiple groups is generated based on a respective cross-component prediction mode, and a prediction sample for the current chroma block is determined as a weighted combination of the prediction samples for the chroma samples in each of the multiple groups.

[0229]

[0230] After (S2550), the process proceeds to (S2599) and ends.

[0230]

[0231] The method 2500 may be adapted as appropriate. Steps in the process 2500 may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0231]

[0232] As shown in Figure 26, the process (2600) may start at (S2601) and proceed to (S2610), where a feature value is determined based on at least one of (i) reconstructed chroma samples in a vicinity of a current chroma block, and (ii) reconstructed luma samples in a vicinity of a luma block co-located with the current chroma block.

[0232]

[0233] In (S2620), based on a feature value threshold, the chroma samples of the current chroma block and the luma samples of the luma block co-located with the current chroma block are grouped into a plurality of groups, where each of the plurality of groups includes a respective chroma sample and a respective luma sample.

[0233]

[0234] At (S2630), a respective cross-component prediction mode for each of the plurality of groups is determined by comparing a respective chroma sample and a respective luma sample for each respective group to the determined feature values.

[0234]

[0235] At (S2640), a predicted sample of the current chroma block is generated based on the determined cross-component prediction mode of the plurality of groups, and coding information indicating the respective cross-component prediction modes for each of the plurality of groups is also generated.

[0235]

[0236] The process then proceeds to (S2699) and ends.

[0236]

[0237] The method 2600 may be adapted as appropriate. Steps in the process 2600 may be modified and / or omitted. Additional steps may be added. Any suitable order of implementation may be used.

[0237]

[0238] The techniques described above may be implemented as computer software using computer readable instructions and may be physically stored on one or more computer readable media. For example, Figure 27 illustrates a computer system (2700) suitable for implementing certain embodiments of the disclosed subject matter.

[0238]

[0239] Computer software may be coded using any suitable machine code or computer language that may be subject to assembly, compilation, linking, or similar mechanisms to produce code that contains instructions that may be executed directly by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc., or instructions that may be executed via interpretation, microcode execution, etc.

[0239]

[0240] The instructions may be executed on various types of computers or components thereof, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0240]

[0241] 27 with respect to the computer system (2700) are exemplary in nature and are not intended to suggest any limitation as to the scope of functionality or application of the computer software implementing the embodiments of the present disclosure, nor should the arrangement of components be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary embodiment of the computer system (2700).

[0241]

[0242] The computer system (2700) may include certain human interface input devices that may be responsive to input by one or more human users, for example, via tactile input (e.g., keystrokes, swipes, data glove movements), auditory input (e.g., voice, clapping), visual input (e.g., gestures), or olfactory input (not shown). Human interface devices may also be used to capture certain media that are not necessarily directly related to conscious human input, such as audio (e.g., speech, music, ambient sounds), images (e.g., scanned images, photographic images obtained from still image cameras), and video (e.g., two-dimensional video, three-dimensional video including stereoscopic pictures).

[0242]

[0243] The input human interface devices may include one or more of (although only one of each is depicted) a keyboard (2701), a mouse (2702), a trackpad (2703), a touch screen (2710), a data glove (not shown), a joystick (2705), a microphone (2706), a scanner (2707), and a camera (2708).

[0243]

[0244] The computer system (2700) may also include certain human interface output devices that may stimulate one or more of the senses of a human user, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (2710), data gloves (not shown), joystick (2705), although there may be haptic feedback devices that do not serve as input devices), auditory output devices (e.g., speakers (2709), headphones (not shown)), visual output devices (e.g., screens (2710), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output, three or more dimensional output by means such as stereoscopic output; virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0244]

[0245] The computer system (2700) may also include human accessible storage devices and their associated media, such as optical media including CD / DVD ROM / RW (2720) using media such as CD / DVD (2721), thumb drives (2722), removable hard drives or solid state drives (2723), legacy magnetic media (not shown) such as tapes and floppy disks (not shown), and specialized ROM / ASIC / PLD based devices such as security dongles (not shown).

[0245]

[0246] Those skilled in the art will also understand that the term "computer-readable medium" as used in connection with the subject matter disclosed herein does not encompass transmission media, carrier waves, or other transitional signals.

[0246]

[0247] The computer system (2700) may also include interfaces to one or more communications networks (2755). The networks may be, for example, wireless, wired, or optical. The networks may further be local, wide area, metropolitan, vehicle industrial, real-time, delay tolerant, etc. Examples of networks include local area networks such as Ethernet, wireless LANs, cellular networks (including GSM, 3G, 4G, 5G, LTE, etc.), TV wired or wireless wide area digital networks (including cable TV, satellite TV, and terrestrial TV), vehicle industrial including CANBus, etc. Particular networks typically require an external network interface adapter attached to a particular general-purpose data port or peripheral bus (2749) (e.g., a USB port on the computer system (2700)); others are commonly integrated into the core of the computer system (2700) by attaching to a system bus as described below (e.g., an Ethernet interface is integrated in a PC computer system, a cellular network interface is integrated in a smartphone computer system). Using any of these networks, the computer system (2700) can communicate with other entities. Such communications can be one-way receive only (e.g., broadcast TV), one-way transmit only (e.g., CANbus to certain CANbus devices), or bi-directional, such as to other computer systems using local or wide area digital networks. Specific protocols and protocol stacks can be used with each of these networks and network interfaces, as described above.

[0247]

[0248] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be attached to a core (2740) of the computer system (2700).

[0248]

[0249] The core (2740) may include one or more central processing units (CPUs) (2741), graphics processing units (GPUs) (2742), specialized programmable processing units in the form of field programmable gate areas (FPGAs) (2743), task specific hardware accelerators (2744), graphics adapters (2750), etc. These devices may be connected via a system bus (2748) along with read only memory (ROM) (2745), random access memory (2746), and internal mass storage devices (e.g., internal non-user accessible hard drives, SSDs, etc.) (2747). In some computer systems, the system bus (2748) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be attached directly to the core's system bus (2748) or via a peripheral bus (2749). In one example, a screen (2710) can be connected to a graphics adapter (2750). Peripheral bus architectures include PCI, USB, etc.

[0249]

[0250] The CPU (2741), GPU (2742), FPGA (2743), and accelerator (2744) may combine to execute certain instructions that may constitute the aforementioned computer code. The computer code may be stored in a ROM (2745) or a RAM (2746). Temporary data may be stored in the RAM (2746), while persistent data may be stored in, for example, an internal mass storage (2747). Fast storage and retrieval from any memory device may be made possible by utilizing a cache memory, which may be closely associated with one or more of the CPU (2741), GPU (2742), mass storage (2747), ROM (2745), RAM (2746), etc.

[0250]

[0251] The computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and computer code can be specially designed and constructed for the purposes of the present disclosure, or they can be of the kind well known and available to those having skill in the computer software arts.

[0251]

[0252] By way of example, and not by way of limitation, a computer system having the architecture (2700), and in particular the core (2740), can provide functionality as a result of operations by a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media can be media associated with user-accessible mass storage as described above, as well as specific storage of the core (2740) of a non-transitory nature, such as mass storage (2747) within the core or ROM (2745). Software implementing various embodiments of the present disclosure can be stored in such devices and executed by the core (2740). The computer-readable media can include one or more memory devices or chips, depending on the particular needs. The software can cause the core (2740) and in particular the processors therein (including a CPU, GPU, FPGA, etc.) to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (2746) and modifying such data structures according to processes defined by the software. Additionally or alternatively, a computer system may provide functionality as a result of logic hardwired or otherwise embedded in circuitry (e.g., accelerator (2744)) that may execute in place of or in conjunction with software to perform a particular process or a particular portion of a particular process described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include circuitry (such as an integrated circuit (IC)) that stores software for execution, circuitry embodying logic for execution, or both, as appropriate. The present disclosure encompasses any appropriate combination of hardware and software.

[0252] Appendix A: Acronyms JEM:joint exploration model VVC:versatile video coding BMS:benchmark set MV: Motion Vector HEVC:High Efficiency Video Coding SEI:Supplementary Enhancement Information VUI:Video Usability Information GOPs:Groups of Pictures TUs:Transform Units PUs:Prediction Units CTUs:Coding Tree Units CTBs:Coding Tree Blocks PBs:Prediction Blocks HRD:Hypothetical Reference Decoder SNR:Signal Noise Ratio CPUs:Central Processing Units GPUs:Graphics Processing Units CRT:Cathode Ray Tube LCD:Liquid-Crystal Display OLED:Organic Light-Emitting Diode CD:Compact Disc DVD:Digital Video Disc ROM:Read-Only Memory RAM:Random Access Memory ASIC:Application-Specific Integrated Circuit PLD:Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile communications LTE: Long-Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI:Peripheral Component Interconnect FPGA:Field Programmable Gate Areas SSD: solid-state drive IC: Integrated Circuit CU: Coding Unit

[0253] While this disclosure describes a number of exemplary embodiments, there are modifications, permutations, and various substitute equivalents which fall within the scope of this disclosure. It will thus be understood that those skilled in the art will be able to devise numerous systems and methods which, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within its spirit and scope.

[0253]

[0254] Additional Notes (Appendix 1) 1. A video decoding method performed in a video decoder, comprising: receiving coded information of a current chroma block and a luma block co-located with the current chroma block; determining a feature value based on at least one of (i) reconstructed chroma samples in a vicinity of the current chroma block, and (ii) reconstructed luma samples in a vicinity of a luma block co-located with the current chroma block; grouping the chroma samples of the current chroma block and the luma samples of a luma block co-located with the current chroma block into a plurality of groups based on the feature value threshold, each of the plurality of groups including a respective chroma sample and a respective luma sample; determining a respective cross-component prediction mode for each of the plurality of groups by comparing the respective chroma samples and the respective luma samples of each respective group to determined feature values; and reconstructing the current chroma block based on the determined cross-component prediction modes of the plurality of groups; The method includes:

[0254] (Appendix 2) 2. The method of claim 1, wherein the step of determining the feature value further comprises: determining the feature value as one of an average value of reconstructed chroma samples in a neighborhood of the current chroma block and an average value of reconstructed luma samples in a neighborhood of the equally positioned luma block; The method includes:

[0255] (Appendix 3) 2. The method of claim 1, wherein the step of determining the feature value further comprises: determining the feature value as one of an average gradient value of reconstructed chroma samples in a neighborhood of the current chroma block and an average gradient value of reconstructed luma samples in a neighborhood of the equally positioned luma block; The method includes:

[0256] (Appendix 4) 2. The method of claim 1, wherein the step of determining the feature value further comprises: determining the feature value as an average value of reconstructed chroma samples in a neighborhood of the current chroma block and reconstructed luma samples in a neighborhood of the equally positioned luma block; The method includes:

[0257] (Appendix 5) 2. The method of claim 1, wherein the grouping step further comprises: determining a characteristic value associated with each luma sample of the co-located luma block; determining whether a characteristic value associated with each luma sample of the co-located luma blocks is greater than the feature value threshold; grouping (i) each luma sample among the luma samples of the equivalently positioned luma block and (ii) each chroma sample among the chroma samples of the current chroma block corresponding to each luma sample into a first group based on a characteristic value associated with the each luma sample being greater than the feature value threshold; and grouping (i) each luma sample among the luma samples of the equivalently positioned luma block and (ii) each chroma sample among the chroma samples of the current chroma block corresponding to each luma sample into a second group based on a characteristic value associated with the each luma sample being less than the feature value threshold; A method comprising:

[0258] (Appendix 6) 6. The method of claim 5, wherein the characteristic value of each luma sample comprises one of: (i) a luma sample value of the each luma sample; and (ii) an average value of reconstructed luma samples in a vicinity of the equally positioned luma block.

[0259] (Appendix 7) 2. The method of claim 1, wherein each cross-component prediction mode includes one of a cross-component linear model (CCLM), a chroma-from-luma (CfL), a convolutional cross-component model (CCCM), a multi-filter linear model (MFLM), a gradient linear model (GLM), a combination of CCLM, CfL, and CCCM, and an angular intra prediction mode.

[0260] (Appendix 8) 2. The method of claim 1, wherein determining a respective cross-component prediction mode for each of the plurality of groups further comprises: determining the respective cross-component prediction modes based on corresponding flags included in the coded information; A method comprising:

[0261] (Appendix 9) 2. The method of claim 1, wherein determining a respective cross-component prediction mode for each of the plurality of groups further comprises: determining whether each of the plurality of groups shares a cross-component prediction mode based on a flag included in the coded information; determining a cross-component prediction mode for each of the plurality of groups in response to the flag indicating that each of the plurality of groups shares a cross-component prediction mode; and in response to the flag indicating that each of the plurality of groups does not share a cross-component prediction mode, determining a respective cross-component prediction mode for each of the plurality of groups based on a corresponding syntax element; A method comprising:

[0262] (Appendix 10) 2. The method of claim 1, wherein determining a respective cross-component prediction mode for each of the plurality of groups further comprises: determining a first cross-component prediction mode for a first group of the plurality of groups based on a first index included in the coded information; and determining a second cross-component prediction mode for a second group of the plurality of groups based on a second index, the second index being determined based on a sum of a delta index included in the coded information and the first index; A method comprising:

[0263] (Appendix 11) 2. The method of claim 1, wherein the step of reconstructing the current chroma block comprises: generating prediction samples for the chroma samples in each of the plurality of groups based on the respective cross-component prediction modes; and applying a filter to prediction samples for the chroma samples in each of the plurality of groups; A method comprising:

[0264] (Appendix 12) 2. The method of claim 1, wherein the step of reconstructing the current chroma block comprises: generating prediction samples for the chroma samples in each of the plurality of groups based on the respective cross-component prediction modes; and determining a prediction sample for the current chroma block as a weighted combination of prediction samples for the chroma samples in each of the plurality of groups; A method comprising:

[0265] (Appendix 13) 1. An apparatus including a processing circuit, the processing circuit comprising: receiving coded information of a current chroma block and a luma block co-located with the current chroma block; determining a feature value based on at least one of (i) reconstructed chroma samples in a vicinity of the current chroma block, and (ii) reconstructed luma samples in a vicinity of a luma block co-located with the current chroma block; grouping the chroma samples of the current chroma block and the luma samples of a luma block co-located with the current chroma block into a plurality of groups based on the feature value threshold, each of the plurality of groups including a respective chroma sample and a respective luma sample; determining a respective cross-component prediction mode for each of the plurality of groups by comparing the respective chroma samples and the respective luma samples of each respective group to determined feature values; and reconstructing the current chroma block based on the determined cross-component prediction modes of the plurality of groups; 23. An apparatus configured to:

[0266] (Appendix 14) 14. The apparatus of claim 13, wherein the processing circuitry: determining the feature value as one of an average value of reconstructed chroma samples in a neighborhood of the current chroma block and an average value of reconstructed luma samples in a neighborhood of the equally positioned luma block; 23. An apparatus configured to:

[0267] (Appendix 15) 14. The apparatus of claim 13, wherein the processing circuitry: determining the feature value as one of an average gradient value of reconstructed chroma samples in a neighborhood of the current chroma block and an average gradient value of reconstructed luma samples in a neighborhood of the equally positioned luma block; 23. An apparatus configured to:

[0268] (Appendix 16) 14. The apparatus of claim 13, wherein the processing circuitry: determining the feature value as an average value of reconstructed chroma samples in a neighborhood of the current chroma block and reconstructed luma samples in a neighborhood of the equally positioned luma block; 23. An apparatus configured to:

[0269] (Appendix 17) 14. The apparatus of claim 13, wherein the processing circuitry: determining a characteristic value associated with each luma sample of the co-located luma block; determining whether a characteristic value associated with each luma sample of the co-located luma blocks is greater than the feature value threshold; grouping (i) each luma sample among the luma samples of the equivalently positioned luma block and (ii) each chroma sample among the chroma samples of the current chroma block corresponding to each luma sample into a first group based on a characteristic value associated with the each luma sample being greater than the feature value threshold; and grouping (i) each luma sample among the luma samples of the equivalently positioned luma block and (ii) each chroma sample among the chroma samples of the current chroma block corresponding to each luma sample into a second group based on a characteristic value associated with the each luma sample being less than the feature value threshold; 23. An apparatus configured to:

[0270] (Appendix 18) 18. The apparatus of claim 17, wherein the characteristic value of each luma sample comprises one of: (i) a luma sample value of the each luma sample; and (ii) an average value of reconstructed luma samples in a vicinity of the equally positioned luma block.

[0271] (Appendix 19) 14. The apparatus of claim 13, wherein each cross-component prediction mode includes one of a cross-component linear model (CCLM), a chroma-from-luma (CfL), a convolutional cross-component model (CCCM), a multi-filter linear model (MFLM), a gradient linear model (GLM), a combination of CCLM, CfL, and CCCM, and an angular intra prediction mode.

[0272] (Appendix 20) 14. The apparatus of claim 13, wherein the processing circuitry: determining the respective cross-component prediction modes based on corresponding flags included in the coded information; 23. An apparatus configured to:

Claims

1. 1. A video decoding method performed in a video decoder, comprising: receiving coded information of a current chroma block and a luma block co-located with the current chroma block; determining a feature value based on at least one of (i) reconstructed chroma samples in a vicinity of the current chroma block, and (ii) reconstructed luma samples in a vicinity of a luma block co-located with the current chroma block; grouping the chroma samples of the current chroma block and the luma samples of a co-located luma block into a plurality of groups based on the feature value threshold, each of the plurality of groups including a respective chroma sample and a respective luma sample; determining a respective cross-component prediction mode for each of the plurality of groups by comparing the respective chroma samples and the respective luma samples of each respective group to determined feature values; and reconstructing the current chroma block based on the determined cross-component prediction modes of the plurality of groups; The method includes:

2. 2. The method of claim 1, wherein the step of determining the feature value further comprises: determining the feature value as one of an average value of reconstructed chroma samples in a neighborhood of the current chroma block and an average value of reconstructed luma samples in a neighborhood of the equally positioned luma block; The method includes:

3. 2. The method of claim 1, wherein the step of determining the feature value further comprises: determining the feature value as one of an average gradient value of reconstructed chroma samples in a neighborhood of the current chroma block and an average gradient value of reconstructed luma samples in a neighborhood of the equally positioned luma block; The method includes:

4. 2. The method of claim 1, wherein the step of determining the feature value further comprises: determining the feature value as an average value of reconstructed chroma samples in a neighborhood of the current chroma block and reconstructed luma samples in a neighborhood of the equally positioned luma block; The method includes:

5. 10. The method of claim 1, wherein the grouping step further comprises: determining a characteristic value associated with each luma sample of the co-located luma block; determining whether a characteristic value associated with each luma sample of the co-located luma block is greater than the feature value threshold; grouping (i) each luma sample among the luma samples of the co-located luma block and (ii) a chroma sample among the chroma samples of the current chroma block corresponding to each luma sample into a first group based on a characteristic value associated with the each luma sample being greater than the feature value threshold; and grouping (i) the respective luma sample among the luma samples of the equivalently positioned luma block and (ii) the chroma sample among the chroma samples of the current chroma block corresponding to the respective luma sample into a second group based on the characteristic value associated with the respective luma sample being less than the feature value threshold; A method comprising:

6. 6. The method of claim 5, wherein the characteristic value of each luma sample comprises one of: (i) a luma sample value of the each luma sample; and (ii) an average value of reconstructed luma samples in a neighborhood of the equally positioned luma block.

7. 2. The method of claim 1, wherein each cross-component prediction mode includes one of a cross-component linear model (CCLM), a chroma-from-luma (CfL), a convolutional cross-component model (CCCM), a multi-filter linear model (MFLM), a gradient linear model (GLM), a combination of CCLM, CfL, and CCCM, and an angular intra prediction mode.

8. 10. The method of claim 1, wherein determining a respective cross-component prediction mode for each of the plurality of groups further comprises: determining the respective cross-component prediction mode based on a corresponding flag included in the coded information; A method comprising:

9. 10. The method of claim 1, wherein determining a respective cross-component prediction mode for each of the plurality of groups further comprises: determining whether each of the plurality of groups share a cross-component prediction mode based on a flag included in the coded information; determining a cross-component prediction mode for each of the plurality of groups in response to the flag indicating that each of the plurality of groups shares a cross-component prediction mode; and determining a respective cross-component prediction mode for each of the plurality of groups based on a corresponding syntax element in response to the flag indicating that each of the plurality of groups does not share a cross-component prediction mode; A method comprising:

10. 10. The method of claim 1, wherein determining a respective cross-component prediction mode for each of the plurality of groups further comprises: determining a first cross-component prediction mode for a first group of the plurality of groups based on a first index included in the coded information; and determining a second cross-component prediction mode for a second group of the plurality of groups based on a second index, the second index being determined based on a sum of a delta index included in the coded information and the first index; A method comprising:

11. 2. The method of claim 1, wherein the step of reconstructing the current chroma block comprises: generating prediction samples for the chroma samples in each of the plurality of groups based on the respective cross-component prediction modes; and applying a filter to prediction samples for the chroma samples in each of the plurality of groups; A method comprising:

12. 2. The method of claim 1, wherein the step of reconstructing the current chroma block comprises: generating prediction samples for the chroma samples in each of the plurality of groups based on the respective cross-component prediction modes; and determining a prediction sample for the current chroma block as a weighted combination of prediction samples for the chroma samples in each of the plurality of groups; A method comprising:

13. 13. Apparatus comprising processing circuitry for carrying out the method of any one of claims 1-12.

14. 1. A video encoding method performed in a video encoder, comprising: creating an encoded bitstream including coded information of a current chroma block and a luma block co-located with the current chroma block; and transmitting the encoded bitstream to a video decoder, the video encoder comprising: Based on the feature value threshold determined based on at least one of (i) reconstructed chroma samples in a vicinity of the current chroma block, and (ii) reconstructed luma samples in a vicinity of a luma block co-located with the current chroma block, the chroma samples of the current chroma block and the luma samples of the luma block co-located with the current chroma block are grouped into a plurality of groups, each of the plurality of groups including a respective chroma sample and a respective luma sample; The respective chroma samples and the respective luma samples of each respective group are compared to the feature values ​​to determine a respective cross-component prediction mode for each of the plurality of groups; and The current chroma block is reconstructed based on the determined cross-component prediction modes of the multiple groups.

Images