Cross-component plane prediction in image and video compression.

Abstract

Aspects of the present disclosure provide a method and apparatus including a processing circuit that decodes prediction information for a chroma block to be reconstructed in a current picture. The prediction information indicates that the chroma block is predicted based on at least a luma block in the current picture using a planar mode. The luma block is reconstructed and co-located with the chroma block. The processing circuit predicts that one or more first chroma samples in the chroma block are based on one or more luma samples in the luma block. The processing circuit predicts a second chroma sample in the chroma block based on the one or more first chroma samples and at least one reference chroma sample of the chroma block. The second chroma sample is different from the one or more first chroma samples. The at least one reference chroma sample is adjacent to the chroma block.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims benefit of priority to U.S. Provisional Application No. 17 / 976,470, entitled "CROSS-COMPONENT PLANAR PREDICTION IN IMAGE AND VIDEO COMPRESSION," filed on October 28, 2022, which claims benefit of priority to U.S. Provisional Application No. 63 / 305,168, entitled "Cross-Component Planar Prediction in Image and Video Compression," filed on January 31, 2022. The disclosures of the prior applications are incorporated herein by reference in their entireties.

[0002] This disclosure describes embodiments that relate generally to video coding. [Background technology]

[0003] The background discussion provided herein is intended to generally present the context of the present disclosure. The inventors' work, to the extent that it is described in this background section, and aspects of the description that may not be admitted as prior art at the time of filing, are not admitted expressly or impliedly as prior art to the present disclosure.

[0004] An uncompressed digital image and / or video may include a sequence of pictures, each having spatial dimensions of, for example, 1920x1080 luma samples and associated chroma samples. The sequence of pictures may have a fixed or variable picture rate (also informally known as frame rate) of, for example, 60 pictures per second, i.e., 60 Hz. Uncompressed images and / or videos have specific bitrate requirements. For example, 1080p60 4:2:0 video (1920x1080 luma sample resolution at a frame rate of 60 Hz) with 8 bits per sample requires a bandwidth approaching 1.5 Gbit / s. One hour of such video requires more than 600 GByte of storage space.

[0005] One objective of image and / or video coding and decoding may be the reduction of redundancy in the input image and / or video signal through compression. Compression may help reduce the aforementioned bandwidth and / or storage space requirements, possibly by more than one order of magnitude. The description herein uses video encoding / decoding as an illustrative example, but the same techniques may be applied to image encoding / decoding in a similar manner without departing from the spirit of this disclosure. Both lossless and lossy compression, and combinations thereof, may be employed. Lossless compression refers to techniques where an exact copy of the original signal may be reconstructed from the compressed original signal. When using lossy compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original signal and the reconstructed signal is small enough to make the reconstructed signal useful for the intended application. For video, lossy compression is widely adopted. The amount of acceptable distortion depends on the application, for example, a user of a particular consumer streaming application may tolerate higher distortion than a user of a television distribution application. The achievable compression ratio may reflect that the higher the tolerable / acceptable distortion, the higher the compression ratio may be.

[0006] Video encoders and decoders can utilize techniques from a number of broad categories, including, for example, motion compensation, transform processing, quantization, and entropy coding.

[0007] Video codec technology can include a technique known as intra-coding. In intra-coding, sample values ​​are represented without reference to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially subdivided into blocks of samples. When all blocks of samples are coded in intra mode, the picture may be an intra picture. Intra pictures and their derivatives, such as independent decoder refresh pictures, are used to reset the decoder state and therefore can be used as the first picture in a coded video bitstream and video session or as a still image. Samples of an intra block undergo a transform and the transform coefficients can be quantized before entropy coding. Intra prediction can be a technique that minimizes sample values ​​in the pre-transform domain. In some cases, the smaller the DC value after the transform and the smaller the AC coefficients, the fewer bits are required for a given quantization step size to represent the block after entropy coding.

[0008] Conventional intra-coding, for example as used in MPEG-2 generation coding techniques, does not use intra-prediction. However, some newer video compression techniques include techniques that attempt to perform prediction based on surrounding sample data and / or metadata obtained during encoding / decoding of a block of data, for example. Such techniques are hereafter referred to as "intra-prediction" techniques. It should be noted that at least in some cases, intra-prediction uses only reference data from the current picture being reconstructed, and not from a reference picture.

[0009] Intra prediction may take many different forms. When more than one of such techniques can be used in a given video coding technique, the particular technique in use may be coded as a particular intra prediction mode that uses the particular technique. In certain cases, an intra prediction mode may have sub-modes and / or parameters that may be coded separately or included in a mode codeword that defines the prediction mode used. Which codeword is used for a given mode, sub-mode, and / or parameter combination may affect the coding efficiency gains via intra prediction, and thus may also affect the entropy coding technique used to convert the codeword into a bitstream.

[0010] A specific mode of intra prediction was introduced in H.264, improved in H.265, and further refined in novel coding techniques such as the Joint Search Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). Neighboring sample values ​​of already available samples can be used to form a predictor block. The sample values ​​of the neighboring samples are copied to the predictor block according to the direction. The reference to the direction in use can be coded in the bitstream or can be predicted itself.

[0011] Referring to FIG. 1A, depicted at the bottom right is a subset of 9 predictor directions known from the 33 possible predictor directions defined in H.265 (corresponding to the 33 angle modes of the 35 intra modes). The point where the arrows converge (101) represents the sample being predicted. The arrows represent the direction in which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from one or more samples to the upper right at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from one or more samples to the lower left of sample (101) at an angle of 22.5 degrees from the horizontal.

[0012] 1A, at the top left is shown a square block (104) of 4×4 samples (indicated by a thick dashed line). The square block (104) contains 16 samples, each labeled with "S", its position in the Y dimension (e.g., row index), and its position in the X dimension (e.g., column index). For example, sample S21 is the second sample (from the top) in the Y dimension and the first sample (from the left) in the X dimension. Similarly, sample S44 is the fourth sample in both the Y and X dimensions in the block (104). Since the block is 4×4 samples in size, S44 is at the bottom right. Also shown are reference samples that follow a similar numbering scheme. The reference samples are labeled R, their Y position (e.g., row index), and X position (column index) relative to the block (104). In both H.264 and H.265, the predicted samples are neighbors of the block being reconstructed, and negative values ​​do not need to be used.

[0013] Intra-picture prediction can work by copying reference sample values ​​from nearby samples indicated by the signaled prediction direction. For example, assume that the coded video bitstream includes signaling for this block indicating a prediction direction that matches the arrow (102), i.e., the sample is predicted from the sample to the upper right at an angle of 45 degrees from the horizontal. In that case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In certain cases, to calculate a reference sample, especially when the orientation is not evenly divisible by 45 degrees, the values ​​of multiple reference samples may be combined, for example by interpolation.

[0015] The number of possible directions is increasing as video coding techniques develop. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013). Currently, JEM / VVC / BMS can support up to 65 directions. Experiments are being carried out to identify the most likely directions, and certain techniques of entropy coding are used to represent those likely directions with a small number of bits, accepting a certain penalty for less likely directions. Furthermore, in some cases the direction itself can be predicted from nearby directions used in nearby, already decoded blocks.

[0016] FIG. 1B shows a schematic diagram (110) showing 65 intra prediction directions with JEM to illustrate the increasing number of prediction directions over time.

[0017] The mapping of intra-prediction direction bits, which represent directions in the coded video bitstream, may vary between video coding techniques. Such mappings may range from simple direct mappings to complex adaptation schemes including codewords, most probable modes, and similar techniques. However, in most cases, there may be certain directions that are statistically less likely to occur in the video content than certain other directions. Since the goal of video compression is to reduce redundancy, these less likely directions are represented by more bits than more likely directions in well-performing video coding techniques.

[0018] Image and / or video coding and decoding can be performed using inter-picture prediction with motion compensation. Motion compensation is a lossy compression technique and may refer to a technique in which blocks of sample data from a previously reconstructed picture or part thereof (reference picture) are used to predict a newly reconstructed picture or part of a picture after being spatially shifted in a direction indicated by a motion vector (hereinafter MV). In some cases, the reference picture may be the same as the picture currently being reconstructed. The MV may have two dimensions X and Y, or three dimensions, with the third dimension being an indication of the reference picture in use (the third dimension may indirectly be a temporal dimension).

[0019] In some video compression techniques, the MV applicable to a particular area of ​​sample data can be predicted from other MVs, e.g., from MVs associated with other areas of sample data that are spatially adjacent to the area being reconstructed and that precede that MV in decoding order. Doing so can significantly reduce the amount of data required to code the MV, thereby eliminating redundancy and increasing the compression ratio. For example, when coding an input video signal derived from a camera (known as natural video), MV prediction can work effectively because there is a statistical likelihood that areas larger than the area to which a single MV is applicable move in similar directions and therefore, in some cases, can be predicted using similar motion vectors derived from MVs of nearby areas. As a result, the detected MV for a given area is similar or the same as the MV predicted from the surrounding MVs, which, after entropy coding, can be represented with fewer bits than would be used if the MV were coded directly. In some cases, MV prediction can be an example of lossless compression of a signal (i.e., MV) derived from an original signal (i.e., a sample stream). In other cases, the MV prediction itself can be lossy, e.g., due to rounding errors when computing a predictor from several surrounding MVs.

[0020] Various MV prediction mechanisms are described in H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016). Among the many MV prediction mechanisms offered by H.265, the one hereafter referred to as "spatial merging" is described with reference to Fig. 2.

[0021] Referring to Figure 2, a current block (201) contains samples that have been discovered by the encoder during the motion search process to be predictable from a previous block of the same size but spatially shifted. Instead of coding its MV directly, the MV can be derived from metadata associated with one or more reference pictures, e.g., from the most recent reference picture (in decoding order), using MVs associated with any one of five surrounding samples, denoted A0, A1, and B0, B1, B2 (202-206, respectively). In H.265, MV prediction can use predictors from the same reference picture that neighboring blocks are using. Summary of the Invention [Means for solving the problem]

[0022] Aspects of the present disclosure provide a method and apparatus for video encoding / decoding. In some examples, an apparatus for video decoding includes a processing circuit. The processing circuit decodes prediction information of a chroma block to be reconstructed in a current picture. The prediction information indicates that the chroma block is predicted based on at least a luma block in the current picture using a planar mode. The luma block has already been reconstructed and is co-located with the chroma block. The processing circuit predicts that one or more first chroma samples in the chroma block are based on one or more luma samples in the luma block. The processing circuit predicts a second chroma sample in the chroma block based on the one or more first chroma samples and at least one reference chroma sample of the chroma block. The second chroma sample is different from the one or more first chroma samples. The at least one reference chroma sample is adjacent to the chroma block.

[0023] In an embodiment, the one or more first chroma samples are chroma samples of a lower right corner in a chroma block. The processing circuit predicts at least one of (i) a chroma sample of a row below the chroma block, or (ii) a chroma sample of a rightmost column of the chroma block based on the chroma sample of the lower right corner and at least one reference chroma sample. In response to the chroma sample of the row below the chroma block being predicted, the processing circuit generates a vertical predictor of the second chroma sample based on a reference chroma sample above in the at least one reference chroma sample and a predicted lower chroma sample within the predicted chroma sample of the row below. The second chroma sample, the reference chroma sample above, and the predicted lower chroma sample are in the same column in the chroma block. In response to the chroma sample in the rightmost column of the chroma block being predicted, the processing circuit generates a horizontal predictor for the second chroma sample based on a left reference chroma sample in the at least one reference chroma sample and a predicted right chroma sample in the predicted chroma sample in the rightmost column, the second chroma sample, the left reference chroma sample, and the predicted right chroma sample being in a same row of the chroma block.

[0024] In an example, the processing circuit predicts a chroma sample in a row below the chroma block and a chroma sample in a rightmost column of the chroma block. The processing circuit predicts a second chroma sample based on a vertical predictor and a horizontal predictor.

[0025] In an example, the width of the chroma block is greater than the height of the chroma block or the height of the chroma block is less than a threshold. The processing circuit predicts a chroma sample in a row below the chroma block and predicts a second chroma sample based on a vertical predictor.

[0026] In an example, the width of the chroma block is less than the height of the chroma block or a threshold. The processing circuit predicts a rightmost column of chroma samples of the chroma block and predicts a second chroma sample based on a horizontal predictor.

[0027] In an example, the processing circuit predicts a chroma sample at the bottom right corner in a chroma block based on at least one of a luma sample at the bottom right corner in the luma block or a luma sample in a luma block co-located with the chroma sample at the bottom right corner.

[0028] In an example, the one or more first chroma samples include (i) a chroma sample in a row below the chroma block, and (ii) a chroma sample in a rightmost column of the chroma block. The processing circuit generates a vertical predictor of the second chroma sample based on an above reference chroma sample in the at least one reference chroma sample and a predicted below chroma sample in the predicted chroma sample in the row below. The second chroma sample, the above reference chroma sample, and the predicted below chroma sample are in the same column in the chroma block. The processing circuit generates a horizontal predictor of the second chroma sample based on a left reference chroma sample in the at least one reference chroma sample and a predicted right chroma sample in the predicted chroma sample in the rightmost column. The second chroma sample, the left reference chroma sample, and the predicted right chroma sample are in the same row of the chroma block. The processing circuit predicts the second chroma sample based on the vertical predictor and the horizontal predictor.

[0029] In an example, the width of the chroma block is greater than the height of the chroma block or the height of the chroma block is less than a threshold. The one or more first chroma samples include a chroma sample in a row below the chroma block. The processing circuit generates a vertical predictor for the second chroma sample based on an above reference chroma sample in the at least one reference chroma sample and a predicted below chroma sample in the predicted chroma sample in the row below. The second chroma sample, the above reference chroma sample, and the predicted below chroma sample are in the same column in the chroma block. The processing circuit predicts the second chroma sample based on the vertical predictor.

[0030] In an example, the width of the chroma block is less than the height of the chroma block or a threshold value. The one or more first chroma samples include a chroma sample in a rightmost column of the chroma block. The processing circuit generates a horizontal predictor for the second chroma sample based on a left reference chroma sample in the at least one reference chroma sample and a predicted right chroma sample in the predicted chroma sample in the rightmost column. The second chroma sample, the left reference chroma sample, and the predicted right chroma sample are in the same row of the chroma block. The processing circuit predicts the second chroma sample based on the horizontal predictor.

[0031] In an example, the processing circuit predicts one or more first chroma samples in a chroma block based on one or more luma samples in the luma block co-located with the one or more first chroma samples using a cross-component linear model prediction (CCLM) mode.

[0032] In an example, the at least one reference chroma sample includes at least one of an above reference chroma sample in a neighbor above the chroma block, or a left reference chroma sample in a neighbor to the left of the chroma block.

[0033] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method for video decoding.

[0034] Further features, nature and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings. [Brief description of the drawings]

[0035] [Figure 1A] FIG. 2 is a schematic diagram of an example subset of intra-prediction modes. [Figure 1B] FIG. 2 is a diagram of an example intra-prediction direction. [Diagram 2]FIG. 2 shows an example of a current block (201) and surrounding samples. [Diagram 3] FIG. 3 is a schematic diagram of an example block diagram of a communication system (300). [Figure 4] FIG. 4 is a schematic diagram of an example block diagram of a communication system (400). [Diagram 5] FIG. 2 is a schematic diagram of an example block diagram of a decoder. [Figure 6] FIG. 2 is a schematic diagram of an example block diagram of an encoder; [Figure 7] 1 is a block diagram illustrating an exemplary encoder. [Figure 8] FIG. 2 is a block diagram illustrating an exemplary decoder. [Figure 9] 4 shows an example reference sample of the current block. [Figure 10] 13 shows examples of nearby reconstructed luma samples and nearby reconstructed chroma samples used in the Cross-Component Linear Model (CCLM) mode. [Figure 11] 13 shows examples of nearby reconstructed luma samples and nearby reconstructed chroma samples used in CCLM mode. [Figure 12] 1 illustrates exemplary samples or sample positions in a chroma block. [Figure 13] 1 shows a flowchart outlining an encoding process according to some embodiments of the present disclosure. [Figure 14] 1 shows a flowchart outlining a decoding process according to some embodiments of the present disclosure. [Figure 15] FIG. 1 is a schematic diagram of a computer system according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] FIG. 3 illustrates an example block diagram of a communication system (300). The communication system (300) includes a plurality of terminal devices that may communicate with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via the network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) implements a unidirectional transmission of data. For example, the terminal device (310) may code video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to the other terminal device (320) via the network (350). The encoded video data may be transmitted in the form of one or more coded video bitstreams. The terminal device (320) may receive the coded video data from the network (350), decode the coded video data to reconstruct the video pictures, and display the video pictures according to the reconstructed video data. Unidirectional data transmission may be common, such as in media serving applications.

[0037] In another example, the communication system (300) includes a second pair of terminal devices (330) and (340) performing bidirectional transmission of coded video data, for example during a video conference. In the case of bidirectional transmission of data, in one example, each of the terminal devices (330) and (340) can code video data (e.g., a stream of video pictures captured by the terminal device) for transmission to the other of the terminal devices (330) and (340) over the network (350). Each of the terminal devices (330) and (340) can also receive coded video data transmitted by the other of the terminal devices (330) and (340), can decode the coded video data to recover the video pictures, and can display the video pictures on an accessible display device according to the recovered video data.

[0038] In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) are illustrated as a server, a personal computer, and a smartphone, respectively, although the principles of the present disclosure may not be so limited. The embodiments of the present disclosure apply with laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. The network (350) represents any number of networks that convey coded video data between the terminal devices (310), (320), (330), and (340), including, for example, wired (wired) and / or wireless communication networks. The communication network (350) may exchange data over circuit-switched and / or packet-switched channels. Exemplary networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For purposes of this description, the architecture and topology of the network (350) may not be important to the operation of the present disclosure unless otherwise described herein below.

[0039] 4 shows a video encoder and video decoder in a streaming environment as an example of an application for the disclosed subject matter. The disclosed subject matter may be equally applicable to other video-enabled applications including, for example, video conferencing, digital television, streaming services, storage of compressed video on digital media including CDs, DVDs, memory sticks, etc.

[0040] The streaming system may include a video source (401) that creates a stream of uncompressed video pictures (402), a capture subsystem (413) that may include, for example, a digital camera. In one example, the stream of video pictures (402) includes samples taken by a digital camera. The stream of video pictures (402) is shown in bold to emphasize its large amount of data compared to the encoded video data (404) (or coded video bitstream) and may be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) may include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as described in more detail below. The encoded video data (404) (or coded video bitstream) is shown in thin to emphasize its small amount of data compared to the stream of video pictures (402) and may be stored in a streaming server (405) for future use. One or more streaming client subsystems, such as the client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include a video decoder (410), for example, within an electronic device (430). The video decoder (410) decodes an input copy (407) of the encoded video data and creates an output stream (411) of video pictures that can be rendered on a display (412) (e.g., a display screen) or other rendering device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., a video bitstream) can be encoded according to a particular video coding / compression standard. Examples of such standards include ITU-T Recommendation H.265.In one example, a developing video coding standard is informally known as Versatile Video Coding (VVC), and the disclosed subject matter may be used in the context of VVC.

[0041] It should be noted that the electronic devices (420) and (430) may include other components (not shown). For example, the electronic device (420) may include a video decoder (not shown), and the electronic device (430) may also include a video encoder (not shown).

[0042] 5 shows an example block diagram of a video decoder (510). The video decoder (510) can be included in an electronic device (530). The electronic device (530) can include a receiver (531) (e.g., receiving circuitry). The video decoder (510) can be used in place of the video decoder (410) in the example of FIG.

[0043] The receiver (531) may receive one or more coded video sequences to be decoded by the video decoder (510). In one embodiment, one coded video sequence is received at a time, and the decoding of each coded video sequence is independent of the decoding of the other coded video sequences. The coded video sequences are received from a channel (501), which may be a hardware / software link to a storage device that stores the coded video data. The receiver (531) receives the coded video data along with other data, e.g., coded audio data and / or auxiliary data streams, which may be forwarded to their respective usage entities (not shown). The receiver (531) may separate the coded video sequences from the other data. To combat network jitter, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter, "parser (520)"). In certain applications, the buffer memory (515) is part of the video decoder (510). In other applications, the buffer memory (515) may be external to the video decoder (510) (not shown). In still other applications, there may be a buffer memory (not shown) external to the video decoder (510), e.g., to combat network jitter, plus another buffer memory (515) internal to the video decoder (510), e.g., to handle playout timing. When the receiver (531) is receiving data from a storage / forwarding device of sufficient bandwidth and controllability, or from an isosynchronous network, the buffer memory (515) may not be necessary or may be small. For use with best-effort packet networks such as the Internet, the buffer memory (515) may be required and may be relatively large, may be advantageously adaptively sized, and may be implemented at least in part in an operating system or similar element (not depicted) external to the video decoder (510).

[0044] The video decoder (510) may include a parser (520) to reconstruct symbols (521) from the coded video sequence. These categories of symbols include information used to manage the operation of the video decoder (510) and information for potentially controlling a rendering device, such as a rendering device (512) (e.g., a display screen) that is not an integral part of the electronic device (530) but may be coupled to the electronic device (530), as shown in FIG. 5. The control information for the rendering device may be in the form of a supplemental enhancement information (SEI) message or a parameter set fragment (not shown) of video usability information (VUI). The parser (520) may parse / entropy decode the received coded video sequence. The coding of the coded video sequence may follow a video coding technique or standard and may follow various principles including variable length coding with or without context dependency, Huffman coding, arithmetic coding, etc. The parser (520) may extract, from the coded video sequence, a set of subgroup parameters for at least one of the subgroups of pixels in the video decoder based on at least one parameter corresponding to the group. The subgroups may include groups of pictures (GOPs), pictures, tiles, slices, macroblocks, coding units (CUs), blocks, transform units (TUs), prediction units (PUs), etc. The parser (520) may also extract information from the coded video sequence, such as transform coefficients, quantizer parameter values, motion vectors, etc.

[0045] The parser (520) can perform entropy decoding / parsing operations on the video sequence received from the buffer memory (515) to produce symbols (521).

[0046] The reconstruction of the symbols (521) can involve several different units, depending on the type of coded video picture or portion thereof (inter-picture and intra-picture, inter-block and intra-block, etc.), as well as other factors. Which units are involved and how can be controlled by subgroup control information parsed by the parser (520) from the coded video sequence. The flow of such subgroup control information between the parser (520) and the following units is not depicted for the sake of illustration.

[0047] Beyond the functional blocks already mentioned, the video decoder (510) may be conceptually subdivided into several functional units, as described below. In an actual implementation operating under commercial constraints, many of these units may interact closely with each other and may be at least partially integrated with each other. However, for purposes of describing the disclosed subject matter, the following conceptual subdivision into functional units is appropriate:

[0048] The first unit is a scalar / inverse transform unit (551), which receives quantized transform coefficients as well as control information from the parser (520) including which transform to use, block size, quantization coefficients, quantization scaling matrices, etc. as symbols (521). The scalar / inverse transform unit (551) can output blocks comprising sample values ​​that can be input to an aggregator (555).

[0049] In some cases, the output samples of the scaler / inverse transform unit (551) may relate to intra-coded blocks. Intra-coded blocks are blocks that do not use prediction information from a previously reconstructed picture, but may use prediction information from a previously reconstructed portion of the current picture. Such prediction information may be provided by an intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) generates a block of the same size and shape as the block being reconstructed using surrounding already reconstructed information fetched from a current picture buffer (558). The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (555) adds, possibly on a sample-by-sample basis, the prediction information generated by the intra-prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551).

[0050] In other cases, the output samples of the scalar / inverse transform unit (551) may relate to an inter-coded, potentially motion-compensated block. In such cases, the motion compensated prediction unit (553) may access the reference picture memory (557) to fetch samples used for prediction. After motion compensating the fetched samples according to the symbols (521) related to the block, these samples may be added by the aggregator (555) to the output of the scalar / inverse transform unit (551) (in this case referred to as residual samples or residual signals) to generate output sample information. The addresses in the reference picture memory (557) from which the motion compensated prediction unit (553) fetches the prediction samples may be controlled by the motion vectors available to the motion compensated prediction unit (553), for example, in the form of symbols (521) that may have X, Y, and reference picture components. Motion compensation may also include interpolation of sample values ​​fetched from the reference picture memory (557) when sub-sample accurate motion vectors are used, motion vector prediction mechanisms, etc.

[0051] The output samples of the aggregator (555) may be subjected to various loop filtering techniques in a loop filter unit (556). Video compression techniques may include in-loop filter techniques controlled by parameters contained in the coded video sequence (also called a coded video bitstream) and provided to the loop filter unit (556) as symbols (521) from the parser (520). Video compression may also be performed in response to meta-information obtained during decoding of the coded picture or previous portions (in decoding order) of the coded video sequence, or in response to previously reconstructed and loop filtered sample values.

[0052] The output of the loop filter unit (556) may be a sample stream that can be stored in a reference picture memory (557) for use in future inter-picture prediction as well as output to a rendering device (512).

[0053] Once a particular coded picture is fully reconstructed, it can be used as a reference picture for future prediction. For example, once a coded picture corresponding to a current picture is fully reconstructed and the coded picture is identified (e.g., by the parser (520)) as a reference picture, the current picture buffer (558) can become part of the reference picture memory (557), and any unused current picture buffer can be reallocated before beginning reconstruction of the next coded picture.

[0054] The video decoder (510) may perform decoding operations according to a given video compression technique or standard, such as ITU-T Rec. H.265. The coded video sequence may conform to the syntax specified by the video compression technique or standard being used, in the sense that the coded video sequence adheres to both the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard. Specifically, the profile may select certain tools from among all tools available in the video compression technique or standard as tools that are only available to them under the profile. Also, what is required for compliance may be that the complexity of the coded video sequence is within a range defined by the level of the video compression technique or standard. In some cases, the level limits the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level may be further limited in some cases by the specification of a hypothetical reference decoder (HRD) and metadata for HRD buffer management signaled within the coded video sequence.

[0055] In one embodiment, the receiver (531) can receive additional (redundant) data along with the encoded video. The additional data may be included as part of the coded video sequence. The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data may be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) enhancement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0056] 6 shows an example block diagram of a video encoder (603). The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used in place of the video encoder (403) of the example of FIG.

[0057] The video encoder (603) can receive video samples from a video source (601) (which is not part of the electronic device (620) in the example of FIG. 6) that can capture video images that are encoded by the video encoder (603). In another example, the video source (601) is part of the electronic device (620).

[0058] The video source (601) may provide a source video sequence that is coded by the video encoder (603) in the form of a digital video sample stream that may be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit, ...), any color space (e.g., BT.601 Y CrCB, RGB, ...), and any suitable sampling structure (e.g., Y CrCb 4:2:0, Y CrCb 4:4:4). In a media serving system, the video source (601) may be a storage device that stores previously prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as multiple individual pictures that convey motion when viewed in sequence. The pictures themselves are organized as a spatial array of pixels, and each pixel may contain one or more samples, depending on the sampling structure, color space, etc. in use. Those skilled in the art can easily understand the relationship between pixels and samples. The following description focuses on samples.

[0059] According to one embodiment, the video encoder (603) may code and compress pictures of a source video sequence into a coded video sequence (643) in real time or under any other time constraints required. Enforcing an appropriate coding rate is one function of the controller (650). In some embodiments, the controller (650) controls and is operatively coupled to other functional units described below, which are not depicted for ease of illustration. Parameters set by the controller (650) may include rate control related parameters (picture skip, quantizer, lambda value for rate distortion optimization techniques, ...), picture size, group of pictures (GOP) layout, maximum motion vector search range, etc. The controller (650) may be configured to have other appropriate functions associated with the video encoder (603) optimized for a particular system design.

[0060] In some embodiments, the video encoder (603) is configured to operate in a coding loop. As an oversimplified explanation, in one example, the coding loop can include a source coder (630) (responsible for creating symbols, such as a symbol stream, based on an input picture to be coded and a reference picture, for example) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to create sample data in a manner similar to what a (remote) decoder would also create. The reconstructed sample stream (sample data) is input to a reference picture memory (634). Since decoding of the symbol stream results in bit-exact results regardless of the location of the decoder (local or remote), the contents of the reference picture memory (634) are also bit-exact between the local and remote encoders. In other words, the predictive part of the encoder "sees" exactly the same sample values ​​as the decoder would "see" when using prediction during decoding as reference picture samples. This basic principle of reference picture synchrony (and the resulting drift when synchrony cannot be maintained, for example due to channel errors) is also used in several related techniques.

[0061] The operation of the "local" decoder (633) may be the same as the operation of a "remote" decoder, such as the video decoder (510), already described in detail above in conjunction with Figure 5. However, and with brief reference to Figure 5, because symbols are available and the encoding / decoding of symbols into a coded video sequence by the entropy coder (645) and parser (520) may be lossless, the entropy decoding portion of the video decoder (510), including the buffer memory (515), and the parser (520), may not be fully implemented in the local decoder (633).

[0062] In one embodiment, the decoder techniques, except for analysis / entropy decoding, present in the decoder are present in the corresponding encoder in the same or substantially the same functional form. Thus, the disclosed subject matter focuses on the operation of the decoder. The description of the encoder techniques can be omitted since they are the inverse of the decoder techniques described generically. Only in certain areas are more detailed descriptions required and are provided below.

[0063] In operation, in some examples, the source coder (630) may perform motion-compensated predictive coding, which predictively codes an input picture with reference to one or more previously coded pictures from a video sequence designated as “reference pictures.” In this manner, the coding engine (632) codes differences between pixel blocks of the input picture and pixel blocks of reference pictures that may be selected as predictive references for the input picture.

[0064] The local video decoder (633) may decode the coded video data of pictures that may be designated as reference pictures based on the symbols created by the source coder (630). The operation of the coding engine (632) may advantageously be a lossy process. When the coded video data may be decoded in a video decoder (not shown in FIG. 6), the reconstructed video sequence may be a copy of the source video sequence, usually with some errors. The local video decoder (633) may replicate the decoding process that may be performed by the video decoder on the reference pictures and store the reconstructed reference pictures in the reference picture memory (634). In this way, the video encoder (603) may locally store copies of reconstructed reference pictures that have common content as the reconstructed reference pictures that will be obtained by the far-end video decoder (without transmission errors).

[0065] The predictor (635) may perform a prediction search for the coding engine (632). That is, for a new picture to be coded, the predictor (635) may search the reference picture memory (634) for sample data (as candidate reference pixel blocks) or specific metadata such as reference picture motion vectors, block shapes, etc., that may serve as suitable prediction references for the new pixels. The predictor (635) may operate on sample blocks on a pixel block by pixel block basis to find suitable prediction references. In some cases, as determined by the search results obtained by the predictor (635), the input picture may have prediction references drawn from multiple reference pictures stored in the reference picture memory (634).

[0066] The controller (650) can manage the coding operations of the source coder (630), including, for example, setting the parameters and subgroup parameters used to encode the video data.

[0067] The output of all the aforementioned functional units may undergo entropy coding in an entropy coder (645), which converts the symbols produced by the various functional units into a coded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0068] The transmitter (640) may buffer the coded video sequence created by the entropy coder (645) and prepare it for transmission over a communication channel (660), which may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (640) may merge the coded video data from the video encoder (603) with other data to be transmitted, such as coded audio data and / or auxiliary data streams (sources not shown).

[0069] The controller (650) can manage the operation of the video encoder (603). During coding, the controller (650) can assign a particular coded picture type to each coded picture, which may affect the coding techniques that may be applied to the respective picture. For example, pictures may often be assigned as one of the following picture types:

[0070] An intra picture (I-picture) may be a picture that can be coded and decoded without using any other picture in a sequence as a source of prediction. Some video codecs allow different types of intra pictures, including, for example, independent decoder refresh ("IDR") pictures. Those skilled in the art are aware of these variations of I-pictures, as well as their respective uses and characteristics.

[0071] A predictive picture (P picture) may be a picture that can be coded and decoded using intra- or inter-prediction, which uses at most one motion vector and reference index to predict sample values ​​for each block.

[0072] A bidirectionally predicted picture (B-picture) may be a picture that can be coded and decoded using intra- or inter-prediction that uses at most two motion vectors and reference indices to predict the sample values ​​of each block. Similarly, a multi-prediction picture may use more than two reference pictures and associated metadata for the reconstruction of a single block.

[0073] A source picture is usually spatially subdivided into multiple sample blocks (e.g., blocks of 4x4, 8x8, 4x8, or 16x16 samples each) and may be coded block by block. A block may be predictively coded with reference to other (already coded) blocks determined by the coding assignment applied to the block's respective picture. For example, a block of an I picture may be non-predictively coded or predictively coded with reference to already coded blocks of the same picture (spatial or intra prediction). A pixel block of a P picture may be predictively coded via spatial prediction with reference to one previously coded reference picture or via temporal prediction. A block of a B picture may be predictively coded via spatial prediction with reference to one or two previously coded reference pictures or via temporal prediction.

[0074] The video encoder (603) may perform coding operations in accordance with a given video coding technique or standard, such as ITU-T Rec. H.265. In its operations, the video encoder (603) may perform various compression operations, including predictive coding operations that exploit temporal and spatial redundancy in the input video sequence. Thus, the coded video data may conform to a syntax specified by the video coding technique or standard being used.

[0075] In one embodiment, the transmitter (640) can transmit additional data along with the encoded video. The source coder (630) may include such data as part of the coded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, etc.

[0076] Video may be captured as multiple source pictures (video pictures) in a time sequence. Intra-picture prediction (often abbreviated as intra-prediction) uses spatial correlation within a given picture, while inter-picture prediction uses correlation (temporal or other) between pictures. In one example, a particular picture being coded / decoded, called the current picture, is divided into blocks. When a block in the current picture is similar to a reference block in a reference picture that was previously coded and is still buffered in the video, the block in the current picture may be coded by a vector called a motion vector. A motion vector points to a reference block in a reference picture and may have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0077] In some embodiments, bi-prediction techniques can be used in inter-picture prediction. According to bi-prediction techniques, two reference pictures, such as a first reference picture and a second reference picture, are used, both of which are prior to the decoding order of the current picture in the video (but the display order may be past and future, respectively). A block in the current picture can be coded by a first motion vector pointing to a first reference block in the first reference picture and by a second motion vector pointing to a second reference block in the second reference picture. A block can be predicted by a combination of the first reference block and the second reference block.

[0078] Furthermore, merge mode techniques can be used in inter-picture prediction to improve coding efficiency.

[0079] According to some embodiments of the present disclosure, prediction, such as inter-picture prediction and intra-picture prediction, is performed on a block-by-block basis. For example, according to the HEVC standard, a picture in a sequence of video pictures is divided into coding tree units (CTUs) for compression, and the CTUs in a picture have the same size, such as 64×64 pixels, 32×32 pixels, 16×16 pixels, etc. In general, a CTU includes three coding tree blocks (CTBs), one luma CTB and two chroma CTBs. Each CTU can be recursively quad-tree partitioned into one or more coding units (CUs). For example, a CTU of 64×64 pixels can be partitioned into one CU of 64×64 pixels, or into four CUs of 32×32 pixels, or into 16 CUs of 16×16 pixels. In one example, each CU is analyzed to determine a prediction type of the CU, such as an inter prediction type or an intra prediction type. A CU is divided into one or more prediction units (PUs) according to temporal predictability and / or spatial predictability. In general, each PU includes one luma prediction block (PB) and two chroma PBs. In one embodiment, prediction operations in coding (encoding / decoding) are performed in units of prediction blocks. Taking a luma prediction block as an example of a prediction block, the prediction block includes a matrix of pixel values ​​(e.g., luma values) of 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0080] 7 shows an example diagram of a video encoder (703). The video encoder (703) is configured to receive a processed block of sample values ​​(e.g., a predictive block) in a current video picture in a sequence of video pictures and encode the processed block into a coded picture that is part of a coded video sequence. In one example, the video encoder (703) is used in place of the video encoder (403) of the example of FIG. 4.

[0081] In an HEVC example, the video encoder (703) receives a matrix of sample values ​​for a processing block, such as a predictive block of 8×8 samples. The video encoder (703) determines whether the processing block is optimally coded using intra mode, inter mode, or bi-predictive mode, for example, using rate-distortion optimization. When the processing block is to be coded in intra mode, the video encoder (703) may encode the processing block into a coded picture using intra prediction techniques, and when the processing block is to be coded in inter mode or bi-predictive mode, the video encoder (703) may encode the processing block into a coded picture using inter prediction techniques or bi-predictive techniques, respectively. In certain video coding techniques, the merge mode may be an inter-picture prediction sub-mode in which a motion vector is derived from one or more motion vector predictors without the aid of coded motion vector components outside the predictors. In certain other video coding techniques, there may be motion vector components applicable to the current block. In one example, the video encoder (703) includes other components, such as a mode decision module (not shown), to determine the mode of the processing block.

[0082] In the example of FIG. 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725), coupled together as shown in FIG.

[0083] The inter-encoder (730) is configured to receive samples of a current block (e.g., a processing block), compare the block to one or more reference blocks in a reference picture (e.g., blocks in a previous picture and a subsequent picture), generate inter-prediction information (e.g., a description of redundancy information, motion vectors, merge mode information due to inter-coding techniques), and calculate an inter-prediction result (e.g., a prediction block) based on the inter-prediction information using any suitable technique. In some examples, the reference picture is a decoded reference picture that is decoded based on the coded video information.

[0084] The intra encoder (722) is configured to receive samples of a current block (e.g., a processing block), possibly compare the block with already coded blocks in the same picture, generate transformed quantized coefficients, and possibly also generate intra prediction information (e.g., intra prediction direction information according to one or more intra encoding techniques). In one example, the intra encoder (722) also calculates intra prediction results (e.g., prediction blocks) based on the intra prediction information and reference blocks in the same picture.

[0085] The generic controller (721) is configured to determine generic control data and control other components of the video encoder (703) based on the generic control data. In one example, the generic controller (721) determines the mode of the block and provides a control signal to the switch (726) based on the mode. For example, when the mode is an intra mode, the generic controller (721) controls the switch (726) to select an intra mode result for use by the residual calculator (723) and controls the entropy encoder (725) to select intra prediction information and include the intra prediction information in the bitstream, and when the mode is an inter mode, the generic controller (721) controls the switch (726) to select an inter prediction result for use by the residual calculator (723) and controls the entropy encoder (725) to select inter prediction information and include the inter prediction information in the bitstream.

[0086] The residual calculator (723) is configured to calculate a difference (residual data) between the received block and a prediction result selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) is configured to operate based on the residual data to encode the residual data to generate transform coefficients. In one example, the residual encoder (724) is configured to transform the residual data from the spatial domain to the frequency domain to generate transform coefficients. The transform coefficients then undergo a quantization process to obtain quantized transform coefficients. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform an inverse transform and generate decoded residual data. The decoded residual data can be used by the intra-encoder (722) and the inter-encoder (730) as appropriate. For example, the inter-encoder (730) can generate decoded blocks based on the decoded residual data and the inter-prediction information, and the intra-encoder (722) can generate decoded blocks based on the decoded residual data and the intra-prediction information. The decoded blocks are appropriately processed to generate decoded pictures, which may be buffered in a memory circuit (not shown) and used as reference pictures in some examples.

[0087] The entropy encoder (725) is configured to format the bitstream to include the encoded block. The entropy encoder (725) is configured to include various information in the bitstream according to an appropriate standard, such as the HEVC standard. In one example, the entropy encoder (725) is configured to include in the bitstream general control data, selected prediction information (e.g., intra-prediction information or inter-prediction information), residual information, and other appropriate information. It should be noted that, in accordance with the disclosed subject matter, no residual information is present when coding a block in a merged sub-mode of either the inter-mode or the bi-prediction mode.

[0088] 8 shows an example diagram of a video decoder (810). The video decoder (810) is configured to receive coded pictures that are part of a coded video sequence and decode the coded pictures to generate reconstructed pictures. In one example, the video decoder (810) is used in place of the video decoder (410) of the example of FIG. 4.

[0089] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872) coupled together as shown in FIG. 8.

[0090] The entropy decoder (871) may be configured to reconstruct from the coded picture certain symbols that represent the syntax elements of which the coded picture is composed. Such symbols may include, for example, prediction information (e.g., intra- or inter-prediction information, etc.) that may identify the mode in which the block is coded (e.g., intra- or inter-prediction modes, inter- or bi-prediction modes of merged or other submodes, etc.) as well as certain samples or metadata used for prediction by the intra- or inter-decoder (872) or inter-decoder (880), respectively. The symbols may also include residual information, for example, in the form of quantized transform coefficients, etc. In one example, when the prediction mode is an inter- or bi-prediction mode, the inter-prediction information is provided to the inter-decoder (880), and when the prediction type is an intra-prediction type, the intra-prediction information is provided to the intra-decoder (872). The residual information may undergo inverse quantization and is provided to the residual decoder (873).

[0091] The inter decoder (880) is configured to receive the inter prediction information and to generate inter prediction results based on the inter prediction information.

[0092] The intra decoder (872) is configured to receive the intra prediction information and to generate a prediction result based on the intra prediction information.

[0093] The residual decoder (873) is configured to perform inverse quantization to extract the inverse quantized transform coefficients, and to process the inverse quantized transform coefficients to transform the residual information from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (to include quantization parameters (QPs)), which may be provided by the entropy decoder (871) (this may be only a small amount of control information, so a data path is not depicted).

[0094] The reconstruction module (874) is configured to combine, in the spatial domain, the residual information output by the residual decoder (873) and the prediction results (possibly output by the inter prediction module or the intra prediction module) to form reconstructed blocks that may become part of a reconstructed picture, which may become part of the reconstructed video. It should be noted that other suitable operations, such as a deblocking operation, may be performed to improve visual quality.

[0095] It should be noted that the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using any suitable technique. In one embodiment, the video encoders (403), (603), and (703) and the video decoders (410), (510), and (810) may be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), and (603) and the video decoders (410), (510), and (810) may be implemented using one or more processors executing software instructions.

[0096] In intra prediction or the intra prediction mode, the sample values of an encoded block can be predicted from already reconstructed neighboring samples, or reconstructed neighboring samples (referred to as reference samples).

[0097] An example of intra prediction is directional intra prediction. In directional intra prediction, samples (e.g., current samples) within the current block can be predicted using reference samples (e.g., prediction samples) or interpolated reference samples. For example, the line connecting the current sample and the prediction sample forms a given angular direction as used in the angular mode.

[0098] In another example of intra prediction, a planar mode based on sample interpolation is used. In the planar mode, one or more key positions within or around the current block can be predicted using neighboring reference samples. Other positions within the current block can be predicted as a linear combination of the samples at one or more key positions and the reference samples. The weights (e.g., combination weights) can be determined according to the position of the current sample within the current block.

[0099] Examples of the planar mode such as in VVC are calculated as follows. predV[x][y]=((H - 1 - y)×p[x][-1]+(y + 1)×p[-1][H])<<Log2(W) Equation 1 predH[x][y]=((W - 1 - x)×p[-1][y]+(x + 1)×p[W][-1])<<Log2(H) Equation 2 pred[x][y]=(predV[x][y]+predH[x][y]+W×H)>>(Log2(W)+Log2(H)+1) Equation 3

[0100] Referring to FIG. 9, a current block (900) includes samples at positions (0,0) to (H-1,W-1) in the current block (900). W and H are the width and height of the current block (900), respectively. As shown in Equations 1 to 3, a predicted sample value pred[x][y] of a current sample at position (x,y) (x=0,1,...,orW-1, and y=0,1,...,orH-1) in the current block (900) can be obtained as a weighted average of reference sample values ​​(e.g., p[-1][y], p[x][-1], p[-1][H], and p[W][-1]) of reference samples located at positions (-1,y), (x,-1), (-1,H), and (W,-1), respectively. The reference samples may include a reference sample located at the same row position (-1,y) as the current sample, a reference sample located at the same column position (x,-1) as the current sample, a reference sample at a bottom left position (-1,H) relative to the current block, and a reference sample at a top right position (W,-1) relative to the current block.

[0101] As described above, the reference sample value of the reference sample may include p[-1][y] of the reference sample at the position (-1, y), p[x][-1] of the reference sample at the position (x, -1), p[-1][H] of the reference sample at the lower left position (-1, H), and p[W][-1] of the reference sample at the position upper right (W, -1). In the example shown in Equation 1, the vertical predictor predV[x][y] is determined based on the reference samples at the positions (x, -1) and (-1, H). In the example shown in Equation 2, the horizontal predictor predH[x][y] is determined based on the reference samples at the positions (-1, y) and (W, -1). In Equation 3, the predicted sample value pred[x][y] is determined based on the average (e.g., weighted average) of the horizontal predictor predH[x][y] and the vertical predictor predV[x][y].

[0102] The cross-component linear model prediction (CCLM) mode is a cross-component prediction method. In CCLM, chroma samples can be predicted based on reconstructed luma samples using a linear model. The linear model can be constructed by already constructed samples in the neighborhood of the current block (e.g., the chroma block to be coded). In some embodiments, the prediction ability is high when the luma channel and the chroma channel are highly linearly correlated.

[0103] In an embodiment, a chroma block (e.g., a current Cb block or a current Cr block) is predicted based on a co-located luma block. A prediction block Pred_C of the chroma block can be derived as follows, and samples Pred_C(x,y) in the prediction block Pred_C of the chroma block can be determined based on samples in the already reconstructed co-located luma block. Pred_C(x,y)=a×Rec_L'(x,y)+b Equation 4

[0104] Pred_C(x,y) represents a predicted chroma sample at sample location (x,y) of a chroma block (e.g., in a chroma channel of a current picture or a chroma picture). Rec_L'(x,y) may be determined from reconstructed samples in a co-located luma block (e.g., a luma channel of a current picture or a luma picture) that has already been reconstructed. Rec_L'(x,y) may represent reconstructed luma samples in a co-located luma block or downsampled luma samples of a co-located luma block.

[0105] In the example, Rec_L' is the already reconstructed co-located luma block, e.g., when the color format is 4:4:4 and the size of the chroma block is identical to the size of the co-located luma block. Thus, Rec_L'(x,y) may represent the reconstructed sample in the co-located luma block, where the reconstructed sample corresponds to the sample position (x,y) of the chroma block.

[0106] In an example, Rec_L' is different from the already reconstructed co-located luma block. The current chroma block is co-located with the co-located luma block, and the luma and chroma channels have different resolutions. In an example, when the color format is 4:2:0, the resolution of the luma block is twice that of the chroma block in both the vertical and horizontal directions. Thus, when the color format is 4:2:0, Rec_L' can be a downsampled block of the corresponding luma block to match the chroma block size during derivation of the linear model. In some embodiments, the co-located luma block is downsampled when the color format is not 4:4:4.

[0107] The parameters (e.g., model parameters) a and b in Equation 4 may represent a gradient and an offset in the linear model shown in Equation 4 and may be referred to as a gradient parameter and an offset parameter, respectively. The parameters a and b in Equation 4 may be derived from reconstructed neighborhood samples (e.g., chroma samples and luma samples) around (i) the current chroma block in the chroma channel and (ii) the co-located luma block in the luma channel. The parameters a and b may be determined using any suitable method.

[0108] In an example, the parameters a and b are determined using classical linear regression theory. A minimal linear least-squares solution between (i) nearby reconstructed luma samples or downsampled samples of nearby reconstructed luma samples and (ii) nearby reconstructed chroma samples can be applied to derive the following parameters a and b:

number

number

[0109] In Equations 5-6, N reconstructed neighboring chroma samples Rec_C(i) and N corresponding luma samples Rec_L'(i) are used. In an example, such as when the color format is 4:4:4, the N corresponding luma samples Rec_L'(i) include N reconstructed neighboring luma samples of a luma block. In an example, such as when the color format is 4:2:0, the N corresponding luma samples Rec_L'(i) include N downsampled samples of the reconstructed neighboring luma samples of a luma block. i may be an integer from 1 to N.

[0110] The N reconstructed neighboring chroma samples used to determine the parameters a and b may include any suitable already reconstructed neighboring samples of a chroma block. The reconstructed neighboring luma samples used to determine the parameters a and b may include any suitable already reconstructed neighboring samples of a co-located luma block.

[0111] The prediction process of the CCLM mode may include (1) downsampling the co-located luma block and the reconstructed neighboring luma samples of the co-located luma block to obtain Rec_L′ and downsampled neighboring luma samples, thus matching the size of the corresponding chroma block, (2) deriving parameters a and b based on the downsampled neighboring luma samples and the reconstructed neighboring chroma samples, e.g., using Equations 5-6, and (3) applying the CCLM model (e.g., Equation 4) to generate a chroma prediction block Pred_C. In some examples, when the spatial resolutions of the co-located luma block and the chroma block are identical and step (2) is based on the reconstructed neighboring luma samples, step (1) is omitted.

[0112] FIG. 10 illustrates an example of reconstructed neighboring luma samples and reconstructed neighboring chroma samples used in CCLM derivation. A chroma block (1000) is being reconstructed. The chroma block (1000) has a width M (e.g., 8) and a height N (e.g., 4). N and M may be positive integers. The chroma block (1000) and a colocated luma block (e.g., colocated luma block) (1001) are used to predict the chroma block (1000). The luma block (1001) includes luma samples (1040). The luma block (1001) may have any suitable width and any suitable height. In the example illustrated in FIG. 10, the luma block (1001) has a width of 2M (e.g., 16) and a height of 2N (e.g., 8).

[0113] The neighboring chroma samples (1010) of the chroma block (1000) (e.g., shaded gray) have already been reconstructed. The neighboring luma samples (1020) of the luma block (1001) have already been reconstructed. The neighboring chroma samples (1010) can include the chroma samples in the top neighbor (1011) and the chroma samples in the left neighbor (1012). The neighboring luma samples (1020) of the luma block (1001) can include the luma samples in the top neighbor (1021) and the luma samples in the left neighbor (1022).

[0114] In the example of Figure 10, the neighboring luma samples (1020) are subsampled or downsampled to generate downsampled neighboring luma samples (1030) (e.g., shaded in gray) to match the number of neighboring chroma samples (e.g., 12). In the example shown in Figure 10, the neighboring chroma samples (1010) and the downsampled neighboring luma samples (1030) can be used to determine the parameters a and b, as shown in Equations 5-6.

[0115] In the example, the chroma samples in the neighborhood of the chroma block (1000) include only the chroma samples in the top neighborhood (1011) and do not include the chroma samples in the left neighborhood (1012). Thus, the luma samples in the neighborhood of the luma block (1001) include only the luma samples in the top neighborhood (1021) and do not include the luma samples in the left neighborhood (1022). As mentioned above, the luma samples in the top neighborhood (1021) can be downsampled (e.g., the downsampled luma samples are shaded in gray) to match the number of chroma samples in the top neighborhood (1011) (e.g., 8).

[0116] In the example, the chroma samples in the neighborhood of the chroma block (1000) include only the chroma samples in the left neighborhood (1012) and do not include the chroma samples in the top neighborhood (1011). Thus, the luma samples in the neighborhood of the luma block (1001) include only the luma samples in the left neighborhood (1022) and do not include the luma samples in the top neighborhood (1021). As described above, the luma samples in the left neighborhood (1022) can be downsampled (e.g., the downsampled luma samples are shaded in gray) to match the number of chroma samples in the left neighborhood (1012) (e.g., 4).

[0117] In some examples, the parameters a and b can be determined using chroma samples in reconstructed blocks neighboring the chroma block (1000) and corresponding luma samples in reconstructed blocks neighboring the luma block (1001).

[0118] The chroma block (1000) may be a Cb block in a Cb channel or a Cr block in a Cr channel. In an example, the parameters a and b may be determined separately for each chroma channel (e.g., Cr or Cb). For example, the parameters a and b of the Cr block are determined based on the chroma reconstructed neighboring samples of the Cr block, and the parameters a and b of the Cb block are determined based on the chroma reconstructed neighboring samples of the Cb block.

[0119] FIG. 10 shows an example of reconstructed neighboring luma samples and reconstructed neighboring chroma samples used in CCLM derivation for a chroma block (1000) having a rectangular shape. The description of FIG. 10 can be adapted for chroma blocks and co-located luma blocks having any shape, such as a square. FIG. 11 shows an example of reconstructed neighboring luma samples (1120) and reconstructed neighboring chroma samples (1110) used in CCLM derivation for a chroma block (1100). The width and height of the chroma block (1100) are W1 (e.g., 8) and H1 (e.g., 8), respectively, with W1 equal to H1. The chroma block (1100) and a co-located luma block (e.g., co-located luma block) (1101) are used to predict the chroma block (1100). The luma block (1101) has a width of 2W1 (e.g., 16) and a height of 2H1 (e.g., 16).

[0120] In some embodiments, the reference samples used to generate the linear model parameters a and b are noisy and / or relatively unrepresentative of the content inside the actual predicted block, and thus the prediction may be suboptimal in terms of coding efficiency.

[0121] This disclosure describes a cross-component planar prediction method in image and video compression. In an embodiment, cross-component prediction (e.g., cross-component linear model (CCLM) mode) and planar mode (e.g., planar intra prediction) are combined in image and video compression.

[0122] According to an embodiment of the present disclosure, cross-component planar prediction may be performed on a chroma block to be encoded (e.g., reconstructed or encoded) as follows: One or more chroma samples in a chroma block may be predicted based on one or more luma samples in a luma block (also referred to as a co-located luma block) that is collocated with the chroma block. The remaining chroma samples in the chroma block may be predicted based on one or more predicted chroma samples in the chroma block and reference chroma samples outside the chroma block, e.g., using a planar mode. The remaining chroma samples are different from the one or more chroma samples. The chroma block and the reference chroma samples may be in a chroma channel (e.g., Cb channel or Cr channel) of the current picture. The luma block is in the current picture. The reference chroma samples may be adjacent to the chroma block. The reference chroma samples may include chroma samples in chroma blocks in the neighborhood of the chroma block to be encoded. For example, the reference chroma samples include an above reference chroma sample above the chroma block and / or a left reference chroma sample to the left of the chroma block. The chroma blocks and luma blocks can have the same resolution or can have different resolutions.

[0123] Because one or more chroma samples of cross-component plane prediction are predicted from luma samples in a co-located luma block in the same picture (e.g., the current picture), cross-component plane prediction may be more accurate than the plane mode without cross-component prediction.

[0124] In an example, a key position (e.g., one or more chroma samples at a key position) of a chroma block is predicted using luma information (e.g., luma samples in a co-located luma block) using cross-component prediction such as CCLM mode. The remaining positions (e.g., all other positions) in the chroma block can then be predicted using the sample values ​​at the key positions of the chroma block and reference samples (e.g., the reference sample above and the reference sample to the left). In an example, a planar gradient effect can be expected for the predicted block of the current block. The planar gradient in cross-component planar prediction can be more accurate than the planar mode without cross-component prediction, and therefore cross-component planar prediction can be more accurate.

[0125] FIG. 12 illustrates example samples or sample positions in a chroma block (1200). The chroma block (1200) is to be coded. In the example, the chroma block (1200) is being reconstructed. A luma block (also called a co-located luma block) (not shown) is co-located with the chroma block (1200). The chroma block (1200) and the luma block are in the current picture. The chroma block (1200) has a width W (e.g., 8) and a height H (e.g., 8). W and H may be any suitable positive integers. W may be the same as H or may be different.

[0126] The reference chroma sample (1210) (shaded in gray) can include a top reference chroma sample (1211) and a left reference chroma sample (1212). The top reference chroma sample (1211) includes the top right reference chroma sample (1204) (called p[W-1][-1]). The left reference chroma sample (1212) includes the bottom left reference chroma sample (1203) (called p[-1][H-1]).

[0127] In the example, the reference chroma sample (1210) further includes the upper left reference chroma sample (1205). In the example, the reference chroma sample (1210) does not include the upper left reference chroma sample (1205).

[0128] The chroma block (1200) can be predicted using cross-component plane prediction. Chroma samples within the chroma block (1200), such as the bottom right corner chroma sample (1202) located at the bottom right corner position of the chroma block (1200), can be predicted based on the luma samples using cross-component prediction techniques (e.g., CCLM mode). Subsequently, using the predicted chroma samples such as the predicted bottom right corner chroma sample (1202) called pred[W-1][H-1], (i) the predicted samples of the chroma samples in the row (1213) below the chroma block (1200) (e.g., pred[x][H-1], where x is an integer from 0 to W-1) and / or (ii) the predicted samples of the chroma samples in the rightmost column (1214) of the chroma block (1200) (e.g., pred[W-1][y], where y is an integer from 0 to H-1) can be generated.

[0129] For example, the predicted chroma sample pred[x][H-1] in the bottom row (1213) is generated based on the average (e.g., linear interpolation) of the predicted bottom right corner chroma sample pred[W-1][H-1] (1202) and the lower left reference chroma sample p[-1][H-1] (1203) as shown in Equation 7. Similarly, the predicted chroma sample pred[W-1][y] in the rightmost column (1214) can be generated using the average (e.g., linear interpolation) of the predicted bottom right corner chroma sample pred[W-1][H-1] (1202) and the upper right reference chroma sample p[W-1][-1] (1204) as shown in Equation 8. pred[x][H-1]=((W-1-x)×p[-1][H-1]+(x+1)×p[W-1][H-1])<<Log2(W) Equation 7 pred[W - 1][y] = ((H - 1 - y) × p[W - 1][-1] + (y + 1) × p[W - 1][H - 1]) << Log2(H) Equation 8

[0130] The weights used in Equations 7 to 8 can be determined by the position of the predicted chroma sample pred[x][H - 1] in the lower row (1213) or the predicted chroma sample pred[W - 1][y] in the rightmost column (1214). The weight (e.g., (W - 1 - x)) of a predictor (or predictor sample) (e.g., p[-1][H - 1]) can increase as the distance between the prediction sample (e.g., pred[x][H - 1]) and the predictor decreases. In the example as shown in Equations 7 to 8, the relationship between (i) the weight of the predictor and (ii) the distance between the prediction sample and the predictor is linear.

[0131] For a chroma sample (1201) located at position (x, y) within a chroma block (1200) (e.g., x = 0, 1,..., W - 1, and y = 0, 1,..., H - 1), the prediction sample (also called the predicted sample) pred[x][y] can be generated by averaging the horizontal predictor predH[x][y] and the vertical predictor predV[x][y], for example, as shown in Equation 9 below. pred[x][y] = (predV[x][y] + predH[x][y] + W × H) >> (Log2(W) + Log2(H) + 1) Equation 9

[0132] The horizontal predictor predH[x][y] of the chroma sample (1201) may be generated based on the left reference chroma sample (1222) (e.g., pred[-1][y]) in the left reference chroma sample (1212) and the predicted right chroma sample (1224) (e.g., p[W-1][y]) in the predicted chroma sample in the rightmost column (1214). For example, the horizontal predictor predH[x][y] is a weighted average or linear interpolation of the left reference chroma sample (1222) (e.g., pred[-1][y]) and the predicted right chroma sample (1224) (e.g., p[W-1][y]) as shown in Equation 10. In the example, the chroma sample (1201), the left reference chroma sample (1222), and the predicted right chroma sample (1224) are in the same row of the chroma block (1200).

[0133] The vertical predictor predV[x][y] of the chroma sample (1201) may be generated based on the upper reference chroma sample (1221) (e.g., pred[x][-1]) in the upper reference chroma sample (1211) and the predicted lower chroma sample (1223) (e.g., p[x][H-1]) in the predicted chroma sample in the lower row (1213). For example, the vertical predictor predV[x][y] is a weighted average or linear interpolation of the upper reference chroma sample (1221) (e.g., pred[x][-1]) and the predicted lower chroma sample (1223) (e.g., p[x][H-1]) as shown in Equation 11. In the example, the chroma sample (1201), the upper reference chroma sample (1221), and the predicted lower chroma sample (1223) are in the same column of the chroma block (1200).

[0134] The weights (e.g., ((W-1-x), (x+1), (H-1-y), and (y+1))) used to generate the horizontal predictor predH[x][y] and the vertical predictor predV[x][y] of Formulas 10 to 11 can be determined by the position of the chroma sample (1201). The weight (e.g., (W-1-x)) of the predictor (or predictor sample) (e.g., p[-1][y]) can increase when the distance between the chroma sample (1201) and the predictor decreases. In the examples shown in Formulas 10 to 11, the relationship between (i) the weight of each predictor and (ii) the distance between the chroma sample (1201) and the predictor is linear. predH[x][y]=((W-1-x)×p[-1][y]+(x+1)×p[W-1][y])<<Log2(W) Formula 10 predV[x][y]=((H-1-y)×p[x][-1]+(y+1)×p[x][H-1])<<Log2(H) Formula 11

[0135] In an embodiment, the predicted chroma samples pred[W-1][y] of the rightmost column (1214) (e.g., all the predicted chroma samples pred[W-1][y] of the entire rightmost column (1214)) and the predicted chroma samples pred[x][H-1] of the bottom row (1213) (e.g., all the predicted chroma samples pred[x][H-1] of the entire bottom row (1213)) can be predicted from the luma samples in the collocated luma block using a cross-component prediction technique (e.g., the CCLM mode). For example, the predicted chroma samples pred[W-1][y] of the entire rightmost column (1214) and the predicted chroma samples pred[x][H-1] of the entire bottom row (1213) are directly obtained from cross-component prediction and are directly predicted from the luma samples in the collocated luma block without using Formulas 7 to 8. Subsequently, the predicted sample pred[x][y] of the chroma sample (1201) can be generated by averaging the horizontal predictor predH[x][y] and the vertical predictor predV[x][y] as described above in Formulas 9 to 11.

[0136] In some examples, the predicted chroma sample (e.g., pred[W-1][H-1]) is directly used to predict another chroma sample (e.g., pred[x][H-1]) or pred[W-1][y]) as described above in Equations 7-11. In examples, the respective residual is added to the predicted chroma sample (e.g., pred[W-1][H-1]) to obtain an updated predicted chroma sample (e.g., a reconstructed sample), and the updated predicted chroma sample is used to predict the other chroma sample (e.g., pred[x][H-1]) or pred[W-1][y]).

[0137] In one embodiment, the horizontal predictor predH[x][y] or the vertical predictor predV[x][y] can be used separately, for example, when the chroma block is rectangular (non-square). For example, when the width W of the chroma block is greater than the height H of the chroma block, only the vertical predictor predV[x][y] is used. For example, when the width W is less than the height H, only the horizontal predictor predH[x][y] is used.

[0138] In one embodiment, if the block size condition is met, the horizontal predictor predH[x][y] or the vertical predictor predV[x][y] can be used. For example, if the width W is less than a width threshold (e.g., a preset constant W0), the horizontal predictor predH[x][y] is enabled. If the height H is less than a height threshold (e.g., a preset constant H0), the vertical predictor predV[x][y] is enabled. If the horizontal predictor predH[x][y] and the vertical predictor predV[x][y] are enabled, the final prediction pred[x][y] can be the average of the horizontal predictor predH[x][y] and the vertical predictor predV[x][y], as described above in Equation 9.

[0139] In the example, if only the vertical predictor predV[x][y] is used, then no prediction is performed on the chroma samples in the rightmost column (1214) except for the chroma sample in the bottom right corner (1202). In the example, if only the horizontal predictor predH[x][y] is used, then no prediction is performed on the chroma samples in the bottom row (1213) except for the chroma sample in the bottom right corner (1202).

[0140] FIG. 13 shows a flow chart outlining a process (encoding process (1300)) according to an embodiment of the present disclosure. The process (1300) can be performed by an apparatus for video encoding, which may include a processing circuit. The processing circuit of the apparatus, e.g., the process (1300), can be configured to be performed by a processing circuit, such as the processing circuit in the terminal devices (310), (320), (330), and (340), the processing circuit performing the functions of a video encoder (e.g., (403), (603), (703)). In some embodiments, the process (1300) is implemented in software instructions, and thus the processing circuit performs the process (1300) when the processing circuit executes the software instructions. The process starts at (S1301) and proceeds to (S1310).

[0141] At (S1310), one or more first chroma samples in a chroma block in the current picture can be predicted based on one or more luma samples in a luma block in the current picture using a planar mode, the luma block having already been reconstructed and co-located with the chroma block.

[0142] In an embodiment, one or more first chroma samples in a chroma block are predicted based on one or more luma samples in a luma block that are co-located with the one or more first chroma samples using a cross-component linear model prediction (CCLM) mode.

[0143] At (S1320), a second chroma sample in the chroma block may be predicted based on the one or more first chroma samples and at least one reference chroma sample of the chroma block. The second chroma sample may be different from the one or more first chroma samples. The at least one reference chroma sample may be adjacent to the chroma block.

[0144] In an example, the at least one reference chroma sample includes at least one of an above reference chroma sample of a top neighbor of the chroma block or a left reference chroma sample of a left neighbor of the chroma block.

[0145] At (S1330), prediction information for the chroma block may be encoded to indicate that the chroma block is predicted based on the luma block using a planar mode. The encoded prediction information may be included in the encoded video bitstream.

[0146] The second chroma sample may be encoded based on the predicted second chroma sample. In an example, no residual is signaled for the second chroma sample in the encoded video bitstream. In an example, a residual of the second chroma sample is signaled in the encoded video bitstream.

[0147] Next, the process (1300) proceeds to (S1399) and ends.

[0148] Process (1300) can be adapted appropriately for various scenarios, and steps within process (1300) can be adjusted accordingly. One or more of the steps within process (1300) can be adapted, omitted, repeated, and / or combined. Any suitable order can be used to perform process (1300). Additional steps can be added.

[0149] In an embodiment, the one or more first chroma samples are chroma samples of the lower right corner in the chroma block. Based on the chroma sample of the lower right corner and at least one reference chroma sample, at least one of (i) a chroma sample of a row below the chroma block, or (ii) a chroma sample of a rightmost column of the chroma block can be predicted. In an example, the chroma sample of the row below the chroma block is predicted, and a vertical predictor of the second chroma sample is generated based on the reference chroma sample above in the at least one reference chroma sample and the predicted lower chroma sample in the predicted chroma sample of the row below. The second chroma sample, the reference chroma sample above, and the predicted lower chroma sample can be in the same column in the chroma block.

[0150] In an example, a chroma sample in a rightmost column of a chroma block is predicted, and a horizontal predictor of a second chroma sample is generated based on a left reference chroma sample in at least one reference chroma sample and a predicted right chroma sample in the predicted chroma sample in the rightmost column. The second chroma sample, the left reference chroma sample, and the predicted right chroma sample can be in the same row of the chroma block.

[0151] In the example, the chroma sample in the row below the chroma block and the chroma sample in the rightmost column of the chroma block are predicted. The second chroma sample can be predicted based on a vertical predictor and a horizontal predictor.

[0152] In an example, the width of a chroma block is greater than the height of the chroma block or the height of the chroma block is less than a threshold. The chroma samples in the row below the chroma block are predicted and the second chroma sample is predicted based on the vertical predictor.

[0153] In an example, the width of the chroma block is less than the height of the chroma block or a threshold value. The chroma sample in the rightmost column of the chroma block is predicted, and the second chroma sample is predicted based on the horizontal predictor.

[0154] In the example, a chroma sample in the bottom right corner in a chroma block is predicted based on at least one of a luma sample in the bottom right corner in a luma block or a luma sample in a luma block that is co-located with the chroma sample in the bottom right corner.

[0155] In one embodiment, the one or more first chroma samples include (i) a chroma sample in a row below the chroma block, and (ii) a chroma sample in a rightmost column of the chroma block. A vertical predictor of the second chroma sample can be generated based on an above reference chroma sample in the at least one reference chroma sample and a predicted below chroma sample in the predicted chroma sample in the row below. The second chroma sample, the above reference chroma sample, and the predicted below chroma sample can be in the same column in the chroma block. A horizontal predictor of the second chroma sample can be generated based on a left reference chroma sample in the at least one reference chroma sample and a predicted right chroma sample in the predicted chroma sample in the rightmost column. The second chroma sample, the left reference chroma sample, and the predicted right chroma sample can be in the same row in the chroma block. The second chroma sample can be predicted based on a vertical predictor and a horizontal predictor.

[0156] In an embodiment, the width of the chroma block is greater than the height of the chroma block or the height of the chroma block is less than a threshold. The one or more first chroma samples include a chroma sample in a row below the chroma block. The vertical predictor of the second chroma sample is generated based on an above reference chroma sample in the at least one reference chroma sample and a predicted below chroma sample in the predicted chroma sample in the row below. The second chroma sample, the above reference chroma sample, and the predicted below chroma sample can be in the same column in the chroma block. The second chroma sample is predicted based on the vertical predictor.

[0157] In one embodiment, the width of the chroma block is less than the height of the chroma block or a threshold value. The one or more first chroma samples include a chroma sample in a rightmost column of the chroma block. The horizontal predictor of the second chroma sample can be generated based on a left reference chroma sample in the at least one reference chroma sample and a predicted right chroma sample in the predicted chroma sample in the rightmost column. The second chroma sample, the left reference chroma sample, and the predicted right chroma sample can be in the same row in the chroma block. The second chroma sample is predicted based on the horizontal predictor.

[0158] FIG. 14 shows a flow chart outlining a process (e.g., a decoding process) (1400) according to one embodiment of the disclosure. The process (1400) can be used in a video decoder. The process (1400) can be performed by an apparatus for video encoding, which can include a receiving circuit and a processing circuit. The processing circuit of the apparatus, e.g., the processing circuit of the terminal devices (310), (320), (330) and (340), the processing circuit performing the function of the video decoder (410), the processing circuit performing the function of the video decoder (510), etc., can be configured to perform the process (1400). In some examples, the process (1400) is used in a video encoder (e.g., the video encoder (403), the video encoder (603)). In an example, the process (1400) is performed by a processing circuit performing the function of the video encoder (e.g., the video encoder (403), the video encoder (603)). In some embodiments, the process (1400) is implemented with software instructions, such that when a processing circuit executes the software instructions, the processing circuit performs the process (1400). The process begins at (S1401) and proceeds to (S1410).

[0159] At (S1410), prediction information for a chroma block to be reconstructed in the current picture may be decoded, where the prediction information indicates that the chroma block is predicted based on at least a luma block in the current picture using a planar mode, where the luma block has already been reconstructed and is co-located with the chroma block.

[0160] At (S1420), one or more first chroma samples in a chroma block may be predicted based on one or more luma samples in a luma block.

[0161] In an example, one or more first chroma samples in a chroma block are identified as being located at one or more respective key positions before being predicted.

[0162] In an embodiment, one or more first chroma samples in a chroma block are predicted based on one or more luma samples in a luma block that are co-located with the one or more first chroma samples using a cross-component linear model prediction (CCLM) mode.

[0163] At (S1430), a second chroma sample in the chroma block may be predicted based on the one or more first chroma samples and at least one reference chroma sample of the chroma block. The second chroma sample may be different from the one or more first chroma samples. In an example, the at least one reference chroma sample is adjacent to the chroma block.

[0164] The second chroma sample is reconstructed based on the predicted second chroma sample. In an example, the reconstructed second chroma sample is a predicted second chroma sample. In an example, the reconstructed second chroma sample is a sum of the predicted second chroma sample and a corresponding residual.

[0165] In an example, the at least one reference chroma sample includes at least one of an above reference chroma sample of a top neighbor of the chroma block or a left reference chroma sample of a left neighbor of the chroma block.

[0166] Thereafter, the process (1400) proceeds to (S1499) and ends.

[0167] Process (1400) can be adapted appropriately for various scenarios, and steps within process (1400) can be adjusted accordingly. One or more of the steps within process (1400) can be adapted, omitted, repeated, and / or combined. Any suitable order can be used to perform process (1400). Additional steps can be added.

[0168] In an embodiment, the one or more first chroma samples are chroma samples of the lower right corner in the chroma block. Based on the chroma sample of the lower right corner and at least one reference chroma sample, at least one of (i) a chroma sample of a row below the chroma block, or (ii) a chroma sample of a rightmost column of the chroma block can be predicted. In an example, the chroma sample of the row below the chroma block is predicted, and a vertical predictor of the second chroma sample is generated based on the reference chroma sample above in the at least one reference chroma sample and the predicted lower chroma sample in the predicted chroma sample of the row below. The second chroma sample, the reference chroma sample above, and the predicted lower chroma sample can be in the same column in the chroma block.

[0169] In an example, a chroma sample in a rightmost column of a chroma block is predicted, and a horizontal predictor of a second chroma sample is generated based on a left reference chroma sample in at least one reference chroma sample and a predicted right chroma sample in the predicted chroma sample in the rightmost column. The second chroma sample, the left reference chroma sample, and the predicted right chroma sample can be in the same row of the chroma block.

[0170] In the example, the chroma sample in the row below the chroma block and the chroma sample in the rightmost column of the chroma block are predicted. The second chroma sample can be predicted based on a vertical predictor and a horizontal predictor.

[0171] In an example, the width of a chroma block is greater than the height of the chroma block or the height of the chroma block is less than a threshold. The chroma samples in the row below the chroma block are predicted and the second chroma sample is predicted based on the vertical predictor.

[0172] In an example, the width of the chroma block is less than the height of the chroma block or a threshold value. The chroma sample in the rightmost column of the chroma block is predicted, and the second chroma sample is predicted based on the horizontal predictor.

[0173] In the example, a chroma sample in the bottom right corner in a chroma block is predicted based on at least one of a luma sample in the bottom right corner in a luma block or a luma sample in a luma block that is co-located with the chroma sample in the bottom right corner.

[0174] In one embodiment, the one or more first chroma samples include (i) a chroma sample in a row below the chroma block, and (ii) a chroma sample in a rightmost column of the chroma block. A vertical predictor of the second chroma sample can be generated based on an above reference chroma sample in the at least one reference chroma sample and a predicted below chroma sample in the predicted chroma sample in the row below. The second chroma sample, the above reference chroma sample, and the predicted below chroma sample can be in the same column in the chroma block. A horizontal predictor of the second chroma sample can be generated based on a left reference chroma sample in the at least one reference chroma sample and a predicted right chroma sample in the predicted chroma sample in the rightmost column. The second chroma sample, the left reference chroma sample, and the predicted right chroma sample can be in the same row in the chroma block. The second chroma sample can be predicted based on a vertical predictor and a horizontal predictor.

[0175] In an embodiment, the width of the chroma block is greater than the height of the chroma block or the height of the chroma block is less than a threshold. The one or more first chroma samples include a chroma sample in a row below the chroma block. The vertical predictor of the second chroma sample is generated based on an above reference chroma sample in the at least one reference chroma sample and a predicted below chroma sample in the predicted chroma sample in the row below. The second chroma sample, the above reference chroma sample, and the predicted below chroma sample can be in the same column in the chroma block. The second chroma sample is predicted based on the vertical predictor.

[0176] In one embodiment, the width of the chroma block is less than the height of the chroma block or a threshold value. The one or more first chroma samples include a chroma sample in a rightmost column of the chroma block. The horizontal predictor of the second chroma sample can be generated based on a left reference chroma sample in the at least one reference chroma sample and a predicted right chroma sample in the predicted chroma sample in the rightmost column. The second chroma sample, the left reference chroma sample, and the predicted right chroma sample can be in the same row in the chroma block. The second chroma sample is predicted based on the horizontal predictor.

[0177] The embodiments of the present disclosure may be used separately or combined in any order. Furthermore, each of the methods (or embodiments), the encoder, and the decoder may be implemented by a processing circuit (e.g., one or more processors, or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0178] The techniques described above may be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 15 illustrates a computer system (1500) suitable for implementing certain embodiments of the disclosed subject matter.

[0179] Computer software can be coded using any suitable machine code or computer language that is amenable to mechanisms such as assembly, compilation, linking, etc. to create code that includes instructions that can be executed by one or more computer central processing units (CPUs), graphics processing units (GPUs), etc. directly, or via interpretation, microcode execution, etc.

[0180] The instructions may be executed on various types of computers or computer components including, for example, personal computers, tablet computers, servers, smart phones, gaming consoles, Internet of Things devices, and the like.

[0181] 15 for computer system (1500) are exemplary in nature and are not intended to suggest any limitation as to the scope of use or functionality of the computer software implementing the embodiments of the present disclosure. The arrangement of components should not be interpreted as having any dependency or requirement regarding any one or combination of components illustrated in the exemplary embodiment of computer system (1500).

[0182] The computer system (1500) may include certain human interface input devices. Such human interface input devices may be responsive to input by one or more human users, for example, via tactile input (keystrokes, swipes, data glove movements, etc.), audio input (voice, clapping, etc.), visual input (gestures, etc.), olfactory input (not shown). Human interface devices may also be used to capture certain media not necessarily directly associated with conscious human input, such as audio (voice, music, ambient sounds, etc.), images (scanned images, photographic images obtained from still image cameras, etc.), video (two-dimensional video, three-dimensional video including stereoscopic video, etc.).

[0183] The input human interface devices may include one or more of a keyboard (1501), a mouse (1502), a trackpad (1503), a touch screen (1510), a data glove (not shown), a joystick (1505), a microphone (1506), a scanner (1507), a camera (1508), etc. (only one of each is shown).

[0184] The computer system (1500) may also include certain human interface output devices. Such human interface output devices may stimulate one or more of the senses of a human user, for example, through haptic output, sound, light, and smell / taste. Such human interface output devices may include haptic output devices (e.g., haptic feedback via a touch screen (1510), data gloves (not shown), or joystick (1505), although there may also be haptic feedback devices that do not function as input devices), audio output devices (such as speakers (1509), headphones (not shown)), visual output devices (such as screens (1510), including CRT screens, LCD screens, plasma screens, OLED screens, each with or without touch screen input capability, each with or without haptic feedback capability, some of which may be capable of outputting two-dimensional visual output, or output in more than three dimensions by means of stereoscopic output, such as virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown)), and printers (not shown).

[0185] The computer system (1500) may also include human accessible storage devices and associated media for such storage devices, such as optical media including CD / DVD ROM / RW (1520) with CD / DVD or similar media (1521), thumb drives (1522), removable hard drives or solid state drives (1523), legacy magnetic media such as tapes or floppy disks (not shown), dedicated ROM / ASIC / PLD based devices such as security dongles (not shown), etc.

[0186] Those skilled in the art should also understand that the term "computer-readable medium" as used in connection with the subject matter of this disclosure does not encompass transmission media, carrier waves, or other transitory signals.

[0187] The computer system (1500) may also include an interface (1554) to one or more communication networks (1555). The networks may be, for example, wireless, wired, optical. The networks may further be local, wide area, metropolitan, vehicular and industrial, real-time, delay tolerant, and the like. Examples of networks include local area networks such as Ethernet, wireless LAN, cellular networks including GSM, 3G, 4G, 5G, LTE, and the like, television wired or wireless wide area digital networks including cable television, satellite television, and terrestrial broadcast television, and vehicular and industrial including CANBus, and the like. Certain networks generally require an external network interface adapter attached to a particular general-purpose data port (e.g., a USB port of the computer system (1500)) or peripheral bus (1549), while other networks are generally integrated into the core of the computer system (1500) by connection to a system bus described below (e.g., an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (1500) can communicate with other entities. Such communications can be unidirectional, receive only (e.g., television broadcast), unidirectional transmit only (e.g., CANbus to a particular CANbus device), or bidirectional to other computer systems using, for example, local-area or wide-area digital networks. Specific protocols and protocol stacks can be used with each of these networks and network interfaces, as described above.

[0188] The aforementioned human interface devices, human accessible storage devices, and network interfaces may be connected to the core (1540) of the computer system (1500).

[0189] The cores (1540) may include one or more central processing units (CPUs) (1541), graphics processing units (GPUs) (1542), dedicated programmable processing units in the form of field programmable gate areas (FPGAs) (1543), hardware accelerators for specific tasks (1544), graphics adapters (1550), etc. These devices may be connected via a system bus (1548) along with read only memory (ROM) (1545), random access memory (1546), internal mass storage (1547) such as internal hard drives, SSDs, etc. that are not user accessible. In some computer systems, the system bus (1548) may be accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripheral devices may be connected directly to the core's system bus (1548) or via a peripheral bus (1549). In one example, a screen (1510) may be connected to the graphics adapter (1550). Architectures for peripheral buses include PCI, USB, etc.

[0190] The CPU (1541), GPU (1542), FPGA (1543), and accelerator (1544) can execute certain instructions that may combine to constitute the aforementioned computer code. The computer code may be stored in a ROM (1545) or a RAM (1546). Temporary data may also be stored in the RAM (1546), and persistent data may be stored, for example, in an internal mass storage (1547). Fast storage and retrieval from any of the memory devices may be enabled by the use of cache memory, which may be closely associated with one or more of the CPU (1541), GPU (1542), mass storage (1547), ROM (1545), RAM (1546), etc.

[0191] The computer-readable medium can bear computer code for performing various computer-implemented operations. The medium and computer code may be those specially designed and constructed for the purposes of the present disclosure, or they may be of the available kind well known to those skilled in the computer software arts.

[0192] By way of example and not limitation, a computer system (1500) having an architecture, and in particular a core (1540), may provide functionality as a result of a processor (or processors) (including CPUs, GPUs, FPGAs, accelerators, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage as described above, as well as media associated with specific storage of the core (1540) that is non-transitory in nature, such as the core internal mass storage (1547) or ROM (1545). Software implementing various embodiments of the present disclosure may be stored in such devices and executed by the core (1540). The computer-readable media may include one or more memory devices or chips according to specific needs. The software may cause the core (1540), and in particular the processors (including CPUs, GPUs, FPGAs, etc.) therein, to perform certain processes or certain portions of certain processes described herein, including defining data structures stored in RAM (1546) and modifying such data structures according to processes defined by the software. Additionally or alternatively, the computer system may provide functionality as a result of logic hardwired or otherwise embodied in circuitry (e.g., accelerator (1544)) that may operate in place of or in conjunction with software to perform certain processes or certain portions of certain processes described herein. References to software may encompass logic, and vice versa, where appropriate. References to computer-readable media may encompass circuitry (such as integrated circuits (ICs)) that store software for execution, circuitry that embodies logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware and software.

[0193] Appendix A: Acronyms JEM: Joint Exploration Model VVC: Versatile Video Coding BMS: Benchmark Set MV: Motion Vector HEVC: High Efficiency Video Coding SEI: Supplemental Extended Information VUI: Video Usability Information GOP: Group of Pictures TU: conversion unit PU: Prediction unit CTU: Coding Tree Unit CTB: coding tree block PB: Predicted block HRD: Hypothetical Reference Decoder SNR: Signal to Noise Ratio CPU: Central Processing Unit GPU: Graphics Processing Unit CRT: cathode ray tube LCD: Liquid crystal display OLED: Organic Light Emitting Diode CD:Compact Disc DVD: Digital Video Disc ROM: Read-Only Memory RAM: Random Access Memory ASIC: Application Specific Integrated Circuit PLD: Programmable Logic Device LAN: Local Area Network GSM: Global System for Mobile Communications LTE: Long Term Evolution CANBus: Controller Area Network Bus USB: Universal Serial Bus PCI: Peripheral Component Interconnect FPGA: Field Programmable Gate Area SSD: Solid State Drive IC: Integrated Circuit CU: coding unit CCLM: Cross-component linear model

[0194] While this disclosure describes several exemplary embodiments, there are alterations, permutations, and various substitute equivalents that fall within the scope of this disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems and methods that, although not explicitly shown or described herein, embody the principles of the present disclosure and are therefore within the spirit and scope of the present disclosure. [Explanation of symbols]

[0195] 101 point, 101 sample, 102 arrow, 103 arrow, 104 square block, 201 current block, 300 communication system, 310 terminal device, 320 terminal device, 330 terminal device, 340 terminal device, 350 communication network, 400 communication system, 401 video source, 402 stream, 403 video encoder, 404 video data, 405 streaming server, 406 client subsystem, 407 copy, 410 video decoder, 411 output stream, 412 display, 413 capture subsystem, 420 electronic device, 430 electronic device, 501 channel, 510 video decoder, 512 rendering device, 515 buffer memory, 520 parser, 521 symbol, 530 electronic device, 531 receiver, 551 scaler / inverse transform unit, 552 intra prediction unit, 553 Motion compensation prediction unit, 555 aggregator, 556 loop filter unit, 557 reference picture memory, 558 current picture buffer, 601 video source, 603 video encoder, 620 electronic device, 630 source coder, 632 coding engine, 633 local decoder, 633 local video decoder, 634 reference picture memory, 635 predictor, 640 transmitter, 643 video sequence, 645 entropy coder, 650 controller, 660 communication channel, 703 video encoder, 721 general controller, 722 intra encoder, 723 residual calculator, 724 residual encoder, 725 entropy encoder, 726 switch, 728 residual decoder, 730 inter encoder, 810 video decoder, 871 entropy decoder, 872 intra decoder, 873 residual decoder, 874 reconstruction module, 880 Inter decoder, 900 blocks, 1000 chroma blocks, 1001 luma blocks, 1001 colocated luma blocks, 1010 chroma samples, 1011 chroma samples, 1012 chroma samples, 1020 luma samples, 1021 luma samples, 1022 luma samples, 1030 luma samples, 1040 luma samples, 1100chroma block, 1101 luma block, 1101 colocated luma block, 1110 chroma sample, 1120 luma sample, 1200 chroma block, 1201 chroma sample, 1202 bottom right corner chroma sample, 1203 bottom left reference chroma sample, 1204 top right reference chroma sample, 1205 top left reference chroma sample, 1210 reference chroma sample, 1211 reference chroma sample, 1212 reference chroma sample, 1213 row, 1214 right most column, 1221 reference chroma sample, 1222 reference chroma sample, 1223 chroma sample, 1224 chroma sample, 1300 process, 1400 process, 1500 computer system, 1501 keyboard, 1502 mouse, 1503 trackpad, 1505 joystick, 1506 microphone, 1507 scanner, 1508 camera, 1509 audio output device speaker, 1510 screen, 1510 touch screen, 1521 media, 1522 thumb drive, 1523 solid state drive, 1540 core, 1541 central processing unit (CPU), 1542 graphics processing unit (GPU), 1543 field programmable gate area (FPGA), 1544 hardware accelerator, 1546 random access memory, 1547 mass storage, 1548 system bus, 1549 peripheral bus, 1550 graphics adapter, 1554 interface, 1555 communication network

Claims

1. 1. A method of video decoding performed by a decoder, comprising: decoding prediction information for a chroma block to be reconstructed in a current picture, the prediction information indicating that the chroma block is predicted based at least on a luma block in the current picture using a planar mode, the luma block being reconstructed and co-located with the chroma block; identifying one or more first chroma samples in the chroma block as being located at one or more respective key positions; predicting the one or more first chroma samples in the chroma block based on one or more luma samples in the luma block; and predicting a second chroma sample in the chroma block based on the predicted one or more first chroma samples and at least one reference chroma sample for the chroma block, wherein the second chroma sample is different from the one or more first chroma samples, and the at least one reference chroma sample is adjacent to the chroma block.

2. the one or more first chroma samples are chroma samples in a lower right corner of the chroma block; The step of predicting the second chroma sample comprises: predicting at least one of (i) a chroma sample in a row below the chroma block or (ii) a chroma sample in a rightmost column of the chroma block based on the chroma sample in the bottom right corner and the at least one reference chroma sample; in response to predicting the chroma sample in the bottom row of the chroma block, generating a vertical predictor for the second chroma sample based on an above reference chroma sample in the at least one reference chroma sample and a predicted below chroma sample in the predicted chroma sample in the bottom row, wherein the second chroma sample, the above reference chroma sample, and the predicted below chroma sample are in a same column in the chroma block; 2. The method of claim 1, further comprising: in response to predicting the chroma sample in the rightmost column of the chroma block, generating a horizontal predictor for the second chroma sample based on a left reference chroma sample in the at least one reference chroma sample and a predicted right chroma sample in the predicted chroma sample in the rightmost column, wherein the second chroma sample, the left reference chroma sample, and the predicted right chroma sample are in a same row of the chroma block.

3. predicting at least one of (i) the chroma samples in the row below the chroma block, or (ii) the chroma samples in the rightmost column of the chroma block comprises predicting the chroma samples in the row below the chroma block and the chroma samples in the rightmost column of the chroma block; predicting the second chroma sample includes predicting the second chroma sample based on the vertical predictor and the horizontal predictor. The method of claim 2.

4. the width of the chroma block is greater than the height of the chroma block or the height of the chroma block is less than a threshold; predicting at least one of (i) the chroma samples in the row below the chroma block, or (ii) the chroma samples in the rightmost column of the chroma block comprises predicting the chroma samples in the row below the chroma block; predicting the second chroma sample includes predicting the second chroma sample based on the vertical predictor. The method of claim 2.

5. the width of the chroma block is less than the height of the chroma block or a threshold; predicting at least one of (i) the chroma samples in the bottom row of the chroma block or (ii) the chroma samples in the rightmost column of the chroma block comprises predicting the chroma samples in the rightmost column of the chroma block; predicting the second chroma sample includes predicting the second chroma sample based on the horizontal predictor. The method of claim 2.

6. The step of predicting the one or more first chroma samples comprises:

3. The method of claim 2, comprising predicting a chroma sample at a bottom right corner of the chroma block based on at least one of a luma sample at a bottom right corner of the luma block or a luma sample in the luma block that is co-located with a chroma sample at the bottom right corner.

7. the one or more first chroma samples include (i) a chroma sample in a row below the chroma block and (ii) a chroma sample in a rightmost column of the chroma block; The step of predicting the second chroma sample comprises: generating a vertical predictor for the second chroma sample based on an above reference chroma sample in the at least one reference chroma sample and a predicted below chroma sample in the predicted chroma sample in the row below, wherein the second chroma sample, the above reference chroma sample, and the predicted below chroma sample are in the same column in the chroma block; generating a horizontal predictor for the second chroma sample based on a left reference chroma sample in the at least one reference chroma sample and a predicted right chroma sample in the predicted chroma sample in a rightmost column, wherein the second chroma sample, the left reference chroma sample, and the predicted right chroma sample are in the same row in the chroma block; predicting the second chroma sample based on the vertical predictor and the horizontal predictor. The method of claim 1.

8. the width of the chroma block is greater than the height of the chroma block or the height of the chroma block is less than a threshold; the one or more first chroma samples include chroma samples in a row below the chroma block; The step of predicting the second chroma sample comprises: generating a vertical predictor for the second chroma sample based on an above reference chroma sample in the at least one reference chroma sample and a predicted below chroma sample in the predicted chroma sample in the row below, wherein the second chroma sample, the above reference chroma sample, and the predicted below chroma sample are in the same column in the chroma block; predicting the second chroma sample based on the vertical predictor. The method of claim 1.

9. the width of the chroma block is less than the height of the chroma block or a threshold; the one or more first chroma samples include chroma samples in a rightmost column of the chroma block; The step of predicting the second chroma sample comprises: generating a horizontal predictor for the second chroma sample based on a left reference chroma sample in the at least one reference chroma sample and a predicted right chroma sample in the predicted chroma sample in a rightmost column, wherein the second chroma sample, the left reference chroma sample, and the predicted right chroma sample are in the same row in the chroma block; predicting the second chroma sample based on the horizontal predictor. The method of claim 1.

10. The step of predicting the one or more first chroma samples comprises:

10. The method of claim 1, comprising predicting the one or more first chroma samples in the chroma block based on the one or more luma samples in the luma block co-located with the one or more first chroma samples using a cross-component linear model prediction (CCLM) mode.

11. the at least one reference chroma sample includes at least one of an upper reference chroma sample in an upper neighborhood of the chroma block or a left reference chroma sample in a left neighborhood of the chroma block; The method of claim 1.

12. An apparatus configured to carry out the method of any one of claims 1 to 11.

13. A computer program product for causing one or more processors to carry out the method of any one of claims 1 to 11.

Images