Method, apparatus, and program for component-dependent intra and inter prediction signal transmission

Abstract

Encoding information of a first color component and a second color component associated with a current block is determined. A first prediction mode is determined to be selected from an intra prediction mode, an inter prediction mode, or a combined intra-inter prediction mode for the first color component associated with the current block. A second prediction mode is determined to be selected from an intra prediction mode, an inter prediction mode, or a combined intra-inter prediction mode for the second color component associated with the current block. The second prediction mode is different from the first prediction mode. The current block is reconstructed based on predicting the first color component of the current block in the first prediction mode and predicting the second color component of the current block in the second prediction mode.

Description

Technical Field

[0001] Incorporation by Reference This application claims the benefit of priority of U.S. Patent Application No. 17 / 982,975, filed Nov. 8, 2022, entitled “Component-Dependent Intra and Inter Prediction Signaling,” which in turn claims the benefit of priority of U.S. Provisional Patent Application No. 63 / 353,716, filed Jun. 20, 2022, entitled “Component-Dependent Intra and Inter Prediction Signaling.” The disclosures of these prior applications are hereby incorporated by reference in their entireties.

[0002] Technical Field This disclosure describes embodiments generally related to video coding.

Background Art

[0003] The background description provided herein is for the purpose of generally presenting the context of the disclosure. Aspects of the work of the inventors named in the present application that are within the scope of the work described in this background section, and aspects of this description that may not be prior art as of the filing date in other respects, are not admitted as prior art to the present disclosure, either expressly or implicitly.

[0004] Uncompressed digital images and / or video can include a series of pictures, each picture having spatial dimensions, for example, of 1920×1080 luminance samples and associated chrominance samples. A series of pictures can have a fixed or variable picture rate (informally also known as the frame rate), for example, 60 pictures per second or 60 Hz. Uncompressed images and / or video have specific bitrate requirements. For example, 8-bit-per-sample 1080p60 4:2:0 video (1920×1080 luminance sample resolution at a 60 Hz frame rate) requires a bandwidth close to 1.5 Gbit / s. One hour of such video requires storage space exceeding 600 GB.

[0005] One purpose of image and / or video encoding and decoding can be the reduction of redundancy in the input image and / or video signal by compression. Compression can help reduce the aforementioned bandwidth and / or memory space requirements, sometimes by more than two orders of magnitude. The description in this document uses video encoding / decoding as an illustrative example, but the same techniques can be applied to image encoding / decoding in a similar manner without departing from the spirit of this disclosure. Both reversible compression and irreversible compression, as well as combinations thereof, can be used. Reversible compression refers to a technique in which an exact copy of the original signal can be reconstructed from the compressed original signal. When using irreversible compression, the reconstructed signal may not be identical to the original signal, but the distortion between the original signal and the reconstructed signal is small enough to make the reconstructed signal useful for its intended purpose. In the case of video, irreversible compression is widely used. The amount of acceptable distortion depends on the application. For example, users of certain consumer streaming applications may tolerate higher distortion than users of television distribution applications. The achievable compression ratio can reflect the fact that higher acceptable / tolerable distortion can result in a higher compression ratio.

[0006] Video encoders and decoders can utilize techniques from several broad categories, including, for example, motion compensation, transform processing, quantization, and entropy encoding.

[0007] Video codec technology can include a technique known as intra coding. In intra coding, sample values are represented without referring to samples or other data from previously reconstructed reference pictures. In some video codecs, a picture is spatially divided into blocks of samples. If all blocks of samples are coded in the intra mode, that picture can be an intra picture. Intra pictures and their derivatives, such as independent decoder refresh pictures, can be used to reset the decoder state and thus can be used as the first picture in an encoded video bitstream and video session or as a still image. Samples of an intra block can be subjected to a transform, and the transform coefficients can be quantized prior to entropy coding. Intra prediction can be a technique that minimizes sample values in the pre-transform region. In some cases, the smaller the post-transform DC value and the smaller the AC coefficients, the fewer bits are required at a given quantization step size to represent the block after entropy coding.

[0008] For example, traditional intra coding used in MPEG-2 generation coding technology does not use intra prediction. However, some newer video compression technologies include techniques that attempt to perform prediction based on, for example, surrounding sample data and / or metadata obtained during the encoding and / or decoding of blocks of data. Such techniques are hereinafter referred to as "intra prediction" techniques. Note that in at least some cases, intra prediction uses only reference data from the current picture being reconstructed and does not use reference data from reference pictures.

[0009] There can be various forms of intra prediction. In a given video coding technology, if two or more such techniques can be used, the specific technique used can be encoded as a specific intra prediction mode that uses that specific technique. In certain cases, an intra prediction mode can have sub - modes and / or parameters, and the sub - modes and / or parameters can be encoded individually or can be included in a mode - codeword that defines the prediction mode used. Which codeword to use for a given combination of mode, sub - mode and / or parameter can affect the coding efficiency gain through intra prediction and can similarly affect the entropy coding technology used to convert the codeword into a bitstream.

[0010] A certain mode of intra prediction was introduced in H.264, refined in H.265, and further refined in newer coding technologies such as the Joint Exploration Model (JEM), Versatile Video Coding (VVC), and Benchmark Set (BMS). A predictor block can be formed using sample values of neighboring samples that are already available. The sample values of the neighboring samples are copied into the predictor block according to a certain direction. The reference to the direction used can be encoded in the bitstream or can itself be predicted.

[0011] Referring to FIG. 1A, in the lower right, a subset of 9 known predictor directions out of 33 possible predictor directions (corresponding to 33 angular modes out of 35 intra - modes) defined in H.265 is depicted. The point (101) where the arrows converge represents the sample to be predicted. The arrows represent the direction in which the sample is predicted. For example, arrow (102) indicates that sample (101) is predicted from the sample(s) in the upper right at an angle of 45 degrees from the horizontal. Similarly, arrow (103) indicates that sample (101) is predicted from the sample(s) in the lower left of sample (101) at an angle of 22.5 degrees from the horizontal.

[0012] Continuing to refer to FIG. 1A, in the upper left, a square block (104) of 4×4 samples is depicted (shown by the thick dashed line). The square block (104) contains 16 samples, and each sample is labeled with its position in the Y dimension (e.g., row index) and its position in the X dimension (e.g., column index) as “S”. For example, sample S21 is the second sample from the top in the Y dimension and the first sample from the left in the X dimension. Similarly, sample S44 is the fourth sample within block (104) in both the Y and X dimensions. Since the block is of size 4×4 samples, S44 is in the lower right. Further, reference samples following a similar numbering scheme are shown. The reference samples are labeled with R and its Y position (e.g., row index) and X position (column index) relative to block (104). In both H.264 and H.265, the predicted samples are in the vicinity of the block being reconstructed, and thus there is no need to use negative values.

[0013] Intra-picture prediction can function by copying the reference sample value from the neighboring samples indicated by the signaled prediction direction. For example, assume that the encoded video bitstream includes signaling indicating the prediction direction that aligns with arrow (102) for this block. That is, the samples are predicted from the upper right sample at an angle of 45 degrees from the horizontal. In that case, samples S41, S32, S23, and S14 are predicted from the same reference sample R05. Then, sample S44 is predicted from reference sample R08.

[0014] In certain cases, especially when the direction is not divisible by 45 degrees, the values of multiple reference samples can be combined, for example, by interpolation, to calculate the reference sample.

[0015] With the development of video coding technology, the number of possible directions has been increasing. In H.264 (2003), nine different directions could be represented. This increased to 33 in H.265 (2013). Currently, JEM / VVC / BMS can support up to 65 directions. Experiments are conducted to identify the most likely directions, and certain techniques in entropy coding are used to represent those likely directions with a few bits while accepting a penalty for some of the less likely directions. Furthermore, the direction itself may be predicted from neighboring directions already decoded in neighboring blocks.

[0016] FIG. 1B shows a schematic diagram (110) depicting 65 intra prediction directions by JEM to show the number of prediction directions increasing over time.

[0017] The mapping of intra prediction direction bits representing directions in the encoded video bitstream can vary for each video coding technology. Such mappings can range from simple direct mappings to complex adaptive schemes related to codewords, most probable modes, and similar techniques. However, in most cases, in video content, there may be certain directions that are statistically less likely to occur than other certain types of directions. Since the goal of video compression is to reduce redundancy, in well - functioning video coding technologies, such less likely methods are represented by a larger number of bits than the more likely directions.

[0018] The encoding and decoding of images and / or videos can be performed using inter-picture prediction with motion compensation. Motion compensation can be an irreversible compression technique, and a block of sample data from a previously reconstructed picture or a part thereof (reference picture) is spatially shifted in the direction indicated by a motion vector (hereinafter, MV) and then used for the prediction of a newly reconstructed picture or picture part. In some cases, the reference picture can be the same as the picture currently being reconstructed. The MV can have two dimensions in X and Y, or three dimensions, where the third dimension is an indication of the reference picture used (which can be, indirectly, the temporal dimension).

[0019] In some video compression techniques, the MV applicable to a certain region of sample data can be predicted from other MVs, for example, from an MV related to another region of sample data that is spatially adjacent to the region being reconstructed and that precedes that MV in decode order. By doing so, the amount of data required for encoding the MV can be significantly reduced, thereby removing redundancy and increasing compression. The reason why MV prediction can function directionally is, for example, that when encoding an input video signal derived from a camera (known as natural video), a larger region than the region where a single MV is applicable moves in a similar direction, and thus, in some cases, there is a statistical likelihood that a similar motion vector derived from the MVs of neighboring regions can be used for prediction. As a result, the MV found for a given region will be similar or identical to the predicted MV from the surrounding MVs, and it can be represented with fewer bits than would be used if the MV were directly encoded after entropy encoding. In some cases, MV prediction can be an example of reversible compression of a signal (i.e., an MV) derived from the original signal (i.e., the sample stream). In other cases, MV prediction itself can be irreversible, for example, due to rounding errors when calculating predictors from some surrounding MVs.

[0020] H.265 / HEVC (ITU-T Rec. H.265, "High Efficiency Video Coding", December 2016) describes various MV prediction mechanisms. Among the many MV prediction mechanisms provided by H.265, a technique hereafter referred to as "spatial merge" is described with reference to FIG. 2.

[0021] Referring to FIG. 2, the current block (201) includes samples found by the encoder during the motion search process that the current block is predictable from a previous block of the same size that has been spatially shifted. Instead of directly coding the MV, the MV can be derived from metadata associated with one or more reference pictures, for example, from the latest reference picture (in decode order), using an MV associated with any of five surrounding samples denoted as A0, A1, and B0, B1, B2 (202 to 206 respectively). In H.265, MV prediction can use predictors from the same reference picture that neighboring blocks are using. SUMMARY OF THE INVENTION PROBLEMS TO BE SOLVED BY THE INVENTION

[0022] Aspects of the present disclosure provide methods and apparatuses for video encoding / decoding. In some examples, an apparatus for video decoding includes a processing circuit. MEANS FOR SOLVING THE PROBLEMS

[0023] According to an aspect of the present disclosure, a method of video decoding executed in a video decoder is provided. In this method, an encoded video bitstream including a current block can be received. The current block can be associated with at least a first color component and a second color component. Encoded information of the first color component and the second color component associated with the current block can be obtained. For the first color component associated with the current block, a first prediction mode can be determined to be selected from an intra prediction mode, an inter prediction mode, or a combined intra-inter prediction mode. For the second color component associated with the current block, a second prediction mode can be determined to be selected from an intra prediction mode, an inter prediction mode, or a combined intra-inter prediction mode. The second prediction mode can be different from the first prediction mode. The current block can be reconstructed based on predicting the first color component of the current block in the first prediction mode and predicting the second color component of the current block in the second prediction mode.

[0024] In some embodiments, the inter prediction mode can include one of an inter prediction mode without a motion vector difference, an inter prediction mode with a motion vector difference, and an inter prediction mode with a motion vector displacement.

[0025] In one example, the first color component can be a luma component and the second color component can be a chroma component.

[0026] In one example, the first color component can be one of a red component, a green component, and a blue component, and the second color component can be another one of a red component, a green component, and a blue component.

[0027] In an embodiment, in response to the first color component being encoded by an inter prediction mode, whether the second prediction mode is an intra prediction mode can be determined based on signaling information included in the encoded information.

[0028] In one embodiment, in response to the first color component being encoded in an inter prediction mode, whether the second prediction mode is a cross-component prediction mode can be determined based on signaling information included in the encoded information.

[0029] The cross-component prediction mode can include one of a cross-component linear mode (CCLM), a chroma from luma mode (CfL), a multi-model CCLM mode, a multi-filter CCLM mode, and a convolutional cross-component model (CCCM).

[0030] The inter prediction mode without a motion vector difference can include one of a merge mode and a near mode. The inter prediction mode with a motion vector difference can include one of an advanced motion vector prediction (AMVP) mode and a new motion vector (NEWMV) mode. The inter prediction mode with a motion vector displacement can include a merge motion vector difference (MMVD) mode.

[0031] The second prediction mode for the second color component is determined based on signaling information included in the encoded information. The signaling information can include a flag that is entropy encoded using a context. The context can be derived from the encoded information based on at least one of the block size of the current block, the first prediction mode, or the value of the flag signaled for neighboring blocks of the current block.

[0032] To determine a second prediction mode based on the TM cost, a plurality of prediction sample values of the template of the current block can be generated based on a plurality of candidate prediction modes, where each of the plurality of prediction sample values can be generated based on each of the plurality of candidate prediction modes. The template of the current block can include samples adjacent to the current block. A plurality of TM costs can be generated based on the plurality of prediction sample values of the template, and each of the plurality of TM costs can be associated with the difference between each of the plurality of prediction sample values and the reconstructed sample values of the template. The second prediction mode can be determined as the candidate prediction mode among the plurality of candidate prediction modes corresponding to the minimum TM cost among the plurality of TM costs.

[0033] In some embodiments, in response to the reconstructed sample values of the template being unavailable, the second prediction mode can be determined as the first prediction mode.

[0034] According to another aspect of the present disclosure, an apparatus is provided. The apparatus includes a processing circuit. The processing circuit can be configured to execute any of the methods for video encoding / decoding.

[0035] Aspects of the present disclosure also provide a non-transitory computer-readable medium storing instructions that, when executed by a computer for video decoding, cause the computer to execute any of the methods for video encoding / decoding.

Brief Description of the Drawings

[0036] Further features, properties, and various advantages of the disclosed subject matter will become more apparent from the following detailed description and the accompanying drawings.

[0037]

Figure 1A

[0038]

Figure 1B

[0039]

Figure 2

[0040]

Figure 3

[0041]

Figure 4

[0042]

Figure 5

[0043]

Figure 6

[0044]

Figure 7

[0045]

Figure 8

[0046]

Figure 9

[0047]

Figure 10

[0048]

Figure 11

[0049]

Figure 12

[0050]

Figure 13

[0051]

Figure 14

[0052]

Figure 15

[0053]

Figure 16A

[0054]

Figure 16B

[0055]

Figure 17

[0056]

Figure 18

[0057]

Figure 19

[0058]

Figure 20

[0059]

Figure 21

[0060] FIG. 3 shows an exemplary block diagram of a communication system (300). The communication system (300) includes a plurality of terminal devices that can communicate with each other, for example, via a network (350). For example, the communication system (300) includes a first pair of terminal devices (310) and (320) interconnected via a network (350). In the example of FIG. 3, the first pair of terminal devices (310) and (320) perform unidirectional data transmission. For example, the terminal device (310) may encode video data (e.g., a stream of video pictures captured by the terminal device (310)) for transmission to the other terminal device (320) via the network (350). The encoded video data can be transmitted in the form of one or more encoded video bitstreams. The terminal device (320) may receive the encoded video data from the network (350), decode the encoded video data to restore the video pictures, and display the video pictures according to the restored video data. Unidirectional data transmission may be common in media service applications and the like.

[0061] In another example, a communication system (300) includes a second pair of terminal devices (330) and (340) that perform bidirectional transmission of, for example, video data encoded during a video conference. For bidirectional data transmission, in one example, each of the terminal devices (330) and (340) may encode video data (e.g., a stream of video pictures captured by a terminal device) for transmission to the other of the terminal devices (330) and (340) via a network (350). Each of the terminal devices (330) and (340) may receive the encoded video data transmitted by the other of the terminal devices (330) and (340), decode the encoded video data to restore the video picture, and display the video picture on an accessible display device according to the restored video data.

[0062] In the example of FIG. 3, the terminal devices (310), (320), (330), and (340) may each be shown as a server, a personal computer, and a smartphone, respectively, but the principles of the present disclosure may not be limited thereto. Embodiments of the present disclosure find applications in laptop computers, tablet computers, media players, and / or dedicated video conferencing facilities. The network (350) represents any number of networks that transmit encoded video data among the terminal devices (310), (320), (330), and (340), including, for example, wired (wired) and / or wireless [wireless] communication networks. The communication network (350) may exchange data in a circuit-switched and / or packet-switched channel. Representative networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of the present discussion, the architecture and topology of the network (350) may not be important for the operation of the present disclosure, unless otherwise described below.

[0063] FIG. 4 shows a video encoder and a video decoder in a streaming environment as an example of an application for the disclosed subject matter. The disclosed subject matter may be equally applicable to other video-enabled applications including, for example, storage of compressed video on digital media including video conferencing, digital TV, streaming services, CDs, DVDs, memory sticks, etc.

[0064] A streaming system can include a video source (401), such as a digital camera, and may include a capture subsystem (413) that generates a stream (402) of, for example, uncompressed video pictures. In one example, the stream of video pictures (402) includes samples captured by the digital camera. The stream of video pictures (402), depicted as a thick line to emphasize the high data volume when compared to the encoded video data (404) (or encoded video bitstream), can be processed by an electronic device (420) that includes a video encoder (403) coupled to the video source (401). The video encoder (403) can include hardware, software, or a combination thereof to enable or implement aspects of the disclosed subject matter, as will be described in more detail below. The encoded video data (404) (or encoded video bitstream), depicted as a thin line to emphasize the lower data volume when compared to the stream of video pictures (402), can be stored in a streaming server (405) for future use. One or more streaming client subsystems, such as client subsystems (406) and (408) of FIG. 4, can access the streaming server (405) to retrieve copies (407) and (409) of the encoded video data (404). The client subsystem (406) can include, for example, a video decoder (410) within an electronic device (430). The video decoder (410) decodes an incoming copy (407) of the encoded video data and generates an outgoing stream (411) of video pictures that can be rendered on a display (412), such as a display screen, or other rendering device (not shown). In some streaming systems, the encoded video data (404), (407), and (409) (e.g., video bitstream) can be encoded according to a particular video encoding / compression standard.Examples of these standards include ITU-T Recommendation H.265. In one example, a video coding standard under development is informally known as VVC. The disclosed subject matter may be used in the context of Versatile Video Coding (VVC).

[0065] Note that electronic devices (420) and (430) can include other components (not shown). For example, electronic device (420) can include a video decoder (not shown), and electronic device (430) can also include a video encoder (not shown).

[0066] FIG. 5 shows an exemplary block diagram of a video decoder (510). The video decoder (510) can be included in an electronic device (530). The electronic device (530) can include a receiver (531) (e.g., a receiving circuit). The video decoder (510) can be used in place of the video decoder (310) in the example of FIG. 4.

[0067] The receiver (531) may receive one or more encoded video sequences to be decoded by the video decoder (510). In certain embodiments, one encoded video sequence is received at a time, and the decoding of each encoded video sequence is independent of the decoding of other encoded video sequences. The encoded video sequence may be received from a channel (501), which may be a hardware / software link to a storage device storing the encoded video data. The receiver (531) may receive the encoded video data together with other data, such as encoded audio data and / or auxiliary data streams, and these data may be transferred to their respective using entities (not shown). The receiver (531) can separate the encoded video sequence from other data. As a network jitter countermeasure, a buffer memory (515) may be coupled between the receiver (531) and the entropy decoder / parser (520) (hereinafter “parser”). In certain applications, the buffer memory (515) is part of the video decoder (510). In other applications, it can be external to the video decoder (510) (not shown). In still other applications, for example, to counter network jitter, there may be a buffer memory (not shown) external to the video decoder (510), and further, for example, to handle playback timing, there may be another buffer memory (515) internal to the video decoder (510). If the receiver (531) is receiving data from a storage / transfer device with sufficient bandwidth and controllability, or from an isochronous network, the buffer memory (515) may not be required or may be small. For use in a best-effort packet network such as the Internet, a buffer memory (515) may be required, may be relatively large, may advantageously be of an adaptable size, and may be implemented, at least in part, in an operating system or similar element (not shown) external to the video decoder (510).

[0068] The video decoder (510) may include a parser (520) for reconstructing symbols (521) from the encoded video sequence. The categories of these symbols include information used to manage the operation of the video decoder (510) and potentially information for controlling a rendering device such as a renderer device (512) (e.g., a display screen). The renderer device can be coupled to the electronic device (530) rather than being an integral part of the electronic device (530) as shown in FIG. 5. The control information for the renderer device(s) may be in the form of a Supplementary Enhancement Information (SEI) message or a Video Usability Information (VUI) parameter set fragment (not shown). The parser (520) can parse / entropy decode the received encoded video sequence. The encoding of the encoded video sequence can follow video encoding techniques or standards and can follow various principles including variable length encoding, Huffman encoding, arithmetic encoding with or without context sensitivity, etc. The parser (520) can extract a set of subgroup parameters for at least one of the subgroups of pixels within the video decoder from the encoded video sequence based on at least one parameter corresponding to the group. The subgroups can include a Group of Pictures (GOP), a picture, a tile, a slice, a macroblock, a Coding Unit (CU), a block, a Transform Unit (TU), a Prediction Unit (PU), etc. The parser (520) can also extract information such as transform coefficients, quantizer parameter values, motion vectors, etc. from the encoded video sequence.

[0069] The parser (520) can perform an entropy decoding / parsing operation on the video sequence received from the buffer memory (515), thereby generating symbols (521).

[0070] The reconstruction of the symbols (521) can involve multiple different units depending on the type of the encoded video picture or its parts (e.g., inter and intra pictures, inter and intra blocks) and other factors. How each unit is involved can be controlled by subgroup control information parsed by the parser (520) from the encoded video sequence. Such a flow of subgroup control information between the parser (520) and the multiple units below is not depicted for clarity.

[0071] In addition to the functional blocks already described, the video decoder (510) can conceptually be divided into several functional units as described below. In a practical implementation operating under commercial constraints, many of these units can interact closely with each other and can be at least partially integrated with each other. However, for the purpose of describing the disclosed subject matter, a conceptual subdivision into the following functional units is appropriate.

[0072] The first unit is the scaler / inverse transform unit (551). The scaler / inverse transform unit (551) receives from the parser (520) the quantized transform coefficients and control information as symbols (singular or plural) (521). The control information includes which transform to use, block size, quantization factor, quantization scaling matrix, etc. The scaler / inverse transform unit (551) can output a block containing sample values that can be input to the aggregator (555).

[0073] In some cases, the output samples of the scaler / inverse transform unit (551) can relate to intra-coded blocks. The intra-coded blocks do not use prediction information from a previously reconstructed picture, but are blocks that can use prediction information from previously reconstructed parts of the current picture. Such prediction information can be provided by the intra-picture prediction unit (552). In some cases, the intra-picture prediction unit (552) uses the surrounding already reconstructed information taken from the current picture buffer (558) to generate a block of the same size and shape as the block being reconstructed. The current picture buffer (558) buffers, for example, a partially reconstructed current picture and / or a fully reconstructed current picture. The aggregator (555) adds, in some cases, for each sample, the prediction information generated by the intra prediction unit (552) to the output sample information provided by the scaler / inverse transform unit (551).

[0074] In other cases, the output samples of the scaler / inverse transform unit (551) can relate to inter-coded and potentially motion-compensated blocks. In such cases, the motion compensation prediction unit (553) can access the reference picture memory (557) to fetch the samples used for prediction. After motion-compensating the fetched samples according to the symbols (521) regarding the block, these samples can be added by the aggregator (555) to the output of the scaler / inverse transform unit (in this case, called the residual samples or residual signal), thereby generating output sample information. The address in the reference picture memory (557) from which the motion compensation unit (553) fetches the prediction samples can be controlled by the motion vectors available to the motion compensation unit (553) in the form of symbols (521). The symbols can have, for example, X, Y, and reference picture components. Motion compensation can include interpolation of the sample values fetched from the reference picture memory (557) when exact motion vectors below the sample are used, a motion vector prediction mechanism, and the like.

[0075] The output samples of the aggregator (555) can be subjected to various loop filtering techniques within the loop filter unit (556). Video compression techniques can include in-loop filter techniques. The in-loop filter techniques are controlled by the parameters included in the encoded video sequence (also called the encoded video bitstream) and are made available to the loop filter unit (556) as symbols (521) from the parser (520). Video compression can also respond to the meta information obtained during the decoding of the previous part (in decoding order) of the encoded picture or encoded video sequence and to the previously reconstructed and loop-filtered sample values.

[0076] The output of the loop filter unit (556) can be a sample stream, which can be output to the rendering device (512) and can also be stored in the reference picture memory (557) for use in future inter-picture prediction.

[0077] Once the encoded image is completely reconstructed, it can be used as a reference picture for future prediction. For example, when the encoded picture corresponding to the current picture is completely reconstructed and the encoded picture is identified as a reference picture (e.g., by the parser (520)), the current picture buffer (558) can become part of the reference picture memory (557), and a fresh current picture buffer can be reallocated before starting the reconstruction of subsequent encoded pictures.

[0078] The video decoder (510) can perform a decoding operation according to a predetermined video compression technique or a standard such as ITU-T Recommendation H.265. The encoded video sequence can conform to the syntax of the video compression technique or standard and the profile documented in the video compression technique or standard, meaning that the encoded video sequence can comply with the syntax defined by the video compression technique or standard being used. Specifically, the profile can select certain tools from all the tools available in the video compression technique or standard as the tools that are only available for use under that profile. For compliance, it may also be necessary that the complexity of the encoded video sequence be within the range defined by the level of the video compression technique or standard. In some cases, the level restricts the maximum picture size, maximum frame rate, maximum reconstruction sample rate (e.g., measured in megasamples per second), maximum reference picture size, etc. The limits set by the level can, in some cases, be further restricted through the hypothetical reference decoder (HRD) specifications and metadata signaled in the encoded video sequence for HRD buffer management.

[0079] In one embodiment, the receiver (531) may receive additional (redundant) data along with the encoded video. The additional data may be included as part of the encoded video sequence(s). The additional data may be used by the video decoder (510) to properly decode the data and / or to more accurately reconstruct the original video data. The additional data can be in the form of, for example, temporal, spatial, or signal-to-noise ratio (SNR) improvement layers, redundant slices, redundant pictures, forward error correction codes, etc.

[0080] FIG. 6 shows an exemplary block diagram of a video encoder (603). The video encoder (603) is included in an electronic device (620). The electronic device (620) includes a transmitter (640) (e.g., a transmission circuit). The video encoder (603) can be used in place of the video encoder (403) in the example of FIG. 4.

[0081] The video encoder (603) can receive video samples from a video source (601) (which is not part of the electronic device (620) in the example of FIG. 6) that can capture the video image to be encoded by the video encoder (603). In another example, the video source (601) is part of the electronic device (620).

[0082] The video source (601) can provide a source video sequence to be encoded by the video encoder (603) in the form of a digital video sample stream that can be of any suitable bit depth (e.g., 8-bit, 10-bit, 12-bit,...), any color space (e.g., BT.601 YCrCB, RGB,...), and any suitable sampling structure (e.g., YCrCb 4:2:0, YCrCb 4:4:4). In a media service system, the video source (601) may be a storage device storing pre-prepared video. In a video conferencing system, the video source (601) may be a camera that captures local image information as a video sequence. The video data may be provided as a plurality of individual pictures that impart motion when viewed in sequence. Each picture itself may be organized as a spatial array of pixels, and each pixel can include one or more samples depending on the sampling structure, color space, etc. in use. One of ordinary skill in the art can readily understand the relationship between pixels and samples. The following description focuses on samples.

[0083] According to one embodiment, the video encoder (603) can encode and compress pictures of the source video sequence in real time or under any other required time constraints to obtain an encoded video sequence (643). Implementing an appropriate encoding speed is one function of the controller (650). In some embodiments, the controller (650) controls other functional units as described below and is functionally coupled to such other functional units. Such couplings are not drawn for clarity. The parameters set by the controller (650) can include parameters related to rate control (picture skip, quantizer, lambda value of rate-distortion optimization techniques, …), picture size, picture group (GOP) layout, maximum motion vector search range, and the like. The controller (650) can be configured to have other suitable functions related to the video encoder (603) optimized for a certain system design.

[0084] In some embodiments, the video encoder (603) is configured to operate in an encoding loop. As a radically simplified explanation, in one example, the encoding loop can include a source encoder (630) (e.g., responsible for generating symbols such as a symbol stream based on an input picture and reference picture(s) to be encoded) and a (local) decoder (633) embedded in the video encoder (603). The decoder (633) reconstructs the symbols to generate sample data in the same manner as a (remote) decoder would also generate. The reconstructed sample stream (sample data) is input into the reference picture memory (634). Since the decoding of the symbol stream yields bit-exact results regardless of the decoder position (local or remote), the content of the reference picture memory (634) is also bit-exact between the local encoder and the remote encoder. In other words, the prediction part of the encoder "sees" the same sample values as reference picture samples that the decoder "sees" when using prediction during decoding. This basic principle of reference picture synchronization (and the resulting drift if synchronization cannot be maintained, e.g., due to channel errors) is also used in some related technologies.

[0085] The operation of the "local" decoder (633) may be the same as that of the "remote" decoder, e.g., the video decoder (410), already described in detail above in connection with FIG. 5. However, also referring momentarily to FIG. 5, since symbols are available and the encoding / decoding of the symbols into an encoded video sequence by the entropy encoder (645) and the parser (420) can be reversible, the entropy decoding part of the video decoder (410) including the buffer memory (415) and the parser (420) may not be fully implemented in the local decoder (633).

[0086] In some embodiments, all decoder technologies except the parse / entropy decoding that exists within the decoder exist in substantially the same functional form within the corresponding encoder. Thus, the disclosed subject matter focuses on decoder operations. The description of encoder technology can be abbreviated since it is the reverse of the decoder technology that is comprehensively described. In certain areas, more detailed descriptions are provided below.

[0087] During operation, in some examples, the source coder (630) can perform motion-compensated predictive coding that predictively codes an input picture by referring to one or more previously coded pictures from the video sequence designated as "reference pictures". In this way, the coding engine (632) codes the difference between a pixel block of the input picture and a pixel block of the reference picture(s) that can be selected as a prediction reference for the input picture.

[0088] The local video decoder (633) can decode the coded video data of a picture that can be designated as a reference picture based on the symbols generated by the source coder (630). The operation of the coding engine (632) can advantageously be a lossy process. When the coded video data can be decoded by a video decoder (not shown in FIG. 6), the reconstructed video sequence can typically be a copy of the source video sequence with some errors. The local video decoder (633) can replicate the decoding process that can be performed on the reference picture by the video decoder and cause the reconstructed reference picture to be stored in the reference picture memory (634). In this way, the video encoder (603) can locally store a copy of the reconstructed reference picture that has (in the absence of transmission errors) the common content as the reconstructed reference picture that would be obtained by a remote video decoder.

[0089] Predictor (635) can perform prediction search for the encoding engine (632). That is, for a new picture to be encoded, predictor (635) can search the reference picture memory (634) to find sample data (as candidate reference pixel blocks) or certain metadata, such as reference picture motion vectors, block shapes, etc., that can serve as appropriate prediction references for the new picture. Predictor (635) can operate on a sample block-by-pixel block basis to find appropriate prediction references. In some cases, depending on what is determined by the search results obtained by predictor (635), the input picture can have prediction references drawn from a plurality of reference pictures stored in the reference picture memory (634).

[0090] Controller (650) may manage the encoding operation of the source encoder (630), including setting parameters and subgroup parameters used for encoding video data, for example.

[0091] The outputs of all the above functional units can undergo entropy encoding in the entropy encoder (645). The entropy encoder (645) converts the symbols generated by the various functional units into an encoded video sequence by applying lossless compression to the symbols according to techniques such as Huffman coding, variable length coding, arithmetic coding, etc.

[0092] The transmitter (640) can buffer the encoded video sequence generated by the entropy encoder (645) and prepare it for transmission via the communication channel (660). The communication channel (660) may be a hardware / software link to a storage device that stores the encoded video data. The transmitter (640) can merge the encoded video data from the video encoder (603) with other data to be transmitted, such as encoded audio data and / or an auxiliary data stream (source not shown).

[0093] The controller (650) may manage the operation of the video encoder (603). During encoding, the controller (650) can assign a certain encoded picture type to each encoded picture. The encoded picture type can affect the encoding technique applicable to each picture. For example, a picture may often be assigned as one of the following picture types.

[0094] An intra picture (I picture) can be encoded and decoded without using other pictures in the sequence as a prediction source. Some video codecs allow different types of intra pictures, including, for example, an Independent Decoder Refresh (IDR) picture. Those skilled in the art recognize these variations of I pictures, as well as their respective uses and characteristics.

[0095] A predicted picture (P picture) can be encoded and decoded using intra prediction or inter prediction that uses at most one motion vector and a reference index to predict the sample values of each block.

[0096] A bi-directional predicted picture (B picture) can be encoded and decoded using intra prediction or inter prediction that uses up to two motion vectors and reference indices to predict the sample values of each block. Similarly, a multi-predicted picture can use three or more reference pictures and associated metadata for the reconstruction of a single block.

[0097] The source picture is typically divided spatially into a plurality of sample blocks (e.g., blocks of 4×4, 8×8, 4×8, or 16×16 samples each) and can be encoded block by block. The blocks can be predicted encoded by referring to other (already encoded) blocks, depending on the encoding assignment applied to each picture of the block. For example, blocks of an I picture may be encoded non-predictively or may be predicted encoded by referring to already encoded blocks of the same picture (spatial prediction or intra prediction). Pixel blocks of a P picture may be predicted encoded via spatial prediction or via temporal prediction by referring to one previously encoded reference picture. Blocks of a B picture may be predicted encoded via spatial prediction or via temporal prediction by referring to one or two previously encoded reference pictures.

[0098] The video encoder (603) can perform an encoding operation according to a predetermined video encoding technology or standard such as ITU-T Recommendation H.265. In that operation, the video encoder (603) can perform various compression operations, including a predictive encoding operation that exploits the temporal and spatial redundancy in the input video sequence. Thus, the encoded video data can conform to the syntax specified by the video encoding technology or standard used.

[0099] In one embodiment, the transmitter (640) may transmit additional data along with the encoded video. The source encoder (630) may include such data as part of the encoded video sequence. The additional data may include temporal / spatial / SNR enhancement layers, other forms of redundant data such as redundant pictures and slices, SEI messages, VUI parameter set fragments, and the like.

[0100] Video may be captured as a plurality of source pictures (video pictures) in a temporal sequence. Intra prediction (often abbreviated as intra prediction) utilizes spatial correlation within a given picture, and inter prediction utilizes (temporal or other) correlation between pictures. In one example, a particular picture to be encoded / decoded, called the current picture, is divided into blocks. If a block within the current picture is similar to a reference block within a reference picture that has been previously encoded and is still in the buffer in the video, that block within the current picture can be encoded by a vector called a motion vector. The motion vector points to the reference block within the reference picture and can have a third dimension that identifies the reference picture if multiple reference pictures are used.

[0101] In some embodiments, dual prediction techniques can be used in inter prediction. According to the dual prediction technique, two reference pictures such as a first reference picture and a second reference picture that both precede the current picture in decode order (but may be past and future respectively in display order) in the video are used. A block within the current picture can be encoded by a first motion vector that points to a first reference block within the first reference picture and a second motion vector that points to a second reference block within the second reference picture. The block can be predicted by a combination of the first reference block and the second reference block.

[0102] Furthermore, in order to improve the coding efficiency, a merge mode technique can be used in inter-picture prediction.

[0103] According to some embodiments of the present disclosure, predictions such as inter-picture prediction and intra-picture prediction are performed in units of blocks. For example, according to the HEVC standard, pictures in a sequence of video pictures are divided into coding tree units (CTUs) for compression, and those CTUs in a picture have the same size such as 64×64 pixels, 32×32 pixels, or 16×16 pixels. Generally, a CTU includes three coding tree blocks (CTBs) which are one luma CTB and two chroma CTBs. Each CTU can be recursively quad-tree divided into one or more coding units (CUs). For example, a 64×64 pixel CTU can be divided into one 64×64 pixel CU, or four 32×32 pixel CUs, or sixteen 16×16 pixel CUs. In one example, each CU is analyzed to determine a prediction type for that CU, such as an inter prediction type or an intra prediction type. The CU is divided into one or more prediction units (PUs) depending on the temporal and / or spatial predictability. Generally, each PU includes a luma prediction block (PB) and two chroma PBs. In some embodiments, the prediction operation in encoding (encoding / decoding) is performed in units of prediction blocks. Using a luma prediction block as an example of a prediction block, the prediction block includes a matrix of values (e.g., luma values) for pixels such as 8×8 pixels, 16×16 pixels, 8×16 pixels, 16×8 pixels, etc.

[0104] FIG. 7 shows an exemplary diagram of a video encoder (703). The video encoder (703) receives a processing block (e.g., a prediction block) of sample values within a current video picture in a sequence of video pictures, and is configured to encode the processing block into an encoded picture that is part of an encoded video sequence. In one example, the video encoder (703) is used instead of the video encoder (403) in the example of FIG. 4.

[0105] In an example of HEVC, the video encoder (703) receives a matrix of sample values for a processing block such as a prediction block of 8×8 samples. The video encoder (703) determines, for example using rate - distortion optimization, which of an intra - mode, an inter - mode, or a bi - prediction mode the processing block is best encoded using. If the processing block is encoded in the intra - mode, the video encoder (703) may use an intra - prediction technique to encode the processing block into the encoded picture; if the processing block is encoded in the inter - mode or the bi - prediction mode, the video encoder (703) may use an inter - prediction technique or a bi - prediction technique, respectively, to encode the processing block into the encoded picture. In some video coding techniques, the merge mode may be an inter - picture prediction sub - mode where the motion vector is derived from one or more motion vector predictors but there is no benefit of the encoded motion vector component outside the predictors. In some other video coding techniques, there may be motion vector components applicable to the target block. In one example, the video encoder (703) includes other components such as a mode - decision module (not shown) for determining the mode of the processing block.

[0106] In the example of FIG. 7, the video encoder (703) includes an inter-encoder (730), an intra-encoder (722), a residual calculator (723), a switch (726), a residual encoder (724), a general controller (721), and an entropy encoder (725) coupled together as shown in FIG. 7.

[0107] The inter-encoder (730) receives samples of the current block (e.g., a processing block), compares the block with one or more reference blocks (e.g., blocks in previous and subsequent pictures) in a reference picture, generates inter-prediction information (e.g., a description of redundant information by inter-encoding techniques, motion vectors, merge mode information), and is configured to calculate an inter-prediction result (e.g., a predicted block) using any suitable technique based on the inter-prediction information. In some examples, the reference picture is a decoded reference picture decoded based on the encoded video information.

[0108] The intra-encoder (722) receives samples of the current block (e.g., a processing block), optionally compares the block with blocks already encoded in the same picture, generates quantized coefficients after transformation, and optionally also generates intra-prediction information (e.g., intra-prediction direction information by one or more intra-encoding techniques). In one example, the intra-encoder (722) also calculates an intra-prediction result (e.g., a predicted block) based on the intra-prediction information and reference blocks in the same picture.

[0109] The overall controller (721) is configured to determine overall control data and control other components of the video encoder (703) based on the overall control data. In one example, the overall controller (721) determines the mode of a block and provides a control signal to the switch (726) based on that mode. For example, when the mode is the intra mode, the overall controller (721) controls the switch (726) to select the intra mode result for use by the residual calculator (723), selects the intra prediction information, and controls the entropy encoder (725) to include the intra prediction information in the bitstream; when the mode is the inter mode, the overall controller (721) controls the switch (726) to select the inter prediction result for use by the residual calculator (723), selects the inter prediction information, and controls the entropy encoder (725) to include the inter prediction information in the bitstream.

[0110] The residual calculator (723) is configured to calculate the difference (residual data) between the received block and the prediction result selected from the intra-encoder (722) or the inter-encoder (730). The residual encoder (724) is configured to encode the residual data based on the residual data to generate a transformation coefficient. In one example, the residual encoder (724) is configured to transform the residual data from the spatial domain to the frequency domain and generate a transformation coefficient. The transformation coefficient is then subjected to quantization processing to obtain a quantized transformation coefficient. In various embodiments, the video encoder (703) also includes a residual decoder (728). The residual decoder (728) is configured to perform inverse transformation to generate decoded residual data. The decoded residual data can be preferably used by the intra-encoder (722) and the inter-encoder (730). For example, the inter-encoder (730) can generate a decoded block based on the decoded residual data and the inter-prediction information, and the intra-encoder (722) can generate a decoded block based on the decoded residual data and the intra-prediction information. The decoded block is preferably processed to generate a decoded picture, and the decoded picture is buffered in a memory circuit (not shown) and can be used as a reference picture in some examples.

[0111] The entropy encoder (725) is configured to format the bitstream to include the encoded blocks. The entropy encoder (725) is configured to include various information in the bitstream according to a suitable standard such as the HEVC standard. In one example, the entropy encoder (725) is configured to include general control data, selected prediction information (e.g., intra prediction information or inter prediction information), residual information, and other suitable information in the bitstream. Note that according to the disclosed subject matter, there is no residual information when encoding a block in either the inter mode or the merge submode of the bi-prediction mode.

[0112] FIG. 8 shows an exemplary diagram of a video decoder (810). The video decoder (810) is configured to receive an encoded picture that is part of an encoded video sequence and decode the encoded picture to generate a reconstructed picture. In one example, the video decoder (810) is used in place of the video decoder (410) in the example of FIG. 4.

[0113] In the example of FIG. 8, the video decoder (810) includes an entropy decoder (871), an inter decoder (880), a residual decoder (873), a reconstruction module (874), and an intra decoder (872) coupled together as shown in FIG. 8.

[0114] The entropy decoder (871) can be configured to reconstruct certain symbols from the encoded picture that represent the syntax elements that the encoded picture is composed of. Such symbols can include, for example, the mode in which a block is encoded (e.g., the latter two in intra mode, inter mode, bi-prediction mode, merge sub-mode or another sub-mode), and prediction information (e.g., intra prediction information or inter prediction information, etc.) that can identify certain samples or metadata used for prediction by the intra decoder (872) or the inter decoder (880) respectively. The symbols can also include, for example, residual information in the form of quantized transform coefficients. In one example, when the prediction mode is inter or bi-prediction mode, the inter prediction information is provided to the inter decoder (880); when the prediction type is intra prediction type, the intra prediction information is provided to the intra decoder (872). The residual information can undergo inverse quantization and is provided to the residual decoder (873).

[0115] The inter decoder (880) is configured to receive the inter prediction information and generate an inter prediction result based on the inter prediction information.

[0116] The intra decoder (872) is configured to receive the intra prediction information and generate a prediction result based on the intra prediction information.

[0117] The residual decoder (873) is configured to perform inverse quantization to extract the dequantized transform coefficients, and process the dequantized transform coefficients to convert the residual information from the frequency domain to the spatial domain. The residual decoder (873) may also require certain control information (including quantization parameter (QP)), and such information may be provided by the entropy decoder (871) (since this is only low-volume control information, the data path is not depicted).

[0118] The reconstruction module (874) is configured to combine, in a spatial domain, the residual information output by the residual decoder (873) and the prediction result (output by an intra or inter prediction module as appropriate) to form a reconstructed block, which may be part of a reconstructed picture, and the reconstructed picture may be part of a reconstructed video. Note that other suitable operations such as a deblocking operation can be performed to improve visual quality.

[0119] Note that the video encoders (403), (603), (703) and the video decoders (410), (510), (810) can be implemented using any suitable technique. In one embodiment, the video encoders (403), (603), (703) and the video decoders (410), (510), (810) can be implemented using one or more integrated circuits. In another embodiment, the video encoders (403), (603), (603), and the video decoders (410), (510), (810) can be implemented using one or more processors that execute software instructions.

[0120] The present disclosure includes embodiments related to flexible selection of intra prediction modes or inter prediction modes for different color components.

[0121] ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC 1 / SC29 / WG11) published the H.265 / HEVC (High Efficiency Video Coding) standard in 2013 (version 1), 2014 (version 2), 2015 (version 3), and 2016 (version 4). In 2015, both standardization organizations jointly established JVET (Joint Video Exploration Team) to explore the possibility of developing the next video coding standard beyond HEVC. In October 2017, both standardization organizations announced a Joint Call for Proposals on Video Compression with Capability beyond HEVC (CfP). By February 15, 2018, 22 CfP responses regarding standard dynamic range (SDR), 12 CfP responses regarding high dynamic range (HDR), and 12 CfP responses regarding 360 video category were respectively submitted. In April 2018, all received CfP responses were evaluated at the 122nd MPEG / 10th JVET meeting. As a result of this meeting, JVET officially launched the standardization process for the next-generation video coding beyond HEVC, and this new standard was named Versatile Video Coding (VVC), and JVET was renamed the Joint Video Experts Team. In 2020, ITU-T VCEG (Q6 / 16) and ISO / IEC MPEG (JTC1 / SC29 / WG11) published the VVC video coding standard (version 1).

[0122] In inter prediction, for each inter-predicted coding unit (CU), motion parameters are required, for example, for the coding features of VVC used for generating inter-predicted samples. The motion parameters can include motion vectors, reference picture indices, reference picture list use indices, and / or additional information. The motion parameters can be signaled explicitly or implicitly. When a CU is coded in skip mode, the CU can be associated with one PU, and significant residual coefficients, coded motion vector deltas, and / or reference picture indices may not be required. When a CU is coded in merge mode, the motion parameters for the CU can be obtained from neighboring CUs. The neighboring CUs can include spatial and temporal candidates, and additional schedules (or additional candidates) as introduced in VVC. The merge mode can be applied not only to skip mode but also to any inter-predicted CU. An alternative to the merge mode is the explicit transmission of motion parameters, where the motion vectors, corresponding reference picture indices for each reference picture list, reference picture list use flags, and / or other required information can be signaled explicitly for each CU.

[0123] In VVC, the VVC Test Model (VTM) reference software can include some new and refined inter-prediction coding tools, which can include one or more of the following: (1) Extended merge prediction (2) Merge motion vector difference (MMVD) (3) AMVP mode using symmetric MVD signaling (4) Affine motion compensation prediction (5) Subblock-based temporal motion vector prediction (SbTMVP) (6) Adaptive motion vector resolution (AMVR) (7) Motion field storage: 1 / 16 luma sample MV memory and 8×8 motion field compression (8) Bi-prediction with CU-level weights (BCW) (9) Bi-directional optical flow (BDOF) (10) Decoder side motion vector refinement (DMVR) (11) Combined inter and intra prediction (CIIP) (12) Geometric partitioning mode (GPM)

[0124] The merge candidate list can be constructed by including five types of candidates as in VTM4. The merge candidate list can be constructed in the following order: (1) Spatial MVP from spatial neighboring CUs (2) Temporal MVP from co-located CUs (3) History-based MVP from the FIFO table (4) Pairwise average MVP (5) Zero MV

[0125] The size of the merge list can be signaled in the slice header. The maximum allowable size of the merge list can be set to 6, as in VTM4. For each CU encoded in merge mode, the index of the best merge candidate can be encoded using, for example, truncated unary binarization. The first bin of the merge index can be encoded using context, and bypass coding can be used for the other bins.

[0126] In spatial candidate derivation, for example, in VVC, the derivation of spatial merge candidates can be the same as or similar to the derivation of spatial merge candidates in HEVC. The maximum number of merge candidates (e.g., four merge candidates) can be selected from among the candidates located at the positions shown in, for example, FIG. 9. As shown in FIG. 9, the current block (901) can include neighboring blocks (902) to (906) located at positions A0, A1, B0, B1, and B2, respectively. The order of derivation of spatial merge candidates can be B1, A1, B0, A0, and B2. The position B2 can be considered only when any of the CUs (or blocks) at positions A0, B0, B1, or A1 are not available (e.g., because the CU belongs to another slice or tile) or when it is intra-coded. After the candidate (or block) at position A1 is added, the addition of the remaining candidates (or blocks) can be subject to a redundancy check. The redundancy check can ensure that candidates with the same motion information are excluded from the merge list so that the coding efficiency is improved. To reduce the computational complexity, the redundancy check does not have to consider all possible candidate pairs. Instead, only the candidate pairs connected by the arrows in FIG. 10 may be considered. For example, the redundancy check can be applied to five candidate pairs such as the candidate pair of A1 and B1 and the candidate pair of A1 and A0. A candidate can be added to the merge list only if the corresponding candidate used for the redundancy check does not contain the same motion information. For example, candidate B0 can be added to the merge list only if the corresponding candidate B1 does not contain the same motion information.

[0127] In the derivation of temporal candidates, only one candidate can be added to the merge list. For example, as shown in FIG. 11, in the derivation of the temporal merge candidate for the current CU (1114), scaled motion vectors can be derived based on the co-located CU (1104) belonging to the co-located reference picture (1112). The reference picture list used for the derivation of the co-located CU (1104) can be explicitly signaled in the slice header. The scaled motion vectors for the temporal merge candidates can be obtained as shown by the dotted line (1102) in FIG. 11, which is scaled from the motion vector of the co-located CU (1104) using the picture order count (POC) distances tb and td. tb can be defined as the POC difference between the reference picture (e.g., Curr_ref) (1106) of the current picture and the current picture (e.g., Curr_pic) (1108). td can be defined as the POC difference between the co-located picture (e.g., Col_ref) (1110) and the co-located picture (e.g., Col_pic) (1112). The reference picture index of the temporal merge candidate can be set to zero.

[0128] The position for the temporal candidate can be selected between candidates C0 and C1 as shown in FIG. 12. For example, if the CU at position C0 is not available, is intra-coded, or is outside the current CTU row, position C1 can be used. Otherwise, position C0 can be used in the derivation of the temporal merge candidate.

[0129] In adaptive reordering of merge candidates with template matching (ARMC-TM), merge candidates can be adaptively reordered using template matching (TM). ARMC-TM can be applied to various prediction modes including the normal merge mode, the TM merge mode, and the affine merge mode (excluding SbTMVP candidates). For the TM merge mode, merge candidates can be reordered before the refinement process.

[0130] After constructing the merge candidate list, the merge candidates can be split into subgroups. The subgroup size can be set to 5 for the normal merge mode and the TM merge mode. For the affine merge mode, the subgroup size can be set to 3. The merge candidates in each subgroup can be sorted in ascending order according to the cost value based on template matching. For simplicity, the merge candidates in the last subgroup instead of the first subgroup may not be sorted.

[0131] The template matching cost of the merge candidates can be measured by the sum of absolute differences (SAD) between the samples of the template of the current block and the corresponding reference samples of the current block. The template can include a set of reconstructed samples adjacent to or close to the current block. The reference samples of the template can be located by the motion information of the merge candidates.

[0132] When the merge candidate uses bidirectional prediction, the reference samples of the template of the merge candidate can also be generated by bidirectional prediction. In the example shown in FIG. 13, the current block (1302) is included in the current picture (1304). The current block (1302) can have a template (1306) located adjacent to the upper side and the left side of the current block (1302). The template (1306) can include the reconstructed neighboring samples of the current block (1302). Based on template matching, the first reference block (1308) in the first reference picture (1310) of the first reference list 0 can be determined within the search range, and the minimum SAD can be determined between the template (1312) of the first reference block (1308) and the template (1306). The first reference block (1308) can be indicated by the MV (1314). Also, based on template matching, the second reference block (1316) in the second reference picture (1318) of the second reference list 1 can be determined within the search range, and the minimum SAD can be determined between the template (1320) of the second reference block (1316) and the template (1306). The second reference block (1316) can be indicated by the MV (1322). Further, based on the first reference block (1308) and the second reference block (1316), the predicted samples of the current block (1302) can be determined.

[0133] For sub-block-based merge candidates where the sub-block size is equal to Wsub×Hsub, the template above the current block can include several sub-templates of size Wsub×1, and the template to the left of the current block can include several sub-templates of size 1×Hsub. As shown in FIG. 14, the motion information of the sub-blocks in the first row and the first column of the current block can be used to derive the reference samples of each sub-template.

[0134] FIG. 14 shows an exemplary derivation of sub-block-based merge candidates. As shown in FIG. 14, a current block (1402) can be included in a current picture (1404). The current block (1402) can include sub-blocks A - G in a first row and a first column. The current block (1402) can include a template (1406) adjacent to an upper side and a left side of the current block (1402). The current block (1402) can include a co-located block (1408) in a reference picture (1410). The co-located block (1408) can include sub-blocks A - G in a first row and a first column corresponding to sub-blocks A - G in the current block (1402). Motion information (e.g., an affine motion vector) of sub-blocks A - G in the first row and the first column of the current block (1402) can be used to derive reference samples of sub-templates (or sub-reference templates) of the co-located block (1408). For example, by applying the motion information of sub-blocks A, E, F, and G of the current block (1402), reference samples of sub-templates arranged adjacent to the left sides of sub-blocks A, E, F, and G of the co-located block (1408) can be derived. The sub-templates adjacent to the left sides of sub-blocks A, E, F, and G of the co-located block (1408) can form a left reference template of the co-located block (1408). By applying the motion information of sub-blocks A, B, C, and D of the current block (1402), reference samples of sub-templates arranged adjacent to the upper sides of sub-blocks A, B, C, and D of the co-located block (1408) can be derived. The sub-templates adjacent to the upper sides of sub-blocks A, B, C, and D of the co-located block (1408) can further form an upper reference template of the co-located block (1408).

[0135] Merge with Motion Vector Difference (MMVD) can be used for certain prediction modes such as skip mode and merge mode that uses motion vector representation methods. In MMVD, merge candidates can be reused as in VVC. A merge candidate can be selected from among the merge candidates and further extended (or refined) by the motion vector representation method. MMVD can provide a new motion vector representation using simplified signaling. The motion vector representation method can include a starting point, the magnitude of the motion, and the direction of the motion.

[0136] MMVD can use a merge candidate list as in VVC. Candidates with the default merge type (e.g., MRG_TYPE_DEFAULT_N) can be considered for the extension of MMVD. In MMVD, the base candidate index can define the starting point. The base candidate index (IDX) can indicate the best candidate among the candidates in a list such as Table 1 below.

Table 1

[0137] When the number of base candidates is equal to 1, the base candidate IDX may not need to be signaled. The distance index can provide information about the magnitude of the motion. The distance index can indicate a predefined distance from the starting point. The predefined distance based on the distance index can be provided in Table 2 below.

Table 2

[0138] The direction index can represent the direction of the MVD based on the starting point. The direction index can represent four directions as shown in Table 3. The MMVD flag can be signaled exactly when the skip flag and the merge flag are transmitted. When the skip flag and the merge flag are true, the MMVD flag can be parsed. When the MMVD flag is equal to 1, the MMVD syntax can be parsed. However, when the MMVD flag is not 1, the AFFINE flag can be parsed. When the AFFINE flag is equal to 1, the AFFINE mode can be applied. However, when the AFFINE flag is not 1, the skip / merge index can be parsed for the skip / merge mode.

Table 3

[0139] Figure 15 shows an exemplary search process for MMVD. As shown in Figure 15, the starting point MV can be indicated by (1511) (e.g., according to the direction IDX and the base candidate IDX), the offset can be indicated by (1512) (e.g., according to the distance IDX and the direction IDX), and the final MV predictor can be indicated by (1513) in Figure 15. In other examples, in Figure 15, the starting point MV can be indicated by (1521) (e.g., according to the direction IDX and the base candidate IDX), the offset can be indicated by (1522) (e.g., according to the distance IDX and the direction IDX), and the final MV predictor can be indicated by 1523.

[0140] Figures 16A and 16B are diagrams showing exemplary search points of MMVD. As shown in Figure 16A, the starting point MV can be indicated by (1611) in the first reference list L0 (for example, according to the direction IDX and the base candidate IDX). In the example of Figure 16A, four search directions of +Y, -Y, +X, and -X are used, and these four search directions can be indexed by 0, 1, 2, and 3. The distance can be indexed by 0 (distance 0 to the starting point MV), 1 (1s (or 1 sample) to the starting point MV), 2 (2s to the starting point MV), 3 (3s to the starting point), etc. Thus, when the direction IDX is 3 and the distance IDX is 2, the final MV predictor is shown as (1615).

[0141] In another example, the search direction and the distance can be combined for indexing. For example, in the second reference list L1, the starting point MV is indicated by (1621) in the second reference list L1 (for example, according to the direction IDX and the base candidate IDX). The search direction and the distance are combined and indexed by 0 to 12 as shown in Figure 16B.

[0142] In template matching based candidate reordering on MMVD and affine MMVD, the MMVD offset can be extended for both the MMVD mode and the affine MMVD mode. In one example, additional refined positions along a diagonal angle of k×π / 8 can be added first. Exemplary additional refined positions can be shown in FIG. 17, and the number of directions can be increased from 4 to 16. Next, based on the SAD cost between the template (e.g., one row above and one column to the left of the current block) for each refined position and the reference of the template, all possible MMVD refined positions (e.g., 16×6) for each base candidate can be reordered. Finally, the top 1 / 8 of the refined positions with the minimum template SAD cost can be retained as the available positions, and thus for MMVD index coding. The MMVD index can be binarized by a code such as a rice code with a parameter equal to 2.

[0143] In another example, in addition to the above MMVD extension, the affine MMVD reordering can also be extended, where additional refined positions along a diagonal angle of k×π / 4 can be added. After reordering, the top 1 / 2 of the refined positions with the minimum template SAD cost can be retained.

[0144] To reduce the cross-component redundancy, similar to in VVC, the cross-component linear model (CCLM) prediction mode can be used. In the CCLM prediction mode, the chroma samples of the current CU can be predicted using the linear model of the following equation (1) based on the reconstructed luma samples of the current CU. pred C (i,j)=α·rec L '(i,j)+β Equation (1) Here, pred C(i, j) represents the predicted chroma sample in the current CU, and rec L (i, j) can represent the downsampled reconstructed luma sample of the current CU. When the chroma subsampling is different from the luma subsampling, as in the YCbCr422 or YCbCr420 format, the chroma CU can have a smaller size than the luma CU. When downsampling the reconstructed luma sample in the current CU, the luma sample and the chroma sample can be matched one-to-one.

[0145] CCLM parameters (such as α and β) can be derived based on at most four neighboring chroma samples and the corresponding downsampled luma samples of the neighboring chroma samples. If the current chroma block has dimensions of W × H, where W is the width of the current chroma block and H is the height of the current chroma block, the width W' of the first reference region and the height H' of the second reference region can be defined as follows: When applying the LM mode, W' = W, H' = H; When applying the LM-A mode, W' = W + H; When applying the LM-L mode, H' = H + W The first reference region can be adjacent to the upper side of the current chroma block. The second reference region can be adjacent to the left side of the current chroma block. In the LM mode, the neighboring chroma samples can be placed in both the first reference region and the second reference region. In the LM-A mode, the neighboring chroma samples can be placed in the first reference region. In the LM-L mode, the neighboring chroma samples can be placed in the second reference region.

[0146] Therefore, the upper neighboring position (or the position of the first reference region) can be denoted as S[0, -1]…S[W' - 1, -1], and the left neighboring position (or the position of the second reference region) can be denoted as S[-1, 0]…S[-1, H' - 1]. Therefore, the positions of the four neighboring chroma samples can be selected as follows: When the LM mode is applied and both the upper and left neighboring samples are available S[W' / 4, -1], S[3*W' / 4, -1], S[-1, H' / 4], S[-1, 3*H' / 4] When the LM-A mode is applied or only the upper neighboring sample is available S[W' / 8, -1], S[3*W' / 8, -1], S[5*W' / 8, -1], S[7*W' / 8, -1] When the LM-L mode is applied or only the left neighboring sample is available S[-1, H' / 8], S[-1, 3*H' / 8], S[-1, 5*H' / 8], S[-1, 7*H' / 8]

[0147] The four luma samples corresponding to the four neighboring chroma samples at the selected positions are downsampled, compared four times, and two larger values x 0A and x 1 A , and two smaller values x 0 B and x 1 B can be found. The chroma sample values of the four neighboring chroma samples corresponding to the four luma samples can be denoted as y 0 A , y 1 A , y 0 B , and y 1 B The parameters X a , X b , Y a , and Y b can be derived from the following equations (2) to (5): X a =(x 0 A +x 1 A +1) >> 1 Equation (2) X b =(x 0B +x 1 B ( + 1) >> 1 Equation (3) Y a = (y 0 A + y 1 A ( + 1) >> 1 Equation (4) Y b = (y 0 B + y 1 B ( + 1) >> 1 Equation (5)

[0148] Finally, the linear model parameters α and β are obtained according to Equation (6) and Equation (7), respectively. [Number]

[0149] Figure 18 shows exemplary positions of neighboring samples to the left and above the current block, and samples of the current block involved in the CCLM prediction mode. As shown in Figure 18, the current chroma CU (1802) can have a size of N × N (e.g., 8 × 8). The corresponding luma CU (1804) of the current chroma CU (1802) can have a size of 2N × 2N (e.g., 16 × 16). The neighboring chroma samples of the current chroma CU (1802) used to derive the linear model parameters α and β can be arranged in the first reference region (or upper neighboring position) (1806) and / or the second reference region (or left neighboring position) (1808). The luma samples corresponding to the neighboring chroma samples (1806) and (1808) can be arranged in the first reference region (1810) and the second reference region (1812), respectively. As shown in Figure 18, the luma samples in the first reference region (1810) and the second reference region (1812) can be downsampled to match one-to-one with the neighboring chroma samples (1806) and (1808). The reconstructed values Rec' of the luma samples in the first reference region (1810) and the second reference region (1812) Land the reconstructed values Rec of the neighboring chroma samples (1806) and (1808) c By applying c and based on formulas (2) to (7), the linear model parameters α and β can be derived. Once the linear model parameters α and β are obtained, the chroma samples in the current chroma CU (1802) can be predicted using the CCLM prediction mode based on the downsampled and reconstructed luma samples in the corresponding luma CU (1804).

[0150] The division operation for calculating parameter α can be implemented using a look-up table. To reduce the memory required to store the table, the diff value (e.g., the difference between the maximum value and the minimum value) and parameter α can be expressed in exponential notation. For example, diff can be approximated with a 4-bit mantissa and an exponent. Thus, the table for 1 / diff can be reduced to 16 elements for 16 significant values as shown in the following formula (8): DivTable[]={0,7,6,5,5,4,4,3,3,2,2,1,1,1,1,0} Formula (8)

[0151] In this way, the effect of reducing the computational complexity and the memory size required to store the necessary table is obtained.

[0152] As shown in FIG. 18, the upper template (e.g., (1806)) and the left template (e.g., (1808)) can be used in combination to calculate the linear model coefficients (e.g., the linear model parameters α and β). The upper template and the left template can alternatively be used in two other LM modes called the LM_A (or LM_T) mode and the LM_L mode.

[0153] In LM_A mode, only the upper template can be used to calculate the linear model coefficients. To obtain more samples, the upper template can be expanded to, for example, (W + H) samples. In LM_L mode, only the left template can be used to calculate the linear model coefficients. To obtain more samples, the left template can be expanded to, for example, (H + W) samples.

[0154] In LM_LT mode, both the left template and the upper template can be used to calculate the linear model coefficients.

[0155] To match the chroma sample positions for a 4:2:0 video sequence, two types of downsampling filters can be applied to the luma samples to achieve a 2:1 downsampling ratio in both the horizontal and vertical directions. The selection of the downsampling filter can be specified by an SPS-level flag. The two downsampling filters can be given as follows in equations (9) and (10) corresponding to "type 0" content and "type 2" content, respectively.

Equation

[0156] Note that when the upper reference line is at the CTU boundary, only one luma line (e.g., a common line buffer in intra prediction) can be used to create the downsampled luma samples.

[0157] The calculation of parameters (such as α and β) can be performed as part of the decoding process and does not have to be limited to the encoder search operation. As a result, the syntax may not be required to convey the α value and β value to the decoder.

[0158] For chroma intra mode coding, a total of 8 intra modes can be allowed for chroma intra mode coding. The 9 intra modes can include 5 conventional intra modes and 3 component cross - linear model modes (e.g., CCLM, LM_A, LM_L). An exemplary chroma mode signaling and derivation process is shown in Table 4. The chroma mode coding of the current chroma block can directly depend on the intra prediction mode of the corresponding luma block. In an I slice, since a separate block partitioning structure can be enabled for luma and chroma components, one chroma block may correspond to multiple luma blocks. Therefore, for the chroma DM mode, the intra prediction mode of the corresponding luma block covering the center position of the current chroma block can be directly inherited. [Table 4]

[0159] In Table 5, a single binarization table can be used regardless of the value of a flag (e.g., sps_cclm_enabled_flag) indicating whether CCLM is enabled. [Table 5]

[0160] As shown in Table 5, the first bin can indicate whether the normal (0) mode or the LM mode (1) is applied. When the first bin indicates the LM mode, the next bin (or the second bin) can indicate whether LM_CHROMA (0) is applied. When the second bin indicates LM_CHROMA, the next single bin (or the third bin) can indicate whether LM_L (0) or LM_A (1) is applied. When the CCLM flag (e.g., sps_cclm_enabled_flag) is 0, the first bin of the binarization table for the corresponding chroma intra mode (e.g., intra_chroma_pred_mode) can be discarded before entropy coding. In other words, the first bin can be presumed to be 0 and thus not coded. The binarization table can be used when the CCLM flag (or sps_cclm_enabled_flag) is equal to 0 or 1. The first two bins in Table 5 can be context-model coded using the corresponding context model, and the remaining bins can be bypass coded.

[0161] Furthermore, to shorten the luma-chroma latency of the dual tree, when the splitting of the 64×64 luma coding tree node is without splitting (and the ISP is not used for 64×64 CUs) or QT, the chroma CUs in the 32×32 / 32×16 chroma coding tree node can be allowed to use CCLM under the following conditions: (1) When the 32×32 chroma node is not split or partitioned by QT splitting, all chroma CUs within the 32×32 node can use CCLM. (2) When the 32×32 chroma node is partitioned horizontally by BT and the 32×16 child node is not split or vertical BT splitting is used, all chroma CUs within the 32×16 chroma node can use CCLM. For other luma and chroma coding tree splitting conditions, CCLM may not be allowed for chroma CUs.

[0162] In related video coding standards, different color components can share the selection of prediction modes. For example, different color components can share the prediction mode selection from intra prediction, inter prediction, or combined intra-inter prediction. The shared prediction mode selection for different color components may reduce the flexibility of the prediction mode selection.

[0163] In the present disclosure, the selection of prediction modes for different color components (or different color channels) of the current block can be determined respectively. In one example, the prediction mode selections for different color components (or different color channels) of the current block can be different. Thus, for each of the color components of the current block, respective prediction modes such as intra prediction, inter prediction, or combined intra-inter prediction can be selected. In one example, the first prediction mode of the first color component of the current block can be determined. The second prediction mode of the second color component of the current block can be determined based on the first color component. The first prediction mode may be the same as or different from the second prediction mode.

[0164] In certain embodiments, the color components of the current block can refer to Y (luma), Cb (chroma blue), and Cr (chroma red). In certain embodiments, the color components of the current block can refer to red, green, and blue. In certain embodiments, the color channels (or components) of the current block can refer to the luma channel and the chroma channel, and the chroma channel can include two color components such as the Cr component and the Cb component.

[0165] In certain embodiments, the color components / channels of the current block can be encoded by a first prediction mode such as the inter prediction mode, and another color component / channel of the current block can be encoded by the same prediction mode as or a different prediction mode from the first prediction mode. For example, the said another color component / channel of the current block can be encoded by either the inter prediction mode or the intra prediction mode.

[0166] In one embodiment, a flag / index can be signaled for a color component / channel of the current block to indicate a prediction mode of the color component / channel. In one embodiment, the flag / index indicates whether the color component / channel associated with the flag / index is encoded by an inter prediction mode or by an intra prediction mode.

[0167] In one example, when the luma component of the current block is encoded by a first prediction mode such as an inter prediction mode, signaling information such as a flag is signaled for one or more of the other chroma components of the current block to indicate whether the one or more of the other chroma components associated with the flag are encoded by a second prediction mode such as an intra prediction mode.

[0168] In one example, when the luma component of the current block is encoded by a first prediction mode such as an inter prediction mode, signaling information such as a flag is signaled for one or more of the other chroma components (e.g., the Cb component or the Cr component) to indicate whether the one or more of the other chroma components associated with the flag are encoded by a cross-component prediction mode. Examples of cross-component prediction modes can include, but are not limited to, the CCLM mode, the CfL mode (e.g., CfL defined in AV1), the multi-model CCLM mode, the multi-filter CCLM mode, the convolutional cross-component model (CCCM) mode, etc. The CCCM mode can be configured to predict chroma samples from the reconstructed luma samples in a similar manner as the CCCM mode.

[0169] In one embodiment, when the color component / channel of the current block is encoded by a specific (or predetermined) coding mode, signaling information such as a flag or an index is signaled for another color component / channel of the current block to indicate whether a specific prediction mode such as an intra prediction mode or an inter prediction mode is applied to the other color component / channel.

[0170] In one example, when a first color component such as a luma component is encoded using an inter prediction mode without signaling a motion vector difference (e.g., merge mode in VVC, NEAR mode in AV1 / AV2), a flag / index is signaled for a second color component such as a chroma component to indicate whether an intra prediction mode (e.g., a certain angular mode or a component cross prediction mode) is applied to the second color component.

[0171] In one example, when a first color component such as a luma component is encoded in an inter prediction mode with a motion vector difference (e.g., AMVP mode in VVC or NEWMV mode in AV1 / AV2), a flag / index is signaled for a second color component such as a chroma component to indicate whether an intra prediction mode (e.g., a component cross prediction mode) is applied to the second color component.

[0172] In one example, when a first color component such as a luma component is encoded in an inter prediction with a motion vector displacement (e.g., MMVD mode in VVC), a flag / index is signaled for a second color component such as a chroma component to indicate whether an intra prediction mode (e.g., a component cross prediction mode) is applied to the second color component.

[0173] In one embodiment, signaling information such as a flag / index indicating whether an intra prediction mode or an inter prediction mode is applied to the second color component / channel of the current block can be entropy encoded using context. The signaling information may be entropy encoded, for example, when the first color component of the current block is encoded by the first prediction mode.

[0174] In one example, the context can be derived from the encoded information. The encoded information can include one or more of the block size of the current block, the first prediction mode of the first color component / channel, and the value of the flag / index signaled to neighboring blocks of the current block, but is not limited thereto.

[0175] In one embodiment, whether to encode the color component of the current block using the inter-mode or the intra-mode can be determined without signaling the encoding information for the color component in the bitstream. The prediction mode of the color component can be derived rather than signaled in the bitstream.

[0176] In one example, when the color component (e.g., luma component) of the current block is encoded using the inter-mode, the prediction mode of the other color component(s) of the current block (e.g., one or both of the chroma components) can be determined based on the template matching (TM) cost of the other color component(s). The TM cost can be determined based on a plurality of candidate prediction modes associated with the other color component(s), and each candidate prediction mode of the plurality of candidate prediction modes corresponding to the lowest TM cost can be selected for the other color component(s).

[0177] The TM cost can be determined based on a template matching process such as template-based intra mode derivation (TIMD). Other suitable template matching processes may be applied to determine the TM cost. In an example of the template matching process, a plurality of predicted sample values of a template of the other color components (e.g., Cb component and Cr component) of the current block can be generated based on a plurality of candidate prediction modes. Each of the plurality of predicted sample values can be generated based on a respective candidate prediction mode. Based on the plurality of predicted sample values of the template, a plurality of TM costs can be determined. Each of the plurality of TM costs can be associated with the difference between a respective predicted sample value of the template and a reconstructed sample value. The best candidate prediction mode among the plurality of candidate prediction modes can be selected for the other color components of the current block. The best candidate prediction mode can correspond to the minimum TM cost among the plurality of TM costs.

[0178] In one example, when the color component of the current block (e.g., the luma component) is encoded by a first prediction mode such as an inter prediction mode, the TM cost can be used to determine whether an intra prediction mode or an inter prediction mode is applied to another color component of the current block (e.g., the Cb component or the Cr component). If the reconstructed template of the current block is not available for the other color component (e.g., the Cb or Cr component) (or if the templates of the other color components (e.g., the Cb component and the Cr component) of the current block do not include the reconstructed samples), the other color component can be encoded using the same inter motion information (or inter prediction mode) as the encoded component (e.g., the encoded luma component). For example, if the luma component of the current block is encoded by a first prediction mode (e.g., an inter prediction mode) and the template of the chroma component of the current block does not include the reconstructed samples, the first prediction mode can be applied to the chroma component.

[0179] FIG. 19 shows a flowchart outlining an exemplary decode process (1900) according to some embodiments of the present disclosure. FIG. 20 shows a flowchart outlining an exemplary encode process (2000) according to some embodiments of the present disclosure. The proposed processes may be used separately or combined in any order. Further, each of the processes (or embodiments), encoders, and decoders may be implemented by a processing circuit (e.g., one or more processors or one or more integrated circuits). In one example, the one or more processors execute a program stored on a non-transitory computer-readable medium.

[0180] The operations of the processes (e.g., (1900) and (2000)) may be combined or arranged in any amount or order as desired. In various embodiments, two or more operations of the processes (e.g., (1900) and (2000)) may be executed in parallel.

[0181] Processes (e.g., (1900) and (2000)) can be used to generate prediction blocks for blocks being reconstructed in block reconstruction and / or encoding. In various embodiments, processes (e.g., (1900) and (2000)) are executed by processing circuits in terminal devices (310), (320), (330) and (340), processing circuits that execute the functions of video encoder (403), processing circuits that execute the functions of video decoder (410), processing circuits that execute the functions of video decoder (510), processing circuits that execute the functions of video encoder (603), etc. In some embodiments, processes (e.g., (1900) and (2000)) are implemented by software instructions, so that when the processing circuit executes the software instructions, the processing circuit executes processes (e.g., (1900) and (2000)).

[0182] As shown in FIG. 19, process (1900) can start from (S1901) and proceed to (S1910). At (S1910), an encoded video bitstream including a current block is received. The current block is associated with at least a first color component and a second color component.

[0183] At (S1920), encoded information of the first color component and the second color component associated with the current block is obtained.

[0184] At (S1930), for the first color component associated with the current block, a first prediction mode selected from an intra prediction mode, an inter prediction mode, or a composite inter-intra prediction mode is determined.

[0185] At (S1940), for the second color component associated with the current block, a second prediction mode selected from an intra prediction mode, an inter prediction mode, or a composite inter-intra prediction mode is determined, and the second prediction mode is different from the first prediction mode.

[0186] In (S1950), the current block is reconstructed based on the prediction of the first color component of the current block in the first prediction mode and the prediction of the second color component of the current block in the second prediction mode.

[0187] In some embodiments, the inter prediction mode includes any one of an inter prediction mode without a motion vector difference, an inter prediction mode with a motion vector difference, and an inter prediction mode with a motion vector displacement.

[0188] In one example, the first color component is a luma component and the second color component is a chroma component.

[0189] In one example, the first color component is any one of a red component, a green component, and a blue component, and the second color component is any one of a red component, a green component, and a blue component.

[0190] In an embodiment, in response to the first color component being encoded in an inter prediction mode, whether the second prediction mode is an intra prediction mode is determined based on signaling information included in the encoded information.

[0191] In an embodiment, in response to the first color component being determined as an inter prediction mode, whether the second prediction mode is a component cross prediction mode is determined based on signaling information included in the encoded information.

[0192] The component cross prediction mode includes any one of a CCLM mode, a CfL mode, a multi-model CCLM mode, a multi-filter CCLM mode, and a CCCM mode.

[0193] The inter prediction mode without a motion vector difference includes any one of a merge mode and a near mode. The inter prediction mode with a motion vector difference includes any one of an AMVP mode and a NEWMV mode. The inter prediction mode with a motion vector displacement includes an MMVD mode.

[0194] The second prediction mode for the second color component is determined based on signaling information included in the encoded information. The signaling information includes a flag entropy encoded using a context. The context is derived from the encoded information based on at least one of the block size of the current block, the first prediction mode, or the value of the flag signaled for neighboring blocks of the current block.

[0195] To determine the second prediction mode based on the TM cost, a plurality of prediction sample values of the template of the current block are generated based on a plurality of candidate prediction modes, and each of the plurality of prediction sample values is generated based on each of the plurality of candidate prediction modes. The template of the current block includes samples adjacent to the current block. A plurality of TM costs are generated based on the plurality of prediction sample values of the template, and each of the plurality of TM costs is associated with the difference between each prediction sample value of the plurality of prediction sample values and the reconstructed sample value of the template. The second prediction mode is determined as the candidate prediction mode among the plurality of candidate prediction modes corresponding to the minimum TM cost among the plurality of TM costs.

[0196] In some embodiments, in response to the reconstructed sample value of the template not being available, the second prediction mode is determined as the first prediction mode.

[0197] After (S1940), proceed to (S1999) and end.

[0198] Process (1900) can be appropriately adapted. The steps of process (1900) can be modified and / or omitted. Additional steps can be added. Any suitable implementation order can be used.

[0199] As shown in FIG. 20, the process (2000) can start from (S2001) and proceed to (S2010). In (S2010), for the first color component associated with the current block, a first prediction mode selected from an intra prediction mode, an inter prediction mode, or a composite inter-intra prediction mode is determined.

[0200] In (S2020), for the second color component associated with the current block, a second prediction mode selected from an intra prediction mode, an inter prediction mode, or a composite inter-intra prediction mode is determined, and the second prediction mode is different from the first prediction mode.

[0201] In (S2030), based on the determined first and second prediction modes, prediction information for the current block is generated. Encoding information is also generated. The encoding information indicates that the second prediction mode is determined based on the first prediction mode.

[0202] Then, the process proceeds to (S2099) and ends.

[0203] The process (2000) can be appropriately adapted. The steps of the process (2000) can be modified and / or omitted. Additional steps can be added. Any suitable implementation order can be used.

[0204] The above-described techniques can be implemented as computer software using computer-readable instructions and can be physically stored on one or more computer-readable media. For example, FIG. 21 shows a computer system (2100) suitable for implementing certain embodiments of the disclosed subject matter.

[0205] Computer software can be coded using any suitable machine code or computer language and be the subject of assembly, compilation, linking, or similar mechanisms to create code containing instructions that can be executed directly or through interpretation, microcode execution, etc. by one or more computer central processing units (CPUs), graphics processing units (GPUs), and the like.

[0206] The instructions can be executed on various types of computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, gaming devices, Internet of Things devices, etc.

[0207] The components shown in FIG. 21 for the computer system (2100) are illustrative in nature and are not intended to imply any limitations on the use or functionality of the computer software implementing the embodiments of the present disclosure. Nor should the configuration of the components be construed as having any dependency or requirement regarding any one or combination of the components shown in the exemplary embodiments of the computer system (2100).

[0208] The computer system (2100) can include certain human interface input devices. Such human interface input devices can respond to input by one or more human users through, for example, tactile input (e.g., keystrokes, swipes, movements of a data glove), voice input (e.g., voice, clapping), visual input (e.g., gestures), olfactory input (not shown). Also, the human interface device can be used to capture certain media that are not necessarily directly related to conscious input by humans, such as voice (e.g., speech, music, ambient sound), images (e.g., scanned images, photographic images obtained from a still image camera), video (e.g., 2D video, 3D video including stereoscopic video).

[0209] The input human interface device may include one or more of a keyboard (2101), a mouse (2102), a trackpad (2103), a touch screen (2110), a data glove (not shown), a joystick (2105), a microphone (2106), a scanner (2107), and a camera (2108) (only one of each is shown).

[0210] The computer system (2100) may also include certain human interface output devices. Such human interface output devices may, for example, stimulate the senses of one or more human users through tactile output, sound, light, and smell / taste. Such human interface output devices may include a tactile output device (e.g., tactile feedback by a touch screen (2110), a data glove (not shown), or a joystick (2105); however, there may also be a tactile feedback device that does not function as an input device), an audio output device (e.g., a speaker (2109), headphones (not shown)), a visual output device (e.g., a screen (2110) including a CRT screen, an LCD screen, a plasma screen, an OLED screen; each may or may not have a touch screen input function, each may or may not have a tactile feedback function, and some of them can output higher than three-dimensional output through means such as two-dimensional visual output or stereoscopic output; virtual reality glasses (not shown), a holographic display, and a smoke tank (not shown)), and a printer (not shown).

[0211] The computer system (2100) can also include an optical medium, such as a CD / DVD ROM / RW (2120), along with a human-accessible memory device and associated media, such as a CD / DVD or similar media (2121), a thumb drive (2122), a removable hard drive or solid state drive (2123), legacy magnetic media, such as tapes and floppy disks (not shown), specialized ROM / ASIC / PLD-based devices, such as security dongles (not shown), and the like.

[0212] One of ordinary skill in the art should also understand that the term "computer-readable medium" as used in connection with the presently disclosed subject matter does not include a transmission medium, a carrier wave, or other transient signals.

[0213] The computer system (2100) can also include an interface (2154) to one or more communication networks (2155). The network can be, for example, wireless, wired, or optical. The network can further be local, wide area, metropolitan area, in-vehicle and industrial, real-time, delay tolerant, etc. Examples of networks include Ethernet [registered trademark], wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., cellular networks including cable TV, satellite TV, terrestrial broadcast TV, TV wired or wireless wide area digital networks including CANBus, in-vehicle and industrial including CANBus, etc. Some types of networks typically require an external network interface adapter attached to a certain type of general-purpose data port or peripheral bus (2149) (such as a USB port of the computer system (2100)). Others are typically integrated into the core of the computer system (2100) by attachment to a system bus as described later (such as an Ethernet interface to a PC computer system or a cellular network interface to a smartphone computer system). Using any of these networks, the computer system (2100) can communicate with other entities. Such communication can be unidirectional, receive only (such as broadcast TV), dedicated unidirectional transmission (such as CANbus to certain CANbus devices), or bidirectional to other computer systems using, for example, local or wide area digital networks. For each of the networks and network interfaces as described above, certain protocols and protocol stacks can be used.

[0214] The aforementioned human interface device, human-accessible storage device, and network interface can be attached to the core (2140) of the computer system (2100).

[0215] The core (2140) can include one or more central processing units (CPUs) (2141), a graphics processing unit (GPU) (2142), a specialized programmable processing device in the form of a field programmable gate array (FPGA) (2143), a hardware accelerator (2144) for certain tasks, a graphics adapter (2150), etc. These devices can be connected through a system bus (2148) together with internal mass storage devices (2147) such as read-only memory (ROM) (2145), random access memory (2146), an internal hard drive not accessible to users, and a solid state drive (SSD). In some computer systems, the system bus (2148) may be accessible in the form of one or more physical plugs to enable expansion by additional CPUs, GPUs, etc. Peripheral devices can be attached directly to the core's system bus (2148) or through a peripheral bus (2149). In one example, a screen (2110) can be connected to the graphics adapter (2150). Architectures for peripheral buses include PCI, USB, etc.

[0216] The CPU (2141), GPU (2142), FPGA (2143), and accelerator (2144) can execute certain instructions that can together constitute the above-described computer code. That computer code can be stored in the ROM (2145) or RAM (2146). Temporary data can also be stored in the RAM (2146), while persistent data can be stored, for example, in the internal mass storage device (2147). High-speed storage and retrieval to / from any of the memory devices can be enabled by using cache memory that can be closely associated with one or more CPUs (2141), GPUs (2142), mass storage devices (2147), ROM (2145), RAM (2146), etc.

[0217] A computer-readable medium can have computer code thereon for performing various computer-implemented operations. The medium and the computer code may be specially designed and constructed for the purposes of this disclosure, or they may be of the kind well known and available to those having skill in the computer software arts.

[0218] By way of example and not limitation, a computer system having an architecture (2100), specifically a core (2140), can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied on one or more tangible computer-readable media. Such computer-readable media can be related to user-accessible mass storage as introduced above, as well as certain storage of the core (2140) of a non-transitory nature such as a mass storage device (2147) inside the core or a ROM (2145). The software implementing various embodiments of the present disclosure can be stored in such a device and executed by the core (2140). The computer-readable media can include one or more memory devices or chips according to specific needs. The software can include defining a data structure stored in a RAM (2146) and modifying such a data structure according to a process defined by the software, and causing a specific process or a specific part described herein to be executed by the core (2140) and specifically a processor (including a CPU, GPU, FPGA, etc.) therein. Additionally or alternatively, the computer system can provide functionality as a result of logic wired within a circuit (e.g., an accelerator (2144)) or otherwise embodied, which can operate instead of or in conjunction with software for executing a specific process or a specific part of a specific process described herein. References to software include logic and vice versa as appropriate. References to computer-readable media can, as appropriate, include a circuit (e.g., an integrated circuit (IC)) storing software for execution, a circuit embodying logic for execution, or both. The present disclosure encompasses any suitable combination of hardware and software.

[0219] Appendix A: Acronyms JEM: joint exploration model VVC: versatile video coding (Multi-purpose video coding) BMS: benchmark set (Benchmark set) MV: Motion Vector (Motion vector) HEVC: High Efficiency Video Coding (High-efficiency video coding) SEI: Supplementary Enhancement Information (Supplementary enhancement information) VUI: Video Usability Information (Video usability information) GOP: Group of Pictures (Picture group) TU: Transform Unit (Transform unit) PU: Prediction Unit (Prediction unit) CTU: Coding Tree Unit (Coding tree unit) CTB: Coding Tree Block (Coding tree block) PB: Prediction Block (Prediction block) HRD: Hypothetical Reference Decoder (Hypothetical reference decoder) SNR: Signal Noise Ratio (Signal-to-noise ratio) CPU: Central Processing Unit (Central processing unit) GPU: Graphics Processing Unit (Graphics processing unit) CRT: Cathode Ray Tube (Cathode ray tube) LCD: Liquid-Crystal Display (Liquid crystal display) OLED: Organic Light-Emitting Diode (Organic light-emitting diode) CD: Compact Disc (Compact disc) DVD: Digital Video Disc (Digital video disc) ROM: Read-Only Memory (Read-Only Memory) RAM: Random Access Memory (Random Access Memory) ASIC: Application-Specific Integrated Circuit (Application-Specific Integrated Circuit) PLD: Programmable Logic Device (Programmable Logic Device) LAN: Local Area Network (Local Area Network) GSM: Global System for Mobile communications (Global System for Mobile Communications) LTE: Long-Term Evolution (Long-Term Evolution) CANBus: Controller Area Network Bus (Controller Area Network Bus) USB: Universal Serial Bus (Universal Serial Bus) PCI: Peripheral Component Interconnect (Peripheral Component Interconnect) FPGA: Field Programmable Gate Areas (Field Programmable Gate Areas) SSD: solid-state drive (Solid State Drive) IC: Integrated Circuit (Integrated Circuit) CU: Coding Unit (Coding Unit)

[0220] Although several exemplary embodiments have been described in this disclosure, there are changes, substitutions, and various alternative equivalents that fall within the scope of this disclosure. Therefore, it will be understood that those skilled in the art can devise many systems and methods that embody the principles of this disclosure and thus fall within the spirit and scope of this disclosure, even though they are not explicitly shown or described in this specification.

Claims

1. A method of decoding performed by a decoder, the method comprising: receiving an encoded video bitstream including a current block, the current block being associated with at least a first color component and a second color component; obtaining encoded information of the first color component and the second color component associated with the current block; determining a first prediction mode selected from an intra prediction mode, an inter prediction mode, or a combined intra-inter prediction mode for the first color component associated with the current block; determining a second prediction mode selected from the intra prediction mode, the inter prediction mode, or the combined intra-inter prediction mode for the second color component associated with the current block, the second prediction mode being different from the first prediction mode; reconstructing the current block based on predicting the first color component of the current block in the first prediction mode and predicting the second color component of the current block in the second prediction mode. A method.

2. The method according to claim 1, wherein the inter prediction mode includes one of an inter prediction mode without a motion vector difference, an inter prediction mode with a motion vector difference, and an inter prediction mode with a motion vector displacement.

3. The method according to claim 1, wherein the first color component is a luminance component and the second color component is a chrominance component.

4. The first color component is one of a red component, a green component, and a blue component, and the second color component is another one of the red component, the green component, and the blue component. The method according to claim 1.

5. Determining the second prediction mode comprises: determining, based on signaling information included in the encoded information, whether the second prediction mode is an intra prediction mode in response to the first color component being encoded by the inter prediction mode. The method according to claim 1.

6. Determining the second prediction mode comprises: In response to the first color component being determined as the inter prediction mode, determining whether the second prediction mode is a component cross prediction mode based on signaling information included in the encoded information, the method according to claim 1.

7. The component cross prediction mode includes one of a component cross linear mode (CCLM), a luma to chroma mode (CfL), a multi-model CCLM mode, a multi-filter CCLM mode, and a convolutional component cross model (CCCM), the method according to claim 6.

8. The inter prediction mode without a motion vector difference includes one of a merge mode and a near mode, The inter prediction mode with a motion vector difference includes one of an advanced motion vector prediction (AMVP) mode and a new motion vector (NEWMV) mode, The inter prediction mode with a motion vector displacement includes a merge motion vector difference (MMVD) mode, The method according to claim 2.

9. The second prediction mode for the second color component is determined based on signaling information included in the encoded information, The signaling information includes a flag that is entropy encoded using a context, The context is derived from the encoded information based on at least one of a block size of a current block, the first prediction mode, or a value of the flag signaled for a neighboring block of the current block, The method according to claim 1.

10. Determining the second prediction mode comprises: Generating a plurality of predicted sample values of a template of a current block based on a plurality of candidate prediction modes, each of the plurality of predicted sample values being generated based on a respective candidate prediction mode of the plurality of candidate prediction modes, the template of the current block including samples adjacent to the current block; Generating a plurality of TM costs based on the plurality of predicted sample values of the template, each of the plurality of TM costs being related to a difference between a respective predicted sample value of the plurality of predicted sample values and a reconstructed sample value of the template; determining the second prediction mode as a candidate prediction mode among the plurality of candidate prediction modes corresponding to the minimum TM cost among the plurality of TM costs; The method according to claim 1. **Claim 11** Determining the second prediction mode comprises: further comprising, in response to the reconstructed sample values of the template being unavailable, determining the second prediction mode as the first prediction mode; The method according to claim 10. **Claim 12** An apparatus having a processing circuit configured to execute the method according to any one of claims 1 to 11. **Claim 13** A computer program for causing a processor to execute the method according to any one of claims 1 to 11.

Images