Cross-component prediction

Abstract

A method is provided for encoding a video sequence into a bitstream and decoding the bitstream to output one or more pictures of the video stream. An exemplary method includes the steps of receiving a video sequence, encoding one or more pictures of the video sequence, and generating a bitstream associated with the encoded pictures, wherein the encoding step includes predicting chroma samples based on luma samples corresponding to chroma samples in the current block using a plurality of cross-component residual models (CCRM).

Description

[Technical Field]

[0001] (Cross-reference of related applications) This disclosure claims the benefit of priority to U.S. Provisional Application No. 63 / 511,659 filed on 2 July 2023, and the benefit of U.S. Application No. 18 / 750,267 entitled “Cross-Component Prediction,” filed on 21 June 2024, both of which are incorporated herein by reference in their entirety.

[0002] This disclosure relates to image processing in general, and more specifically to a cross-component prediction technique used to predict a chroma sample based on a collated luma sample. [Background technology]

[0003] Video is a series of still pictures (or "frames") that capture visual information. To reduce memory usage and transmission bandwidth, video can be compressed before storage or transmission and decompressed before display. The compression process is usually called encoding, and the decompression process is usually called decoding. There are various video coding formats that use standardized video coding techniques, most commonly based on prediction, transformation, quantization, entropy coding, and in-loop filtering. Video coding standards that specify a particular video coding format, such as the High Efficiency Video Codec (HEVC / H.265) standard, the Versatile Video Coding (VVC / H.266) standard, and the AVS standard, are developed by standardization organizations. As increasingly advanced video coding techniques are adopted in video standards, the coding efficiency of new video coding standards increases. [Overview of the project]

[0004] Embodiments of this disclosure provide a method and apparatus for predicting a chromatic sample based on a collated chromatic sample.

[0005] A first aspect of the present disclosure provides a method for encoding a video sequence into a bitstream, comprising the steps of: receiving a video sequence; encoding one or more pictures of the video sequence; and generating a bitstream associated with the encoded pictures, wherein the encoding step includes the step of predicting chroma samples based on luma samples corresponding to chroma samples in the current block using a plurality of cross-component residual models (CCRMs).

[0006] A second aspect of the present disclosure provides a method for decoding a bitstream to output one or more pictures of a video stream, comprising the steps of receiving a bitstream and decoding one or more pictures using coded information of the bitstream, wherein the decoding step includes predicting a chroma sample based on a luma sample corresponding to a chroma sample in the current block using a plurality of cross-component residual models (CCRMs).

[0007] In a third aspect of the present disclosure, an apparatus is provided for encoding a video sequence into a bitstream, comprising: a receiving module configured to receive the video sequence; an encoding module configured to encode one or more pictures of the video sequence; and a generating module configured to generate a bitstream associated with the encoded pictures, wherein the encoding module is configured to predict a chroma sample based on a luma sample corresponding to a chroma sample in the current block using a plurality of cross-component residual models (CCRM).

[0008] A fourth aspect of the present disclosure provides an apparatus for decoding a bitstream and outputting one or more pictures of a video stream, comprising: a receiving module configured to receive a bitstream; and a decoding module configured to decode one or more pictures using coded information of the bitstream, wherein the decoding module is configured to predict a chroma sample based on a luma sample corresponding to a chroma sample in the current block using a plurality of cross-component residual models (CCRMs).

[0009] A fifth aspect of the present disclosure provides an electronic device comprising one or more processors and a computer-readable storage medium communicatively coupled to the one or more processors, wherein the computer-readable storage medium is executable by the one or more processors and stores computer-readable instructions that, when executed by the one or more processors, perform the method described in the first or second aspect.

[0010] A sixth aspect of this disclosure provides a non-temporary, computer-readable storage medium for storing a video bitstream. Once the bitstream is encoded by an encoder, the encoder is instructed to perform the method described in the first aspect.

[0011] A seventh aspect of this disclosure provides a non-temporary, computer-readable storage medium for storing a video bitstream. Once the bitstream is decoded by a decoder, the decoder is instructed to perform the method described in the second aspect.

[0012] In an eighth aspect of this disclosure, a computer program product is provided which includes computer program instructions, the computer program instructions enabling a computer to perform the method described in the first or second aspect.

[0013] In a ninth aspect of the present disclosure, a computer program is provided that enables a computer to execute the method described in the first aspect or the second aspect.

Brief Description of the Drawings

[0014] Embodiments and various aspects of the present disclosure are shown in the following detailed description and the accompanying drawings. The various features shown in the drawings are not drawn to scale.

[0015] [Figure 1] FIG. 1 is a schematic diagram showing an exemplary system for preprocessing and encoding image data according to some embodiments of the present disclosure.

[0016] [Figure 2A] FIG. 2 is a schematic diagram showing an exemplary encoding process of a hybrid video coding system consistent with an embodiment of the present disclosure.

[0017] [Figure 2B] FIG. 3 is a schematic diagram showing another exemplary encoding process of a hybrid video coding system consistent with an embodiment of the present disclosure.

[0018] [Figure 3A] FIG. 4 is a schematic diagram showing an exemplary decoding process of a hybrid video coding system consistent with an embodiment of the present disclosure.

[0019] [Figure 3B] FIG. 5 is a schematic diagram showing another exemplary decoding process of a hybrid video coding system consistent with an embodiment of the present disclosure.

[0020] [Figure 4] FIG. 6 is a block diagram of an exemplary apparatus for preprocessing or encoding image data according to some embodiments of the present disclosure.

[0021] [Figure 5]Examples used to derive linear model parameters according to some embodiments of this disclosure are shown.

[0022] [Figure 6] This is a decoder block diagram of a block-based hybrid video coding system according to some embodiments of the present disclosure.

[0023] [Figure 7] Four Sobel-based gradient patterns for gradient linear models (GLMs) according to some embodiments of this disclosure are shown.

[0024] [Figure 8] The spatial portion of a convolutional filter according to some embodiments of this disclosure is shown.

[0025] [Figure 9] The following shows a reference region (and its padding) used to derive filter coefficients according to some embodiments of this disclosure.

[0026] [Figure 10] The following are some undownsampled Luma samples according to some embodiments of this disclosure.

[0027] [Figure 11] The following are spatial samples used in gradient and position-based convolutional cross-component models (GL-CCCM) according to some embodiments of this disclosure.

[0028] [Figure 12] The process of the cross-component residual model (CCRM) on the decoder side according to some embodiments of this disclosure is shown.

[0029] [Figure 13] The correspondence between chromatic and luma samples according to some embodiments of this disclosure is shown.

[0030] [Figure 14] This disclosure shows another correspondence between chromatic and luma samples according to some embodiments of this disclosure.

[0031] [Figure 15] This disclosure describes a process for using multiple models in CCRM according to some embodiments of this disclosure.

[0032] [Figure 16] The following are exemplary methods for encoding a video sequence into a bitstream, according to some embodiments of the present disclosure.

[0033] [Figure 17] Substeps of the method shown in Figure 16, according to some embodiments of this disclosure, are shown.

[0034] [Figure 18] Substeps of the method shown in Figure 16, according to some embodiments of this disclosure, are shown.

[0035] [Figure 19] The present disclosure describes exemplary methods for decoding a bitstream and outputting one or more pictures of a video stream, according to some embodiments of this disclosure.

[0036] [Figure 20] The following are exemplary methods for encoding a video sequence into a bitstream, according to some embodiments of the present disclosure.

[0037] [Figure 21] The present disclosure describes exemplary methods for decoding a bitstream and outputting one or more pictures of a video stream, according to some embodiments of this disclosure.

[0038] [Figure 22]The following are exemplary methods for encoding a video sequence into a bitstream, according to some embodiments of the present disclosure.

[0039] [Figure 23] The present disclosure describes exemplary methods for decoding a bitstream and outputting one or more pictures of a video stream, according to some embodiments of this disclosure.

[0040] [Figure 24] The following are exemplary methods for encoding a video sequence into a bitstream, according to some embodiments of the present disclosure.

[0041] [Figure 25] The present disclosure describes exemplary methods for decoding a bitstream and outputting one or more pictures of a video stream, according to some embodiments of this disclosure. [Modes for carrying out the invention]

[0042] Herein, exemplary embodiments shown in the accompanying drawings will be described in detail. The following description refers to the accompanying drawings, in which, unless otherwise noted, the same number in different drawings represents the same or similar elements. The embodiments described below in the description of exemplary embodiments do not represent all embodiments consistent with the present disclosure. Rather, these embodiments are merely examples of apparatus and methods consistent with aspects related to the present disclosure enumerated in the accompanying claims. Specific aspects of the present disclosure will be described in more detail below. In the event of any conflict between terms and / or definitions incorporated by reference and those provided herein, the terms and definitions provided herein shall prevail.

[0043] The Joint Video Expert Team (JVET), comprised of the ITU-T Video Coding Expert Group (ITU-TVCEG) and the ISO / IEC Video Expert Group (ISO / IECMPEG), is currently developing the Multipurpose Video Coding (VVC / H.266) standard. The VVC standard aims to double the compression efficiency of its predecessor, the High Efficiency Video Coding (HEVC / H.265) standard. In other words, the goal of VVC is to achieve the same subjective quality as HEVC / H.265 with half the bandwidth.

[0044] To achieve this goal, JVET has been developing a technology that surpasses HEVC since 2015, using the Joint Search Model (JEM) reference software. As coding techniques were incorporated into JEM, JEM achieved significantly higher coding performance than HEVC. In October 2017, VCEG and MPEG issued a Joint Request for Proposal (CfP), formally commencing the development of a next-generation video compression standard that surpasses HEVC. In April 2018, responses to the CfP were evaluated at the JVET conference in San Diego, and the formal development process for the VVC standard commenced in April 2018.

[0045] The VVC standard has been progressing steadily since April 2018, continuously incorporating more coding technologies to provide better compression performance. VVC is based on the same hybrid video coding system used in modern video compression standards such as HEVC, H.264 / AVC, MPEG2, and H.263.

[0046] Figure 1 is a block diagram showing a system 100 for preprocessing and encoding image data according to some disclosed embodiments. Image data may include images (also called “pictures” or “frames”), a set of images, or video. An image is a still picture. A set of images may or may not be spatially or temporally related. Video is a set of images arranged in chronological order.

[0047] As shown in Figure 1, the system 100 includes a source device 120 that provides encoded video data to be decoded at a later time by a destination device 140. In accordance with the disclosed embodiments, the source device 120 and the destination device 140 may each include any of a wide range of devices, including desktop computers, notebook (e.g., laptop) computers, servers, tablet computers, set-top boxes, mobile phones, vehicles, cameras, image sensors, robots, televisions, wearable devices (e.g., smartwatches or wearable cameras), display devices, digital media players, video game consoles, video streaming devices, and the like. The source device 120 and the destination device 140 may be equipped for wireless or wired communication.

[0048] Referring to Figure 1, the source device 120 may include an image / video preprocessor 122, an image / video encoder 124, and an output interface 126. The destination device 140 may include an input interface 142, an image / video decoder 144, and one or more machine vision applications 146. The image / video preprocessor 122 preprocesses image data, i.e., images or videos, to generate an input bitstream for the image / video encoder 124. The image / video encoder 124 encodes the input bitstream and outputs the encoded bitstream 162 via the output interface 126. The encoded bitstream 162 is transmitted via the communication medium 160 and received by the input interface 142. The image / video decoder 144 then decodes the encoded bitstream 162 to generate decoded data that can be used by the machine vision application 146.

[0049] More specifically, the source device 120 may further include various devices (not shown) for providing source image data to be preprocessed by the image / video preprocessor 122. Devices for providing source image data may include image / video capture devices such as cameras, image / video archive or storage devices containing previously captured images / videos, or image / video feed interfaces for receiving images / videos from image / video content providers.

[0050] The image / video encoder 124 and the image / video decoder 144 can each be implemented as one or more suitable encoder or decoder circuits, including one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. If encoding or decoding is partially implemented in software, the image / video encoder 124 or image / video decoder 144 can perform the technology according to this disclosure by storing instructions for the software in a suitable non-temporary computer-readable medium and executing these instructions in hardware using one or more processors. The image / video encoder 124 or image / video decoder 144 can each be included in one or more encoders or decoders, and any of them can be integrated as part of a combined encoder / decoder (CODEC) in their respective devices.

[0051] The image / video encoder 124 and image / video decoder 144 may operate according to any video coding standard, such as Advanced Video Coding (AVC), High Efficiency Video Coding (HEVC), Multipurpose Video Coding (VVC), AOMedia Video 1 (AV1), Joint Photo Professional Group (JPEG), and Video Professional Group (MPEG). Alternatively, the image / video encoder 124 and image / video decoder 144 may be customized devices that do not conform to existing standards. Although not shown in Figure 1, in some embodiments, the image / video encoder 124 and image / video decoder 144 may be integrated with an audio encoder and decoder, respectively, and may include a suitable MUX-DEMUX unit or other hardware and software to handle the encoding of both audio and video, either in a common data stream or in separate data streams.

[0052] The output interface 126 may include any type of medium or device capable of transmitting the encoded bitstream 162 from the source device 120 to the destination device 140. For example, the output interface 126 may include a transmitter or transceiver configured to transmit the encoded bitstream 162 directly from the source device 120 to the destination device 140 in real time. The encoded bitstream 162 may be modulated according to a communication standard such as a wireless communication protocol and transmitted to the destination device 140.

[0053] The communication medium 160 may include a temporary medium such as a wireless broadcast or a wired network transmission. For example, the communication medium 160 may include a radio frequency (RF) spectrum or one or more physical transmission lines (e.g., a cable). The communication medium 160 may form part of a packet-based network such as a local area network, a wide area network, or a global network such as the Internet. In some embodiments, the communication medium 160 may include a router, a switch, a base station, or any other equipment that may be useful in facilitating communication from the source device 120 to the destination device 140. For example, a network server (not shown) may receive an encoded bitstream 162 from the source device 120, for example, via network transmission, and provide the encoded bitstream 162 to the destination device 140.

[0054] The communication medium 160 may also be in the form of a storage medium (e.g., a non-temporary storage medium), such as a hard disk, flash drive, compact disc, digital video disc, Blu-ray disc, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded image data. In some embodiments, a computing device of a media manufacturing facility, such as a disc stamping machine, may receive encoded image data from the source device 120 and generate a disc containing the encoded video data.

[0055] The input interface 142 may include any type of medium or device capable of receiving information from the communication medium 160. The received information includes an encoded bitstream 162. For example, the input interface 142 may include a receiver or transceiver configured to receive the encoded bitstream 162 in real time.

[0056] The machine vision application 146 includes various hardware or software for utilizing the decoded image data generated by the image / video decoder 144. For example, the machine vision application 146 may include a display device for displaying the decoded image data to a user, and may include any of the various display devices such as a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, organic light-emitting diode (OLED) display, or other types of display devices. As another example, the machine vision application 146 may include one or more processors configured to use the decoded image data to perform various machine vision applications such as object recognition and tracking, face recognition, image matching, image / video search, augmented reality, robot vision and navigation, autonomous driving, 3D structure construction, stereo support, and motion tracking.

[0057] Next, exemplary image data encoding and decoding techniques will be explained in relation to Figures 2A-2B and 3A-3B.

[0058] Figure 2A shows a schematic diagram of an exemplary encoding process 200A consistent with embodiments of the present disclosure. For example, the encoding process 200A can be performed by an encoder, for example, the image / video encoder 124 in Figure 1. As shown in Figure 2A, the encoder can encode a video sequence 202 into a video bitstream 228 according to process 200A. The video sequence 202 may include a series of pictures (referred to as “original pictures”) arranged in chronological order. Each original picture in the video sequence 202 can be divided by the encoder into a basic processing unit, a basic processing subunit, or a region for processing. In some embodiments, the encoder can perform process 200A at the level of a basic processing unit for each original picture in the video sequence 202. For example, the encoder can perform process 200A iteratively, in which case the encoder can encode one basic processing unit in one iteration of process 200A. In some embodiments, the encoder can perform process 200A in parallel for the regions of each original picture in the video sequence 202.

[0059] In Figure 2A, the encoder can generate predicted data 206 and predicted BPU 208 by sending the basic processing unit of the original picture of the video sequence 202 (called the "original BPU") to the prediction stage 204. The encoder can generate residual BPU 210 by subtracting the predicted BPU 208 from the original BPU. The encoder can generate quantization conversion coefficients 216 by sending the residual BPU 210 to the conversion stage 212 and the quantization stage 214. The encoder can generate the video bitstream 228 by sending the predicted data 206 and quantization conversion coefficients 216 to the binary coding stage 226. Components 202, 204, 206, 208, 210, 212, 214, 216, 226, and 228 may be called the "forward path". During process 200A, after the quantization stage 214, the encoder can generate a reconstructed residual BPU 222 by sending the quantization conversion coefficients 216 to the inverse quantization stage 218 and the inverse conversion stage 220. The encoder can then generate a prediction criterion 224, which will be used for the next iteration of process 200A in the prediction stage 204, by adding the reconstructed residual BPU 222 to the prediction BPU 208. Components 218, 220, 222, and 224 of process 200A may be referred to as the “reconstruction path”. The reconstruction path can be used to ensure that both the encoder and decoder use the same reference data for prediction.

[0060] The encoder can iteratively perform process 200A to encode each original BPU of the original picture (in the forward path) and generate a predictive criterion 224 for encoding the next original BPU of the original picture (in the reconstruction path). After encoding all original BPUs of the original picture, the encoder can proceed to encoding the next picture in the video sequence 202.

[0061] Referring to process 200A, the encoder can receive a video sequence 202 generated by a video capture device (e.g., a camera). As used herein, the term “receive” may mean any act by any means of receiving, inputting, acquiring, retrieving, obtaining, reading, accessing, or inputting data.

[0062] In prediction stage 204, in the current iteration, the encoder receives the original BPU and prediction criterion 224 and can perform prediction operations to generate prediction data 206 and prediction BPU 208. The prediction criterion 224 can be generated from the reconstruction path in the previous iteration of process 200A. The objective of prediction stage 204 is to reduce information redundancy by extracting prediction data 206 that can be used to reconstruct the original BPU as prediction BPU 208 from the prediction data 206 and prediction criterion 224.

[0063] Ideally, the predicted BPU 208 can be identical to the original BPU. However, due to non-ideal prediction and reconstruction operations, the predicted BPU 208 is usually slightly different from the original BPU. To record such differences, the encoder can generate the residual BPU 210 by subtracting the predicted BPU 208 from the original BPU after generating it. For example, the encoder can subtract the pixel values ​​(e.g., grayscale or RGB values) of the predicted BPU 208 from the corresponding pixel values ​​of the original BPU. Each pixel in the residual BPU 210 may have a residual value resulting from such a subtraction between the corresponding pixels in the original BPU and the predicted BPU 208. Compared to the original BPU, the predicted data 206 and residual BPU 210 may have fewer bits, but they can be used to reconstruct the original BPU without a significant loss of quality. Thus, the original BPU is compressed.

[0064] To further compress the residual BPU210, in the conversion stage 212, the encoder can reduce the spatial redundancy of the residual BPU210 by decomposing it into a set of two-dimensional "base patterns," each associated with a "conversion coefficient." The base patterns can have the same size (e.g., the size of the residual BPU210). Each base pattern can represent the fluctuating frequency component (e.g., brightness fluctuation frequency) of the residual BPU210. No base pattern can be reproduced from any combination of any other base patterns (e.g., a linear combination). In other words, the decomposition allows the fluctuations of the residual BPU210 to be decomposed into the frequency domain. Such a decomposition is analogous to the discrete Fourier transform of a function, where the base patterns are analogous to the basis functions of the discrete Fourier transform (e.g., trigonometric functions), and the conversion coefficients are analogous to the coefficients associated with the basis functions.

[0065] Different transformation algorithms can use different base patterns. Various transformation algorithms can be used in the transformation stage 212, such as discrete cosine transform and discrete sine transform. The transformation in the transformation stage 212 is reversible; that is, the encoder can reconstruct the residual BPU 210 by performing the inverse operation of the transformation (called the "inverse transformation"). For example, to reconstruct the pixels of the residual BPU 210, the inverse transformation might involve multiplying the values ​​of the corresponding pixels in the base pattern by their respective associated coefficients and adding their products to produce a weighted sum. In video coding standards, both the encoder and decoder can use the same transformation algorithm (and therefore the same base pattern). Therefore, the encoder can record only the transformation coefficients, and the decoder can reconstruct the residual BPU 210 from these transformation coefficients without receiving the base pattern from the encoder. While the transformation coefficients may have fewer bits compared to the residual BPU 210, they can be used to reconstruct the residual BPU 210 without a significant loss of quality. Thus, the residual BPU 210 is further compressed.

[0066] The encoder can further compress the conversion coefficients in the quantization stage 214. In the conversion process, different base patterns can represent different fluctuation frequencies (e.g., brightness fluctuation frequencies). Generally, the human eye is better at recognizing low-frequency fluctuations, so the encoder can ignore information about high-frequency fluctuations without causing significant quality degradation during decoding. For example, in the quantization stage 214, the encoder can generate quantization conversion coefficients 216 by dividing each conversion coefficient by an integer value (called a "quantization parameter") and rounding the quotient to its nearest integer. After such an operation, some conversion coefficients for high-frequency base patterns may be converted to zero, and conversion coefficients for low-frequency base patterns may be converted to smaller integers. The encoder can ignore quantization conversion coefficients 216 with zero values, thereby further compressing the conversion coefficients. The quantization process is also reversible, in which the quantization conversion coefficients 216 can be reconstructed into conversion coefficients in the inverse operation of quantization (called "inverse quantization").

[0067] Because the encoder ignores the remainder of such division in rounding operations, the quantization stage 214 can be irreversible. Typically, the quantization stage 214 can cause the greatest information loss in process 200A. The greater the information loss, the fewer bits the quantization conversion coefficient 216 may require. To obtain different levels of information loss, the encoder can use different values ​​for the quantization parameters or any other parameters of the quantization process.

[0068] In the binary coding stage 226, the encoder can encode the predicted data 206 and quantization conversion coefficients 216 using binary coding techniques such as entropy coding, variable-length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other reversible or irreversible compression algorithm. In some embodiments, in addition to the predicted data 206 and quantization conversion coefficients 216, the encoder can encode other information in the binary coding stage 226, such as the prediction mode used in the prediction stage 204, parameters of the prediction operation, the conversion type in the conversion stage 212, parameters of the quantization process (e.g., quantization parameters), and encoder control parameters (e.g., bitrate control parameters). The encoder can use the output data from the binary coding stage 226 to generate a video bitstream 228. In some embodiments, the video bitstream 228 may be further packetized for network transmission.

[0069] Referring to the reconstruction path of process 200A, in the inverse quantization stage 218, the encoder can generate reconstruction transformation coefficients by performing inverse quantization on the quantization transformation coefficients 216. In the inverse transformation stage 220, the encoder can generate reconstruction residual BPU 222 based on the reconstruction transformation coefficients. The encoder can add the reconstruction residual BPU 222 to the prediction BPU 208 to generate a prediction criterion 224 to be used in the next iteration of process 200A.

[0070] It should be noted that other variations of process 200A can be used to encode video sequence 202. In some embodiments, the stages of process 200A may be executed in a different order by the encoder. In some embodiments, one or more stages of process 200A may be combined into a single stage. In some embodiments, a single stage of process 200A may be divided into multiple stages. For example, the conversion stage 212 and the quantization stage 214 may be combined into a single stage. In some embodiments, process 200A may include additional stages. In some embodiments, process 200A may omit one or more stages in Figure 2A.

[0071] Figure 2B shows a schematic diagram of another exemplary coding process 200B consistent with embodiments of the present disclosure. For example, coding process 200B can be performed by an encoder, e.g., the image / video encoder 124 in Figure 1. Process 200B may be a modification of process 200A. For example, process 200B may be used by an encoder compliant with a hybrid video coding standard (e.g., the H.26x series). Compared to process 200A, the forward path of process 200B further includes a mode determination stage 230 and divides the prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044. The reconstruction path of process 200B further includes a loop filter stage 232 and a buffer 234.

[0072] Generally, prediction techniques can be classified into two types: spatial prediction and temporal prediction. Spatial prediction (e.g., intra-picture prediction or "intra-prediction") can predict the current BPU by using pixels from one or more already coded neighboring BPUs in the same picture. That is, the prediction criterion 224 in spatial prediction can include neighboring BPUs. Spatial prediction can reduce the inherent spatial redundancy of a picture. Temporal prediction (e.g., inter-picture prediction or "inter-prediction") can predict the current BPU by using regions from one or more already coded pictures. That is, the prediction criterion 224 in temporal prediction can include coded pictures. Temporal prediction can reduce the inherent temporal redundancy of a picture.

[0073] Referring to process 200B, in the forward path, the encoder performs prediction operations in the spatial prediction stage 2042 and the temporal prediction stage 2044. For example, in the spatial prediction stage 2042, the encoder can perform intra-prediction. For the original BPU of the picture being encoded, the prediction criterion 224 can include one or more adjacent BPUs that are encoded (in the forward path) and reconstructed (in the reconstruction path) in the same picture. The encoder can generate the prediction BPU 208 by extrapolating adjacent BPUs. Extrapolation techniques may include, for example, linear extrapolation or interpolation, or polynomial extrapolation or interpolation. In some embodiments, the encoder can perform extrapolation at the pixel level, such as by extrapolating the values ​​of the corresponding pixels for each pixel of the prediction BPU 208. The adjacent BPU used for extrapolation can be positioned from various directions relative to the original BPU, such as vertically (e.g., above the original BPU), horizontally (e.g., to the left of the original BPU), diagonally (e.g., below left, below right, above left, or above right of the original BPU), or in any direction defined by the video coding standard used. In intra-prediction, the prediction data 206 may include, for example, the location (e.g., coordinates) of the adjacent BPU used, the size of the adjacent BPU used, the extrapolation parameters, and the orientation of the adjacent BPU used relative to the original BPU.

[0074] As another example, in the temporal prediction stage 2044, the encoder can perform interpretation. For the original BPU of the current picture, the prediction criterion 224 may include one or more pictures (referred to as "reference pictures") that have been encoded (in the forward path) and reconstructed (in the reconstruction path). In some embodiments, the reference pictures may be encoded and reconstructed for each BPU. For example, the encoder may generate a reconstructed BPU by adding the reconstructed residual BPU 222 to the prediction BPU 208. Once all the reconstructed BPUs for the same picture have been generated, the encoder can generate a reconstructed picture as a reference picture. The encoder may perform a "motion estimation" operation to search for a matching region within the range of the reference picture (referred to as the "search window"). The position of the search window in the reference picture may be determined based on the position of the original BPU in the current picture. For example, the search window may be centered at a position in the reference picture that has the same coordinates as the original BPU in the current picture and may extend outward by a predetermined distance. When the encoder identifies a region in the search window that is similar to the original BPU (for example, by using a pixel recursion algorithm, a block matching algorithm, etc.), the encoder can determine such a region as a matching region. The matching region may have different dimensions from the original BPU (for example, smaller than, equal to, larger than, or different in shape from the original BPU). Because the reference picture and the current picture are separated in time on the timeline, the matching region can be considered to "move" to the position of the original BPU over time. The encoder can record the direction and distance of such movement as a "motion vector". If multiple reference pictures are used, the encoder can search for a matching region for each reference picture and determine the associated motion vector. In some embodiments, the encoder can assign weights to the pixel values ​​of the matching region of each matching reference picture.

[0075] Motion estimation can be used to identify various types of motion, such as translation, rotation, and zooming. In interpretation, the prediction data 206 may include, for example, the location (e.g., coordinates) of the matching region, the motion vector associated with the matching region, the number of reference pictures, and the weights associated with the reference pictures.

[0076] To generate a predicted BPU 208, the encoder can perform a “motion compensation” operation. Motion compensation can be used to reconstruct the predicted BPU 208 based on prediction data 206 (e.g., motion vectors) and prediction criteria 224. For example, the encoder can move the matching region of a reference picture according to the motion vector, thereby allowing the encoder to predict the original BPU of the current picture. If multiple reference pictures are used, the encoder can move the matching region of the reference pictures according to the respective motion vector and average pixel value of the matching region. In some embodiments, if the encoder has assigned weights to the pixel values ​​of the matching region of each matching reference picture, the encoder can add the weighted sum of the pixel values ​​of the moved matching region.

[0077] In some embodiments, the interpretation may be unidirectional or bidirectional. Unidirectional interpretation can use one or more reference pictures that are in the same time direction relative to the current picture. Unidirectional interpretation uses a reference picture that precedes the current picture. Bidirectional interpretation can use one or more reference pictures that are in both directions in the time direction relative to the current picture.

[0078] Continuing to refer to the forward path of process 200B, after the spatial prediction 2042 and the temporal prediction stage 2044, in the mode determination stage 230, the encoder can select a prediction mode (e.g., either intra-prediction or inter-prediction) for the current iteration of process 200B. For example, the encoder can perform a rate distortion optimization technique, in which the encoder can select a prediction mode based on the bit rate of a candidate prediction mode and the distortion of the reconstructed reference picture in the candidate prediction mode to minimize the value of the cost function. Depending on the selected prediction mode, the encoder can generate the corresponding prediction BPU 208 and prediction data 206.

[0079] In the reconstruction path of process 200B, if the intra-prediction mode is selected in the forward path, after generating the prediction criterion 224 (e.g., the current BPU encoded and reconstructed in the current picture), the encoder can send the prediction criterion 224 directly to the spatial prediction stage 2042 for later use (e.g., to extrapolate the next BPU of the current picture). If the inter-prediction mode is selected in the forward path, after generating the prediction criterion 224 (e.g., the current picture encoded and reconstructed with all BPUs), the encoder can send the prediction criterion 224 to the loop filter stage 232, where the encoder can apply a loop filter to the prediction criterion 224 to reduce or remove distortions (e.g., blocking artifacts) introduced by inter-prediction. In the loop filter stage 232, the encoder can apply various loop filtering techniques, such as deblocking, sample-adaptive offset, or adaptive loop filtering. Loop-filtered reference pictures can be stored in buffer 234 (or “Decoded Picture Buffer”) for later use (for example, as inter-prediction reference pictures for future pictures in video sequence 202). The encoder may store one or more reference pictures in buffer 234 for use in the temporal prediction stage 2044. In some embodiments, the encoder may encode the loop filter parameters (e.g., the strength of the loop filter) along with the quantization transformation coefficients 216, the prediction data 206, and other information in the binary coding stage 226.

[0080] In some embodiments, the input video sequence 202 is processed block by block according to the encoding process 200B. In VVC, the coded tree unit (CTU) is the largest block unit and can be 128 × 128 larger chroma samples (and corresponding chroma samples depending on the chroma format). The CTU can be further divided into coding units (CUs) by a quadtree, binary tree, or ternary tree. At the leaf nodes of the partitioned structure, coded information such as the coding mode (intra-mode or inter-mode), motion information when intercoded (reference index, motion vector difference, etc.), and quantized conversion coefficients 216 are transmitted. When intra-prediction (also called spatial prediction) is used, the current block is predicted using spatially adjacent samples. When inter-prediction (also called temporal prediction or motion-compensated prediction) is used, the current block is predicted using samples from an already coded picture called a reference picture. Inter-prediction may use unidirectional or bidirectional prediction. In unidirectional prediction, only one motion vector pointing to one reference picture is used to generate a prediction signal for the current block. On the other hand, in bidirectional prediction, two motion vectors, each pointing to its own reference picture, are used to generate a prediction signal for the current block. The motion vectors and reference index are sent to the decoder to identify where the prediction signal(s) for the current block came from. After intra-prediction or inter-prediction, the mode determination stage 230 selects the best prediction mode for the current block, for example, based on a rate-distortion optimization method. Based on the best prediction mode, a prediction BPU 208 is generated and subtracted from the input video block.

[0081] Continuing to refer to Figure 2B, the predicted residual BPU 210 is sent to the transformation stage 212 and the quantization stage 214 to generate the quantization transformation coefficients 216. The quantization transformation coefficients 216 are then dequantized in the dequantization stage 218 and inversely transformed in the inverse transformation stage 220 to obtain the reconstructed residual BPU 222. The predicted BPU 208 and the reconstructed residual BPU 222 are added together before loop filtering to form a prediction criterion 224, which is used to provide a reference sample for the intra-prediction. Loop filtering such as deblocking, sample adaptive offset (SAO), and adaptive loop filtering (ALF) may be applied to the prediction criterion 224 in the loop filtering stage 232, thereby forming a reconstructed block that is stored in buffer 234 and used to provide a reference sample for the intra-prediction. The coded information generated in the mode determination stage 230, such as the coding mode (intra or inter-predictive), intra-predictive mode, motion information, and quantization residual coefficients, is sent to the binary coding stage 226 to further reduce the bitrate before being packed into the output video bitstream 228.

[0082] Figure 3A shows a schematic diagram of an exemplary decoding process 300A consistent with embodiments of the present disclosure. For example, the decoding process 300A can be performed by a decoder, e.g., the image / video decoder 144 in Figure 1. Process 300A may be a decompression process corresponding to the compression process 200A in Figure 2A. In some embodiments, process 300A may be analogous to the reconstruction path of process 200A. The decoder (e.g., the image / video decoder 144 in Figure 1) can decode the video bitstream 228 into a video stream 304 according to process 300A. The video stream 304 may be very similar to the video sequence 202. However, due to the loss of information in the compression and decompression processes (e.g., the quantization stage 214 in Figures 2A and 2B), the video stream 304 is usually not identical to the video sequence 202. Similar to processes 200A and 200B in Figures 2A and 2B, the decoder can execute process 300A at the level of the basic processing unit (BPU) for each picture encoded in the video bitstream 228. For example, the decoder can execute process 300A iteratively, in which case the decoder can decode one basic processing unit in one iteration of process 300A. In some embodiments, the decoder can execute process 300A in parallel for each region of each picture encoded in the video bitstream 228.

[0083] In Figure 3A, the decoder can send a portion of the video bitstream 228 associated with the basic processing unit of the encoded picture (called the "encoded BPU") to the binary decoding stage 302. In the binary decoding stage 302, the decoder can decode this portion into prediction data 206 and quantization conversion coefficients 216. The decoder can generate a reconstructed residual BPU 222 by sending the quantization conversion coefficients 216 to the inverse quantization stage 218 and the inverse transformation stage 220. The decoder can generate a prediction BPU 208 by sending the prediction data 206 to the prediction stage 204. The decoder can generate a prediction criterion 224 by adding the reconstructed residual BPU 222 to the prediction BPU 208. In some embodiments, the prediction criterion 224 can be stored in a buffer (e.g., a decoded picture buffer in computer memory). The decoder can send the prediction criterion 224 to the prediction stage 204 to perform a prediction operation in the next iteration of process 300A.

[0084] The decoder can decode each encoding BPU of the encoded picture by iteratively performing process 300A and generate a prediction criterion 224 to encode the next encoding BPU of the encoded picture. After decoding all encoding BPUs of the encoded picture, the decoder can output the picture to the video stream 304 for display and proceed to decode the next encoded picture in the video bitstream 228.

[0085] In the binary decoding stage 302, the decoder can perform the inverse operation of the binary coding technique used by the encoder (e.g., entropy coding, variable-length coding, arithmetic coding, Huffman coding, context-adaptive binary arithmetic coding, or any other lossless compression algorithm). In some embodiments, in the binary decoding stage 302, the decoder can decode other information in addition to the predicted data 206 and quantization conversion coefficients 216, such as the prediction mode, parameters of the prediction operation, conversion type, parameters of the quantization process (e.g., quantization parameters), and encoder control parameters (e.g., bitrate control parameters). In some embodiments, if the video bitstream 228 is transmitted in packet form over the network, the decoder can depacketize the video bitstream 228 before sending it to the binary decoding stage 302.

[0086] Figure 3B shows a schematic diagram of another exemplary decoding process 300B consistent with embodiments of the present disclosure. For example, decoding process 300B can be performed by a decoder, e.g., the image / video decoder 144 in Figure 1. Process 300B may be a modification of process 300A. For example, process 300B may be used by a decoder compliant with a hybrid video coding standard (e.g., the H.26x series). Compared to process 300A, process 300B additionally divides the prediction stage 204 into a spatial prediction stage 2042 and a temporal prediction stage 2044, and further includes a loop filter stage 232 and a buffer 234.

[0087] In process 300B, for the encoding base processing unit ("current BPU") of the encoded picture being decoded ("current picture"), the prediction data 206 decoded by the decoder from the binary decoding stage 302 may contain various types of data depending on which prediction mode was used by the encoder to encode the current BPU. For example, if intra-prediction was used by the encoder to encode the current BPU, the prediction data 206 may include a prediction mode indicator (e.g., a flag value) indicating intra-prediction, parameters of the intra-prediction operation, etc. Parameters of the intra-prediction operation may include, for example, the location (e.g., coordinates) of one or more adjacent BPUs used as reference, the size of the adjacent BPUs, extrapolation parameters, the orientation of the adjacent BPUs relative to the original BPU, etc. As another example, if inter-prediction was used by the encoder to encode the current BPU, the prediction data 206 may include a prediction mode indicator (e.g., a flag value) indicating inter-prediction, parameters of the inter-prediction operation, etc. Parameters for interpretation operation may include, for example, the number of reference pictures associated with the current BPU, the weights associated with each reference picture, the location (e.g., coordinates) of one or more matching regions in each reference picture, and one or more motion vectors associated with each matching region.

[0088] Based on the prediction mode indicator, the decoder can decide whether to perform a spatial prediction (e.g., intra-prediction) in the spatial prediction stage 2042 or a temporal prediction (e.g., inter-prediction) in the temporal prediction stage 2044. Details of performing such spatial or temporal predictions are shown in Figure 2B and will not be repeated below. After performing such spatial or temporal predictions, the decoder can generate a prediction BPU 208. The decoder can generate a prediction criterion 224 by adding the prediction BPU 208 and the reconstructed residual BPU 222, as shown in Figure 3A.

[0089] In process 300B, the decoder can send the prediction criterion 224 to the spatial prediction stage 2042 or the temporal prediction stage 2044 to perform a prediction operation in the next iteration of process 300B. For example, if the current BPU is decoded using intra-prediction in the spatial prediction stage 2042, after generating the prediction criterion 224 (e.g., the decoded current BPU), the decoder can send the prediction criterion 224 directly to the spatial prediction stage 2042 for later use (e.g., to extrapolate the next BPU of the current picture). If the current BPU is decoded using inter-prediction in the temporal prediction stage 2044, after generating the prediction criterion 224 (e.g., the reference picture with all BPUs decoded), the encoder can send the prediction criterion 224 to the loop filter stage 232 to reduce or remove distortion (e.g., blocking artifacts). The decoder can apply a loop filter to the prediction criterion 224 in the manner shown in Figure 2B. The loop-filtered reference picture can be stored in buffer 234 (e.g., a decoded picture buffer in computer memory) for later use (e.g., as an inter-prediction reference picture for future encoded pictures of the video bitstream 228). The decoder may store one or more reference pictures in buffer 234 for use in the temporal prediction stage 2044. In some embodiments, the prediction data 206 may further include loop filter parameters (e.g., loop filter strength) if the prediction mode indicator indicates that inter-prediction was used to encode the current BPU. The reconstructed picture from buffer 234 is also transmitted to a display such as a television, PC, smartphone, or tablet for viewing and viewing by the end user.

[0090] Returning to Figure 1, each of the image / video preprocessor 122, image / video encoder 124, and image / video decoder 144 can be implemented as any suitable hardware, software, or combination thereof. Figure 4 is a block diagram of an exemplary apparatus 400 for processing image data, consistent with embodiments of the present disclosure. For example, apparatus 400 may be a preprocessor, encoder, or decoder. As shown in Figure 4, apparatus 400 may include a processor 402. When the processor 402 executes the instructions described herein, apparatus 400 can become a dedicated machine for preprocessing, encoding, and / or decoding image data. The processor 402 may be any type of circuit capable of manipulating or processing information. For example, the processor 402 may include any number and any combination of such components, including a central processing unit (or "CPU"), graphics processing unit (or "GPU"), neural processing unit ("NPU"), microcontroller unit ("MCU"), optical processor, programmable logic controller, microcontroller, microprocessor, digital signal processor, intellectual property (IP) core, programmable logic array (PLA), programmable array logic (PAL), generic array logic (GAL), complex programmable logic device (CPLD), field programmable gate array (FPGA), system-on-a-chip (SoC), and application-specific integrated circuit (ASIC). In some embodiments, the processor 402 may be a set of processors grouped as a single logic component. For example, as shown in Figure 4, the processor 402 may include multiple processors, including processor 402a, processor 402b, and processor 402n.

[0091] The device 400 may also include a memory 404 configured to store data (e.g., instruction sets, computer code, intermediate data, etc.). For example, as shown in Figure 4, the stored data may include program instructions (e.g., program instructions for implementing stages of process 200A, 200B, 300A, or 300B) and processing data (e.g., video sequence 202, video bitstream 228, or video stream 304). The processor 402 can access the program instructions and processing data (e.g., via bus 410), execute the program instructions, and perform operations or manipulations on the processing data. The memory 404 may include a high-speed random-access memory or a non-volatile memory. In some embodiments, the memory 404 may include any number and any combination of random-access memory (RAM), read-only memory (ROM), optical disks, magnetic disks, hard drives, solid-state drives, flash drives, security digital (SD) cards, memory sticks, compact flash (CF) cards, etc. Memory 404 may be a group of memories grouped as a single logical component (not shown in Figure 4).

[0092] Bus 410 may be a communication device that transfers data between components in device 400, such as an internal bus (e.g., CPU memory bus) or an external bus (e.g., a universal serial bus port, a peripheral component interconnection express port).

[0093] To facilitate explanation without creating ambiguity, the processor 402 and other data processing circuits are collectively referred to as “data processing circuits” in this disclosure. The data processing circuits may be implemented entirely in hardware, or as a combination of software, hardware, or firmware. Furthermore, the data processing circuits may be a single, independent module, or may be combined whole or partially with any other component of the device 400.

[0094] The device 400 may further include a network interface 406 for providing wired or wireless communication to a network (e.g., the Internet, an intranet, a local area network, a mobile communication network, etc.). In some embodiments, the network interface 406 may include any number and any combination of a network interface controller (NIC), radio frequency (RF) module, transponder, transceiver, modem, router, gateway, wired network adapter, wireless network adapter, Bluetooth adapter, infrared adapter, near-field communication ("NFC") adapter, cellular network chip, etc.

[0095] In some embodiments, the apparatus 400 may further include a peripheral interface 408 for providing connectivity to one or more peripheral devices. As shown in Figure 4, peripheral devices may include, but are not limited to, cursor control devices (e.g., mouse, touchpad, or touchscreen), keyboards, displays (e.g., cathode ray tube displays, liquid crystal displays, or light-emitting diode displays), video input devices (e.g., cameras, or input interfaces connected to video archives), and the like.

[0096] It should be noted that a video codec (for example, a codec that runs processes 200A, 200B, 300A, or 300B) may be implemented as any combination of any software or hardware modules in device 400. For example, some or all stages of processes 200A, 200B, 300A, or 300B may be implemented as one or more software modules of device 400, such as program instructions that can be loaded into memory 404. As another example, some or all stages of processes 200A, 200B, 300A, or 300B may be implemented as one or more hardware modules of device 400, such as dedicated data processing circuits (e.g., FPGA, ASIC, NPU, etc.).

[0097] Various cross-component prediction techniques can be used in video encoding and decoding to reduce signal redundancy between different components (e.g., luminous and chroma samples).

[0098] For example, VVC employs a cross-component linear model (CCLM) that represents the relationship between chroma and chroma components using a linear model. In this model, the chroma samples of a block can be predicted from collated reconstructed chroma samples using a linear model, as shown in Equation 1.

number

number

number

[0099] In the context of this disclosure, unless otherwise specified, "pred C "(x,y)" or "predC(x,y)" represents the predicted chroma value of a chroma sample with coordinates (x,y). Similarly, "pred L "(x,y)" or "predL(x,y)" represents the predicted luma value of a luma sample with coordinates (x,y). Also, "rec C "(x,y)" or "recC(x,y)" represents the reconstructed chroma value of a chroma sample with coordinates (x,y), and "rec L"(x,y)" or "recL(x,y)" represents the reconstructed luma value of a luma sample with coordinates (x,y).

[0100] VVC specifies three CCLM modes: CCLM_LT, CCLM_L, and CCLM_T. These three modes differ in the position of the reconstructed neighbor samples used to derive the linear model parameters (α and β). The upper reconstructed neighbor samples are involved in CCLM_T mode, and the left reconstructed neighbor samples are involved in CCLM_L mode. In CCLM_LT mode, both the upper and left reconstructed neighbor samples are used.

[0101] In Chroma Intra Mode signaling, a flag indicating whether or not CCLM is applied is first signaled. If the flag is signaled as true, then it is further signaled which of the three CCLM modes is applied.

[0102] In some embodiments, downsampling of reconstructed lumar samples is used for cross-component prediction. Specifically, two types of downsampling filters, shown in Equations 2 and 3, can be applied to lumar samples to match the chroma sample positions of video sequences in 4:2:0 or 4:2:2 color formats, both having a 2:1 downsampling ratio horizontally and vertically. Based on the SPS level flag, a two-dimensional 6-tap or 5-tap filter is applied to the lumar sample in the current block and its adjacent lumar samples. If the SPS level flag is equal to 1, the prediction process operates to design for chroma sample positions that are not vertically shifted relative to the corresponding lumar sample positions, and a 5-tap filter is specified to be used. If the SPS level flag is equal to 0, the prediction process operates to design for chroma sample positions that are shifted down by 0.5 lumar samples relative to the corresponding lumar sample positions, and a 6-tap filter is specified to be used. An exception is raised if the top row of the current block is a CTU boundary. In this case, to avoid using two or more luma lines above the CTU boundary, a one-dimensional three-tap filter, as shown in Equation 4, can be applied to the upper adjacent luma samples.

number

[0103] The downsampling process using the above filters can be expressed by the following equations, where equations 5, 6, and 7 correspond to the filters in equations 2, 3, and 4, respectively.

number

number

[0104] The linear model parameters α and β are derived based on reconstructed adjacent chroma samples and their corresponding reconstructed luma samples, which are downsampled on both the encoder and decoder sides in the case of non-4:4:4 color formats to avoid signaling overhead.

[0105] In the initial version of the CCLM mode, a linear least mean squared error (LMMSE) estimator was used to derive the parameters.

number

number

[0106] In some embodiments, some method is used to increase or decrease the number of samples to ensure that the number of samples used to derive the linear model parameters is a power of two.

[0107] However, in the final design, only four samples are involved to reduce the computational complexity. For a W×H chroma block, the four samples used in the CCLM_LT mode are the samples at positions W / 4 and 3W / 4 on the upper boundary, and H / 4 and 3H / 4 on the left boundary. In the CCLM_T mode and the CCLM_L mode, the upper and left boundaries are extended to the size of (W + H) samples, and the four samples used for deriving the model parameters are at positions (W + H) / 8, 3(W + H) / 8, 5(W + H) / 8, and 7(W + H) / 8. For example, in the case of an 8x8 chroma CU, the samples used are indicated by circles in Figure 6.

[0108] The four downsampled reconstructed adjacent luma samples at the selected positions are compared four times,

Number

Number

Number

Number

Number

Number

Number

number

[0109] Finally, the linear model parameters α and β are obtained by the following equations.

number

[0110] The division operation to calculate the parameter α is implemented using a lookup table. To reduce the memory required to store the table, the diff value (the difference between the maximum and minimum values) and the parameter α are expressed in exponential notation. For example, diff is approximated by a 4-bit effective part and an exponent. As a result, the table for 1 / diff is reduced to 16 elements for 16 mantissa values, as shown in the following equation. DivTable [ ] = {0,7,6,5,5,4,4,3,3,2,2,1,1,1,1,0} (Formula 13)

[0111] This has two advantages: it reduces the complexity of the calculations and also reduces the amount of memory needed to store the necessary tables.

[0112] In accordance with the disclosed embodiments, a multi-model CCLM (MMLM) may be used for cross-component prediction. Specifically, the CCLM included in the VVC is extended by introducing a multi-model to the CU. That is, the samples within the CU are divided into different groups, and each group has a linear model for prediction. Depending on the adjacent reconstructed samples used for model derivation, there are also different modes of the multi-model CCLM: MMLM_LT, MMLM_L, and MMLM_T. The difference between the three modes lies in the position of the reconstructed adjacent samples used for deriving the linear model parameters (α and β), which is the same as the difference between the CCLM_LT, CCLM_L, and CCLM_T modes. In each MMLM mode, there may be two or more linear models between the luma and chroma in a single block. First, the reconstructed adjacent samples are classified into two classes by a threshold which is the mean of the values ​​of the reconstructed adjacent luma samples. Next, each class is treated as an independent training set, and a linear model is derived using the LMMSE method described above. Next, the reconstructed luma samples of the current block are classified based on the same rule. Finally, the reconstructed luma samples allow for predictions to be made to the chroma samples in different classes and in different ways.

[0113] In accordance with the disclosed embodiments, a gradient linear model (GLM) method may be used. Compared to CCLM, GLM derives a linear model using the gradient of the Luma samples instead of downsampling the reconstructed Luma samples. In other words, in the CCLM process, the gradient G is used instead of the filters in Equations 2, 3, and 4. Other design elements of CCLM (e.g., parameter derivation, linear transformation of predicted samples) remain unchanged.

[0114] Two modes of GLM are supported: a 2-parameter GLM mode and a 3-parameter GLM mode.

[0115] Compared to CCLM, the two-parameter GLM derives a linear model using luma sample gradients instead of downsampled luma values. Specifically, applying the two-parameter GLM replaces the input to the CCLM process, i.e., the downsampled luma samples L, with luma sample gradients G. Other parts of the CCLM (e.g., parameter derivation, linear transformation of predicted samples) remain unchanged. C = α·G + β (Equation 14)

[0116] In a 3-parameter GLM, chroma samples can be predicted based on both the gradient of the luma sample and downsampled luma values ​​with different parameters. The model parameters for the 3-parameter GLM are derived from adjacent 6x6 samples by an MSE minimization method based on LDL decomposition, as used in CCCM. C=α0·G+α1·L+α2·β (Formula 15)

[0117] The gradient G can be calculated using one of the four Sobel-based gradient patterns shown in Figure 7 and the following equation.

number

[0118] Using the gradient patterns described above, the gradient G can be calculated by the following equation, where equations 20, 21, 22, and 23 correspond to the gradient patterns of equations 16, 17, 18, and 19, respectively.

number

[0119] The linear model parameters α and β are derived in the same way as CCLM, for example, by the LMMSE method, based on the corresponding gradients G of the reconstructed adjacent chroma samples and the collated reconstructed chroma samples on both the encoder and decoder sides. Then, the chroma samples of a block can be predicted by the linear model from the gradients of the collated reconstructed chroma samples as follows: Nod C (i,j) = α·G L (i,j)+β (Equation 24)

[0120] Regarding signaling, when CCLM mode is enabled for the current CU, two flags are individually signaled for the Cb and Cr components to indicate whether GLM is enabled for the component. If GLM is enabled for a component, one more syntax element is further signaled to select one of four gradient patterns for gradient calculation. In some embodiments, a 2-bit fixed-length code is used to encode the syntax element.

[0121] In some embodiments, GLM is available only in certain CCLM modes. For example, GLM is available only in CCLM_LT mode, i.e., in CCLM_LT mode, several syntax elements are signaled to indicate whether GLM is enabled and which gradient pattern is used. When GLM is enabled in CCLM_LT mode, in the parameter derivation process of the linear model, the gradient G of the upper and left adjacent reconstructed luma samples is used to replace the downsampled reconstructed adjacent luma samples, and only the signal-linear model is used in the current block. When GLM is enabled in CCLM_LT mode, the original CCLM_LT mode applies. There are no changes for the other CCLM modes (i.e., CCLM_L, CCLM_T, and the three MMLM modes). As another example, GLM is available only in CCLM_LT and MMLM_LT modes. As yet another example, GLM is available only in CCLM_LT, CCLM_L, and CCLM_T modes. As yet another example, GLM is available in all six CCLM modes.

[0122] When applying GLM to MMLM mode, the multi-model GLM (MMGLM) method is used. In MMGLM mode, there may be two or more linear models between the gradient G and chroma in a single block. First, the gradients of reconstructed neighboring samples are classified into two classes by a threshold, which is the average of the gradient values ​​of the reconstructed neighboring chroma samples. Next, each class is treated as an independent training set, and a linear model is derived using the LMMSE method described above. Then, the gradients of the reconstructed chroma samples in the current block are classified based on the same rule. Finally, the gradients of the reconstructed chroma samples are used to make predictions for the chroma samples in different ways for different classes.

[0123] In some embodiments, the GLM method supports 16 gradient patterns. That is, the gradient G can be calculated using one of the 16 gradient patterns, as shown in the following equation. Syntax elements are signaled to indicate which gradient pattern is used.

number

[0124] In some embodiments, the linear model is derived using a combination of downsampled reconstructed chroma samples and the gradient of the reconstructed chroma samples. The linear model parameters are derived using the same method as CCLM, e.g., the LMMSE method, based on reconstructed adjacent chroma samples on both the encoder and decoder sides, the corresponding gradient G of the collated reconstructed chroma samples, and the downsampled reconstructed chroma samples. The value of the downsampled reconstructed chroma sample is obtained by one of the downsampled filters described above. Then, from the gradient of the collated reconstructed chroma sample and the value of the downsampled reconstructed chroma sample, the linear model can predict the chroma sample of a block as follows:

number

[0125] In some embodiments, a flag is signaled to indicate whether the GLM method of Equation 24 or Equation 41 is used.

[0126] In accordance with the disclosed embodiments, a convolutional cross-component intra-predictive model may be used for cross-component prediction. In this method, a convolutional cross-component model (CCCM) is applied to predict chroma samples from reconstructed luma samples, in a similar spirit to that performed by the current CCLM mode. As with CCLM, when chroma subsampling is used, the reconstructed luma samples are downsampled to match a low-resolution chroma grid. As with CCLM, top, left, or top and left reference samples are used as templates for model derivation.

[0127] Furthermore, similar to CCLM, there is an option to use a single-model or multi-model variant of CCCM. In the multi-model variant, two models are used: one derived for samples exceeding the mean Luma reference value, and another derived for the remaining samples (following the spirit of the CCLM design). The multi-model CCCM mode may be selected for PUs with at least 128 available reference samples.

[0128] In convolutional cross-component intra-predictive models, various types of convolutional filters can be used. A 7-tap convolutional filter consists of a 5-tap positive-sign shape-space component, a nonlinear term, and a bias term. As shown in Figure 8, the input to the filter's 5-tap spatial component consists of the predicted chroma sample and a collated central (C) chroma sample, along with its adjacent samples to the north (N), south (S), left (W), and east (E), as shown below.

[0129] The nonlinear term P is expressed as the central luma sample C raised to the power of 2 and scaled to the range of sample values ​​in the content. P = (C * C + midVal) >> bitDepth (Equation 42)

[0130] In other words, for 10-bit content, the calculation is as follows: P = ( C*C + 512 ) >> 10 (Formula 43)

[0131] The bias term B represents the scalar offset between the input and output (similar to the offset term in CCLM) and is set to an intermediate chroma value (512 for 10-bit content).

[0132] The filter output is calculated as a convolution of the filter coefficients ci and the input value, and clipped to the range of valid chroma samples. predChromaVal = c0C + c1N + c2S + c3E + c4W + c5P + c6B (Equation 44)

[0133] Filter coefficient C i This is calculated by minimizing the MSE between the predicted chroma samples and the reconstructed chroma samples within the reference region. Figure 9 shows a reference region consisting of six rows of chroma samples at the top and left of the PU. The reference region is extended by 1 PU width to the right and 1 PU height down from the PU boundary. The region is adjusted to include only available samples. To support the "side samples" of the plus-shaped spatial filter, the extension to the region shown in blue is necessary and padding is applied where unavailable samples are located.

[0134] Minimizing the MSE is performed by computing the autocorrelation matrix for the Luma input and the cross-correlation vector between the Luma input and the Chroma output. The autocorrelation matrix is ​​LDL decomposed, and the final filter coefficients are computed using the back substitution method. This process closely follows the computation of ALF filter coefficients in ECM, but LDL decomposition is chosen over Cholesky decomposition to avoid the use of square root operations. In some embodiments, the filter coefficients can be computed using an approach based on Gaussian elimination.

[0135] The autocorrelation matrix is ​​calculated using reconstructed values ​​of lumens and chroma samples. Since these samples are full-range (e.g., 0 to 1023 for 10-bit content), the values ​​of the autocorrelation matrix are relatively large. This necessitates high-bit-depth operations during the calculation of model parameters. It has been proposed to remove fixed offsets from the lumens and chroma samples in each PU of each model. This reduces the magnitude of the values ​​used for model creation and may reduce the precision required for fixed-point arithmetic. As a result, it has been proposed to use 16-bit decimal precision instead of the 22-bit precision of the original CCCM implementation.

[0136] For simplicity, the reference sample value just outside the upper left corner of the PU is used as the offset (offsetLuma, offsetCb, and offsetCr). The sample values ​​used in both model creation and final prediction (i.e., Luma and Chroma in the reference region, and Luma in the current PU) are subtracted by these fixed values ​​as follows: C ’ = C - offsetLuma (Formula 45) N ’ = N - offsetLuma (Equation 46) S ’ = S - offsetLuma (Equation 47) E ’ = E - offsetLuma (Equation 48) W ’ = W - offsetLuma (Equation 49) P ’ = nonLinear(C ’ (Formula 50) B = midValue = 1 << (bitDepth - 1) (Equation 51) The chroma value is predicted using the following formula, where offsetChroma is equal to offsetCr for the Cr component and offsetCb for the Cb component. predChromaVal = c0C' + c1N' + c2S' + c3E' + c4W' + c5P' + c6B + offsetChroma (Formula 52)

[0137] To avoid any additional sample-level manipulation, the luma offset is removed during the interpolation of luma reference samples. This can be achieved, for example, by replacing the rounding term used in the interpolation of luma reference samples with an updated offset that includes both the rounding term and offsetLuma. The chroma offset can be removed by directly subtracting the chroma offset from the reference chroma sample. Alternatively, the effect of the chroma offset can be removed from the cross-component vector, thereby achieving the same result. To add the chroma offset back into the output of the convolutional prediction operation, the chroma offset is added to the bias term of the convolutional model.

[0138] While division is required for parameter calculations in CCCM models, it is not always considered beneficial to perform division. Division can be replaced with multiplication (with the scale factor) and shift operations, where the scale factor and shift number are calculated based on the denominator, similar to how they are used in CCLM parameter calculations.

[0139] A method for signaling CCCM is proposed in accordance with the disclosed embodiments. The use of CCCM mode is signaled by a CABAC-coded PU level flag. To support this, a new CABAC context is included. With regard to signaling, CCCM is considered a submode of CCLM; that is, the CCCM flag is signaled only when the intra-prediction mode is LM_CHROMA.

[0140] In accordance with the disclosed embodiments, undownsampled chroma samples can be used in CCCM. A CCCM mode is used that has a 3x2 filter using undownsampled chroma samples, comprising a 6-tap spatial term, four nonlinear terms, and a bias term, as shown in Equation 53. The above 6-tap spatial term corresponds to six adjacent chroma samples (i.e., L0, L1, ..., L5) around the predicted chroma sample (i.e., C), and the above four nonlinear terms are derived from samples L0, L1, L2, and L3, as shown in Figure 10.

number

[0141] In accordance with the disclosed embodiments, a gradient and position-based convolutional cross-component model (GL-CCCM) may be used. This method maps the lumens to chromens using a filter with an input consisting of one spatial lumens sample, two gradient values, two positional information, a nonlinear term, and a bias term. The GL-CCCM method uses gradients and positional information instead of the four spatial neighboring samples used in the CCCM filter. The GL-CCCM filter used for prediction is given by the following equation:

number

number

[0142] Figure 11 shows a spatial sample used in a gradient and position-based convolutional cross-component model (GL-CCCM).

[0143] Furthermore, Y and X are the spatial coordinates of the central Luma sample.

[0144] The remaining parameters are the same as those in the CCCM tool. The reference area for parameter calculations is the same as in the CCCM method.

[0145] Mode usage is signaled by a CABAC-coded PU level flag. Regarding signaling, GL-CCCM is considered a submode of CCCM; that is, the GL-CCCM flag is signaled only when the original CCCM flag is true.

[0146] Similar to CCCM, the GL-CCCM tool has the following six modes for calculating parameters: • Single model GL-CCCM from top and left-hand templates • Single model GL-CCCM from the template above • Single model GL-CCCM from the template on the left. • Multi-model GL-CCCM from the top and left-hand templates • Multi-model GL-CCCM from the template above • Multimodel GL-CCCM from the template on the left

[0147] The encoder performs a SATD search against six GL-CCCM modes, along with the existing CCCM mode, to find the best candidate for a complete RD test.

[0148] In accordance with the disclosed embodiments, the cross-component residual model (CCRM) method may be used. CCRM is a cross-component prediction tool for inter-slicing that predicts chroma samples from reconstructed luma samples when the current block uses inter-prediction or intra-block copying (IBC). Figure 12 shows the CCRM process on the decoder side. Specifically, the current block first obtains the predicted luma value predL for the luma sample in the current block and the predicted chroma value predC for the chroma sample in the current block using inter-prediction mode or IBC mode (where predC can represent the predicted Cb value or predicted Cr value, also called the “original” predicted chroma value). Here, for brevity, the predicted luma value and predicted chroma value may be referred to as “predicted values” and can be distinguished depending on the context. By performing the CCRM method, the cross-component model is derived from predL and predC. Next, the derived cross-component model is applied to the reconstructed luma value recL of the luma sample in the current block, obtained by adding the luma residual to predL, to generate another predicted chroma value predC' (also called the predicted chroma value) of the chroma sample in the current block. Finally, predC and predC' are merged (i.e., weighted averaged) to obtain the final predicted chroma value precC of the chroma sample in the current block, as shown in Equation 57. * A (also called the "final" predicted chroma value) is generated. Here, according to the CCRM proposal, w0 is equal to 1, w1 is equal to 3, offset is equal to 2, and shift is equal to 2. According to the embodiment of the example, CCRM does not have a fusion process, and predC' is used as the final predicted chroma value of the chroma sample in the current block.

number

[0149] In some embodiments, an n-tap filter may be used as a cross-component model in CCRM, which includes n-2 spatial luma samples, a nonlinear term, and a bias term. The value of n and the position of the luma samples used are determined based on the color format of the image.

[0150] For a 4:4:4 color format, the value of n is equal to 3, which means that only one spatial chroma sample is used in the cross-component model. For example, a chroma sample with coordinates (i,J) can be predicted as shown in Equation 58, where i and j represent the horizontal and vertical distances between the current chroma sample and the chroma sample at the top-left corner of the current frame, respectively. In Equation 58, recL(i,J) is the reconstructed chroma value of the spatial chroma sample corresponding to the current chroma sample, and P is,

number

number

number

number

[0151] For a 4:2:2 color format, the value of n is equal to 5, which means that three spatial chroma samples are used in the cross-component model. For example, a chroma sample with coordinates (i,j) can be predicted as shown in Equation 59, where recL(2i,j), recL(2i-1,j), and recL(2i+1,j) are the reconstructed chroma values ​​of the spatial chroma sample corresponding to the current chroma sample, and p is,

number

number

number

number

[0152] For the most widely used 4:2:0 video color format, the value of n and the position of the luma sample used are further determined based on the flag sps_chroma_vertical_collocated_flag. If the flag is equal to 1, it specifies that the prediction process should operate to design for chroma sample positions that are not vertically shifted relative to the corresponding luma sample positions. If the flag is equal to 0, it specifies that the prediction process should operate to design for chroma sample positions that are shifted downward by 0.5 luma samples relative to the corresponding luma sample positions.

[0153] For a 4:2:0 video color format, if sps_chroma_vertical_collocated_flag is equal to 1, the value of n is equal to 7, which means that five spatial chroma samples are used in the cross-component model. For example, a chroma sample with coordinates (i,j) can be predicted as shown in Equation 60, where recL(2i,2j), recL(2i-1,2j), recL(2i+1,2j), recL(2i,2j+1), and recL(2i,2j-1) are the reconstructed chroma values ​​of the five spatial chroma samples corresponding to the current chroma sample, as shown in Figure 13, and P is,

number

number

number

number

[0154] For a 4:2:0 video color format, if sps_chroma_vertical_collocated_flag is equal to 0, the value of becomes equal to 8, meaning that six spatial chroma samples are used in the cross-component model. For example, a chroma sample with coordinates (i,j) can be predicted as shown in Equation 61. Here, according to the example embodiment, recL(2i,2j), recL(2i-1,2j), and recL(2i+1,2j), recL(2i,2j+1), recL(2i,2j+1), and recL(2i+1,2j+1) are the reconstructed chroma values ​​of the six spatial chroma samples corresponding to the current chroma sample, as shown in Figure 14, where P is,

number

number

number

[0155] According to the example embodiment, when applying the CCRM model to predict the current chroma sample, an offset is subtracted from each reconstructed chroma value of the chroma sample. For example, by using the offset, equation 61 can be rewritten as equation 62, where P is

number

number

[0156] The model parameters (filter coefficients) are derived using Gaussian elimination without division to minimize the mean squared error (MSE), based on predL and predC.

[0157] During the derivation and application of the CCRM model, if the luma sample to be used is outside the current block, the nearest luma sample within the block will be used instead.

[0158] If the current block contains more than 256 chroma samples, the chroma samples are downsampled to obtain 256 chroma samples for the derivation of the CCRM model.

[0159] If the LumaCbf flag is not equal to zero, the TU level flag is signaled to the bitstream to indicate whether the CCRM method is used for interpredictive mode or IBC mode coded blocks.

[0160] Conventional CCRM approaches can eliminate redundancy between the luma and chroma components of an interconnected block using a single conventional model. However, this single model may not always be suitable for all samples within a coding block. In some embodiments of this disclosure, particularly when the coding block has complex texture features, using multiple models may be more reliable in describing the relationship between luma and chroma samples within the coding block.

[0161] Furthermore, current CCRM approaches may simply use luma samples as input, without utilizing gradient or positional information. Predicting chroma samples using only the reconstructed luma values ​​of luma samples can make it difficult to obtain highly accurate predictions. Generally speaking, more accurate predictions can be obtained by considering more information, such as the gradient information and positional information of the current sample.

[0162] In some embodiments, multiple models can be used for CCRM. Figure 15 shows a process for using multiple models for CCRM according to some embodiments of this disclosure. As can be understood, some of the following embodiments may be described in relation to the encoding procedure, but their descriptive aspects are also applicable to the decoding procedure. Specifically, both the encoding and decoding sides can predict the chroma sample in the same manner, and the encoding side can generate a residual signal based on the predicted chroma value of the chroma sample, and the decoding side can reconstruct the chroma sample using the predicted chroma value and the residual signal. The following embodiments may be described in relation to Figure 15 and other figures.

[0163] As shown in Figure 15, when applying CCRM, two or more models are used for a single block. As described above, the conventional CCRM approach eliminates redundancy between the luma and chroma components of the interconnected block with a single conventional model. As can be understood, chroma samples and collated luma samples(s) can be descriptive aspects of a pixel. The movement of a pixel between luma samples shows some correlation with the movement between chroma samples. Therefore, luma sample motion estimates can be reused for chroma samples, and CCRM can be used to derive such correlations. However, the correlation may not show a linear relationship between samples of different amplitudes (e.g., luma values ​​of luma samples). For example, a model applied to weak amplitude samples may not be suitable for strong amplitude samples. Therefore, the conventional single-model scheme may not be suitable for all samples within a coding block. Thus, in some embodiments of this disclosure, particularly when the coding block has complex texture features, using multiple models may be more reliable in describing the relationship between luma and chroma samples within the coding block. Conventional CCRM approaches using a single model inevitably generate residual signals represented by a large amount of data. The inventors of this disclosure have identified the shortcomings of conventional solutions and overcome the technical bias that the use of CCRM must follow simple rules, thereby improving coding efficiency.

[0164] Figure 16 shows an exemplary method 1600 for encoding a video sequence into a bitstream according to some embodiments of the present disclosure. As shown in Figure 16, the method 1600 comprises the following steps 1602 to 1606, which may be implemented by one or more processors associated with an encoder (e.g., the image / video encoder 124 in Figure 1, or the device 400 in Figure 4).

[0165] In step 1602, the encoder can receive the video sequence.

[0166] In step 1604, the encoder can encode one or more pictures of the video sequence. Specifically, the encoder can predict chroma samples based on luma samples corresponding to chroma samples in the current block using multiple cross-component residual models (CCRMs).

[0167] As can be understood, luma samples and chroma samples are different representative aspects of the current (coding) block. Luma and chroma samples may show correlation in motion estimation, and luma samples can be used to predict the corresponding chroma samples. As described above, CCRM can be used to eliminate redundancy between luma and chroma components for an intercoding block. However, since the correlation between samples may change depending on the sample values, a single CCRM may not be sufficient to explain the relationships between all samples. In some embodiments, two or more CCRMs can be used to predict chroma samples based on luma samples.

[0168] In step 1606, the encoder generates a bitstream associated with the encoded picture. The bitstream may contain the encoding result generated in step 1604.

[0169] As mentioned above, when applying CCRM, it has been proposed that there may be two or more models within a single block. Figure 17 shows some substeps of the method shown in Figure 16 according to some embodiments of the present disclosure. In some embodiments, the step of predicting a luma sample based on a chroma sample corresponding to a chroma sample in step 1604 may include substeps 1702 and 1704 as shown in Figure 17, and these substeps may be implemented by an encoder.

[0170] In substep 1702, the encoder can classify chroma samples into multiple classes. For example, the number of models in a block may be equal to m. For instance, when deriving a CCRM model, the chroma samples in the current block can be classified into m classes based on the predicted luma values ​​of the corresponding luma samples via multiple thresholds. Multiple CCRMs corresponding to multiple classes can each be trained based on the chroma samples and their corresponding luma samples. Each class determined in substep 1702 can be treated as an independent training set to derive the corresponding CCRM model.

[0171] In substep 1704, the encoder can generate predicted chroma values ​​for the target chroma sample based on the corresponding luma samples among the chroma samples, using multiple CCRMs. Then, when applying these models, the chroma samples within the current block can be classified through the same threshold based on the reconstructed luma values ​​of the corresponding luma samples. Finally, different classes of chroma samples are predicted by applying different CCRM models to the reconstructed luma values ​​of the luma samples. The multi-model CCRM process on the decoder side is shown in Figure 15.

[0172] Figure 18 shows some substeps of the method shown in Figure 16 according to some embodiments of the present disclosure. As shown in Figure 18, in some embodiments, substep 1704, which generates a predicted chroma value for a target chroma sample, may further include the following substeps 1802 and 1804, which may be implemented by an encoder.

[0173] In substep 1802, the encoder can classify the target chroma sample into one of several classes based on the reconstructed luma value of the luma sample corresponding to the target chroma sample. As described above, the reconstructed luma value of the luma sample (shown as recL in Figure 15) can be used to further classify the chroma sample before applying the model. Specifically, the target chroma sample can be classified into one of several classes based on a comparison of the reconstructed luma value of the luma sample with a threshold. The threshold can be associated with the predicted or reconstructed luma values ​​of at least some of the luma samples in the current block. In some embodiments, the threshold may be the average of the predicted luma values ​​of all luma samples in the current block, or the average of the reconstructed luma values ​​of all luma samples in the current block.

[0174] In substep 1804, the encoder can generate a predicted chroma value for the target chroma sample based on the reconstructed luma value of the luma sample corresponding to the target chroma sample, using one of several CCRMs corresponding to one of several classes.

[0175] In some embodiments, as shown in Figure 15, two models (i.e., Model 1 and Model 2) can be used for the current block in CCRM. In substep 1702, the encoder can classify the target chroma sample into one of several classes based on the predicted luma value of the luma sample corresponding to the target chroma sample (shown as predL in Figure 15). Here, the “target” sample or “objective” sample refers to the sample under study and may be different. When deriving the CCRM model, the current chroma sample can be classified based on the predicted luma value of one corresponding luma sample.

[0176] In some embodiments, the target chroma sample can be classified into one of several classes based on a comparison between the predicted chroma value of the chroma sample and a threshold. The threshold can be associated with the predicted or reconstructed chroma values ​​of at least some of the chroma samples in the current block. In some embodiments, the several classes may include two classes, and the threshold may be the average of the predicted chroma values ​​of all chroma samples in the current block, or the average of the reconstructed chroma values ​​of all chroma samples in the current block.

[0177] In some embodiments, for a target chroma sample having coordinates (i,j), the corresponding luma sample can be determined as a luma sample having coordinates (2i,2j) in a 4:2:0 color format. For example, predL(2i,2j) can be used for classification in a 4:2:0 color format, where predL(2i,2j) is the predicted luma value of the luma sample having coordinates (2i,2j). That is, the target chroma sample can be classified into one of several classes based on a comparison between predL(2i,2j) and a threshold.

[0178] The threshold TH can be the mean of the predicted chroma values ​​of all chroma samples in the current block, i.e., the mean of predL, or the mean of the predicted chroma values ​​of some of the chroma samples in the current block. If predL(2i,2j) is greater than (or equal to) TH, the chroma sample at (i,j) falls into the first category; otherwise, the chroma sample at (i,j) falls into the second category. After classifying each chroma sample in the current block using the same rules, two CCRM models can be derived for each of the two categories.

[0179] When applying the CCRM model, the current chroma sample can be classified based on the reconstructed chroma value of a corresponding chroma sample via the same threshold TH. That is, if recL(2i,2j), which represents the reconstructed chroma value of a chroma sample with coordinates (2i,2j), is greater than (or equal to) TH, then the chroma sample at (i,j) falls into the first category, and the CCRM model derived for the first category is applied to obtain predC'(i,j). Otherwise, the chroma sample at (i,j) falls into the second category, and the CCRM model derived for the second category is applied to obtain predC'(i,j). This represents the predicted chroma value of the chroma sample with coordinates (i,j).

[0180] In some embodiments, for a target chroma sample having coordinates (i,j), the luma sample corresponding to that target chroma sample can be determined as a luma sample having coordinates (i,j) in a 4:4:4 color format. Specifically, in the derivation of the model, predL(i,j) can be used for comparison with a classification threshold. Also, in the application of the model, recL(i,j) can be used for comparison with a classification threshold. predL(i,j) is the predicted luma value of the luma sample having coordinates (i,j), and recL(i,j) is the reconstructed luma value of the luma sample having coordinates (i,j).

[0181] In some embodiments, for a target chroma sample having coordinates (i,j), the luma sample corresponding to that target chroma sample may be determined as a luma sample having coordinates (2i,j) in a 4:2:2 color format. Specifically, in the derivation of the model, predL(2i,j) can be used for comparison with a classification threshold. Also, in the application of the model, recL(2i,j) can be used for comparison with a classification threshold. predL(2i,j) is the predicted luma value of the luma sample having coordinates (2i,j), and recL(2i,j) is the reconstructed luma value of the luma sample having coordinates (2i,j).

[0182] In some embodiments, when classifying current chroma samples, an average or weighted average value associated with several corresponding chroma samples can be used.

[0183] In some embodiments, for a target chroma sample having coordinates (i,j), in response to a 4:2:0 color format and a prediction of a chroma sample such that the chroma sample does not shift vertically relative to the corresponding luma sample, the luma sample corresponding to the target chroma sample is determined as a luma sample having coordinates (2i,2j) and (2i,2j+1). For example, for a 4:2:0 color format where sps_chroma_vertical_collocated_flag is equal to 1, for a chroma sample having coordinates (i,j), in the derivation of the model,

number

number

[0184] In some embodiments, for a target chroma sample having coordinates (i,j), in response to a 4:2:0 color format and a prediction of a chroma sample such that the chroma sample does not shift vertically relative to the corresponding luma sample, the luma sample corresponding to the target chroma sample is determined as a luma sample having coordinates (2i,2j), (2i-1,2j), (2i+1,2j), (2i,2j+1), (2i-1,2j+1), and (2i+1,2j+1). For example, for a 4:2:0 color format where sps_chroma_vertical_collocated_flag is equal to 1, in the derivation of the model for a chroma sample having coordinates (i,j),

number

[0185] In some embodiments, for a target chroma sample having coordinates (i, j), in response to a prediction of the chroma sample that is shifted downward by 0.5 in luma sample units with respect to the 4:2:0 color format and the luma sample corresponding to the chroma sample, the luma sample corresponding to the target chroma sample is determined as luma samples having coordinates (2i, 2j), (2i - 1, 2j), and (2i + 1, 2j). For example, for a 4:2:0 color format where sps_chroma_vertical_collocated_flag is equal to 0, for a chroma sample having coordinates (i, j), in the derivation of the model, [Number] can be compared with the classification threshold using. Also, in the application of the model, [Number] can be compared with the classification threshold using.

[0186] In some embodiments, for a target chroma sample having coordinates (i,j), in response to a 4:2:0 color format and a prediction of a chroma sample such that the chroma sample is shifted downward by 0.5 units per luma sample relative to the corresponding luma sample, the luma samples corresponding to the target chroma sample are determined as luma samples having coordinates (2i,2j), (2i,2j-1), and (2i,2j+1). For example, for a 4:2:0 color format where sps_chroma_vertical_collocated_flag is equal to 0, in the derivation of the model for a chroma sample having coordinates (i,j),

number

number

[0187] In some embodiments, for a target chroma sample having coordinates, in response to a 4:2:0 color format and a prediction of a chroma sample such that the chroma sample is shifted downward by 0.5 units per luma sample relative to the corresponding luma sample, the luma sample corresponding to the target chroma sample is determined as a luma sample having coordinates (2i,2j), (2i-1,2j), (2i+1,2j), (2i,2j+1), and (2i,2j-1). For example, for a 4:2:0 color format where sps_chroma_vertical_collocated_flag is equal to 0, in the derivation of the model for a chroma sample having coordinates,

number

number

[0188] In some embodiments, when deriving the CCRM model, the current chroma sample can be alliteratively classified based on the reconstructed ruma value of the ruma sample corresponding to the current chroma sample. That is, the predicted ruma value of the ruma sample used for classifying the chroma sample is replaced with the reconstructed ruma value of the corresponding ruma sample. For example, when deriving the CCRM model, predL(2i,2j) can be replaced with recL(2i,2j).

[0189] In some embodiments, the threshold TH described above may be the average of the reconstructed luma values ​​of all luma samples in the current block, i.e., the average of recL, or the average of a portion of the reconstructed luma values ​​of the luma samples in the current block.

[0190] Referring further to Figure 15, in some embodiments, a target CCRM among multiple CCRMs can be trained based on the original predicted chroma values ​​of a chroma sample set classified into the class corresponding to the target CCRM (shown as predC in Figure 15) and the predicted luma values ​​of luma samples corresponding to the chroma sample set (shown as predL in Figure 15). As can be understood, the original predicted chroma values ​​can be generated by interpretation against a reference picture or by intrablock copying (IBC).

[0191] In some embodiments, a chroma sample can be predicted by multiple CCRMs based on the corresponding chroma sample, in response to a decision to predict the chroma sample by two or more CCRMs. For example, the encoder can generate a flag in step 1604 indicating whether or not to predict the chroma sample by one or more CCRMs, and signal this to the decoder side in the bitstream generated in step 1606. Specifically, a TU level flag can be signaled to the bitstream to indicate whether multiple models or a single model are used in the CCRMs. In some embodiments, if the number of chroma samples exceeds a threshold, it may be determined that the chroma sample is predicted by two or more CCRMs. Specifically, if the number of samples in the current block is less than (or equal to) the threshold, only a single-model CCRM can be used. Otherwise, a TU level flag is signaled to the bitstream to indicate whether multiple-model CCRMs or a single-model CCRM are used.

[0192] In some embodiments, in step 1604, the encoder can merge the predicted chroma value of the target chroma sample with the original predicted chroma value of the target chroma sample to obtain the final predicted chroma value. As previously mentioned, the original predicted chroma value can be generated by interpretation with respect to a reference picture or by intrablock copying (IBC). In some embodiments, in step 1604, the encoder can generate the residual chroma value of the target chroma sample based on the final predicted chroma value.

[0193] In some embodiments, at step 1604, the encoder can filter the final predicted chroma value with a low-pass filter to obtain a filtered predicted chroma value of the target chroma sample. In some embodiments, the encoder can further generate a residual chroma value of the target chroma sample based on the filtered predicted chroma value. In some embodiments, the coefficients of the low-pass filter are determined based on the predicted luma value and the reconstructed luma value of the luma sample corresponding to the target chroma sample. In some embodiments, the encoder can generate a flag indicating whether to filter with the low-pass filter.

[0194] In some embodiments, first, a low-pass filter can be applied to the predicted chroma value. Next, the filtered predicted chroma value can be fused with the original predicted chroma value. Specifically, at step 1604, the encoder can filter the predicted chroma value with a low-pass filter to obtain a filtered predicted chroma value of the target chroma sample. Thereafter, the encoder can fuse the filtered predicted chroma value of the target chroma sample with the original predicted chroma value of the target chroma sample to obtain a final predicted chroma value.

[0195] FIG. 19 shows an exemplary method 1900 for decoding a bitstream to output one or more pictures of a video stream according to some embodiments of the present disclosure. As shown in FIG. 19, method 1900 can include the following steps 1902 and 1904, and these steps can be implemented by one or more processors associated with a decoder (e.g., the image / video decoder 144 in FIG. 1, or the apparatus 400 in FIG. 4).

[0196] At step 1902, the decoder can receive a bitstream.

[0197] In step 1904, the decoder can decode one or more pictures using the encoded information of the bitstream. Specifically, the decoder can predict chroma samples based on luma samples corresponding to chroma samples in the current block using multiple cross-component residual models (CCRMs).

[0198] In some embodiments, the decoder can predict the chroma sample using the same method as described above in relation to the encoder. The decoder can also reconstruct the chroma sample using the predicted chroma value and residual signal generated by the encoder.

[0199] In some embodiments, a non-temporary, computer-readable storage medium for storing the bitstream is also provided. The bitstream can be encoded and decoded according to any of the methods described above.

[0200] In some embodiments, gradient information or positional information can be used in the CCRM model. Figure 20 shows an exemplary method 2000 for encoding a video sequence into a bitstream according to some embodiments of the present disclosure. As shown in Figure 20, method 2000 comprises the following steps 2002 to 2006, which may be implemented by one or more processors associated with an encoder (e.g., the image / video encoder 124 in Figure 1, or the device 400 in Figure 4).

[0201] In step 2002, the encoder can receive the video sequence.

[0202] In step 2004, the encoder can encode one or more pictures of the video sequence. Specifically, the encoder can predict the chroma sample based on the gradient of the luma sample corresponding to the chroma sample in the current block using a cross-component residual model (CCRM).

[0203] In step 2006, the encoder generates a bitstream associated with the encoded picture. The bitstream may contain the encoding result generated in step 2004.

[0204] In some embodiments, the luma samples and chroma samples are organized in a 4:2:0 color format, and the target luma sample having coordinates (i,j) can be predicted based on the gradient of the luma sample having coordinates (i,j). Specifically, a gradient-based CCRM method is proposed for the 4:2:0 color format, which uses the gradient of the reconstructed luma value of the corresponding luma sample to obtain a CCRM prediction. For example, the chroma sample is predicted using Equation 63.

number

[0205] In some embodiments, G(i,j) can be determined based on the reconstructed luma values ​​of luma samples having coordinates (2i-1,2j), (2i+1,2j), (2i-1,2j+1), and (2i+1,2j+1). Specifically, G(i,j) is determined based on the following equation:

number

[0206] In some embodiments, G(i,j) can be determined based on the reconstructed luma values ​​of luma samples having coordinates (2i,2j), (2i,2j+1), (2i-1,2j), (2i+1,2j), (2i-1,2j+1), and (2i+1,2j+1). Specifically, G(i,j) is determined based on the following formula:

number

[0207] The model parameters c0, c1, and B can be derived based on the gradients of the original predicted chroma value of the chroma sample predC and the predicted luma value of the luma sample corresponding to the chroma sample predL. Specifically, the parameters c0, c1, and B are:

number

[0208] In some embodiments, G'(i,j) may be determined based on the predicted luma values ​​of luma samples having coordinates (2i-1,2j), (2i+1,2j), (2i-1,2j+1), and (2i+1,2j+1).

number

[0209] In some embodiments, G'(i,j) may be determined based on the predicted luma values ​​of luma samples having coordinates (2i,2j), (2i,2j+1), (2i-1,2j), (2i+1,2j), (2i-1,2j+1), and (2i+1,2j+1).

number

[0210] In some embodiments, an index that may be a flag is signaled to the bitstream to indicate which of the gradients represented by the various formulas above is being used.

[0211] In some embodiments, multiple gradients of the reconstructed luma values ​​of the corresponding luminance samples are used to obtain a predicted value of CCRM as expressed in Equation 72.

number

[0212] In some embodiments, the chroma sample in the current block is further predicted based on the reconstructed luma value of the luma sample corresponding to this chroma sample. Thus, in CCRM, both the reconstructed luma value of the corresponding luma sample and the gradient of the reconstructed luma value are used to obtain the predicted chroma value of the current chroma sample, as expressed in Equation 73 (i.e., a gradient term is added to Equation 62). For example, the luma samples corresponding to the chroma sample are luma samples with coordinates (2i,2j), (2i-1,2j), (2i+1,2j), (2i,2j+1), (2i-1,2j+1), and (2i+1,2j+1).

number

number

[0213] For example, one of the four gradients corresponding to equations 64 through 67 can be used. In another example, two gradients corresponding to equations 64 and 65 are used. In some embodiments, P may be a nonlinear term of the gradient.

[0214] In some embodiments, positional information is used for CCRM prediction as expressed by Equation 74 (i.e., the positional term is added to Equation 62), where X and Y represent the horizontal and vertical distances between the current chroma sample and the chroma sample at the upper left corner of the current block, respectively.

number

[0215] The above equation 74 is generated by adding the position term "c6*X+c7*Y" to equation 62.

[0216] In some embodiments, gradient information, positional information, and the reconstructed luma values ​​of the corresponding luma samples are all used in CCRM prediction, as shown in Equation 75.

number

[0217] The above equation 75 is generated by adding the position term and gradient term "c6*X+c7*Y+c8*G0(i,j)+c9*G1(i,j)+···" to equation 62.

[0218] In the above embodiment, the positions of different terms in the CCRM prediction formula can be freely changed.

[0219] In some embodiments, in step 2204, the encoder can merge the predicted chroma value of the target chroma sample with the original predicted chroma value of the target chroma sample to obtain the final predicted chroma value. As mentioned above, the original predicted chroma value can be generated by interpretation with respect to a reference picture or by intrablock copying (IBC). In some embodiments, the encoder can generate the residual chroma value of the chroma sample based on the final predicted chroma value.

[0220] Figure 21 shows an exemplary method 2100 for decoding a bitstream to output one or more pictures of a video stream, according to some embodiments of the present disclosure. As shown in Figure 21, the method 2100 may include the following steps 2102 and 2104, which may be implemented by one or more processors associated with a decoder (e.g., the picture / video decoder 144 in Figure 1, or the device 400 in Figure 4).

[0221] In step 2102, the decoder can receive the bitstream.

[0222] In step 2104, the decoder can decode one or more pictures using the encoded information of the bitstream. Specifically, the decoder can predict chroma samples based on the gradients of the luma samples corresponding to the chroma samples in the current block using multiple cross-component residual models (CCRMs).

[0223] In some embodiments, the decoder can predict the chroma sample using the same method as described above in relation to the encoder. The decoder can also reconstruct the chroma sample using the predicted chroma value and residual signal generated by the encoder.

[0224] In some embodiments, a non-temporary, computer-readable storage medium for storing the bitstream is also provided. The bitstream can be encoded and decoded according to any of the methods described above.

[0225] In some embodiments, a low-pass filter may be used in the CCRM.

[0226] Figure 22 shows an exemplary method 2200 for encoding a video sequence into a bitstream according to some embodiments of the present disclosure. As shown in Figure 22, the method 2200 comprises the following steps 2202 to 2206, which may be implemented by one or more processors associated with an encoder (e.g., the image / video encoder 124 in Figure 1, or the device 400 in Figure 4).

[0227] In step 2202, the encoder can receive the video sequence.

[0228] In step 2204, the encoder can encode one or more pictures of the video sequence. Specifically, the encoder can obtain filtered predicted chroma values ​​by filtering the predicted chroma values ​​of the chroma samples in the current block with a low-pass filter, and these predicted chroma values ​​are generated by one of the cross-component residual models (CCRM) based on the luma samples corresponding to the chroma samples. Specifically, the predicted chroma values ​​of a chroma sample can be predicted by the CCRM based on the predicted luma values ​​and reconstructed luma values ​​of the luma samples corresponding to the chroma samples.

[0229] In step 2206, the encoder generates a bitstream associated with the encoded picture. The bitstream may contain the encoding result generated in step 2204.

[0230] In some embodiments, it is proposed to use a low-pass filter to refine the predicted chroma value of the chroma sample obtained by CCRM. In some embodiments, the low-pass filter is applied to the above predicted chroma value predC' obtained by applying the CCRM model to the reconstructed chroma value of the chroma sample. In some embodiments, the low-pass filter is applied to the above final chroma value predC'' obtained by fusing the original predicted chroma value predC and the predicted chroma value predC' (also called the time-predicted chroma value in this context), where predC is obtained using interprediction mode or IBC mode.

[0231] In some embodiments, the low-pass filter is a 3x3 tap filter, as shown in Equation 76. Equation 77 shows how the filter is applied to obtain the filtered predicted chroma value predC''(i,j). For chroma samples that are within the current block but not on the top / left boundary of the current block, the filtering window is only concerned with the predicted chroma value of the chroma sample. For samples that are on the top / left boundary of the current block, the filtering window may be concerned with the reconstructed chroma values ​​of adjacent chroma samples.

number

[0232] In some embodiments, the filter coefficients are derived based on the predicted luma value predL of the luma sample corresponding to the chroma sample, and the reconstructed luma value recL of the luma sample corresponding to the chroma sample.

[0233] In some embodiments, a flag is signaled to indicate whether or not to apply a filter. In some embodiments, the filter can only be applied to blocks that use a multi-model CCRM.

[0234] In some embodiments, in step 2204, the encoder can generate residual chroma values ​​of the chroma sample based on the filtered predicted chroma values.

[0235] Figure 23 shows an exemplary method 2300 for decoding a bitstream to output one or more pictures of a video stream, according to some embodiments of the present disclosure. As shown in Figure 23, method 2300 may include the following steps 2302 and 2304, which may be implemented by one or more processors associated with a decoder (e.g., the picture / video decoder 144 in Figure 1, or the device 400 in Figure 4).

[0236] In step 2302, the decoder can receive the bitstream.

[0237] In step 2304, the decoder can decode one or more pictures using the encoded information of the bitstream. Specifically, the decoder can filter the predicted chroma values ​​of the chroma samples in the current block with a low-pass filter to obtain filtered predicted chroma values, which are generated by a cross-component residual model (CCRM) based on the chroma samples corresponding to the chroma samples.

[0238] In some embodiments, the decoder can filter the predicted chroma values ​​of the chroma sample using the same method as described above in relation to the encoder. The decoder can also reconstruct the chroma sample using the predicted chroma values ​​and residual signals generated by the encoder.

[0239] In some embodiments, a non-temporary, computer-readable storage medium for storing the bitstream is also provided. The bitstream can be encoded and decoded according to any of the methods described above.

[0240] In some embodiments, cross-component model inheritance may be used for CCRM.

[0241] Figure 24 shows an exemplary method 2400 for encoding a video sequence into a bitstream according to some embodiments of the present disclosure. As shown in Figure 24, the method 2400 comprises the following steps 2402 to 2406, which may be implemented by one or more processors associated with an encoder (e.g., the image / video encoder 124 in Figure 1, or the device 400 in Figure 4).

[0242] In step 2402, the encoder can receive the video sequence.

[0243] In step 2404, the encoder can encode one or more pictures of the video sequence. Specifically, the encoder can build a model based on a first component of the current and use this model to predict a second component of the current. This model may be one of the models described above, for example, a cross-component linear model (CCLM), a multi-model CCLM (MMLM), a gradient linear model, a convolutional cross-component intra-predictive model, or a cross-component residual model (CCRM).

[0244] In step 2406, the encoder generates a bitstream associated with the encoded picture. The bitstream may contain the encoding result generated in step 2404.

[0245] In some embodiments, the first component is a luma sample and the second component is a chroma sample. For example, it is proposed to build a model for a predicted sample and a reconstructed sample and reuse the above model between different components. Specifically, a model can be built with a predicted luma sample as input and a reconstructed luma sample as output. Model parameters can be derived based on the predicted luma block and the reconstructed luma block of the current coding block. All of the above models and parameter derivation methods, including linear models and conventional modes, can be used. Next, this model can predict the predicted chroma value of a chroma sample based on the original predicted chroma value of the chroma sample, the original predicted chroma value being interpredicted against a reference picture or generated by intrablock copying (IBC). This model can be applied to the predicted chroma sample to derive a reconstructed chroma sample. In some embodiments, the encoder can merge the derived reconstructed chroma sample with the predicted chroma sample to obtain the final predicted chroma sample.

[0246] Figure 25 shows an exemplary method 2500 for decoding a bitstream to output one or more pictures of a video stream, according to some embodiments of the present disclosure. As shown in Figure 25, the method 2500 may include the following steps 2502 and 2504, which may be implemented by one or more processors associated with a decoder (e.g., the picture / video decoder 144 in Figure 1, or the device 400 in Figure 4).

[0247] In step 2502, the decoder can receive the bitstream.

[0248] In step 2504, the decoder can decode one or more pictures using the encoded information of the bitstream. Specifically, the decoder builds a model based on the first component of the current and uses this model to predict the second component of the current.

[0249] In some embodiments, the decoder can construct a model using the same method as described above in relation to the encoder. The decoder can also reconstruct the chroma sample using the predicted chroma values ​​and residual signals generated by the encoder.

[0250] In some embodiments, a non-temporary, computer-readable storage medium for storing the bitstream is also provided. The bitstream can be encoded and decoded according to any of the methods described above.

[0251] The embodiments described in this disclosure can be freely combined.

[0252] In some embodiments, a non-temporary, computer-readable storage medium for storing the bitstream is also provided. The bitstream can be encoded and decoded according to the disclosed cross-component prediction method.

[0253] In some embodiments, non-temporary computer-readable storage media containing instructions are also provided, which can be executed by a device (such as the disclosed encoder and decoder) to perform the methods described above. Common forms of non-temporary media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tapes, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, physical media having a pattern of holes, RAM, PROMs, and EPROMs, flash EPROMs or any other flash memory, NVRAMs, caches, registers, any other memory chips or cartridges, and networked versions thereof. The device may include one or more processors (CPUs), input / output interfaces, network interfaces, and / or memory.

[0254] Embodiments may be further described using the following clauses. 1. A method for encoding a video sequence into a bitstream, The steps include receiving a video sequence and The steps include encoding one or more pictures of the aforementioned video sequence, The steps include generating a bitstream associated with the encoded picture, The step of encoding one or more pictures of the video sequence is: A method comprising the step of predicting a chroma sample based on a luma sample corresponding to a chroma sample in the current block using multiple cross-component residual models (CCRM). 2. The method according to Clause 1, wherein the chroma sample is predicted by the plurality of CCRMs based on the chroma sample corresponding to the chroma sample, in response to a decision to predict the chroma sample by two or more CCRMs. 3. The method according to Clause 2, wherein, in response to the number of chroma samples exceeding a threshold, it is determined that the chroma samples are predicted by two or more CCRMs. 4. The step of encoding one or more pictures of the video sequence is: The method according to Clause 2, further comprising the step of generating a flag indicating whether or not to predict the chroma sample by one or more CCRMs. 5. The step of predicting the chroma sample based on the luma sample corresponding to the chroma sample is: A step of classifying the chroma sample into a plurality of classes, wherein the plurality of CCRMs corresponding to the plurality of classes are each trained based on the chroma sample and the corresponding luma sample. The method according to any one of the claims 1 to 4, comprising the step of generating a predicted chroma value for the target chroma sample based on a chroma sample corresponding to the target chroma sample among the chroma samples using the plurality of CCRMs. 6. The step of classifying the chroma samples into the plurality of classes is: The method according to Clause 5, comprising the step of classifying the target chroma sample into one of the plurality of classes based on the predicted luma value of the luma sample corresponding to the target chroma sample. 7. The method according to Clause 6, wherein the target chroma sample is classified into one of the plurality of classes based on a comparison between the predicted luma value of the luma sample and a threshold associated with the predicted or reconstructed luma values ​​of at least some of the luma samples in the current block. 8. The method according to Clause 7, wherein the plurality of classes comprises two classes, and the threshold is the average of the predicted luma values ​​of all luma samples in the current block, or the average of the reconstructed luma values ​​of all luma samples in the current block. 9. The method according to Clause 7, wherein, with respect to a target chroma sample having coordinates (i,j), the luma sample corresponding to the target chroma sample is determined as a luma sample having coordinates (i,j) for a 4:4:4 color format, or as a luma sample having coordinates (2i,j) for a 4:2:2 color format. 10. The method according to Clause 7, wherein, for a target chroma sample having coordinates (i,j), the luma sample corresponding to the target chroma sample is determined as a luma sample having coordinates (2i,2j) for a 4:2:0 color format. 11. The method according to clause 10, wherein the target chroma sample is classified into one of the plurality of classes based on a comparison of predL(2i,2j) with the threshold, and predL(2i,2j) is the predicted chroma value of the chroma sample having the coordinates (2i,2j). 12. The method according to Clause 7, wherein, for a target chroma sample having coordinates (i,j), in response to a 4:2:0 color format and a prediction of the chroma sample such that the chroma sample does not shift vertically relative to the corresponding luma sample, the luma sample corresponding to the target chroma sample is determined as a luma sample having coordinates (2i,2j) and (2i,2j+1). 13. The target chroma sample is classified into one of the multiple classes based on a comparison between the threshold and the following formula:

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0255] It should be noted that relational terms such as “first” and “second” used herein are used solely to distinguish one entity or operation from another, and do not require or imply any actual relationship or order between these entities or operations. Furthermore, the words “comprising,” “having,” “containing,” and “including,” as well as other similar forms, are intended to be open-ended in that they are equivalent in meaning, and that one or more items following any one of these words do not exhaustively list such one or more items, nor are they limited to the one or more items listed.

[0256] As used herein, unless otherwise specified, the term “or” encompasses all possible combinations, unless impractical. For example, if it is stated that a database may contain A or B, then unless otherwise specified or impractical, the database may contain A, or B, or A and B. As a second example, if it is stated that a database may contain A, B, or C, then unless otherwise specified or impractical, the database may contain A, or B, or C, or A and B, or A and C, or B and C, or A and B and C.

[0257] It will be understood that the embodiments described above may be implemented by hardware, software (program code), or a combination of hardware and software. If implemented by software, it may be stored in the computer-readable medium described above. When executed by a processor, the software may perform the methods disclosed herein. The computing units and other functional units described herein may be implemented by hardware, software, or a combination of hardware and software. It will also be understood by those skilled in the art that several of the above modules / units may be integrated into a single module / unit, and each of the above modules / units may be further divided into several submodules / subunits.

[0258] In the above specification, embodiments have been described with reference to numerous specific details that may vary depending on the embodiment. Certain adaptations and modifications may be made to the above embodiments. Other embodiments may become apparent to those skilled in the art given the specifications and practices of this disclosure disclosed herein. This specification and examples are to be considered merely illustrative, and the true scope and spirit of this disclosure is intended to be shown by the appended claims. Furthermore, the order of steps shown in the figures is for illustrative purposes only and is not intended to limit the steps to any particular order. Thus, those skilled in the art will understand that these steps may be performed in different orders while carrying out the same method.

[0259] Exemplary embodiments are disclosed in the drawings and specification. However, many variations and modifications can be made to these embodiments. Accordingly, certain terms are used, but only in a general and descriptive sense, and not as limiting.

Claims

1. A method for encoding a video sequence into a bitstream, The steps include receiving a video sequence and The steps include encoding one or more pictures of the aforementioned video sequence, The steps include generating a bitstream associated with the encoded picture, The step of encoding one or more pictures of the video sequence is: A method comprising the step of predicting a chroma sample based on a luma sample corresponding to a chroma sample in the current block using multiple cross-component residual models (CCRMs).

2. The method according to claim 1, wherein the chroma sample is predicted by the plurality of CCRMs based on the luma sample corresponding to the chroma sample, in response to a decision to predict the chroma sample by two or more CCRMs.

3. The step of predicting the chroma sample based on the luma sample corresponding to the chroma sample is: A step of classifying the chroma sample into a plurality of classes, wherein the plurality of CCRMs corresponding to the plurality of classes are each trained based on the chroma sample and the corresponding luma sample. The method according to claim 1 or 2, comprising the step of generating a predicted chroma value for the target chroma sample based on a chroma sample corresponding to the target chroma sample among the chroma samples using the plurality of CCRMs.

4. The step of classifying the chroma samples into the plurality of classes is: The method according to claim 3, further comprising the step of classifying the target chroma sample into one of the plurality of classes based on the predicted luma value of the luma sample corresponding to the target chroma sample.

5. The method according to claim 4, wherein the target chroma sample is classified into one of the plurality of classes based on a comparison between the predicted luma value of the luma sample and a threshold associated with the predicted or reconstructed luma values ​​of at least some of the luma samples in the current block.

6. The step of encoding one or more pictures of the video sequence is: The method according to any one of claims 3 to 5, further comprising the step of fusing the predicted chroma value of the target chroma sample with the original predicted chroma value of the target chroma sample which is interpredicted with respect to a reference picture or generated by intrablock copying (IBC) to obtain a final predicted chroma value.

7. The step of encoding one or more pictures of the video sequence is: The method according to claim 6, further comprising the step of generating residual chroma values ​​of the target chroma sample based on the final predicted chroma values.

8. The step of encoding one or more pictures of the video sequence is: The method according to claim 6, further comprising the step of filtering the final predicted chroma value with a low-pass filter to obtain the filtered predicted chroma value of the target chroma sample.

9. The step of encoding one or more pictures of the video sequence is: The steps include: filtering the predicted chroma value with a low-pass filter to obtain the filtered predicted chroma value of the target chroma sample; The method according to any one of claims 3 to 5, further comprising the step of fusing the filtered predicted chroma value of the target chroma sample with the original predicted chroma value of the target chroma sample which is interpredicted with respect to a reference picture or generated by intrablock copying (IBC) to obtain a final predicted chroma value.

10. A method for decoding a bitstream and outputting one or more pictures of a video stream, The steps include receiving a bitstream, The step of decoding one or more pictures using the encoded information of the bitstream, The step of decoding one or more pictures using the encoded information of the bitstream is: A method comprising the step of predicting a chroma sample based on a luma sample corresponding to a chroma sample in the current block using multiple cross-component residual models (CCRMs).

11. The method according to claim 10, wherein the chroma sample is predicted by the plurality of CCRMs based on the luma sample corresponding to the chroma sample, in response to a decision to predict the chroma sample by two or more CCRMs.

12. The step of predicting the chroma sample based on the luma sample corresponding to the chroma sample is: A step of classifying the chroma sample into a plurality of classes, wherein the plurality of CCRMs corresponding to the plurality of classes are each trained based on the chroma sample and the corresponding luma sample. The method according to claim 10 or 11, comprising the step of generating a predicted chroma value for the target chroma sample based on a chroma sample corresponding to the target chroma sample among the chroma samples using the plurality of CCRMs.

13. The step of classifying the chroma samples into the plurality of classes is: The method according to claim 12, further comprising the step of classifying the target chroma sample into one of the plurality of classes based on the predicted luma value of the luma sample corresponding to the target chroma sample.

14. The method according to claim 13, wherein the target chroma sample is classified into one of the plurality of classes based on a comparison between the predicted luma value of the luma sample and a threshold associated with the predicted or reconstructed luma values ​​of at least some of the luma samples in the current block.

15. The step of decoding one or more pictures using the encoded information of the bitstream is: The method according to any one of claims 12 to 14, further comprising the step of fusing the predicted chroma value of the target chroma sample with the original predicted chroma value of the target chroma sample which is interpredicted with respect to a reference picture or generated by intrablock copying (IBC) to obtain a final predicted chroma value.

16. The step of decoding one or more pictures using the encoded information of the bitstream is: The steps include receiving the residual chromatic value of the target chromatic sample, The method according to claim 15, further comprising the step of generating a chroma value for the target chroma sample based on the residual chroma value and the final predicted chroma value.

17. The step of decoding one or more pictures using the encoded information of the bitstream is: The method according to claim 15, further comprising the step of filtering the final predicted chroma value with a low-pass filter to obtain the filtered predicted chroma value of the target chroma sample.

18. The step of decoding one or more pictures using the encoded information of the bitstream is: The steps include: filtering the predicted chroma value with a low-pass filter to obtain the filtered predicted chroma value of the target chroma sample; The method according to any one of claims 12 to 14, further comprising the step of fusing the filtered predicted chroma value of the target chroma sample with the original predicted chroma value of the target chroma sample which is interpredicted with respect to a reference picture or generated by intrablock copying (IBC) to obtain a final predicted chroma value.

19. A device for encoding a video sequence into a bitstream, A receiving module configured to receive a video sequence, An encoding module configured to encode one or more pictures of the aforementioned video sequence, A generation module configured to generate a bitstream associated with the encoded picture, The aforementioned encoding module is An apparatus configured to predict a chroma sample based on a luma sample corresponding to a chroma sample in the current block, using multiple cross-component residual models (CCRMs).

20. The apparatus according to claim 19, wherein the chroma sample is predicted by the plurality of CCRMs based on the luma sample corresponding to the chroma sample, in response to a decision to predict the chroma sample by two or more CCRMs.

21. The aforementioned encoding module is The classification of the chroma samples into multiple classes, wherein the multiple CCRMs corresponding to the multiple classes are each trained based on the chroma samples and the corresponding chroma samples. The apparatus according to claim 19 or 20, configured to generate a predicted chroma value for the target chroma sample based on a chroma sample corresponding to the target chroma sample among the chroma samples, using the plurality of CCRMs.

22. The aforementioned encoding module is The apparatus according to claim 21, configured to classify the target chroma sample into one of the plurality of classes based on the predicted luma value of the luma sample corresponding to the target chroma sample.

23. The apparatus according to claim 22, wherein the target chroma sample is classified into one of the plurality of classes based on a comparison between the predicted luma value of the luma sample and a threshold associated with the predicted or reconstructed luma values ​​of at least some of the luma samples in the current block.

24. The aforementioned encoding module is The apparatus according to any one of claims 21 to 23, wherein the predicted chroma value of the target chroma sample is fused with the original predicted chroma value of the target chroma sample, which is interpredicted with respect to a reference picture or generated by intrablock copying (IBC), to obtain a final predicted chroma value.

25. The aforementioned encoding module is The apparatus according to claim 24, configured to generate residual chromatic values ​​of the target chromatic sample based on the final predicted chromatic values.

26. The aforementioned encoding module is The apparatus according to claim 24, configured to filter the final predicted chroma value with a low-pass filter to obtain the filtered predicted chroma value of the target chroma sample.

27. The aforementioned encoding module is The predicted chroma values ​​are filtered with a low-pass filter to obtain the filtered predicted chroma values ​​of the target chroma sample. The apparatus according to any one of claims 21 to 23, configured to fuse the filtered predicted chroma value of the target chroma sample with the original predicted chroma value of the target chroma sample which is interpredicted with respect to a reference picture or generated by intrablock copying (IBC) to obtain a final predicted chroma value.

28. A device for decoding a bitstream and outputting one or more pictures of a video stream, A receiving module configured as a bitstream, A decoding module configured to decode one or more pictures using the coded information of the bitstream, The decoding module is, An apparatus configured to predict a chroma sample based on a luma sample corresponding to a chroma sample in the current block, using multiple cross-component residual models (CCRMs).

29. The apparatus according to claim 28, wherein the chroma sample is predicted by the plurality of CCRMs based on the luma sample corresponding to the chroma sample, in response to a decision to predict the chroma sample by two or more CCRMs.

30. The decoding module is, The classification of the chroma samples into multiple classes, wherein the multiple CCRMs corresponding to the multiple classes are each trained based on the chroma samples and the corresponding chroma samples. The apparatus according to claim 28 or 29, configured to generate a predicted chroma value for the target chroma sample based on a chroma sample corresponding to the target chroma sample among the chroma samples, using the plurality of CCRMs.

31. The decoding module is, The apparatus according to claim 30, configured to classify the target chroma sample into one of the plurality of classes based on the predicted luma value of the luma sample corresponding to the target chroma sample.

32. The apparatus according to claim 31, wherein the target chroma sample is classified into one of the plurality of classes based on a comparison between the predicted luma value of the luma sample and a threshold associated with the predicted or reconstructed luma values ​​of at least some of the luma samples in the current block.

33. The decoding module is, The apparatus according to any one of claims 30 to 32, configured to fuse the predicted chroma value of the target chroma sample with the original predicted chroma value of the target chroma sample, which is interpredicted with respect to a reference picture or generated by intrablock copying (IBC), to obtain a final predicted chroma value.

34. The decoding module is, The residual chromatic values ​​of the target chromatic sample are received, The apparatus according to claim 33, configured to generate a chroma value for the target chroma sample based on the residual chroma value and the final predicted chroma value.

35. The decoding module is, The apparatus according to claim 33, configured to filter the final predicted chroma value with a low-pass filter to obtain the filtered predicted chroma value of the target chroma sample.

36. The decoding module is, The predicted chroma values ​​are filtered with a low-pass filter to obtain the filtered predicted chroma values ​​of the target chroma sample. The apparatus according to any one of claims 30 to 32, wherein the filtered predicted chroma value of the target chroma sample is fused with the original predicted chroma value of the target chroma sample, which is interpredicted with respect to a reference picture or generated by intrablock copying (IBC), to obtain a final predicted chroma value.

37. One or more processors, A computer-readable storage medium is provided which is communicably coupled to one or more processors, The computer-readable storage medium is an electronic device that stores computer-readable instructions that, when executed by the one or more processors, perform the method according to any one of claims 1 to 18.

38. A non-temporary, computer-readable storage medium for storing a video bitstream, wherein the bitstream, once encoded by an encoder, causes the encoder to perform the method according to any one of claims 1 to 9.

39. A non-temporary, computer-readable storage medium for storing a video bitstream, wherein, when the bitstream is decoded by a decoder, the decoder causes the decoder to perform the method according to any one of claims 10 to 18.

40. A computer program product comprising computer program instructions, wherein the computer program instructions enable a computer to perform the method according to any one of claims 1 to 18.

41. A computer program that enables a computer to perform the method described in any one of claims 1 to 18.

Images