Apparatus, method and computer program for video encoding and decoding

Abstract

A method comprising: receiving a picture block unit of a frame, the picture block unit comprising samples in a first chroma channel, a second chroma channel and a luma channel; defining co-located reference areas on a luma component of the luma channel, a chroma component of the first chroma channel and a chroma component of the second chroma channel; performing motion-compensated prediction on the luma component, the first chroma component and the second chroma component using a reference frame and motion vectors; obtaining a first mapping function and a second mapping function using the co-located luma prediction and chroma predictions, the first mapping function mapping the luma component into the first chroma component, the second mapping function mapping the luma component into the second chroma component; reconstructing a luma residual; and obtaining a first chroma prediction by using the first function and the luma residual, and a second chroma prediction by using the second function and the luma residual.

Description

Technical Field

[0001] This invention relates to apparatus, methods, and computer programs for video encoding and decoding. Background Technology

[0002] In video decoding, video and image samples are typically decoded using color representations such as YUV or YCbCr, consisting of one luma channel and two chroma channels. In these cases, the luma channel, which primarily represents the lighting of the scene, is usually decoded at a specific resolution, while the chroma channels, which generally represent the differences between certain color components, are decoded at a second resolution, typically lower than the resolution of the luma signal. The intention of this differential representation is to decorrelate the color components and to compress the data more efficiently.

[0003] In the Universal Video Decoding (VVC / H.266) standard, a cross-component linear model (CCLM) is used as the linear model for predicting samples in the chroma channels (e.g., Cb and Cr). Model parameters are derived based on reconstructed samples in the neighborhood of the chroma block, co-located neighboring samples in the luma block, and reconstructed samples within the co-located luma block. An improved version of cross-component prediction, known as CCCM, uses a two-dimensional (2D) filter kernel to derive the luma-to-chroma model. The filter coefficients are the decoder-side derived using the input data and the reconstructed set of chroma samples.

[0004] Inter-frame decoding in modern video decoding standards (such as H.266) uses single-tree motion-compensated prediction: regions from one or more reference frames are copied to the target frame using subpixel interpolation. Additional tools, such as weighted prediction (WP), bidirectional prediction with CU-level weights (BCW), combined inter- and intra-frame prediction (CIIP), or local illumination compensation (LIC), are used to account for variations in illumination and / or color differences between the reference and target frames. Single-tree decoding signals a set of partitioning information per block, and this partitioning is shared between the luma and chroma components. Due to the limited number of chroma-specific decoding tools, inter-frame decoding typically implicitly prioritizes luma blocks over chroma blocks. Even with chroma-specific tools such as CCCM or CCLM, they are not very efficient because chroma blocks in the reference frame are generally of higher quality than cross-component blocks in intra-frame prediction. To benefit from cross-component tools in inter-frame prediction, a method is needed to model chroma primarily using co-located motion-compensated luma and chroma. Summary of the Invention

[0005] To at least alleviate the aforementioned problems, this paper introduces an enhancement method for achieving a better cross-component residual model for prediction.

[0006] The scope of protection sought by the various embodiments of the present invention is set forth in the independent claims. Embodiments and features described herein that are not within the scope of the independent claims (if any) are to be interpreted as examples that may be used to understand the various embodiments of the invention.

[0007] According to some embodiments, a method is provided for directly mapping luma residuals to chroma prediction. Specifically, this mapping can be applied to single-tree inter-frame decoding blocks that share the same partitioning between luma and chroma blocks. An intuitive way of performing this mapping after decoding the luma residuals of a single-tree inter-frame block is also disclosed. The method can also be extended to dual-tree blocks or any other partitioning scheme. Additional methods for improving the robustness of the mapping process are also provided. Alternative configurations such as chroma-to-chroma prediction are also described.

[0008] According to a first aspect, a method includes: receiving an image block unit of a frame, the image block unit including samples in a first chroma channel, a second chroma channel, and a luminance channel; defining a co-located reference region on the luminance component of the luminance channel, the chroma component of the first chroma channel, and the chroma component of the second chroma channel; performing motion-compensated prediction on the luminance component, the first chroma component, and the second chroma component using the reference frame and a motion vector; obtaining a first mapping function and a second mapping function using the co-located luminance prediction and chroma prediction, the first mapping function mapping the luminance component to the first chroma component, and the second mapping function mapping the luminance component to the second chroma component; reconstructing a luminance residual; and obtaining a first chroma prediction by using the first function and the luminance residual, and obtaining a second chroma prediction by using the second function and the luminance residual.

[0009] According to a second aspect, an apparatus includes: components for receiving image block units of a frame, the image block units including samples in a first chroma channel, a second chroma channel, and a luminance channel; components for defining co-located reference regions on the luminance component of the luminance channel, the chroma component of the first chroma channel, and the chroma component of the second chroma channel; components for performing motion-compensated predictions on the luminance component, the first chroma component, and the second chroma component using the reference frame and a motion vector; components for obtaining a first mapping function and a second mapping function using the co-located luminance and chroma predictions, the first mapping function mapping the luminance component to the first chroma component, and the second mapping function mapping the luminance component to the second chroma component; components for reconstructing a luminance residual; and components for obtaining a first chroma prediction using the first function and the luminance residual, and obtaining a second chroma prediction using the second function and the luminance residual.

[0010] According to an embodiment, the apparatus includes a component for downsampling a co-located luminance component to match a subsampled chromaticity grid.

[0011] According to an embodiment, the device includes a component for filtering co-located luminance components using a one-dimensional filter or a two-dimensional filter.

[0012] According to an embodiment, the apparatus includes: a component for mapping a luminance component to a first chromaticity component or a second chromaticity component, or mapping a first chromaticity component to a second chromaticity component, using a first mapping function or a second mapping function.

[0013] According to an embodiment, the apparatus includes a component for mapping a first chromaticity component to a second chromaticity component using a first mapping function or a second mapping function.

[0014] According to an embodiment, the apparatus includes a component for signaling the use of the cross-component residual model (CCRM) from the encoder to the decoder at the transform unit, prediction unit, decoding unit, decoding tree unit, strip, frame, or sequence level.

[0015] According to an embodiment, the apparatus includes a component for cropping or spatially filtering luminance residuals before applying a cross-component model.

[0016] According to an embodiment, the apparatus includes components for determining the type and amount of filtering to be applied to the luminance residual for a first chromaticity component and a second chromaticity component, respectively.

[0017] According to an embodiment, the apparatus includes a component for determining the type and amount of filtering to be applied to the luminance residual based on the output of a first mapping function or the output of a second mapping function.

[0018] According to an embodiment, the apparatus includes components for applying different types and amounts of filtering to the luminance residuals at the sub-block or sample level.

[0019] According to an embodiment, the apparatus includes a component for using different mapping functions in block prediction based on one or more thresholds derived from luminance, luminance residual, or chrominance.

[0020] According to an embodiment, the apparatus includes a component for mapping chromaticity components to luminance components.

[0021] According to an embodiment, the apparatus includes a component for mapping a first chromaticity component to a second chromaticity component.

[0022] According to an embodiment, the apparatus includes a component for upsampling the input of a mapping function to match a brightness grid.

[0023] According to an embodiment, the apparatus includes a component for upsampling the output of a mapping function to match a brightness grid.

[0024] According to an embodiment, the apparatus includes a component for mapping a first chromaticity component to a second chromaticity component.

[0025] As described above, the device and the computer-readable storage medium on which code is stored are therefore arranged to perform the above-described methods and one or more embodiments thereon. Attached Figure Description

[0026] To better understand the invention, reference will now be made to the accompanying drawings by way of example, wherein:

[0027] Figure 1 An electronic device employing an embodiment of the present invention is illustrated schematically;

[0028] Figure 2 A user equipment suitable for employing embodiments of the present invention is illustrated schematically;

[0029] Figure 3 The electronic device employing an embodiment of the present invention using wireless and wired network connections is further illustrated schematically;

[0030] Figure 4a and Figure 4b An encoder and decoder suitable for implementing embodiments of the present invention are schematically illustrated;

[0031] Figure 5 The location of the sample used to derive the parameters for the cross-component linear model is shown;

[0032] Figure 6a and Figure 6b Examples of classifying luminance samples into two classes are shown in the sample domain and spatial domain, respectively.

[0033] Figure 7 An example of four reference rows adjacent to the prediction block is shown;

[0034] Figure 8 The matrix-weighted intra-frame prediction process is illustrated.

[0035] Figure 9 A flowchart of a method for predicting samples in at least one color channel according to an embodiment of the present invention is shown;

[0036] Figure 10 An example of a co-located reference sample region consisting of reconstructed luminance and chrominance samples defined for both luminance and chrominance, according to an embodiment of the present invention, is shown;

[0037] Figure 11 Various examples of filter kernel dimensions based on one method are shown;

[0038] Figure 12The top and left neighbor blocks used in the combined inter-frame and intra-frame prediction (CIIP) weight derivation according to one method are shown;

[0039] Figure 13 A schematic diagram of an example multimedia communication system in which various embodiments can be implemented is shown. Detailed Implementation

[0040] The following section describes in more detail suitable devices and possible mechanisms for predicting chromaticity samples. For this, please refer first to... Figure 1 and Figure 2 ,in Figure 1 A block diagram of a video decoding system, which is shown as an exemplary device or electronic device 50 according to an example embodiment, incorporates a codec according to an embodiment of the invention. Figure 2 The layout of the device according to an example embodiment is shown. The following will explain... Figure 1 and Figure 2 Components.

[0041] Electronic device 50 may be, for example, a mobile terminal or user equipment of a wireless communication system. However, it should be understood that embodiments of the present invention can be implemented in any electronic device or apparatus that may require encoding and decoding, or encoding or decoding, video images.

[0042] Device 50 may include a housing 30 for incorporation and protection of the device. Device 50 may also include a display 32 in the form of a liquid crystal display. In other embodiments of the invention, the display may be any suitable display technology suitable for displaying images or video. Device 50 may also include a keypad 34. In other embodiments of the invention, any suitable data interface or user interface mechanism may be employed. For example, the user interface may be implemented as a virtual keyboard or data entry system as part of a touch-sensitive display.

[0043] The device may include a microphone 36 or any suitable audio input that may be a digital or analog signal input. The device 50 may also include an audio output device, which in embodiments of the invention may be any of the following: headphones 38, a speaker, or an analog or digital audio output connection. The device 50 may also include a battery (or, in other embodiments of the invention, the device may be powered by any suitable mobile energy device, such as a solar cell, fuel cell, or spring generator). The device may also include a camera capable of recording or capturing images and / or video. The device 50 may also include an infrared port for short-range line-of-sight communication with other devices. In other embodiments, the device 50 may also include any suitable short-range communication solution, such as Bluetooth wireless connectivity or USB / FireWire wired connectivity.

[0044] The device 50 may include a controller 56, a processor, or processor circuitry for controlling the device 50. The controller 56 may be connected to a memory 58, which in embodiments of the invention may store data in the form of image and audio data and / or may also store instructions for implementation on the controller 56. The controller 56 may also be connected to a codec circuit 54, which is adapted to perform encoding and decoding of audio and / or video data or to assist in encoding and decoding performed by the controller.

[0045] The device 50 may also include a card reader 48 and a smart card 46, such as a UICC and a UICC reader, for providing user information and suitable for providing authentication information for user authentication and authorization at the network.

[0046] Device 50 may include a radio interface circuit 52 connected to a controller and adapted to generate wireless communication signals, such as for communicating with a cellular communication network, a wireless communication system, or a wireless local area network. Device 50 may also include an antenna 44 connected to the radio interface circuit 52 for transmitting radio frequency signals generated at the radio interface circuit 52 to other devices and for receiving radio frequency signals from other devices.

[0047] Apparatus 50 may include a camera capable of recording or detecting individual frames, which are then passed to codec 54 or a controller for processing. The apparatus may receive video image data for processing from another device before transmission and / or storage. Apparatus 50 may also receive images for encoding / decoding wirelessly or via a wired connection. The structural elements of apparatus 50 described above represent examples of components used to perform the corresponding functions.

[0048] about Figure 3 This illustrates an example of a system in which embodiments of the invention can be utilized. System 10 includes multiple communication devices that can communicate via one or more networks. System 10 may include any combination of wired or wireless networks, including but not limited to wireless cellular telephone networks (such as GSM, UMTS, CDMA networks, etc.), wireless local area networks (WLANs) such as those defined by any of the IEEE 802.x standards, Bluetooth personal area networks, Ethernet LANs, token ring LANs, wide area networks, and the Internet.

[0049] System 10 may include wired and wireless communication devices and / or apparatus 50 suitable for implementing embodiments of the present invention.

[0050] For example, Figure 3The system shown illustrates a representation of mobile phone network 11 and Internet 28. Connectivity to Internet 28 can include, but is not limited to, long-range wireless connections, short-range wireless connections, and various wired connections, including but not limited to telephone lines, cable lines, power lines, and similar communication paths.

[0051] The example communication devices shown in System 10 may include, but are not limited to, electronic devices or apparatus 50, a combination of a personal digital assistant (PDA) and a mobile phone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, and a laptop computer 22. When carried by a mobile individual, apparatus 50 may be stationary or mobile. Apparatus 50 may also be located in a mode of transport, including but not limited to cars, trucks, taxis, buses, trains, ships, airplanes, bicycles, motorcycles, or any similar suitable mode of transport.

[0052] The embodiments can also be implemented in set-top boxes, i.e., digital TV receivers, which may or may not have a display or wireless capability; in tablets or (laptop) personal computers (PCs), which have a combination of hardware or software or encoder / decoder implementations; in various operating systems; and in chipsets, processors, DSPs and / or embedded systems that provide hardware / software-based encoding.

[0053] Some or additional devices can send and receive calls and messages, and communicate with service providers to base station 24 via wireless connection 25. Base station 24 can connect to network server 26 that allows communication between mobile phone network 11 and Internet 28. The system may include additional communication equipment and various types of communication devices.

[0054] Communication devices can communicate using various transmission technologies, including but not limited to Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), Universal Mobile Telecommunications System (UMTS), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Transmission Control Protocol-Internet Protocol (TCP-IP), Short Message Service (SMS), Multimedia Messaging Service (MMS), email, Instant Messaging Service (IMS), Bluetooth, IEEE 802.11, and any similar wireless communication technologies. In various embodiments of the invention, the communication devices can communicate using various media, including but not limited to radio, infrared, laser, cable connections, and any suitable connection.

[0055] In telecommunications and data networks, a channel can refer to a physical channel or a logical channel. A physical channel can refer to a physical transmission medium such as a wire, while a logical channel can refer to a logical connection on a multiplexed medium capable of transmitting several logical channels. A channel can be used to transmit information signals (e.g., bit streams) from one or more senders (or transmitters) to one or more receivers.

[0056] The MPEG-2 transport stream (TS), as defined in ISO / IEC 13818-1 or equivalently in ITU-T Recommendation H.222.0, is a format used to carry audio, video, and other media, as well as program metadata or other metadata, in a multiplexed stream. A packet identifier (PID) is used to identify the elementary stream (also known as a packetized elementary stream) within the TS. Therefore, logical channels within an MPEG-2 TS can be considered as corresponding to specific PID values.

[0057] Available media file format standards include the ISO Basic Media File Format (ISO / IEC 14496-12, which can be abbreviated as ISOBMFF) and the file format for NAL unit structured video (ISO / IEC 14496-15), which is derived from ISOBMFF.

[0058] A video codec consists of an encoder that transforms the input video into a compressed representation suitable for storage / transmission and a decoder that decompresses the compressed video representation back into a visual form. Video encoders and / or video decoders can also be separate entities; that is, they do not need to form a codec. Typically, the encoder discards some information from the original video representation in order to represent the video in a more compact form (i.e., at a lower bit rate).

[0059] Typical hybrid video encoders (such as many encoder implementations of ITU-T H.263 and H.264) decode video information in two stages. First, a first pixel value in a specific picture region (or “block”) is predicted, for example, by a motion compensation component (which identifies and indicates a region in one of the previously encoded video frames that closely corresponds to the block being encoded) or by a spatial component (using pixel values ​​around the block encoded in a specified manner). Second, the prediction error is encoded, i.e., the difference between the predicted pixel block and the original pixel block. This is typically done by transforming the difference in pixel values ​​using a specified transform (e.g., Discrete Cosine Transform (DCT) or a variant thereof), quantizing the coefficients, and entropy encoding the quantization coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and the size (file size or transmission bit rate) of the resulting encoded video representation.

[0060] In temporal prediction, the prediction source is the previously decoded image (i.e., the reference image). In intra-block copy (IBC, also known as intra-block copy prediction), prediction is applied similarly to temporal prediction, but the reference image is the current image, and only previously decoded samples can be referenced during the prediction process. Inter-layer or inter-view prediction can be applied similarly to temporal prediction, but the reference images are decoded images from another scalable layer or another view, respectively. In some cases, inter-frame prediction can refer to temporal prediction only, while in others, inter-frame prediction can collectively refer to temporal prediction and any of intra-block copy, inter-layer, and inter-view predictions, provided they are performed using the same or similar procedures as temporal prediction. Inter-frame prediction or temporal prediction is sometimes referred to as motion compensation or motion-compensated prediction.

[0061] Motion compensation can be performed with full-sample or subsample precision. In the case of full-sample precise motion compensation, motion can be represented as a motion vector with integer values ​​for horizontal and vertical displacements, and the motion compensation process uses those displacements to effectively replicate samples from a reference image. In the case of subsample precise motion compensation, the motion vector is represented by fractional or decimal values ​​for the horizontal and vertical components of the motion vector. When the motion vector references a non-integer position in the reference image, a subsample interpolation process is typically invoked to compute predicted sample values ​​based on the reference sample and selected subsample positions. The subsample interpolation process typically includes horizontal filtering that compensates for horizontal offsets relative to the full-sample position, followed by vertical filtering that compensates for vertical offsets relative to the full-sample position. However, vertical processing can also be performed before horizontal processing in some environments.

[0062] Inter-frame prediction (also known as temporal prediction, motion compensation, or motion-compensated prediction) reduces temporal redundancy. In inter-frame prediction, the prediction source is the previously decoded image. Intra-frame prediction leverages the fact that adjacent pixels within the same image may be correlated. Intra-frame prediction can be performed in the spatial domain or the transform domain; that is, it can predict sample values ​​or transform coefficients. Intra-frame prediction is typically used for intra-frame decoding, where inter-frame prediction is not applied.

[0063] One outcome of the encoding process is the set of encoded parameters, such as motion vectors and quantization transform coefficients. These parameters can be entropy-coded more efficiently if many parameters are predicted first from spatially or temporally neighboring parameters. For example, motion vectors can be predicted from spatially neighboring motion vectors, and only the difference relative to the motion vector predictor can be encoded. The encoding of parameters and the prediction of intra-frame predictions can be collectively referred to as intra-image prediction.

[0064] Figure 4a and Figure 4bAn encoder and decoder suitable for employing embodiments of the present invention are illustrated. The video codec consists of an encoder that transforms an input video into a compressed representation suitable for storage / transmission, and a decoder capable of decompressing the compressed video representation back into a visual form. Typically, the encoder discards and / or loses some information from the original video sequence in order to represent the video in a more compact form (i.e., at a lower bit rate). Figure 4a An example of the encoding process is shown. Figure 4a The image to be encoded (I) is shown. n ); Predictive representation of image patches (P' n Prediction error signal (D) n ); Reconstruct the prediction error signal (D' n ); Preliminary reconstructed image (I' n ); final reconstructed image (R' n Transform (T) and inverse transform (T) -1 ); Quantization (Q) and inverse quantization (Q) -1 ); Entropy coding (E); Reference frame memory (RFM); Inter-frame prediction (P) inter Intra-frame prediction (P) intra ); Mode selection (MS) and filtering (F).

[0065] Figure 4b An example of the decoding process is shown in the figure. Figure 4b The predicted representation of the image patch (P') is shown. n ); Reconstruct the prediction error signal (D' n ); Preliminary reconstructed image (I' n ); final reconstructed image (R' n Inverse transform (T) -1 Inverse quantization (Q) -1 Entropy decoding (E) -1 Reference Frame Memory (RFM); Prediction (inter-frame or intra-frame) (P); and Filtering (F).

[0066] Many hybrid video encoders encode video information in two stages. First, a first pixel value in a particular picture region (or “block”) is predicted, for example, by a motion compensation component (which identifies and indicates a region in one of the previously encoded video frames that closely corresponds to the block being encoded) or by a spatial component (using pixel values ​​around the block encoded in a specified manner). In this first stage, predictive coding, such as so-called sample prediction and / or so-called syntax prediction, can be applied.

[0067] In sample prediction, pixel or sample values ​​are predicted for a specific region or "block" of an image. These pixel or sample values ​​can be predicted using one or more of the following mechanisms: motion compensation or intra-frame prediction.

[0068] Motion compensation mechanisms (also known as inter-frame prediction, temporal prediction, motion-compensated temporal prediction, or motion-compensated prediction, or MCP) involve finding and indicating regions in one of the previously encoded video frames that closely correspond to the block being decoded. Inter-frame prediction can reduce temporal redundancy.

[0069] Intra-frame prediction (where pixel or sample values ​​can be predicted by spatial mechanisms) involves discovering and indicating spatial region relationships. Intra-frame prediction leverages the fact that neighboring pixels within the same image may be related. Intra-frame prediction can be performed in the spatial domain or the transform domain; that is, it can predict sample values ​​or transform coefficients. Intra-frame prediction is typically used in intra-frame coding, where inter-frame prediction is not applied.

[0070] In syntax prediction, also known as parameter prediction, syntax elements and / or syntax element values ​​and / or variables derived from earlier encoded syntax elements (de) and / or earlier derived variables are predicted from the syntax elements. A non-restrictive example of syntax prediction is provided below.

[0071] In motion vector prediction, motion vectors, for example, used for inter-frame and / or inter-view prediction, can be encoded relative to block-specific predicted motion vectors. In many video codecs, predicted motion vectors are created in a predefined manner, such as by calculating the median of encoded or decoded motion vectors from adjacent blocks. Another way to create motion vector predictions (sometimes called Advanced Motion Vector Prediction (AMVP)) is to generate a list of candidate predictions from adjacent and / or co-located blocks in a time-referenced picture and signal the selected candidate as a motion vector predictor. In addition to predicting motion vector values, reference indices from previously encoded / decoded pictures can be predicted. Reference indices are typically predicted from adjacent and / or co-located blocks in a time-referenced picture. Differential coding of motion vectors is typically disabled across stripe boundaries.

[0072] Block partitioning can be predicted, for example, from CTU to CU and down to PU.

[0073] In filter parameter prediction, filter parameters, for example, used for sample adaptive offset, can be predicted.

[0074] Prediction methods that use image information from previously encoded images are also called inter-frame prediction methods, or temporal prediction and motion compensation. Prediction methods that use image information within the same image are also called intra-frame prediction methods.

[0075] Secondly, the prediction error, i.e., the difference between the predicted pixel block and the original pixel block, is encoded. This is typically accomplished by transforming the difference in pixel values ​​using a specified transform (e.g., Discrete Cosine Transform (DCT) or a variant thereof), quantizing the coefficients, and entropy encoding the quantization coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (image quality) and the size of the resulting encoded video representation (file size or transmission bit rate). The video codec can also provide a transform skip mode that the encoder can optionally use. In transform skip mode, the prediction error is encoded in the sample domain, for example, by deriving the sample-by-sample difference relative to some neighboring samples and encoding the sample-by-sample difference using an entropy encoder.

[0076] Entropy coding / decoding can be performed in several ways. For example, context-based coding / decoding can be applied, where both the encoder and decoder modify the context state of the coding parameters based on previously encoded / decoded coding parameters. Context-based coding can be, for example, Context Adaptive Binary Arithmetic Coding (CABAC), Context-Based Variable-Length Coding (CAVLC), or any similar entropy coding. Entropy coding / decoding can be performed alternatively or additionally using variable-length coding schemes (such as Huffman coding / decoding or exponential Golomb coding / decoding). Decoding the coding parameters of a bitstream or codeword derived from entropy coding can be referred to as parsing.

[0077] A phrase along a bitstream (e.g., a bitstream indication) can be defined as referring to out-of-band transmission, signaling, or storage in a way that associates out-of-band data with the bitstream. Decoding along a bitstream or similar phrase can refer to decoding the referenced out-of-band data associated with the bitstream (which can be obtained from out-of-band transmission, signaling, or storage). For example, a bitstream indication can refer to metadata in a container file encapsulating the bitstream.

[0078] The H.264 / AVC standard was developed by the Video Coding Experts Group (VCEG) of the Telecommunication Standardization Sector of the International Telecommunication Union (ITU-T) and the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard was published by both parent standardization organizations and is known as ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10, also called MPEG-4 Part 10 Advanced Video Coding (AVC). Several versions of the H.264 / AVC standard have existed, integrating new extensions or features into the specification. These extensions include Scalable Video Coding (SVC) and Multi-View Video Coding (MVC).

[0079] Version 1 of the High Efficiency Video Coding (H.265 / HEVC, also known as HEVC) standard was developed by the Joint Collaboration Team on Video Coding (JCT-VC) of VCEG and MPEG. This standard was published by both parent standardization organizations and is known as ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2, also called MPEG-H Part 2 High Efficiency Video Coding (HEVC). Subsequent versions of H.265 / HEVC include Scalable, Multiple View, Fidelity Range, 3D, and Screen Content Coding Extensions, which can be abbreviated as SHVC, MV-HEVC, REXT, 3D-HEVC, and SCC, respectively.

[0080] Universal Video Coding (VVC) (MPEG-I Part 3), also known as ITU-T H.266, is a video compression standard developed by the Joint Video Experts Group (JVET) of the Moving Picture Experts Group (MPEG) (formally ISO / IEC JTC1SC29 WG11) and the Video Coding Experts Group (VCEG) of the International Telecommunication Union (ITU), as a successor to HEVC / H.265.

[0081] This section describes some key definitions, bitstreams, and encoding structures and concepts of H.264 / AVC and HEVC, using video encoders, decoders, encoding methods, decoding methods, and bitstream structures as examples, wherein the described embodiments can be implemented. Some key definitions, bitstreams, and encoding structures and concepts of H.264 / AVC are the same as those in HEVC, and therefore are described together below. Aspects of the invention are not limited to H.264 / AVC or HEVC, but are described with respect to a possible basis on which the invention can be partially or completely implemented.

[0082] Similar to many earlier video coding standards, H.264 / AVC and HEVC specify the bitstream syntax and semantics, as well as the decoding process for error-free bitstreams. No encoding process is specified, but the encoder must generate a compliant bitstream. Bitstream and decoder consistency can be verified using a hypothetical reference decoder (HRD). The standard includes encoding tools to help handle transmission errors and data loss, but using these tools during encoding is optional, and no decoding process is specified for erroneous bitstreams.

[0083] The basic units used as input to the H.264 / AVC or HEVC encoder and output to the H.264 / AVC or HEVC decoder are images, respectively. The image given as input to the encoder can also be called the source image, and the image decoded by the decoder can be called the decoded image.

[0084] The source image and the decoded image each consist of one or more sample arrays, such as one of the following sample array sets: -Luminance (Y) only (monochrome). - Brightness and two chromaticities (YCbCr or YCgCo). - Green, blue, and red (GBR, also known as RGB). - indicates an array of other unspecified monochromatic or trichromatic stimulus samples (e.g., YZX, also known as XYZ).

[0085] In H.264 / AVC and HEVC, a picture can be a frame or a field. A frame consists of a matrix of luminance samples and possibly corresponding chrominance samples. A field is a collection of alternating sample rows of a frame and can be used as encoder input when the source signal is interleaved. The chrominance sample array may not exist (and therefore monochromatic sampling can be used), or the chrominance sample array may be double-sampled compared to the luminance sample array. The chrominance format can be summarized as follows: - In monochrome sampling, there is only one sample array, which can be nominally regarded as a brightness array. - In 4:2:0 sampling, each of the two chroma arrays has half the height and half the width of the luminance array. In 4:2:2 sampling, each of the two chroma arrays has the same height and half the width of the luminance array. - In 4:4:4 sampling, when no separate color plane is used, each of the two chroma arrays has the same height and width as the luminance array.

[0086] In H.264 / AVC and HEVC, sample arrays can be encoded as separate color planes into a bitstream, and the individually encoded color planes can be decoded separately from the bitstream. When using separate color planes, each of them is processed individually (by the encoder and / or decoder) into an image with monochrome sampling.

[0087] A partition can be defined as dividing a set into subsets such that every element in the set is in exactly one of the subsets.

[0088] When describing HEVC encoding and / or decoding operations, the following terminology can be used. A coded block can be defined as an N×N sample block for a certain value of N, such that the division from the decoder block to the coded block is a partition. A decoder block (CTB) can be defined as an N×N sample block for a certain value of N, such that the division from the component to the decoder block is a partition. A coded tree unit (CTU) can be defined as a decoder block for luma samples, two corresponding coded tree blocks for chroma samples of an image with three sample arrays, or a sample decoder block for a monochrome image or an image encoded using three separate color planes and a syntax structure for encoding the samples. A decoder unit (CU) can be defined as a coded block for luma samples, two corresponding coded blocks for chroma samples of an image with three sample arrays, or a sample coded block for a monochrome image or an image encoded using three separate color planes and a syntax structure for encoding the samples. The CU with the largest allowed size can be named LCU (Maximum Decoded Unit) or Decoded Tree Unit (CTU), and video images are divided into non-overlapping LCUs.

[0089] A CU (Currency Unit) consists of one or more prediction units (PUs) and one or more transformation units (TUs). The PUs define the prediction process for samples within the CU, and the TUs define the prediction error coding process for samples within the CU. Typically, a CU consists of square sample blocks with a size selectable from a predefined set of possible CU sizes. Each PU and TU can be further subdivided into smaller PUs and TUs to increase the granularity of the prediction and prediction error coding processes, respectively. Each PU has associated prediction information defining what type of prediction will be applied to the pixels within the PU (e.g., motion vector information for inter-frame prediction PUs and intra-frame prediction directionality information for intra-frame prediction PUs).

[0090] Each TU can be associated with information describing the prediction error decoding process of the samples within that TU (including, for example, DCT coefficient information). Typically, at the CU level, signaling is used to indicate whether prediction error coding is applied for each CU. If no prediction error residual is associated with a CU, then it can be assumed that no TU exists for that CU. Typically, signaling is used in the bitstream to indicate the partitioning of the image into CUs and the partitioning of CUs into PUs and TUs, thereby allowing the decoder to reproduce the intended structure of these units.

[0091] In HEVC, an image can be partitioned into tiles, which are rectangular and contain an integer number of LCUs. In HEVC, partitioning into tiles forms a regular grid, where the height and width of each tile differ from the previous one LCU at their maximum values. In HEVC, a stripe is defined as an integer number of decoder tree units contained in an independent stripe segment, plus all subsequent dependent stripe segments (if any) preceding the next independent stripe segment within the same access unit. In HEVC, a stripe segment is defined as an integer number of decoder tree units consecutively ordered in a tile scan and contained in a single NAL unit. Dividing each image into stripe segments is partitioning. In HEVC, an independent stripe segment is defined as a stripe segment whose syntax element values ​​in its segment header are not inferred from the values ​​of previous stripe segments; and a dependent stripe segment is defined as a stripe segment whose syntax element values ​​in its segment header are inferred from the values ​​of previous independent stripe segments in the decoding order. In HEVC, a stripe header is defined as the header of an independent stripe segment, which is either the current stripe segment or an independent stripe segment preceding the currently dependent stripe segment. The stripe header is defined as a portion of an encoded stripe segment containing data elements related to the first or all decode tree units represented in the stripe segment. If a tile is not used, the CUs are scanned in the raster scan order of the LCUs within the tile or the image. Within an LCU, the CUs have a specific scan order.

[0092] The decoder reconstructs the output video by applying an encoder-like prediction component to form a predictive representation of pixel blocks (using motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (the inverse operation of prediction error encoding to recover the quantized prediction error signal in the spatial pixel domain). After applying the prediction and prediction error decoding components, the decoder sums the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) may also apply additional filtering components to improve the quality of the output video before passing it for display and / or storing it as a predictive reference for upcoming frames in a video sequence.

[0093] Filtering can include, for example, one or more of the following: deblocking, sample adaptive offset (SAO), and / or adaptive loop filtering (ALF). H.264 / AVC includes deblocking, while HEVC includes both deblocking and SAO.

[0094] In a typical video codec, motion information is indicated by motion vectors associated with each motion-compensated image block (such as a prediction unit). Each of these motion vectors represents the displacement of the image block in the picture to be encoded (on the encoder side) or decoded (on the decoder side) relative to a prediction source block in one of the previously encoded or decoded pictures. To efficiently represent motion vectors, they are typically encoded differentially relative to block-specific predicted motion vectors. In a typical video codec, predicted motion vectors are created in a predefined manner, such as by calculating the median of the encoded or decoded motion vectors of neighboring blocks. Another way to create motion vector predictions is to generate a list of candidate predictions from neighboring and / or co-located blocks in a temporal reference picture and signal the selected candidates as motion vector predictors. In addition to predicting motion vector values, it is possible to predict which reference picture(s) will be used for motion-compensated prediction, and this prediction information can be represented, for example, by reference indices of previously encoded / decoded pictures. Reference indices are typically predicted from neighboring and / or co-located blocks in a temporal reference picture. Furthermore, typical high-efficiency video codecs employ additional motion information encoding / decoding mechanisms, often referred to as merging / merging modes. This involves predicting and using all motion field information, including motion vectors and corresponding reference image indices for each available list of reference images, without any modification / correction. Similarly, motion field information prediction is performed using motion field information from adjacent and / or co-located blocks in the temporal reference images, and the motion field information used is signaled in a list populated with motion field candidate information from available adjacent / co-located blocks.

[0095] In a typical video codec, the prediction residuals after motion compensation are first transformed using a transform kernel (such as DCT) and then encoded. This is because, in many cases, some correlation still exists between the residuals and the transform, and this method helps to reduce this correlation and provide more efficient encoding.

[0096] Video coding standards and specifications allow encoders to divide coded images into coded stripes or similar. Typically, intra-image prediction across stripe boundaries is disabled. Therefore, a stripe can be viewed as a way to break down a coded image into independently decodeable pieces. In H.264 / AVC and HEVC, intra-image prediction across stripe boundaries is disabled. Therefore, a stripe can be viewed as a way to break down a coded image into independently decodeable pieces, and thus, a stripe is often considered the basic unit used for transmission. In many cases, the encoder can indicate in the bitstream which types of intra-image prediction are disabled across stripe boundaries, and the decoder operation takes this information into account, for example, when inferring which prediction sources are available. For example, if neighboring CUs reside in different stripes, samples from neighboring CUs can be considered unavailable for intra-frame prediction.

[0097] The basic units used for the output of an H.264 / AVC or HEVC encoder and the input of an H.264 / AVC or HEVC decoder are Network Abstraction Layer (NAL) units. For transmission over packet-oriented networks or storage in structured files, NAL units can be encapsulated within packets or similar structures. Bytestream formats have been specified in H.264 / AVC and HEVC for transmission or storage environments that do not provide framing structures. Bytestream formats separate NAL units from each other by appending start codes to the beginning of each NAL unit. To avoid erroneous detection of NAL unit boundaries, the encoder runs a byte-oriented start code emulation prevention algorithm, which adds emulation prevention bytes to the NAL unit payload when start codes are present in other cases. Start code emulation prevention can always be performed to enable direct gateway operation between packet-oriented and stream-oriented systems, regardless of whether a bytestream format is in use. An NAL unit can be defined as a syntax structure and bytes, the syntax structure containing an indication of the type of data to be followed, and the bytes containing data in RBSP form, embellished with emulation prevention bytes where necessary. The Raw Byte Sequence Payload (RBSP) can be defined as a syntax structure containing an integer number of bytes encapsulated in NAL units. The RBSP can be empty, or it can be a string of data bits containing syntax elements, followed by the RBSP stop bit, and then a number of subsequent bits that are zero or equal to 0.

[0098] A NAL cell consists of a header and a payload. In H.264 / AVC and HEVC, the NAL cell header indicates the type of NAL cell.

[0099] In HEVC, a two-byte NAL cell header is used for all specified NAL cell types. The NAL cell header contains a reserved bit, a 6-bit NAL cell type indicator, a 3-bit nuh_temporal_id_plus1 indicator for the time level (which may need to be greater than or equal to 1), and a 6-bit nuh_layer_id syntax element. The temporal_id_plus1 syntax element can be considered as a time identifier for the NAL cell and can be used to derive a zero-based TemporalId variable as follows: TemporalId = temporal_id_plus1 - 1. The abbreviation TID can be used interchangeably with the TemporalId variable. A TemporalId equal to 0 corresponds to the lowest time level. The value of temporal_id_plus1 must be non-zero to avoid start code simulation involving two NAL cell header bytes. A bitstream is produced by excluding all VCL NAL cells with a TemporalId greater than or equal to the selected value and including all other VCL NAL cells that maintain consistency. Therefore, images with TemporalId equal to tid_value are not used as inter-frame prediction references for any images with TemporalId greater than tid_value. A sublayer, or temporal sublayer, can be defined as a temporally scalable layer (or temporal layer TL) of a temporally scalable bitstream, consisting of VCL NAL units with a TemporalId variable of a specific value and associated non-VCL NAL units. nuh_layer_id can be understood as a scalability layer identifier.

[0100] NAL units can be classified into Video Coding Layer (VCL) NAL units and non-VCL NAL units. VCL NAL units are typically coding stripe NAL units. In HEVC, a VCL NAL unit contains syntax elements representing one or more CUs.

[0101] Non-VCL NAL units can be, for example, one of the following types: sequence parameter set, picture parameter set, supplementary enhancement information (SEI) NAL unit, access unit delimiter, sequence end NAL unit, bitstream end NAL unit, or padding data NAL unit. A parameter set may be required to reconstruct the decoded picture, while many other non-VCL NAL units are not necessary for reconstructing the decoded sample values.

[0102] Parameters that remain unchanged across the encoded video sequence can be included in the sequence parameter set. In addition to parameters that may be needed during decoding, the sequence parameter set may optionally include Video Availability Information (VUI), which includes parameters important for buffering, picture output timing, rendering, and resource reservation. In HEVC, the sequence parameter set RBSP includes parameters that can be referenced by one or more picture parameter sets RBSP or one or more SEI NAL units containing buffer periodic SEI messages. Picture parameter sets contain such parameters that may remain unchanged across several encoded pictures. Picture parameter sets RBSP may include parameters that can be referenced by the encoded stripe NAL units of one or more encoded pictures.

[0103] In HEVC, a Video Parameter Set (VPS) can be defined as a grammatical structure containing grammatical elements applied to zero or more complete encoded video sequences, as determined by the content of grammatical elements found in the PPS, which are referenced by grammatical elements found in the PPS, which are referenced by grammatical elements found in the header of each strip segment.

[0104] A video parameter set (RBSP) may include parameters referenced by one or more sequence parameter sets (RBSPs).

[0105] The relationship and hierarchy between the Video Parameter Set (VPS), Sequence Parameter Set (SPS), and Picture Parameter Set (PPS) can be described as follows. The VPS resides above the SPS in the parameter set hierarchy and in the context of scalability and / or 3D video. The VPS can include parameters common to all slices across all (scalability or view) layers in a complete encoded video sequence. The SPS includes parameters common to all slices in a specific (scalability or view) layer in a complete encoded video sequence and can be shared by multiple (scalability or view) layers. The PPS includes parameters common to all slices in a specific layer representation (a representation of a scalability or view layer within an access unit) and may be shared by all slices in multiple layer representations.

[0106] A VPS can provide information about the dependencies between layers in a bitstream, as well as much other information that can be applied to all stripes across all (scalability or view) layers in a complete encoded video sequence. A VPS can be viewed as consisting of two parts: a base VPS and a VPS extension, where the VPS extension is optional.

[0107] Out-of-band transmission, signaling, or storage may be used additionally or alternatively for purposes other than tolerating transmission errors, such as facilitating access or session negotiation. For example, sample entries for tracks in a file conforming to the ISO Basic Media File Format may include parameter sets, while coded data in the bitstream is stored elsewhere in the file or in another file. Phrases along the bitstream (e.g., along the bitstream indicator) or along the bitstream decoding unit (e.g., along the coded tile indicator) may be used in the embodiments of the claims and described in such a way that out-of-band data is associated with the bitstream or decoding unit, respectively, to refer to out-of-band transmission, signaling, or storage. The phrase "along the bitstream" or "along the bitstream decoding unit" or the like may refer to decoding the out-of-band data (which may be obtained from out-of-band transmission, signaling, or storage) associated with the bitstream or decoding unit, respectively.

[0108] The SEI NAL unit can contain one or more SEI messages. It does not need to be used for decoding the output image, but it can assist in related processes such as image output timing, rendering, error detection, error hiding, and resource preservation.

[0109] An encoded image is an encoded representation of an image.

[0110] In HEVC, an encoded picture can be defined as the encoded representation of a picture containing all decoder tree units. In HEVC, an access unit (AU) can be defined as a set of NAL units associated with each other according to a specified classification rule, which are sequential in decoding order and contain at most one picture with any specific value of nuh_layer_id. In addition to VCL NAL units containing encoded pictures, access units can also contain non-VCL NAL units. The specified classification rule can, for example, associate pictures with the same output time or picture output count value into the same access unit.

[0111] A bitstream can be defined as a sequence of bits in the form of a NAL unit stream or a byte stream, which forms a representation of the associated data that constitutes an encoded picture and one or more encoded video sequences. A second bitstream may follow the first bitstream in the same logical channel, such as in the same file or within the same connection of a communication protocol. A primary stream (in the context of video coding) can be defined as a sequence of one or more bitstreams. The end of the first bitstream can be indicated by a specific NAL unit, which may be called the End-of-Batch (EOB) NAL unit and is the last NAL unit of the bitstream. In HEVC and its current draft extensions, the EOB NAL unit is required to have a nuh_layer_id equal to 0.

[0112] In H.264 / AVC, a encoded video sequence is defined as a sequence of consecutive access units in the decoding order from the first IDR access unit (inclusive) to the next IDR access unit (exclusive) or to the end of the bitstream (whichever comes first).

[0113] In HEVC, a coded video sequence (CVS) can be defined, for example, as a sequence of access units that, in decoding order, contains IRAP access units with a NoRaslOutputFlag equal to 1, followed by zero or more access units with a NoRaslOutputFlag not equal to 1, including all subsequent access units up to but not including any subsequent access units with a NoRaslOutputFlag equal to 1. An IRAP access unit can be defined as an access unit where the base layer picture is an IRAP picture. For each IDR picture, each BLA picture, and each IPAP picture, the value of NoRaslOutputFlag is equal to 1, where the IRAP picture is the first picture in a specific layer of the bitstream in the decoding order, and the first IRAP picture in the decoding order following the end of a sequence of NAL units with the same nuh_layer_id value. There is a component that provides the value of HandleCraAsBlaFlag to the decoder from an external entity (e.g., a player or receiver) that can control the decoder. HandleCraAsBlaFlag can be set to 1, for example, by a player that seeks a new position in the bitstream or tunes to the broadcast and begins decoding, then starts decoding from the CRA image. When HandleCraAsBlaFlag is equal to 1 for a CRA image, the CRA image is processed and decoded as if it were a BLA image.

[0114] In HEVC, an encoded video sequence may be additionally or alternatively (as explained above) designated as the end when a specific NAL unit (which may be referred to as the end-of-sequence (EOS) NAL unit) appears in the bitstream and has a nuh_layer_id equal to 0.

[0115] A Group of Pictures (GOP) and its characteristics can be defined as follows. A GOP can be decoded regardless of whether any previous pictures have been decoded. An open GOP is a group of pictures in which, when decoding begins with the initial intra-picture of the open GOP, pictures preceding the initial intra-picture in the output order may not be decoded correctly. In other words, the pictures in an open GOP can refer to (in inter-frame prediction) pictures belonging to previous GOPs. The HEVC decoder can identify the intra-picture that starts an open GOP because a specific NAL unit type (CRA NAL unit type) is available for its coding stripe. A closed GOP is a group of pictures in which all pictures can be decoded correctly when decoding begins with the initial intra-picture of the closed GOP. In other words, no picture in a closed GOP refers to any picture in a previous GOP. In H.264 / AVC and HEVC, a closed GOP can start with an IDR picture. In HEVC, a closed GOP can also start with a BLA_W_RADL or BLA_N_LP picture. Compared to closed GOP coding structures, open GOP coding structures are potentially more efficient in terms of compression due to the greater flexibility in the selection of reference images.

[0116] A decoded picture buffer (DPB) can be used in the encoder and / or decoder. There are two reasons for buffering decoded pictures: for reference in inter-frame prediction and for reordering decoded pictures in the output order. Because H.264 / AVC and HEVC offer considerable flexibility for both reference picture tagging and output reordering, separate buffers for reference and output pictures could be wasteful of memory resources. Therefore, the DPB can include a unified decoded picture buffering process for both reference and output reordering. When a decoded picture is no longer used as a reference and is no longer needed for output, it can be removed from the DPB.

[0117] In many coding modes of H.264 / AVC and HEVC, reference pictures used for inter-frame prediction are indicated by indices pointing to a list of reference pictures. Variable-length coding can be used to encode these indices, which typically results in smaller indices having smaller values ​​for the corresponding syntax elements. In H.264 / AVC and HEVC, two lists of reference pictures (reference picture list 0 and reference picture list 1) are generated for each bidirectional prediction (B) stripe, and one list of reference pictures (reference picture list 0) is formed for each inter-frame coded (P) stripe.

[0118] Many coding standards (including H.264 / AVC and HEVC) can have a decoding process that derives a reference picture index to a list of reference pictures, which can be used to indicate which of a number of reference pictures was used for inter-frame prediction of a particular block. The reference picture index can be encoded into the bitstream by the encoder in some inter-frame coding mode, or it can be derived, for example (by the encoder and decoder), using neighboring blocks in some other inter-frame coding mode.

[0119] Motion parameter types or motion information may include, but are not limited to, one or more of the following types: - An indication of the prediction type (e.g., intra-frame prediction, one-way prediction, two-way prediction) and / or the number of reference images; - Indication of prediction direction, such as inter-frame (i.e., temporal) prediction, inter-layer prediction, inter-view prediction, view composition prediction (VSP), and inter-component prediction (which can be indicated based on a reference image and / or based on the prediction type, and in some embodiments, inter-view and view composition predictions can be considered together as a single prediction direction), and / or - Indication of reference picture type, such as short-term reference picture and / or long-term reference picture and / or interlayer reference picture (which may be indicated, for example, based on reference picture). - Reference indexes of the reference image list and / or any other identifiers of the reference images (which may be indicated, for example, by reference images, and whose type may depend on the prediction direction and / or the type of reference images, and may be accompanied by other relevant information, such as the similarities to which the reference image list or reference index applies); - Horizontal motion vector components (which may be indicated, for example, according to a prediction block or according to a reference index or similar); - Vertical motion vector components (which may be indicated, for example, based on a prediction block or a reference index or similar); - One or more parameters, such as the image order count difference and / or relative camera interval between an image containing or associated with motion parameters and its reference image, which can be used to scale the horizontal motion vector component and / or the vertical motion vector component during one or more motion vector prediction processes (where said one or more parameters may be indicated, for example, according to each reference image or each reference index or the like); - The coordinates of the block to which motion parameters and / or motion information are applied, for example, the coordinates of the top left sample of the block in the brightness sample cell; - The extent (e.g., width and height) to which motion parameters and / or motion information are applied to the block.

[0120] Compared to previous video coding standards, the Universal Video Codec (H.266 / VVC) introduces several new encoding tools, such as the following: Intra-frame prediction - 67-frame intra-mode with wide-angle mode extension - Block size and mode-dependent 4-tap interpolation filter -Location-dependent intra-prediction combination (PDPC) Cross-component linear model intra-frame prediction (CCLM) - Multi-reference row intra-frame prediction - Intra-frame sub-partitioning - Weighted intra-frame prediction with matrix multiplication Image prediction - Block motion replication with spatial, temporal, historical, and pairwise average merge candidates -Affine motion inter-frame prediction - Sub-block-based temporal motion vector prediction -Adaptive motion vector resolution - 8×8 block-based motion compression for time motion prediction - High-precision (1 / 16 pixel) motion vector storage and motion compensation with 8-tap interpolation filtering for the luminance component and 4-tap interpolation filtering for the chrominance component. - Triangle division - Combined intra-frame and inter-frame prediction - Merging with MVD (MMVD) -Symmetric MVD encoding -Bidirectional optical flow -Decoder-side motion vector refinement - Bidirectional prediction with CU-level weights Transformation, quantization, and coefficient encoding - Multiple master transform options including DCT2, DST7, and DCT8 - Secondary transformation for low-frequency regions - Sub-block transformation for inter-frame prediction residuals - Dependency quantization with a maximum QP increasing from 51 to 63 - Transform coefficient encoding with symbolic data hiding -Transform skip residual coding Entropy coding - Arithmetic coding engine with adaptive dual-window probability update In-loop filter -Intra-circumferential shaping - Deblocking filter with strong, longer filter strength -Sample adaptive offset - Adaptive Loop Filter Screen content encoding: - Current image reference with reference area limitations 360-degree video encoding -Horizontal orbital motion compensation Advanced syntax and parallel processing Reference image management with direct reference image list signaling - A tile group with rectangular tile groups

[0121] Partitioning in VVC is performed similarly to HEVC, where each image is divided into Code Tree Units (CTUs). Images can also be divided into stripes, tiles, bricks, and sub-images. CTUs can be split into smaller CUs using a quadtree structure. Each CU can be partitioned using quadtrees and nested multi-type trees, including ternary and binary splits. However, specific rules exist for inferring partitions within image boundaries, and redundant splitting patterns are not allowed in nested multi-type partitioning.

[0122] Among the new encoding tools listed above, a cross-component linear model (CCLM) prediction mode is used in VVC to reduce cross-component redundancy. Specifically, chroma samples based on reconstructed luminance samples from the same CU are predicted using the following linear model: pred C (i,j)=α·rec L ′(i,j)+β (Equation 1a)

[0123] Among them, pred C (i,j) represents the predicted chromaticity sample in CU, and rec L ′(i,j) represents the downsampled reconstructed brightness sample of the same CU.

[0124] Alternatively, the following equation can be used for CCLM: pred C (i,j)=α·rec′ L (i,j) >> k + β (Equation 1b) The >> operation represents shifting the value k to the right by a number of bits.

[0125] The CCLM parameters (α and β) are derived using at most four adjacent chroma samples and their corresponding downsampled luminance samples. Assuming the current chroma block size is W×H, then W' and H' are set to... -W' = W, H' = H, when applying LM mode; -W' = W+H, when applying LM-A mode; -H' = H+W, when applying LM-L mode;

[0126] In this paper, LM-A mode refers to linear model_above, where only the upper template (i.e., sample values ​​from neighboring positions above the CU) is used to compute linear model coefficients. To obtain more samples, the upper template is extended to (W+H). LM-L mode refers to linear model_left, where only the left template (i.e., sample values ​​from neighboring positions to the left of the CU) is used to compute linear model coefficients. To obtain more samples, the left template is extended to (H+W). For non-square blocks, the upper template is extended to W+W, and the left template is extended to H+H.

[0127] The upper neighbor positions are denoted as S[0,-1]…S[W'-1,-1], and the left neighbor positions are denoted as S[-1,0]…S[-1,H'-1]. Then, four samples are selected as... -S[W' / 4,-1],S[3*W' / 4,-1],S[-1,H' / 4],S[-1,3*H' / 4], when LM mode is applied and the samples above and to the left of the neighboring sample are available; -S[W' / 8,-1],S[3*W' / 8,-1],S[5*W' / 8,-1],S[7*W' / 8,-1],when applying LM-A mode or only the upper neighboring sample is available; -S[-1,H' / 8],S[-1,3*H' / 8],S[-1,5*H' / 8],S[-1,7*H' / 8],when applying LM-L mode or only left neighboring samples are available;

[0128] Four neighboring luminance samples at the selected location are downsampled and compared four times to find two smaller values: x0A and x1A, and two larger values: x0B and x1B. Their corresponding chromaticity sample values ​​are represented as y0A, y1A, y0B, and y1B. Subsequently, xA, xB, yA, and yB are derived as: X a =(x 0 A +x 1 A +1)>>1;X b =(x 0 B +x 1 B +1)>>1;Y a =(y 0 A +y 1 A +1)>>1;Y b =(y 0 B +y 1B +1)>>1(Equation 2)

[0129] Finally, the linear model parameters α and β are obtained according to the following equation: β=Y b -α·X b (Equation 4)

[0130] Figure 5 This shows an example of the positions of the left and top samples and the current block sample involved in CCLM mode.

[0131] A lookup table is used to perform the division operation on the parameter α. To reduce the memory required to store this table, the diff values ​​(the difference between the maximum and minimum values) and the parameter α are represented by exponent symbols. For example, the diff is approximated using a 4-bit significant part and an exponent. Therefore, for the 16 values ​​of the significant part, the table used for 1 / diff is reduced to 16 elements, as follows: DivTable[] = {0,7,6,5,5,4,4,3,3,2,2,1,1,1,1,0} (Equation 5)

[0132] This provides the benefit of reducing computational complexity and the memory size required to store the necessary tables.

[0133] In addition to the top and left templates being used together to calculate the coefficients of the linear model, they can also be used alternatively for two other LM modes, referred to as the LM_A mode and the LM_L mode.

[0134] In LM_A mode, only the upper template is used to calculate the linear model coefficients. To obtain more samples, the upper template is extended to (W+H). In LM_L mode, only the left template is used to calculate the linear model coefficients. To obtain more samples, the left template is extended to (H+W).

[0135] For non-square blocks, expand the top template to W+W and the left template to H+H.

[0136] To match the chroma sample positions used in a 4:2:0 video sequence, two types of downsampling filters are applied to the luminance samples to achieve a 2:1 downsampling ratio in both the horizontal and vertical directions. The selection of the downsampling filters is specified by the SPS level flag. The two downsampling filters are as follows, corresponding to "Type 0" and "Type 2" respectively:

[0137] Note that when the upper reference line is at the CTU boundary, only one luminance line (the universal line buffer in intra-frame prediction) is used to downsample the luminance sample.

[0138] This parameter calculation is performed as part of the decoding process, not just as part of the encoder's search operation. As a result, no syntax is used to pass the α and β values ​​to the decoder.

[0139] For chroma intra-mode coding, a total of eight intra-modes are allowed. These modes include five traditional intra-modes and three cross-component linear model modes (CCLM, LM_A, and LM_L). The chroma mode signaling and derivation process are shown in Table 1. Chroma mode coding directly depends on the intra-prediction mode of the corresponding luma block. Due to the separate block partitioning structure for luma and chroma components enabled in the I-strip, one chroma block can correspond to multiple luma blocks. Therefore, for chroma DM mode, the intra-prediction mode of the corresponding luma block covering the center position of the current chroma block is directly inherited. Table 1

[0140] Regardless of the value of sps_cclm_enabled_flag, a single binary table is used, as shown in Table 2. Table 2

[0141] In Table 2, the first binary bit indicates whether it is normal (0) or LM mode (1). If it is LM mode, the next binary bit indicates whether it is LM_CHROMA (0). If it is not LM_CHROMA, the next 1 binary bit indicates whether it is LM_L (0) or LM_A (1). In this case, when sps_cclm_enabled_flag is 0, the first binary bit of the binarization table used for the corresponding intra_chroma_pred_mode can be discarded before entropy encoding. Or, in other words, the first binary bit is inferred to be 0 and therefore not encoded. This single binarization table is used for both cases where sps_cclm_enabled_flag equals 0 and 1. The first two binary bits in Table 2 are context-encoded using their own context model, and the remaining binary bits are bypass-encoded.

[0142] Additionally, to reduce luma-chroma latency in dual-tree systems, when 64x64 luma decoding tree nodes are partitioned using unsplit (and intra-fraction sub-partitions (ISP) are not used for 64x64 CUs) or QT, chroma CUs in 32x32 / 32x16 chroma decoding tree nodes are allowed to use CCLM in the following manner: - If the 32x32 chroma node is not split or partitioned by QT, then all chroma CUs in the 32x32 node can use CCLM. - If a 32x32 chroma node is partitioned using horizontal BT, and the 32x16 child node is not split or is split using vertical BT, then all chroma CUs in the 32x16 chroma node can use CCLM.

[0143] CCLM is not allowed for chroma CU under all other luminance and chroma decoding tree splitting conditions.

[0144] Multi-model LM (MMLM)

[0145] The CCLM included in VVC is extended by adding three multi-model LM (MMLM) modes. In each MMLM mode, the reconstructed neighbor samples are classified into two classes using a threshold that serves as the mean of the brightness reconstructed neighbor samples. A linear model for each class is derived using the least mean square (LMS) method. For the CCLM mode, the LMS method is also used to derive the linear model. Figure 6a and Figure 6b Two luminance-to-chromaticity models are shown, one in the sample domain and one in the spatial domain, for a luminance (Y) threshold of 17. Each luminance-to-chromaticity model has its own linear model parameters α and β. Figure 6b As can be seen, each luminance-to-chrominance model corresponds to a spatial fragment of the content (i.e., they correspond to different objects or textures in the scene).

[0146] Convolutional Cross-Component Model (CCCM)

[0147] An improved version of cross-component prediction, called CCCM, uses a 2D filter kernel to derive the luma-to-chroma model. The filter coefficients are derived on the decoder side using the input data and a reconstructed set of chroma samples. For the filter coefficient derivation, as follows... Figure 10 As shown, a co-located reference sample region (composed of reconstructed luminance and chrominance samples) is defined for both luminance and chrominance, where the commonly used 4:2:0 chrominance downsampling has been applied. The reference sample region for a given block could be, for example, six lines above and to the left of the given block, such as... Figure 10 As shown, however, any number of reference rows can be used (which can be implemented by both the encoder and decoder). Typically, the reference samples can contain any chroma and luminance samples that have already been reconstructed by both the encoder and decoder. Once the reference samples are determined, the filter coefficients can be derived using, for example, different types of linear regression tools such as ordinary least squares estimation, orthogonal matching pursuit, optimized orthogonal matching pursuit, ridge regression, or minimum absolute shrinkage and selection operators.

[0148] Figure 11The examples shown illustrate various sizes of filter kernels, such as 1x3 (1D vertical), 3x1 (1D horizontal), 3x3, 7x7, or any other size, and how they can be shaped (by selecting only a subset of all possible kernel locations) as cross-shaped or rhomboid (e.g., Figure 11 (as shown) or any given shape. When referencing samples within the filter kernel, use the following symbols: North (above), East (right), South (below), West (left), and Center, as shown. Figure 11 The letters N, E, S, W, and C are used to represent this.

[0149] Here, the overall method of reconstructing chroma samples using the convolution between the filter kernel obtained from the decoder side and the input dataset is called Convolutional Cross-Component Model (CCCM). The following steps can be applied to perform a CCCM operation: 1. Define a co-located reference area for the luminance and chrominance components. 2. Downsample luminance samples to match the chromaticity grid (optional). 3. Scan the luminance and chromaticity samples of the reference region and collect available statistics (such as autocorrelation matrix and cross-correlation vector) based on the filter shape. 4. Solve for the filter coefficients by minimizing the squared error (or any other metric) based on available statistics (such as the autocorrelation matrix and cross-correlation vector). 5. The predicted chromaticity patch is calculated by convolving the downsampled luminance samples with the filter kernel.

[0150] A potentially downsampled luminance sample can be defined as a 2D array Y(x,y) indexed using horizontal x-coordinates and vertical y-coordinates. Let us also define the co-located chrominance samples as a 2D array C(x,y) and the filter kernel (i.e., coefficients) as a 3x3 array F(i,j). At the sample level, the convolution between Y and F can be defined as...

[0151] When other data terms, such as nonlinear square root terms, are used, the attached convolution becomes in, As part of the linear equation used to solve the 2D filter coefficients in step 4 above, filter coefficients located outside the 2D filter kernel were also obtained.

[0152] Similarly, the following can be used to add bias terms to the convolution.

[0153] F and F were obtained using reconstructed samples near the chroma blocks. (i.e., CCCM coefficients). However, in cases where the immediate neighborhood cannot describe the cross-component relationship between luminance and chromaticity, it may be necessary to consider CCCM coefficients derived from more distant samples.

[0154] Multi-reference line (MRL) intra-frame prediction

[0155] Multi-reference line (MRL) intra-prediction uses more reference lines for intra-prediction. Figure 7 The example depicts four reference lines, where samples from segments A and F are not obtained from reconstructed neighboring samples, but are instead filled with the nearest samples from segments B and E, respectively. HEVC intra-prediction uses the nearest reference line (i.e., reference line 0). In MRL, two additional lines are used (reference line 1 and reference line 3).

[0156] The index (mrl_idx) of the selected reference line is signaled and used to generate the intra-predictor. For reference line idx greater than 0, only the additional reference line mode is included in the MPM list, and only the mpm index is signaled without any remaining modes. The reference line index is signaled before the intra-predictor mode, and if a non-zero reference line index is signaled, the planar mode is excluded from the intra-predictor mode.

[0157] MRL is disabled for the first line of a block within the CTU to prevent the use of extended reference samples outside the current CTU line. Additionally, PDPC is disabled when additional lines are used. For MRL mode, the derivation of the DC value in the intra-prediction mode for non-zero reference line indices is aligned with the derivation of the DC value at reference line index 0. MRL requires three neighboring lumen reference lines stored in the CTU to generate predictions. The CCLM tool also requires three neighboring lumen reference lines for its downsampling filters. The definition of MRL using the same three lines is aligned to CCLM to reduce the decoder's storage requirements.

[0158] Intra-frame sub-partition (ISP)

[0159] Intra-Frame Sub-Partitioning (ISP) divides the luma intra-prediction block vertically or horizontally into 2 or 4 sub-partitions based on the block size. For example, the minimum block size for ISP is 4x8 (or 8x4). If the block size is larger than 4x8 (or 8x4), then the corresponding block is divided into 4 sub-partitions. It has been noted that M×128 (where M≤64) and 128×N (where N≤64) ISP blocks can generate potential problems with 64×64 VDPUs. For example, in a single-tree case, an M×128 CU has an M×128 luma TB and two corresponding M / 2×64 chroma TBs. If the CU uses ISP, the luma TB will be divided into four M×32 TBs (horizontal splitting is possible only), each smaller than a 64×64 block. However, in the current design of ISP, the chroma block is not partitioned. Therefore, the two chroma components will have a size larger than 32×32 blocks. Similarly, a similar case can be created using ISP with 128×N CUs. Therefore, both of these cases are issues with the 64×64 decoder pipeline. For this reason, the CU size of the ISP can be limited to a maximum of 64×64. All sub-partitions satisfy the condition of having at least 16 samples.

[0160] Matrix-weighted intra-frame prediction (MIP)

[0161] Matrix-weighted intra-prediction (MIP) is a new intra-prediction technique added to VVC. For a sample predicting a rectangular block of width W and height H, MIP takes one row of the H reconstructed neighboring boundary samples to the left of the block and one row of the W reconstructed neighboring boundary samples above the block as input. If reconstructed samples are unavailable, they are generated during the regular intra-prediction process. The generation of the predicted signal is based on three steps: averaging, matrix-vector multiplication, and linear interpolation, as follows: Figure 8 As shown.

[0162] Inter-frame prediction in VVC

[0163] The merge list may include the following candidates: 1) Spatial MVP from adjacent CUs 2) Time MVP from co-located CU 3) Historical MVPs from FIFO tables 4) Pairwise average MVP (using candidates already in the list) 5) Zero MV.

[0164] The merged mode width motion vector difference (MMVD) refers to the signaling notification of the MVD and resolution index after signaling merging.

[0165] In symmetric MVD, with bidirectional prediction, the motion information of list-1 is derived from the motion information of list-0.

[0166] In affine prediction, several motion vectors are indicated / signed for different corners of a block, which are used to derive the motion vectors of sub-blocks. In affine merging prediction, affine motion information of a block is generated based on the normal or affine motion information of neighboring blocks.

[0167] In sub-block-based temporal motion vector prediction, the motion vector of the current block's sub-block is predicted from the appropriate sub-block in the reference frame, indicated by the motion vectors of spatially neighboring blocks (if available).

[0168] In Adaptive Motion Vector Resolution (AMVR), the accuracy of the MVD is communicated to each CU via a signal.

[0169] In bidirectional forecasting with CU-level weights, the index indicates the weight value of the weighted average of the two forecast blocks.

[0170] Two-way optical flow (BDOF) refines motion vectors in the case of two-way prediction. BDOF generates two prediction blocks using motion vectors fed by a signal. Motion refinement is then computed to minimize the error between the two prediction blocks using its gradient value. The final prediction block is refined using the motion refinement and the gradient value.

[0171] Two-way prediction (BCW) and weighted prediction (WP) with CU-level weights

[0172] In HEVC, a bidirectional prediction signal is generated by averaging two prediction signals obtained from two different reference images and / or using two different motion vectors. In VVC, the bidirectional prediction mode is extended beyond simple averaging to allow a weighted average of the two prediction signals. P bi-pred =((8-w)*P0+w*P1+4)>>3

[0173] Five weights are allowed in weighted average bidirectional prediction, w∈{-2,3,4,5,10}. For each bidirectional prediction CU, the weight w is determined in one of two ways: 1) for non-merged CUs, the weight index is signaled after the motion vector difference; 2) for merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is applied only to CUs with 256 or more luminance samples (i.e., CU width multiplied by CU height is greater than or equal to 256). For low-latency images, all 5 weights are used. For non-low-latency images, only 3 weights are used (w∈{3,4,5}).

[0174] At the encoder, a fast search algorithm is applied to find the weight index without significantly increasing the encoder complexity. These algorithms are summarized below. - When combined with AMVR, if the current image is a low-latency image, the unequal weights are conditionally checked only for 1-pixel and 4-pixel motion vector precision. - When combined with affine mode, affine ME will be performed for unequal weights if and only if the affine mode is selected as the current best mode. - When the two reference images in bidirectional prediction are the same, only conditionally check for unequal weights.

[0175] Under certain conditions, unequal weights are not searched, depending on the POC distance between the current image and its reference image, the QP encoding, and the time level.

[0176] The BCW weight index is encoded using a context-coded binary bit followed by a bypass-coded binary bit. The first context-coded binary bit indicates whether equal weights are used; if unequal weights are used, then the bypass-coded binary bit is used to signal which unequal weight is used.

[0177] Weighted Prediction (WP) is an encoding tool supported by the H.264 / AVC and HEVC standards to efficiently encode video content using fading. Support for WP will also be added to the VVC standard. WP allows signaling of weighting parameters (weights and offsets) for each reference picture in each of the reference picture lists L0 and L1. The corresponding reference picture's weights and offsets are then applied during motion compensation. WP and BCW are designed for different types of video content. To avoid interaction between WP and BCW, which would complicate the VVC decoder design, if the CU uses WP, the BCW weight index is not signaled, and w is inferred to be 4 (i.e., equal weights are applied). For merging CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. This can be applied to normal merging mode and inherited affine merging mode. For constructed affine merging mode, affine motion information is constructed based on motion information from up to 3 blocks. The BCW index for the CU using the constructed affine merging mode is simply set to be equal to the BCW index of the first control point MV.

[0178] In VVC, CIIP and BCW cannot be applied together to a CU. When a CU is encoded using CIIP mode, the BCW index of the current CU is set to 2, for example, with equal weights.

[0179] Combined inter-frame and intra-frame prediction (CIIP)

[0180] In VVC, when a CU is encoded in merged mode, if the CU contains at least 64 luma samples (i.e., the CU width multiplied by the CU height is equal to or greater than 64), and if both the CU width and CU height are less than 128 luma samples, an additional flag is signaled to indicate whether a combined inter-frame and intra-frame prediction (CIIP) mode is applied to the current CU. As its name suggests, CIIP prediction combines the inter-frame prediction signal with the intra-frame prediction signal. The inter-frame prediction signal P in CIIP mode... inter Derived using the same inter-frame prediction process applied to the regular merging mode; and the intra-frame prediction signal P intra This is derived after a standard intra-frame prediction process with a planar pattern. Then, a weighted average is used to combine the intra-frame and inter-frame prediction signals, where the weight values ​​depend on the top and left neighboring blocks (in...). Figure 12 The encoding pattern (described in the text) is calculated as follows: - If the top neighbor is available and intra-coded, set isIntraTop to 1; otherwise, set isIntraTop to 0. - If the left neighbor is available and intra-coded, set isIntraLeft to 1; otherwise, set isIntraLeft to 0. - If (isIntraLeft+isIntraTop) equals 2, then set wt to 3; Otherwise, if (isIntraLeft+isIntraTop) equals 1, then set wt to 2; Otherwise, set wt to 1.

[0181] CIIP predictions are formed as follows: P CIIP =((4-wt)*P inter +wt*P intra +2)>>2

[0182] Localized Illumination Compensation (LIC)

[0183] LIC is an inter-frame prediction technique used to model the local illumination variation between the current block and its predicted block, as a function of the local illumination variation between the current block template and the reference block template. The parameters of the function can be represented by the scale α and the offset β, forming a linear equation, i.e., α*p[x]+β, to compensate for the illumination variation, where p[x] is the reference sample pointed to by MV at position x on the reference image. Since α and β can be derived based on the current block template and the reference block template, no signaling overhead is required for them, except for signaling the LIC flag for AMVP mode to indicate the use of LIC.

[0184] The local illumination compensation proposed in JVET-O0066 is used in ECM for unidirectional prediction of inter-frame CU, with the following modifications. • Intra-neighbor samples can be used to derive LIC parameters; • Disable LIC for blocks with fewer than 32 luminance samples; • For both non-subblock and affine modes, LIC parameters are derived based on the template block sample corresponding to the current CU, rather than the partial template block sample corresponding to the first top-left 16x16 cell. • Samples of the reference block template are generated by using a MC with block MV without rounding it to integer pixel precision.

[0185] Inter-frame coding in modern video coding standards (such as H.266) uses single-tree motion-compensated prediction: regions from one or more reference frames are copied to the target frame using subpixel interpolation. Additional tools, such as Weighted Prediction (WP), Bidirectional Prediction with CU-level weights (BCW), Combined Inter-Frame and Intra-Frame Prediction (CIIP), or Local Illumination Compensation (LIC), are used to account for variations in illumination and / or chromaticity between the reference and target frames. Single-tree coding signals a set of partition information based on blocks, and this partitioning is shared between the luma and chroma components. Due to the limited number of chroma-specific coding tools, inter-frame coding typically implicitly sets higher priority for luma blocks compared to chroma. Even with chroma-specific tools such as CCCM or CCLM, they are not very efficient because chroma blocks in the reference frame are generally of higher quality than cross-component blocks in intra-frame prediction. To benefit from cross-component tools in inter-frame prediction, there should be a way to model chroma primarily using luma and chroma with co-located motion compensation.

[0186] We now introduce an improved cross-component residual model for prediction.

[0187] According to an embodiment, a method is provided to directly map luma residuals to chroma predictions. Specifically, this mapping can be applied in single-tree inter-frame coded blocks, where the same partition is shared between luma and chroma blocks. An intuitive way of performing this mapping after decoding the luma residuals of a single-tree inter-frame block is disclosed. The method can also be extended to dual-tree blocks or any other partitioning scheme. Alternative configurations such as chroma-to-chroma are also possible.

[0188] Single-tree coding shares the same partitions and motion between the luma (Y) and chroma (Cb, Cr) components. Typically, the reconstruction process for a single-tree inter-frame coded block is as follows: First, motion-compensated prediction is performed using a reference frame and sub-pixel motion vectors; second, additional adjustments are made using tools such as WP, BCW, CIIP, or LIC; and third, residuals are applied. These steps can be applied sequentially to the luma and chroma components. Cross-component prediction has demonstrated the ability to locally model chroma using linear or nonlinear models that map the luma component to chroma. Such a model can be formulated as: C=f(Y)+∈ c , Where C is chromaticity, f(·) is the function that maps luminance to chromaticity, and ∈ c This is the residual chroma. In existing cross-component tools (such as CCCM), the function f(·) is obtained using the nearest reference row, and the co-located luminance value is used as input to apply the function f(·). Typically, cross-component tools are used for dual-tree or single-tree intra-frame coding. For single-tree inter-frame coding, the function f(·) can be obtained using the co-located luminance after inter-frame prediction, including motion compensation and any optional additional tools such as WP / BCW / CIIP / LIC, but the application of the luminance residual ∈ y Previously, the final chromaticity prediction mapped the luminance residual into the chromaticity prediction. C=f(Y+∈ y )+∈ c .

[0189] The parameters of the model f(·) can be obtained using CCCM, CCLM, or any other cross-component method. The model can be one-dimensional (1D) or two-dimensional (2D) and can include nonlinear terms.

[0190] According to the embodiment, there are four main steps in the cross-component residual model (CCRM). These steps are listed below: 1) Perform motion compensation prediction on the luminance component Y and chrominance components Cb and Cr (optionally including additional refinement such as WP / BCW / CIIP / LIC). 2) Using co-located luminance prediction and chrominance prediction, functions f are obtained for the chrominance components Cb and Cr, respectively. Cb f Cr . 3) Reconstructing the brightness residual ∈ y . 4) Obtain chromaticity prediction C Cb =f Cb (Y+∈ y ) and C Cr =f Cr (Y+∈ y ).

[0191] Signaling for CCRM can be accomplished using, for example, TU / PU / CU level flags (a combined flag for Cb and Cr) or separate flags for Cb and Cr. Alternatively, tools providing additional predictive refinement can be applied after CCRM. In this case, the steps would be changed as follows: 1. Motion compensation prediction is performed using additional refinement for the luminance component Y and chrominance components Cb and Cr, where additional refinement such as WP / BCW / CIIP / LIC is applied only to the luminance component Y. 2. (Unchanged) Using the co-located luminance prediction and chrominance prediction, obtain the function f for the chrominance components Cb and Cr respectively. Cb f Cr . 3. (Unchanged) Reconstructed brightness residual ∈ y . 4. (Unchanged) Obtain chromaticity prediction chromaticity prediction C Cb =f Cb (Y+∈ y ) and C Cr =f Cr (Y+∈ y ). 5. Apply additional refinements, such as WP / BCW / CIIP / LIC for Cb and Cr prediction.

[0192] Following the last step, which is either step 4 or 5, depending on the two cases above, the decoder can reconstruct and add the chroma residual ∈ Cb ,∈ Cr This allows for further refinement of the reconstruction. The above list predicts chromaticity from luminance, but CCRM can also be configured to perform chromaticity-to-chromaticity or chromaticity-to-luminance prediction, as will be shown later.

[0193] Even after adding luminance residuals, the derived model f(·) should maintain its stability. Step 4a (an alternative to step 4 above) can be used to mitigate potential problems caused by large values ​​or high variance of the luminance residuals: 4a. Obtain chromaticity predictions. and in This represents the filtered version of the original luminance residual. Such filtering can be, for example, a cropping of a large value or spatial operation such as a low-pass filter, or multiplying the residual by a predetermined or signaled value, such as 0.5 or 0.75. The filtering need not be the same for the chromaticity components Cb and Cr.

[0194] In some cases, performing step 4 as a weighted sum of the original chromaticity prediction and the CCRM prediction may be beneficial, for example, as follows: 4b. Obtain chromaticity prediction and in and The initial inter-frame predicted chroma block is defined, and α and β are the mixing weights defined for the chroma components Cb and Cr, respectively.

[0195] As another example, step 4 can be performed by applying only the mapping function f(·) to the luminance residual, for example, as follows: 4c. Obtain chromaticity prediction and in and The initial inter-frame predicted chroma block is defined, and α and β are the mixing weights defined for the chroma components Cb and Cr, respectively.

[0196] Any combination of steps 4, 4a, 4b, and 4c above is possible; for example, a weighted summation w0*4a + w1*4b can be used, where w0 and w1 are the weights. The choice between steps 4, 4a, 4b, and 4c can occur at the sub-block level or even at the sample level. A further extension of the above method is to include chromaticity residuals in the model, resulting in chromaticity-to-chromaticity prediction: or in and Map Cr to Cb and Cb to Cr respectively. Typically, the Cb to Cr model is used due to the processing order. In some cases, it may be advantageous to derive a weighted sum of two different cross-component models as follows: or Where α and β are in C Cb With C Cr They are independent of each other. When the processing order of the components allows for the prediction of luminance from chroma, such a predictor can be designed as follows: Y=f(C+∈ c ), Where C can be Cb or Cr. In chromaticity-to-luminance prediction, the input or output of f(·) can be upsampled to match the luminance grid. Alternatively, we can also use two chromaticity components to predict the luminance as follows: Y=α*f0(Cb+∈ Cb )+β*f1(Cr+∈ Cr ), Where α and β are weights, and f0 and f1 map Cb to Y and Cr to Y, respectively.

[0197] Some example implementations will be described below.

[0198] In an embodiment, the co-located luminance component can be downsampled to match the subsampled chrominance grid.

[0199] In the embodiments, any 1D or 2D filter can be used to filter the co-located luminance component (including the residual).

[0200] In an embodiment, the mapping function f(·) can be obtained using any method used to derive a cross-component model such as CCCM or CCLM.

[0201] In the embodiments, the mapping function f(·) can contain any number of filter taps and can contain nonlinear terms, such as quadratic terms.

[0202] In the embodiments, the mapping function f(·) can be 1D or 2D.

[0203] In this embodiment, the mapping function f(·) can map luminance to chrominance or chrominance to chrominance. A final prediction can be obtained using any weighted sum of the two versions.

[0204] In an embodiment, the use of CCRM can be signaled from the encoder to the decoder at the TU, PU, ​​CU, CTU, strip, frame, or sequence level.

[0205] In an embodiment, the use of CCRM can be signaled individually or in combination for different components such as Cb, Cr, and Y.

[0206] In this embodiment, the luminance residuals can be clipped or spatially filtered before applying the cross-component model.

[0207] In the embodiment, the type and amount of filtering applied to the luminance residual in step 4a can be determined for Cb and Cr respectively.

[0208] In an embodiment, the type and amount of filtering applied to the luminance residual in step 4a can be determined based on the output of f(·).

[0209] In an embodiment, the type and amount of filtering applied to the luminance residual in step 4a can be applied differently at the sub-block or sample level.

[0210] In an embodiment, the mixed weights α and β of step 4b can be defined independently or jointly for Cb and Cr.

[0211] In an embodiment, the mixed weights α and β of step 4b can be defined at the sub-block level or even at the sample level.

[0212] In an embodiment, the encoder can signal the mixing weights α and β of step 4b to the decoder at the TU, CU, CTU, strip, frame, or sequence level.

[0213] In an embodiment, block prediction can be obtained as a weighted sum of different mapping functions, such as the mapping functions described in steps 4, 4a, and 4b.

[0214] In this embodiment, block prediction can use different mapping functions at the sub-block level.

[0215] In an embodiment, block prediction can use different mapping functions based on thresholds derived from luminance, luminance residuals, or chromaticity. For example, a threshold (based on, for example, the average luminance over the block) can be set, and different mapping functions are derived and used below or above that threshold.

[0216] In this embodiment, the mapping function f(·) maps chroma to luminance. In chroma-to-luminance prediction, the input of f(·) can be upsampled to match the luminance grid. In chroma-to-luminance prediction, the output of f(·) can be upsampled to match the luminance grid.

[0217] Figure 9 The diagram illustrates a method according to one aspect, wherein the method includes: receiving (900) image block units of a frame, the image block units including samples in a first chroma channel, a second chroma channel, and a luminance channel; defining (902) co-located reference regions on the luminance component of the luminance channel, the chroma component of the first chroma channel, and the chroma component of the second chroma channel; performing (904) motion-compensated prediction on the luminance component, the first chroma component, and the second chroma component using the reference frame and a motion vector; obtaining (906) a first mapping function and a second mapping function using the co-located luminance prediction and chroma prediction, the first mapping function mapping the luminance component to the first chroma component, and the second mapping function mapping the luminance component to the second chroma component; reconstructing (908) a luminance residual; and obtaining (910) a first chroma prediction by using the first function and the luminance residual, and obtaining a second chroma prediction by using the second function and the luminance residual.

[0218] An apparatus according to one aspect includes: components for receiving image block units of a frame, the image block unit including samples in a color channel including at least one chroma channel and a luminance channel; components for determining an intra-frame prediction direction based on the samples of the luminance channel of the image block unit; components for selecting a low-frequency inseparable transform index for a low-frequency inseparable transform using the determined intra-frame prediction direction; and components for performing a low-frequency inseparable transform using the selected index.

[0219] According to an embodiment, the apparatus includes a component for downsampling a sample of the luminance channel to correspond to the sample size of the chrominance sample before determining the filter coefficients.

[0220] In another aspect, an apparatus is provided, comprising: at least one processor and at least one memory, wherein code is stored on the at least one memory, the code, when executed by the at least one processor, causes the apparatus to perform at least: receiving components of an image patch unit of a frame, the image patch unit including samples in a color channel comprising at least one chroma channel and a luminance channel; reconstructing samples of the luminance channel of the image patch unit; determining an intra-frame prediction direction based on the samples of the luminance channel of the image patch unit; selecting a low-frequency inseparable transform index for a low-frequency inseparable transform using the determined intra-frame prediction direction; and performing a low-frequency inseparable transform using the selected index.

[0221] Such a device may include, for example, in Figure 1 , Figure 2 , Figure 4a and Figure 4b The functional units disclosed in any of the embodiments for implementing the embodiments.

[0222] Such a device also includes code stored in the at least one memory, which, when executed by the at least one processor, causes the device to perform one or more embodiments disclosed herein.

[0223] Figure 13This is a graphical representation of an example multimedia communication system in which various embodiments can be implemented. Data source 1510 provides a source signal in analog, uncompressed digital, or compressed digital formats, or any combination of these formats. Encoder 1520 may include or be connected to preprocessing, such as data format conversion and / or filtering of the source signal. Encoder 1520 encodes the source signal into an encoded media bitstream. It should be noted that the bitstream to be decoded can be received directly or indirectly from a remote device located within virtually any type of network. Alternatively, the bitstream can be received from local hardware or software. Encoder 1520 is capable of encoding more than one media type (such as audio and video), or may require more than one encoder 1520 to encode different media types of the source signal. Encoder 1520 may also receive synthesized input, such as graphics and text, or encoded bitstreams of synthesized media. In the following description, the processing of only one encoded media bitstream of one media type is considered for simplicity. However, it should be noted that live broadcast services typically include several streams (typically at least one audio, video, and text caption stream). It should also be noted that a system may include many encoders, but only one encoder 1520 is shown in the figure for simplicity, without lacking generality. It should also be understood that although the text and examples contained herein may specifically describe the encoding process, those skilled in the art will understand that the same concepts and principles apply to the corresponding decoding process, and vice versa.

[0224] Encoded media bitstreams can be transmitted to storage device 1530. Storage device 1530 may include any type of mass storage for storing the encoded media bitstreams. The format of the encoded media bitstreams in storage device 1530 may be a basic self-contained bitstream format, or one or more encoded media bitstreams may be encapsulated into a container file, or the encoded media bitstreams may be encapsulated into a fragment format suitable for DASH (or a similar streaming system) and stored as a sequence of fragments. If one or more media bitstreams are encapsulated in a container file, a file generator (not shown) may be used to store one or more media bitstreams in the file and create file format metadata that can also be stored in the file. Encoder 1520 or storage device 1530 may include a file generator, or a file generator may be operatively attached to encoder 1520 or storage device 1530. Some systems operate "in-place," omitting storage and transmitting the encoded media bitstream directly from encoder 1520 to sender 1540. The encoded media bitstream can then be transmitted to sender 1540, also known as a server, as needed. The format used in transmission can be a basic self-contained bitstream format, a packet stream format, a fragment format suitable for DASH (or a similar streaming system), or one or more encoded media bitstreams can be encapsulated into a container file. Encoder 1520, storage device 1530, and server 1540 can reside in the same physical device, or they can be included in separate devices. Encoder 1520 and server 1540 can operate with live content, in which case the encoded media bitstream is typically not permanently stored, but rather cached for short periods in the content encoder 1520 and / or server 1540 to smooth out variations in latency, transmission delay, and encoded media bitrate.

[0225] Server 1540 uses a communication protocol stack to send encoded media bitstreams. This stack can include, but is not limited to, one or more of Real-Time Transport Protocol (RTP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Transmission Control Protocol (TCP), and Internet Protocol (IP). When the communication protocol stack is packet-oriented, server 1540 encapsulates the encoded media bitstream into packets. For example, when using RTP, server 1540 encapsulates the encoded media bitstream into RTP packets according to the RTP payload format. Typically, each media type has a dedicated RTP payload format. It should be noted again that the system can contain more than one server 1540, but for simplicity, the following description considers only one server 1540.

[0226] If the media content is encapsulated in a container file for storage device 1530 or for inputting data to sender 1540, sender 1540 may include or be operatively attached to a "transmit file parser" (not shown). Specifically, if the container file is not transmitted in this way, but at least one of the contained encoded media bitstreams is encapsulated for transmission via a communication protocol, the send file parser locates the appropriate portion of the encoded media bitstream to be transmitted via the communication protocol. The send file parser can also help create the correct format for the communication protocol, such as packet headers and payloads. The multimedia container file may contain encapsulation instructions, such as prompt tracks in ISOBMFF, for encapsulating at least one of the contained media bitstreams over the communication protocol.

[0227] Server 1540 may or may not be connected to gateway 1550 via a communication network (e.g., a CDN, the Internet, and / or a combination of one or more access networks). The gateway may also be, or alternatively, referred to as a middlebox. For DASH, the gateway may be an edge server (of a CDN) or a web proxy. Note that a system can typically include any number of gateways or the like, but for simplicity, the following description considers only one gateway 1550. Gateway 1550 can perform different types of functions, such as converting packet streams to another communication protocol stack according to one, merging and forking data streams, and manipulating data streams according to downlink and / or receiver capabilities, such as controlling the bit rate of forwarded streams based on prevailing downlink network conditions. In various embodiments, gateway 1550 may be a server entity.

[0228] The system includes one or more receivers 1560, which are typically capable of receiving, demodulating, and decapsulating transmitted signals into an encoded media bitstream. The encoded media bitstream can be transferred to a recording storage device 1570. The recording storage device 1570 may include any type of mass storage for storing the encoded media bitstream. The recording storage device 1570 may optionally or additionally include computational memory, such as random access memory. The format of the encoded media bitstream in the recording storage device 1570 may be a basic self-contained bitstream format, or one or more encoded media bitstreams may be encapsulated into a container file. If multiple encoded media bitstreams (e.g., audio and video streams) are associated with each other, then a container file is typically used, and the receiver 1560 includes or is attached to a container file generator that produces container files from the input streams. Some systems operate "in-place," omitting the recording storage device 1570 and transferring the encoded media bitstream directly from the receiver 1560 to the decoder 1580. In some systems, only the most recent portion of the recording stream (e.g., the most recent 10-minute extract of the recording stream) is retained in the recording storage device 1570, while any earlier recorded data is discarded from the recording storage device 1570.

[0229] Encoded media bitstreams can be transferred from recording storage device 1570 to decoder 1580. If many encoded media bitstreams (e.g., audio and video streams) are associated with each other and encapsulated in a container file, or if a single media bitstream is encapsulated in a container file, for example for easier access, a file parser (not shown) is used to decapsulate each encoded media bitstream from the container file. Recording storage device 1570 or decoder 1580 may include a file parser, or a file parser may be attached to recording storage device 1570 or decoder 1580. It should also be noted that the system may include many decoders, but only decoder 1570 is discussed here for simplicity, without lack of generality.

[0230] The encoded media bitstream can be further processed by decoder 1570, the output of which is one or more uncompressed media streams. Finally, for example, renderer 1590 can reproduce the uncompressed media stream using speakers or a display. Receiver 1560, recording storage device 1570, decoder 1570, and renderer 1590 can reside in the same physical device, or they can be included in separate devices.

[0231] Sender 1540 and / or gateway 1550 can be configured to perform switching between different representations, such as switching between different viewports of 360-degree video content, view switching, bitrate adaptation, and / or fast startup, and / or sender 1540 and / or gateway 1550 can be configured to select the representation to send. Switching between different representations can occur for various reasons, such as in response to a request from receiver 1560 or general conditions of the network transmitting the bitstream, such as throughput. In other words, receiver 1560 can initiate a switching between representations. Requests from the receiver can be, for example, requests for segments or sub-segments from representations different from earlier representations, requests for changes to the scalability layer and / or sub-layers being sent, or changes to rendering devices with different capabilities compared to a previous one. Requests for segments can be HTTP GET requests. Requests for segments can be HTTP GET requests with byte ranges. Additionally or alternatively, bit rate adjustment or bit rate adaptation can be used, for example, in streaming services to provide so-called fast start, where the bit rate of the transmitted stream is lower than the channel bit rate after the start of or random access streaming, so as to start playback immediately and achieve a buffer occupancy level that tolerates occasional packet delays and / or retransmissions. Bit rate adaptation may include multiple representation or layer-up switching and representation or layer-down switching operations occurring in various orders.

[0232] Decoder 1580 can be configured to switch between different representations, such as for switching between different viewports of 360-degree video content, view switching, bitrate adaptation, and / or fast startup, and / or decoder 1580 can be configured to select the representation to be transmitted. Switching between different representations can occur for a variety of reasons, such as to achieve faster decoding operations or to adapt the transmitted bitstream (e.g., according to bitrate) to the general conditions of the network transmitting the bitstream (e.g., throughput). For example, if the device including decoder 1580 is multitasking and uses computational resources for purposes other than decoding video bitstreams, faster decoding operations may be required. In another example, faster decoding operations may be required when content is played back at a faster speed than normal playback speed, for example, two or three times faster than the usual real-time playback rate.

[0233] In the foregoing, some embodiments have been described with reference to and / or using the terminology of HEVC and / or VVC. It should be understood that the embodiments can be implemented similarly to any video encoder and / or video decoder.

[0234] Having described the example embodiments with reference to the encoder above, it is to be understood that the resulting bitstream and decoder may have corresponding elements therein. Similarly, having described the example embodiments with reference to the decoder, it is to be understood that the encoder may have a structure and / or computer program for generating the bitstream to be decoded by the decoder. For example, some embodiments relating to generating prediction blocks as part of encoding have been described. Embodiments can be similarly implemented by generating prediction blocks as part of decoding, wherein the encoding parameters decoded from the bitstream differ from the encoding parameters determined by the encoder, such as horizontal and vertical offsets.

[0235] The embodiments of the present invention described above are based on separate encoder and decoder devices to aid in understanding the processes involved. However, it should be understood that the device, structure, and operation can be implemented as a single encoder-decoder device / structure / operation. Furthermore, the encoder and decoder may share some or all common elements.

[0236] While the examples above describe embodiments of the invention operating within a codec in an electronic device, it should be understood that the invention as defined in the claims can be implemented as part of any video codec. Therefore, for example, embodiments of the invention can be implemented in a video codec that enables video encoding via a fixed or wired communication path.

[0237] Therefore, user equipment may include video codecs, such as those described in the embodiments of the present invention above. It should be understood that the term "user equipment" is intended to cover any suitable type of wireless user equipment, such as mobile phones, portable data processing devices, or portable web browsers.

[0238] In addition, the components of a Public Land Mobile Network (PLMN) may also include video codecs as described above.

[0239] Generally, various embodiments of the present invention can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, while others may be implemented in firmware or software executable by a controller, microprocessor, or other computing device, but the invention is not limited thereto. While various aspects of the invention may be shown and described as block diagrams, flowcharts, or using some other graphical representation, it should be understood that these blocks, apparatuses, systems, techniques, or methods described herein may be implemented, in non-limiting examples, as hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0240] Embodiments of the present invention can be implemented by computer software executed by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Furthermore, it should be noted in this regard that any block of the logical flow in the figures may represent a program step, or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips, or in memory blocks implemented within a processor, magnetic media (such as hard disks or floppy disks), and optical media (such as, for example, DVDs and their data variants, CDs).

[0241] The memory can be of any type suitable for the local technical environment and can be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. As a non-limiting example, the data processor can be of any type suitable for the local technical environment and can include one or more of general-purpose computers, special-purpose computers, microprocessors, digital signal processors (DSPs), and processors based on multi-core processor architectures.

[0242] Embodiments of the present invention can be implemented in various components such as integrated circuit modules. The design of integrated circuits is a highly automated process. Complex and powerful software tools can be used to transform logic-level designs into semiconductor circuit designs ready to be etched and formed on a semiconductor substrate.

[0243] Programs such as those provided by Synopsys, based in Mountain View, California, automatically route conductors and locate components on semiconductor chips using well-established design rules and a library of pre-stored design modules. Once the design for the semiconductor circuit is complete, the resulting design, in a standardized electronic format (e.g., Opus, GDSII, etc.), can be transferred to a semiconductor manufacturing facility or "fab" for fabrication.

[0244] The foregoing description provides a complete and informative description of exemplary embodiments of the invention through exemplary and non-limiting examples. However, various modifications and adaptations may become apparent to those skilled in the art when read in conjunction with the accompanying drawings and appended claims, given the foregoing description. Nevertheless, all such modifications and similar alterations to the teachings of the invention will still fall within the scope of the invention.

Claims

1. An apparatus for decoding, comprising: Components for receiving image block units of frames, the image block units including samples in a first chroma channel, a second chroma channel and a luminance channel; A component for defining a co-located reference region on the luminance sample of the luminance channel, the first chroma sample of the first chroma channel, and the second chroma sample of the second chroma channel; A component for performing motion-compensation prediction on the luminance sample, the first chromaticity sample, and the second chromaticity sample in the co-located reference region; Components for obtaining a first mapping function using the motion-compensated predicted luminance sample and the first chroma sample, and obtaining a second mapping function using the motion-compensated predicted luminance sample and the second chroma sample, wherein the first mapping function maps the luminance sample to the first chroma sample, and the second mapping function maps the luminance sample to the second chroma sample; Components used to reconstruct luminance residuals; as well as A component for obtaining a first chromaticity prediction by using the first mapping function and the luminance residual, and for obtaining a second chromaticity prediction by using the second mapping function and the luminance residual.

2. The apparatus according to claim 1, further comprising: A component for downsampling luminance samples in the co-located reference region to match the subsampled chromaticity grid.

3. The apparatus according to any one of claims 1 to 2, further comprising: A component for filtering brightness samples in the co-located reference region using a one-dimensional or two-dimensional filter; as well as A component for performing texture analysis, wherein the texture analysis includes gradient calculation for the brightness sample.

4. The apparatus according to any one of claims 1 to 2, further comprising: A component used to signal the use of cross-component residual models from the encoder to the decoder at the transform unit, prediction unit, decoding unit, decoding tree unit, strip, frame, or sequence level.

5. The apparatus according to any one of claims 1 to 2, comprising: A component for cropping or spatially filtering the luminance residual before obtaining a first chromaticity prediction by using the first mapping function and the luminance residual and obtaining a second chromaticity prediction by using the second mapping function and the luminance residual.

6. The apparatus according to any one of claims 1 to 2, further comprising: A component for determining the type and amount of filtering to be applied to the luminance residual for the first chromaticity sample and the second chromaticity sample, respectively.

7. The apparatus according to any one of claims 1 to 2, further comprising: A component for determining the type and amount of filtering to be applied to the luminance residual based on the output of the first mapping function or the output of the second mapping function.

8. The apparatus according to claim 7, further comprising: A component for applying the type and amount of the filter differently to the brightness residual at the sub-block or sample level.

9. The apparatus according to any one of claims 1 to 2, further comprising: A component for using different mapping functions based on one or more thresholds derived from the luminance channel, luminance residual, or the first and second chroma channels.

10. The apparatus according to any one of claims 1 to 2, further comprising: A component used to map the first and second chromaticity samples to luminance samples.

11. The apparatus according to any one of claims 1 to 2, comprising at least one of the following: Components for upsampling the inputs of the first and second mapping functions to match the brightness grid; and A component used to upsample the outputs of the first and second mapping functions to match the brightness grid.

12. The apparatus according to any one of claims 1 to 2, further comprising: A component used to map the first chromaticity sample to the second chromaticity sample.

13. A method for decoding, comprising: The image block unit of the received frame includes samples in a first chroma channel, a second chroma channel, and a luminance channel; A co-located reference region is defined on the luminance sample of the luminance channel, the first chroma sample of the first chroma channel, and the second chroma sample of the second chroma channel; Motion-compensation prediction is performed on the luminance sample, the first chromaticity sample, and the second chromaticity sample in the co-located reference region; A first mapping function is obtained using the motion-compensated predicted luminance sample and the first chromaticity sample, and a second mapping function is obtained using the motion-compensated predicted luminance sample and the second chromaticity sample, wherein the first mapping function maps the luminance sample to the first chromaticity sample, and the second mapping function maps the luminance sample to the second chromaticity sample; Reconstructing the brightness residual; as well as A first chromaticity prediction is obtained by using the first mapping function and the luminance residual, and a second chromaticity prediction is obtained by using the second mapping function and the luminance residual.

14. The method of claim 13, further comprising: The luminance samples in the co-located reference region are downsampled to match the chromaticity grid of the subsampled region.

Images