Apparatus, method, and computer program for video coding and decoding

Abstract

12. A method for predicting target samples of at least one color channel of an image block unit using a cross-component prediction model. The method includes: receiving (1200) image block units of a frame, the image block units including samples in color channels, the color channels including at least one chrominance channel and one luminance channel; reconstructing (1202) samples of the luminance channel of the image block unit; determining (1204) a reference area for predicting target samples of at least one color channel of the image block unit, the reference area including one or more of reference samples in adjacent blocks in a current color channel / frame, reference samples adjacent to a co-located block in a reference color channel / frame, and / or reference samples inside the co-located block in a reference color channel / frame; determining (1206) gradient values ​​and / or position values ​​of spatial samples in the reference area to be used in a cross-component prediction model; and predicting (1208) the target samples of at least one color channel of the image block unit using the cross-component prediction model based on at least the gradient values ​​and / or the position values ​​of spatial samples in the reference area.

Description

[Technical Field]

[0001] The present invention relates to an apparatus, a method, and a computer program for video coding and decoding. [Background technology]

[0002] In video coding, video and image samples are typically encoded using color representations such as YUV or YCbCr, which consist of one luminance (luma) channel and two chrominance (chroma) channels. In these cases, the luminance channel, which primarily represents the illumination of the scene, is typically coded at a particular resolution, while the chrominance channels, which typically represent the differences between specific color components, are coded at a second resolution, often lower than that of the luminance signal. The intent of this type of differential representation is to decorrelate the color components, allowing the data to be compressed more efficiently.

[0003] A cross-component linear model (CCLM) and a convolutional cross-component model (CCCM) are used to predict samples in chroma channels (e.g., Cb and Cr) by cross-channel correlation (e.g., using luma samples). The model parameters are derived based on neighboring reconstructed samples of the chroma block, co-located adjacent samples in the luma block, and reconstructed samples within the co-located luma block.

[0004] Because cross-component prediction models (e.g., CCLM and CCCM) use spatial samples to derive model parameters, cross-component prediction models may not always perform optimally for certain content if spatial correlation is not present or is not the main contributor to cross-channel correlation. Summary of the Invention

[0005] Now, to at least alleviate the above-mentioned problems, an enhanced method is introduced herein to better model cross-channel correlation in the cross-component model.

[0006] The scope of protection sought for various embodiments of the invention is set forth in the independent claims. The embodiments and features described herein that do not fall within the scope of the independent claims, if any, should be interpreted as examples useful for understanding various embodiments of the invention.

[0007] A method according to a first aspect includes receiving an image block unit of a frame, the image block unit including samples in color channels, the color channels including at least one chrominance channel and one luminance channel; reconstructing the samples of the luminance channel of the image block unit; determining a reference area for predicting target samples of at least one color channel of the image block unit, the reference area including one or more of reference samples in adjacent blocks in a current color channel / frame, reference samples adjacent to a co-located block in a reference color channel / frame, and / or reference samples inside the co-located block in a reference color channel / frame; determining gradient values ​​and / or position values ​​of spatial samples in the reference area to be used in a cross-component prediction model; and predicting the target samples of at least one color channel of the image block unit using the cross-component prediction model based on at least the gradient values ​​and / or the position values ​​of the spatial samples in the reference area.

[0008] An apparatus according to a second aspect includes: means for receiving an image block unit of a frame, the image block unit including samples in color channels, the color channels including at least one chrominance channel and one luminance channel; means for reconstructing samples of the luminance channel of the image block unit; means for determining a reference area for predicting target samples of at least one color channel of the image block unit, the reference area including one or more of reference samples in adjacent blocks in a current color channel / frame, reference samples adjacent to a co-located block in a reference color channel / frame, and / or reference samples inside the co-located block in a reference color channel / frame; means for determining gradient values ​​and / or position values ​​of spatial samples in the reference area to be used in a cross-component prediction model; and means for predicting the target samples of at least one color channel of the image block unit using the cross-component prediction model based on at least the gradient values ​​and / or the position values ​​of spatial samples in the reference area.

[0009] According to one embodiment, the apparatus comprises means for using said gradient values ​​and / or said position values ​​of spatial samples together with or instead of spatial sample values ​​in a cross-component prediction model.

[0010] According to one embodiment, the apparatus includes means for using a combination of said gradient values ​​and said position values ​​of spatial samples in a cross-component prediction model, together with or instead of the spatial sample values.

[0011] According to one embodiment, the apparatus includes means for indicating the use of gradient values ​​and / or position values ​​of spatial samples when signaling within or along a bitstream including predicted image block units.

[0012] According to one embodiment, the device is configured to use the following cross-component prediction model: pred=P0·C+P1·N+P2·S+P3·W+P4·E+…+P n ·grad hor +P n+1 ·grad ver +P n+2 ·grad diag +…+P M-1 β and means for predicting the target sample of at least one color channel of an image block unit using where pred is the predicted chroma sample, P is the model coefficient for each tap of the M-tap filter, (C,N,S,W,E) are example spatial samples at locations (center, north, south, west, east), and grad hor , grad ver , grad diag are the directional gradient values ​​in the horizontal, vertical, and diagonal directions, respectively, and β is the offset value.

[0013] According to one embodiment, the apparatus comprises means for using the sum or average of gradient values ​​in different directions as parameters of a cross-component prediction model.

[0014] According to one embodiment, the apparatus includes means for using one or more of the gradient values ​​in a non-linear function and using the output of the non-linear function as a parameter of a cross-component prediction model.

[0015] According to one embodiment, the apparatus includes means for using higher order derivatives of the gradient values ​​as parameters of a cross-component prediction model.

[0016] According to one embodiment, the device is configured to use the following cross-component prediction model: pred=P0·C+P1·N+P2·S+P3·W+P4·E+…+P n X+P n+1 Y+…+P M-1 β where pred is a predicted chroma sample, P is a model coefficient for each tap of an M-tap filter, (C,N,S,W,E) are example spatial samples at locations (center, north, south, west, east), X and Y are the horizontal and vertical coordinates of the center sample, respectively, and β is an offset value.

[0017] According to one embodiment, the device comprises means for using the horizontal and / or vertical location of the central sample.

[0018] According to one embodiment, the apparatus comprises means for using the horizontal and / or vertical locations of spatially adjacent samples together with or instead of the location information of the central sample.

[0019] According to one embodiment, the apparatus includes means for using one or more of the position values ​​in a non-linear function and using the output of the non-linear function as parameters of a cross-component prediction model.

[0020] According to one embodiment, the apparatus includes means for determining, using a rate-distortion optimization (RDO) algorithm, a combination of a filter and the gradient values ​​and / or the position values ​​of spatial samples to be used together with or instead of spatial sample values ​​in a cross-component prediction model.

[0021] Thus, as described above, devices and computer-readable storage media having code stored thereon are configured to perform one or more of the above-described methods and related embodiments.

[0022] For a better understanding of the present invention, reference will now be made, by way of example, to the accompanying drawings in which: [Brief explanation of the drawings]

[0023] [Figure 1]1 is a diagram illustrating a schematic diagram of an electronic device utilizing an embodiment of the present invention. [Figure 2] FIG. 1 illustrates a schematic diagram of user equipment suitable for utilizing embodiments of the present invention; [Figure 3] FIG. 10 further illustrates, in a schematic manner, electronic devices utilizing embodiments of the present invention connected using wireless and wired network connections. [Figure 4a] 1 illustrates schematically an encoder and decoder suitable for implementing embodiments of the present invention; [Figure 4b] 1 illustrates schematically an encoder and decoder suitable for implementing embodiments of the present invention; [Figure 5] FIG. 1 shows the sample locations used to derive the parameters of the cross-component linear model (CCLM). [Figure 6a] FIG. 1 illustrates an example of classifying luma samples into two classes in the sample domain and spatial domain, respectively. [Figure 6b] FIG. 1 illustrates an example of classifying luma samples into two classes in the sample domain and spatial domain, respectively. [Figure 7] FIG. 10 illustrates an example of a co-located reference sample area consisting of reconstructed luma and chroma samples defined for both luma and chroma in a convolutional cross-component model (CCCM). [Figure 8] 10A and 10B are diagrams illustrating various examples of filter kernel dimensions in CCCM. [Figure 9] FIG. 10 is a diagram showing an example of four reference lines adjacent to a prediction block. [Figure 10] FIG. 1 illustrates a matrix weighted intra prediction process. [Figure 11] FIG. 1 illustrates an example of a low frequency non-separable transform (LFNST) process. [Figure 12] 3 is a flow diagram of a method for predicting samples in at least one color channel according to an embodiment of the present invention. [Figure 13]1 is a schematic diagram of an example multimedia communication system in which various embodiments may be implemented; DETAILED DESCRIPTION OF THE INVENTION

[0024] The following describes in more detail suitable apparatus and possible mechanisms for chroma sample prediction. In this regard, reference is first made to Figures 1 and 2, where Figure 1 shows a block diagram of a video coding system according to an example embodiment as a schematic block diagram of an example apparatus or electronic device 50 that can incorporate a codec according to an embodiment of the present invention. Figure 2 shows the layout of an apparatus according to an example embodiment. The elements of Figures 1 and 2 are now described.

[0025] Electronic device 50 may be, for example, a mobile terminal or user equipment of a wireless communication system, but it will be appreciated that embodiments of the present invention may be implemented in electronic devices or apparatuses that may require encoding and / or decoding of video images.

[0026] The device 50 may include a housing 30 for housing and protection of the device. The device 50 may further include a display 32 in the form of a liquid crystal display. In other embodiments of the invention, the display may be any suitable display technology suitable for displaying images or video. The device 50 may further include a keypad 34. In other embodiments of the invention, any suitable data or user interface mechanism may be utilized. For example, the user interface may be implemented as a virtual keyboard or data entry system as part of a touch-sensitive display.

[0027] The device may include a microphone 36 or any suitable audio input, which may be a digital or analog signal input. The device 50, in embodiments of the present invention, may further include an audio output device, which may be any one of earphones 38, a speaker, or an analog or digital audio output connection. The device 50 may further include a battery (or in other embodiments of the present invention, the device may be powered by any suitable portable energy device, such as a solar cell, a fuel cell, or a clockwork generator). The device may further include a camera capable of recording or capturing images and / or video. The device 50 may further include an infrared port for short-range line-of-sight communication with other devices. In other embodiments, the device 50 may further include any suitable short-range communication solution, such as a Bluetooth wireless connection, a USB / Firewire wired connection, etc.

[0028] The device 50 may include a controller 56, processor, or processor circuitry for controlling the device 50. The controller 56, in embodiments of the present invention, may be connected to a memory 58 that may store data, both in the form of image and audio data, and / or may further store instructions for execution by the controller 56. The controller 56 may further be connected to a codec circuit 54 suitable for performing coding and decoding of audio and / or video data or for assisting in the coding and decoding performed by the controller.

[0029] The device 50 may further include a card reader 48 and a smart card 46, for example a UICC and UICC reader suitable for providing user information and providing authentication information for authentication and authorization of the user on the network.

[0030] The device 50 may include a radio interface circuit 52 connected to the controller and suitable for generating radio communication signals for communication with, for example, a cellular communication network, a radio communication system, or a wireless local area network. The device 50 may further include an antenna 44 connected to the radio interface circuit 52 for transmitting radio frequency signals generated by the radio interface circuit 52 to other devices and receiving radio frequency signals from other devices.

[0031] The device 50 may include a camera capable of recording or detecting individual frames, which are then passed to a codec 54 or controller for processing. The device may receive video image data for processing from another device before transmission and / or storage. The device 50 may further receive images for coding / decoding wirelessly or via a wired connection. The structural elements of the device 50 described above represent examples of means for performing the corresponding functions.

[0032] With reference to Figure 3, an example of a system in which embodiments of the present invention may be utilized is shown. System 10 includes a number of communication devices that may communicate over one or more networks. System 10 may include any combination of wired or wireless networks, including, but not limited to, wireless cellular telephone networks (such as GSM, UMTS, CDMA networks, etc.), wireless local area networks (WLANs) such as those defined by any of the IEEE 802.x standards, Bluetooth personal area networks, Ethernet local area networks, token ring local area networks, wide area networks, and the Internet.

[0033] System 10 may include both wired and wireless communication devices and / or apparatus 50 suitable for implementing embodiments of the present invention.

[0034] 3 shows a cellular network 11 and a representation of the Internet 28. Connections to the Internet 28 can include, but are not limited to, long-range wireless connections, short-range wireless connections, and various wired connections, including, but not limited to, telephone lines, cable lines, power lines, and similar communication paths.

[0035] Exemplary communication devices shown in system 10 may include, but are not limited to, electronic devices or apparatus 50, a combination personal digital assistant (PDA) and mobile phone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, and a notebook computer 22. The apparatus 50 may be stationary or portable, carried by a person on the move. The apparatus 50 may also be located in a vehicle, including, but not limited to, an automobile, truck, taxi, bus, train, boat, airplane, bicycle, motorcycle, or similar suitable vehicle.

[0036] Embodiments may also be implemented in set-top boxes, i.e., digital TV receivers that may or may not have display or wireless capabilities, in tablets or (laptop) personal computers (PCs) with hardware, software, or combinations of encoder / decoder implementations, in various operating systems, and in chipsets, processors, DSPs, and / or embedded systems that provide hardware / software-based coding.

[0037] Some or further devices may send and receive calls and messages and communicate with a service provider via a wireless connection 25 to a base station 24. The base station 24 may be connected to a network server 26 that enables communication between the cellular network 11 and the Internet 28. The system may include additional communication devices and communication devices of various types.

[0038] The communication devices may communicate using a variety of transmission technologies, including, but not limited to, Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), Universal Mobile Telecommunications System (UMTS), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Transmission Control Protocol-Internet Protocol (TCP-IP), Short Message Service (SMS), Multimedia Message Service (MMS), email, Instant Message Service (IMS), Bluetooth, IEEE 802.11, and similar wireless communication technologies. Communication devices implementing various embodiments of the present invention may communicate using a variety of mediums, including, but not limited to, radio, infrared, laser, cable connections, and any suitable connection.

[0039] In telecommunications and data networks, a channel can refer to either a physical channel or a logical channel. A physical channel can refer to a physical transmission medium such as a wire, while a logical channel can refer to a logical connection over a multiplexed medium that can carry several logical channels. A channel can be used to carry an information signal, e.g., a bit stream, from one or several senders (or transmitters) to one or several receivers.

[0040] The MPEG-2 Transport Stream (TS), specified in ISO / IEC 13818-1 or the equivalent ITU-T Recommendation H.222.0, is a format for carrying audio, video, and other media, as well as program or other metadata, in a multiplexed stream. Packet Identifiers (PIDs) are used to identify elementary streams (also known as packetized elementary streams) within a TS. Thus, logical channels within an MPEG-2 TS can be considered to correspond to specific PID values.

[0041] Available media file format standards include the ISO Base Media File Format (ISO / IEC 14496-12, sometimes abbreviated as ISOBMFF) and the File Format for NAL Unit Structured Video (ISO / IEC 14496-15), which is derived from ISOBMFF.

[0042] A video codec consists of an encoder that converts input video into a compressed representation suitable for storage / transmission, and a decoder that decompresses the compressed video representation back into a displayable format. The video encoder and / or video decoder may also be separate from each other, i.e., they do not need to form a codec. Generally, an encoder discards some information from the original video sequence in order to represent the video in a more compact form (i.e., at a lower bit rate).

[0043] A typical hybrid video encoder, e.g., many encoder implementations of ITU-T H.263 and H.264, encodes video information in two phases. First, pixel values ​​of a particular picture area (or "block") are predicted, for example, by motion compensation (finding and identifying an area in one of the previously coded video frames that closely corresponds to the block being coded) or by spatial means (using pixel values ​​around the block being coded in a specified manner). Second, the prediction error, i.e., the difference between the predicted block of pixels and the original block of pixels, is coded. This is typically done by transforming the pixel value differences using a specified transform (e.g., a discrete cosine transform (DCT) or a variant thereof), quantizing the coefficients, and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (picture quality) and the size of the resulting coded video representation (file size or transmission bit rate).

[0044] In temporal prediction, the source of prediction is a previously decoded picture (also known as a reference picture). In intra block copy (IBC; also known as intra block copy prediction), prediction is applied in the same way as temporal prediction, but the reference picture is the current picture, and only previously decoded samples can be referenced in the prediction process. Inter-layer or inter-view prediction can be applied in the same way as temporal prediction, but the reference picture is a decoded picture from another scalable layer or another view, respectively. In some cases, inter-prediction can refer only to temporal prediction, but in other cases, inter-prediction can refer jointly to temporal prediction and any of intra block copy, inter-layer, and inter-view prediction, provided that they are performed in the same or similar process as temporal prediction. Inter-prediction or temporal prediction is sometimes referred to as motion compensation or motion-compensated prediction.

[0045] Motion compensation can be performed with either full-sample accuracy or sub-sample accuracy. For full-sample accurate motion compensation, motion can be represented as integer-valued motion vectors for horizontal and vertical displacements, and the motion compensation process uses those displacements to effectively copy samples from a reference picture. For sub-sample accurate motion compensation, motion vectors are represented with fractional or decimal values ​​for their horizontal and vertical components. When a motion vector points to a non-integer position within a reference picture, a sub-sample interpolation process is typically invoked to calculate predicted sample values ​​based on the reference sample and the selected sub-sample position. The sub-sample interpolation process typically consists of horizontal filtering, which compensates for the horizontal offset relative to the full sample position, followed by vertical filtering, which compensates for the vertical offset relative to the full sample position. However, vertical processing can also occur before horizontal processing, depending on the environment.

[0046] Inter-prediction, sometimes called temporal prediction, motion compensation, or motion-compensated prediction, reduces temporal redundancy. In inter-prediction, the source of prediction is a previously decoded picture. Intra-prediction exploits the fact that adjacent pixels in the same picture are likely to be correlated. Intra-prediction can be performed in the spatial domain or the transform domain, i.e., either sample values ​​or transform coefficients can be predicted. Intra-prediction is generally used in intra-coding, where inter-prediction is not applied.

[0047] One result of the coding procedure is a set of coding parameters, such as motion vectors and quantized transform coefficients. Many parameters can be entropy coded more efficiently if they are first predicted from spatially or temporally neighboring parameters. For example, motion vectors can be predicted from spatially neighboring motion vectors, and only the difference relative to the motion vector predictor can be coded. Coding parameter prediction and intra-prediction are sometimes collectively referred to as in-picture prediction.

[0048] Figures 4a and 4b show an encoder and decoder suitable for use with embodiments of the present invention. A video codec consists of an encoder that converts an input video into a compressed representation suitable for storage / transmission, and a decoder that can decompress the compressed video representation back into a displayable format. Generally, an encoder discards and / or loses some information of the original video sequence in order to represent the video in a more compact form (i.e., at a lower bit rate). An example of the encoding process is shown in Figure 4a. Figure 4a shows the image (I) to be encoded. n ), the predicted representation of the image block (P' n ), prediction error signal (D n ), the reconstructed prediction error signal (D' n ), the preliminary reconstructed image (I' n ), the final reconstructed image (R' n ), transformation (T) and inverse transformation (T -1 ), quantization (Q) and dequantization (Q-1 ), Entropy Encoding (E), Reference Frame Memory (RFM), Inter Prediction (P inter ), intra prediction (P intra ), mode selection (MS), and filtering (F).

[0049] An example of the decoding process is shown in Figure 4b. Figure 4b shows a predicted representation (P' n ), the reconstructed prediction error signal (D' n ), the preliminary reconstructed image (I' n ), the final reconstructed image (R' n ), inverse transformation (T -1 ), inverse quantization (Q -1 ), entropy decoding (E -1 ), Reference Frame Memory (RFM), Prediction (either Inter or Intra) (P), and Filtering (F).

[0050] Many hybrid video encoders encode video information in two phases. First, pixel values ​​of a particular picture area (or "block") are predicted, for example, by motion compensation (finding and identifying an area in one of the previously coded video frames that closely corresponds to the block being coded) or by spatial means (using pixel values ​​around the block being coded in a specified manner). Second, the prediction error, i.e., the difference between the predicted block of pixels and the original block of pixels, is coded. This is typically done by transforming the pixel value differences using a specified transform (e.g., a discrete cosine transform (DCT) or a variant thereof), quantizing the coefficients, and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and the size of the resulting coded video representation (file size or transmission bit rate). Video codecs may also provide a transform skip mode that the encoder can choose to use. In transform skip mode, the prediction error is coded in the sample domain, for example, by deriving a sample-by-sample difference value relative to a particular neighboring sample and coding the sample-by-sample difference value with an entropy coder.

[0051] Entropy coding / decoding can be performed in many ways. For example, context-based coding / decoding can be applied, in which both the encoder and the decoder modify the context state of coding parameters based on previously coded / decoded coding parameters. Context-based coding can be, for example, context-adaptive binary arithmetic coding (CABAC) or context-based variable length coding (CAVLC) or any similar entropy coding. Alternatively or additionally, entropy coding / decoding can be performed using a variable length coding scheme such as Huffman coding / decoding or exponential-Golomb coding / decoding. Decoding coding parameters from an entropy-coded bitstream or codeword is sometimes called parsing.

[0052] The phrase along the bitstream (e.g., indicating along the bitstream) can be defined to refer to out-of-band transmission, signaling, or storage in a manner that the out-of-band data is associated with the bitstream. A phrase such as decoding along the bitstream can refer to decoding of the referenced out-of-band data (which may be obtained from out-of-band transmission, signaling, or storage) associated with the bitstream. For example, an indication along the bitstream can refer to metadata within a container file that encapsulates the bitstream.

[0053] The H.264 / AVC standard was developed by the Video Coding Experts Group (VCEG) of the Telecommunication Standardization Sector of the International Telecommunication Union (ITU-T) and the Joint Video Team (JVT) of the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard is published by both standards bodies and is known as ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10, also known as MPEG-4 Part 10 Advanced Video Coding (AVC). There have been numerous versions of the H.264 / AVC standard that incorporate new extensions or features of this specification. These extensions include scalable video coding (SVC) and multiview video coding (MVC).

[0054] Version 1 of the High Efficiency Video Coding (H.265 / HEVC, also known as HEVC) standard was developed by the Joint Collaborative Team on Video Coding (JCT-VC) of VCEG and MPEG. The standard, published by both bodies, is known as ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2, also known as MPEG-H Part 2 High Efficiency Video Coding (HEVC). Later versions of H.265 / HEVC include scalable, multiview, range-fidelity, three-dimensional, and screen content coding extensions, sometimes abbreviated as SHVC, MV-HEVC, REXT, 3D-HEVC, and SCC, respectively.

[0055] Versatile Video Coding (VVC) (MPEG-I Part 3), also known as ITU-T H.266, is a video compression standard developed by the Joint Video Experts Team (JVET) of the Moving Picture Experts Group (MPEG) (formally ISO / IEC JTC1 SC29 WG11) and the Video Coding Experts Group (VCEG) of the International Telecommunications Union (ITU), and is the successor to HEVC / H.265.

[0056] Some important definitions, bitstream and coding structures, and concepts of H.264 / AVC, as well as HEVC, are described in this section as examples of video encoders, decoders, encoding methods, decoding methods, and bitstream structures on which embodiments can be implemented. Some of the important definitions, bitstream and coding structures, and concepts of H.264 / AVC are the same as those of HEVC, and therefore, they are described collectively below. Aspects of the present invention are not limited to H.264 / AVC or HEVC, but rather, a description of one possible basis on which the present invention can be partially or fully realized is given.

[0057] Like many earlier video coding standards, H.264 / AVC and HEVC specify bitstream syntax and semantics, as well as the decoding process for error-free bitstreams. The encoding process is not specified, but encoders must produce compliant bitstreams. Bitstream and decoder conformance can be verified using a hypothetical reference decoder (HRD). The standard includes coding tools to help address transmission errors and losses, but the use of the tools in encoding is optional, and the decoding process is not specified for erroneous bitstreams.

[0058] The basic unit of input to an H.264 / AVC or HEVC encoder and output of an H.264 / AVC or HEVC decoder, respectively, is a picture. A picture provided as input to an encoder is sometimes called a source picture, and a picture decoded by a decoder is sometimes called a decoded picture.

[0059] The source picture and the decoded picture each consist of one or more sample arrays, such as one of the following sets of sample arrays: - Luma(Y) only (single color). - Luma and two chroma (YCbCr or YCgCo). - Green, Blue, and Red (GBR, also known as RGB). - Arrays representing other unspecified monochromatic or tristimulus color samplings (e.g., YZX, also known as XYZ).

[0060] In H.264 / AVC and HEVC, a picture can be either a frame or a field. A frame contains a matrix of luma samples and possibly corresponding chroma samples. A field is a set of alternating sample rows of a frame if the source signal is interlaced and can be used as the encoder input. The chroma sample array may be absent (and thus monochrome sampling may be used), or the chroma sample array may be subsampled when compared to the luma sample array. Chroma formats can be summarized as follows: - In monochromatic sampling, there is only one sample array, which can nominally be thought of as the luma array. - In 4:2:0 sampling, each of the two chroma arrays has half the height and half the width of the luma array. - In 4:2:2 sampling, each of the two chroma arrays has the same height as the luma array and half the width of the luma array. - In 4:4:4 sampling, if separate color planes are not used, each of the two chroma arrays has the same height and width as the luma array.

[0061] In H.264 / AVC and HEVC, it is possible to code the sample arrays as separate color planes into the bitstream and decode each of the separately coded color planes from the bitstream. When separate color planes are used, each of them is processed separately (by the encoder and / or decoder) as a picture with monochromatic sampling.

[0062] Partitioning can be defined as dividing a set into subsets such that each element of the set is present in exactly one of the subsets.

[0063] The following terminology may be used when describing HEVC encoding and / or decoding operations: A coding block may be defined as an N×N block of samples for some value of N, such that the division of a coding tree block into coding blocks is a partition. A coding tree block (CTB) may be defined as an N×N block of samples for some value of N, such that the division of a component into coding tree blocks is a partition. A coding tree unit (CTU) may be defined as a coding tree block of luma samples, two corresponding coding tree blocks of chroma samples for a picture with three sample arrays, or a coding tree block of samples for a monochrome picture or a picture coded using three separate color planes and a syntax structure used to code the samples. A coding unit (CU) may be defined as a coding block of luma samples, two corresponding coding tree blocks of chroma samples for a picture with three sample arrays, or a coding block of samples for a monochrome picture or a picture coded using three separate color planes and a syntax structure used to code the samples. A CU of the largest allowed size can be named an LCU (Largest Coding Unit) or a Coding Tree Unit (CTU), and a video picture is divided into non-overlapping LCUs.

[0064] A CU consists of one or more prediction units (PUs), which define the prediction process for samples within the CU, and one or more transform units (TUs), which define the prediction error coding process for samples within the CU. Generally, a CU consists of a square block of samples with a size selectable from a predefined set of possible CU sizes. Each PU and TU can be further split into smaller PUs and TUs to increase the granularity of the prediction and prediction error coding processes, respectively. Each PU has associated prediction information (e.g., motion vector information for an inter-predicted PU and intra-prediction direction information for an intra-predicted PU) that defines what type of prediction is applied to pixels within that PU.

[0065] Each TU may be associated with information (e.g., including DCT coefficient information) that describes the prediction error decoding process for samples within the TU. Whether prediction error coding is applied for each CU is typically signaled at the CU level. If there is no prediction error residual associated with a CU, the CU can be considered to have no TUs. The division of an image into CUs and the division of CUs into PUs and TUs is typically signaled in the bitstream, allowing a decoder to recreate the intended structure of these units.

[0066] In HEVC, a picture can be partitioned into tiles, which are rectangular and contain an integer number of LCUs. In HEVC, the partitioning into tiles forms a regular grid, and the heights and widths of the tiles differ from each other by at most one LCU. In HEVC, a slice is defined as an integer number of coding tree units contained in one independent slice segment and all subsequent dependent slice segments (if any) that precede the next independent slice segment (if any) within the same access unit. In HEVC, a slice segment is defined as an integer number of coding tree units that are consecutively ordered in a tile scan and contained in a single NAL unit. The division of each picture into slice segments is a partitioning. In HEVC, an independent slice segment is defined as a slice segment in which the values ​​of syntax elements in its slice segment header are not inferred from the values ​​of previous slice segments, and a dependent slice segment is defined as a slice segment in which the values ​​of some syntax elements in its slice segment header are inferred from the values ​​of previous independent slice segments in decoding order. In HEVC, a slice header is defined to be the slice segment header of an independent slice segment that is either the current slice segment or an independent slice segment preceding the current dependent slice segment, and a slice segment header is defined to be the part of a coded slice segment that contains data elements associated with the first or all coding tree units represented in the slice segment. CUs are scanned in raster scan order for LCUs within tiles, or within a picture if tiles are not used. Within an LCU, CUs have a specific scan order.

[0067] The decoder reconstructs the output video by applying prediction means similar to the encoder to form a predictive representation of pixel blocks (using motion or spatial information produced by the encoder and stored in the compressed representation) and prediction error decoding (the inverse operation of prediction error coding, which recovers the quantized prediction error signal in the spatial pixel domain). After applying the prediction and prediction error decoding means, the decoder sums the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) may also apply additional filtering means to improve the quality of the output video, before sending it for display and / or storing it as a predictive reference for upcoming frames of the video sequence.

[0068] The filtering may include, for example, one or more of the following: deblocking, sample adaptive offset (SAO), and / or adaptive loop filtering (ALF). H.264 / AVC includes deblocking, while HEVC includes both deblocking and SAO.

[0069] In a typical video codec, motion information is indicated by a motion vector associated with each motion-compensated image block, such as a prediction unit. Each of these motion vectors represents the displacement between an image block of a picture to be coded (at the encoder side) or decoded (at the decoder side) and a prediction source block of one of the previously coded or decoded pictures. To efficiently represent motion vectors, they are generally differentially coded with respect to a block-specific predicted motion vector. In a typical video codec, the predicted motion vector is generated in a predefined manner, for example, by calculating the median value of the encoded or decoded motion vectors of neighboring blocks. Another way to generate a motion vector prediction is to generate a list of candidate predictions from neighboring and / or co-located blocks in a temporal reference picture and signal the selected candidate as a motion vector predictor. In addition to predicting the motion vector value, it is also possible to predict which reference picture will be used for motion-compensated prediction, and this prediction information can be represented, for example, by a reference index of a previously coded / decoded picture. Reference indexes are generally predicted from neighboring and / or co-located blocks in temporal reference pictures. Moreover, common high-efficiency video codecs often utilize an additional motion information coding / decoding mechanism called merging mode, in which all motion field information, including motion vectors and corresponding reference picture indexes for each available reference picture list, is predicted and used without any transformation / modification. Similarly, motion field information prediction is performed using motion field information of neighboring and / or co-located blocks in temporal reference pictures, and the used motion field information is signaled between motion field candidate lists filled with the motion field information of available neighboring / co-located blocks.

[0070] In common video codecs, the prediction residual after motion compensation is first transformed with a transform kernel (such as DCT) and then coded. This is because there is often still some correlation between the residuals, and transforms can often help to reduce this correlation and allow for more efficient coding.

[0071] Video coding standards and specifications may allow encoders to divide coded pictures into coded slices, etc. In-picture prediction is generally disabled across slice boundaries. Thus, slices can be viewed as a way of splitting coded pictures into independently decodable pieces. In H.264 / AVC and HEVC, in-picture prediction may be disabled across slice boundaries. Thus, slices can be viewed as a way of splitting coded pictures into independently decodable pieces, and therefore slices are often considered the basic unit of transmission. Often, encoders can indicate in the bitstream what type of in-picture prediction is stopped across slice boundaries, and decoder operations take this information into account when concluding, for example, which prediction sources are available. For example, if neighboring CUs reside in different slices, samples from neighboring CUs may be considered unavailable for intra prediction.

[0072] The basic unit of output of an H.264 / AVC or HEVC encoder and input of an H.264 / AVC or HEVC decoder, respectively, is the Network Abstraction Layer (NAL) unit. For transport over packet-oriented networks or storage in structured files, NAL units may be encapsulated in packets or similar structures. A byte stream format is specified in H.264 / AVC and HEVC for transmission or storage environments that do not provide a framing structure. The byte stream format separates NAL units from each other by prefixing each NAL unit with a start code. To avoid false detection of NAL unit boundaries, encoders implement a byte-oriented start code emulation prevention algorithm that adds an emulation prevention byte to the NAL unit payload if the start code occurs otherwise. To enable direct gateway operation between packet-oriented and stream-oriented systems, start code emulation prevention can always be performed regardless of whether the byte stream format is used. A NAL unit can be defined as a syntax structure containing bytes that contain an indication of the type of data that follows and that data in the form of an RBSP, optionally interspersed with emulation prevention bytes. A Raw Byte Sequence Payload (RBSP) can be defined as a syntax structure that contains an integer number of bytes encapsulated in a NAL unit. An RBSP has the form of a data bit that is either empty or contains a syntax element, followed by an RBSP stop bit, followed by a string of zero or more subsequent bits equal to 0.

[0073] A NAL unit consists of a header and a payload. In H.264 / AVC and HEVC, the NAL unit header indicates the type of the NAL unit.

[0074] In HEVC, a two-byte NAL unit header is used for all specified NAL unit types. The NAL unit header contains one reserved bit, a six-bit NAL unit type indication, a three-bit nuh_temporal_id_plus1 indication at the temporal level (which may be required to be greater than or equal to 1), and a six-bit nuh_layer_id syntax element. The temporal_id_plus1 syntax element can be considered as a temporal identifier for the NAL unit, and the zero-based TemporalId variable can be derived as follows: TemporalId = temporal_id_plus1 - 1. The abbreviation TID can be used interchangeably with the TemporalId variable. A TemporalId equal to 0 corresponds to the lowest temporal level. The value of temporal_id_plus1 is required to be non-zero to avoid start code emulation, which involves two NAL unit header bytes. A bitstream created by excluding all VCL NAL units with a TemporalId greater than or equal to the selected value and including all other VCL NAL units remains conformant. As a result, pictures with TemporalId equal to tid_value do not use pictures with TemporalId greater than tid_value as inter-prediction references. A sub-layer or temporal sub-layer can be defined to be a temporal scalable layer (or temporal layer, TL) of a temporal scalable bitstream consisting of VCL NAL units with a particular value of the TemporalId variable and associated non-VCL NAL units. nuh_layer_id can be understood as a scalability layer identifier.

[0075] NAL units can be classified into video coding layer (VCL) NAL units and non-VCL NAL units. VCL NAL units are generally coded slice NAL units. In HEVC, a VCL NAL unit contains a syntax element that represents one or more CUs.

[0076] A non-VCL NAL unit may be, for example, one of the following types: sequence parameter set, picture parameter set, supplemental enhancement information (SEI) NAL unit, access unit delimiter, end of sequence NAL unit, end of bitstream NAL unit, or filler data NAL unit. While parameter sets may be required for the reconstruction of a decoded picture, many of the other non-VCL NAL units are not required for the reconstruction of decoded sample values.

[0077] Parameters that remain unchanged throughout a coded video sequence may be included in a sequence parameter set. In addition to parameters that may be required by the decoding process, a sequence parameter set may optionally include video usability information (VUI), which includes parameters that may be important for buffering, picture output timing, rendering, and resource reservation. In HEVC, a sequence parameter set RBSP includes parameters that may be referenced by one or more picture parameter sets RBSP or one or more SEI NAL units containing buffering period SEI messages. A picture parameter set includes such parameters that are likely to remain unchanged for several coded pictures. A picture parameter set RBSP may include parameters that may be referenced by coded slice NAL units of one or more coded pictures.

[0078] In HEVC, a video parameter set (VPS) can be defined as a syntax structure containing syntax elements that apply to zero or more entire coded video sequences as determined by the content of syntax elements found in the SPS referenced by syntax elements found in the PPS referenced by syntax elements found in each slice segment header.

[0079] A video parameter set RBSP may contain parameters that may be referenced by one or more sequence parameter sets RBSP.

[0080] The relationship and hierarchy among video parameter sets (VPSs), sequence parameter sets (SPSs), and picture parameter sets (PPSs) can be described as follows: A VPS exists one level above an SPS in the parameter set hierarchy, and in the context of scalability and / or 3D video. A VPS can contain parameters common to all slices across all (scalability or view) layers throughout a coded video sequence. An SPS contains parameters common to all slices of a particular (scalability or view) layer throughout a coded video sequence and may be shared by multiple (scalability or view) layers. A PPS contains parameters that are common to all slices in a particular layer representation (one scalability or view layer representation of one access unit) and may be shared by all slices in multiple layer representations.

[0081] The VPS can provide information about layer dependency relationships in the bitstream, as well as many other pieces of information that are applicable to all slices across all (scalability or view) layers in the entire coded video sequence. The VPS can be thought of as including two parts: a basic VPS and a VPS extension, which can optionally be present.

[0082] Out-of-band transmission, signaling, or storage may additionally or alternatively be used for purposes other than transmission error tolerance, such as facilitating access or session negotiation. For example, a sample entry for a track in a file conforming to the ISO Base Media File Format may include a parameter set, while the coded data in the bitstream is stored elsewhere in the file or in another file. The phrase along the bitstream (e.g., an indication along the bitstream) or along a coding unit of the bitstream (e.g., an indication along the coding tile) may be used in the claims and described embodiments to refer to out-of-band transmission, signaling, or storage such that the out-of-band data is associated with the bitstream or coding unit, respectively. Phrases such as decoding along the bitstream or decoding along a coding unit of the bitstream may refer to decoding of the referenced out-of-band data (which may be retrieved from out-of-band transmission, signaling, or storage) associated with the bitstream or coding unit, respectively.

[0083] An SEI NAL unit may contain one or more SEI messages, It is not required for decoding the output picture, but It can assist in related processes such as picture output timing, rendering, error detection, error concealment, and resource reservation.

[0084] A coded picture is a coded representation of a picture.

[0085] In HEVC, a coded picture can be defined as a coded representation of a picture that includes all of the coding tree units of the picture. In HEVC, an access unit (AU) can be defined as a set of NAL units that are associated with each other according to specified classification rules, are consecutive in decoding order, and include at most one picture with a particular value of nuh_layer_id. In addition to containing VCL NAL units of a coded picture, an access unit can also contain non-VCL NAL units. The specified classification rules can, for example, associate pictures with the same output time or picture output count value with the same access unit.

[0086] A bitstream can be defined as a sequence of bits in the form of a NAL unit stream or byte stream that forms a representation of coded pictures and associated data that form one or more coded video sequences. A first bitstream can be followed by a second bitstream within the same logical channel, such as within the same file or connection of a communication protocol. An elementary stream (in the context of video coding) can be defined as a sequence of one or more bitstreams. The end of a first bitstream can be indicated by a specific NAL unit, sometimes called the end-of-bitstream (EOB) NAL unit, which is the last NAL unit of the bitstream. HEVC and its latest draft extensions require that the EOB NAL unit have nuh_layer_id equal to 0.

[0087] In H.264 / AVC, a coded video sequence is defined to be a sequence of consecutive access units in decoding order, starting from an IDR access unit (including itself) to the next IDR access unit or to the end of the bitstream (excluding itself), whichever occurs first.

[0088] In HEVC, a coded video sequence (CVS) may be defined as a sequence of access units consisting, for example, in decoding order, of an IRAP access unit with NoRaslOutputFlag equal to 1 followed by zero or more access units that are not IRAP access units with NoRaslOutputFlag equal to 1, including all subsequent access units up to (but not including) the subsequent access unit that is an IRAP access unit with NoRaslOutputFlag equal to 1. An IRAP access unit may be defined as an access unit whose base layer picture is an IRAP picture. The value of NoRaslOutputFlag is equal to 1 for each IDR picture, each BLA picture, and each IRAP picture that is the first picture of that particular layer in the bitstream in decoding order, and is equal to 1 for the first IRAP picture that follows the end-of-sequence NAL unit with the same value of nuh_layer_id in decoding order. There may be means for providing the value of HandleCraAsBlaFlag to the decoder from an external entity, such as a player or receiver, that can control the decoder. HandleCraAsBlaFlag may be set to 1, for example, by a player that seeks to a new position in the bitstream or tunes to a broadcast, starts decoding, and then starts decoding from a CRA picture. If HandleCraAsBlaFlag is equal to 1 for a CRA picture, the CRA picture is treated and decoded as if it were a BLA picture.

[0089] In HEVC, a coded video sequence may additionally or alternatively (to the above specification) be specified to end when a specific NAL unit, which may be called an end-of-sequence (EOS) NAL unit, appears in the bitstream and has a nuh_layer_id equal to 0.

[0090] Groups of pictures (GOPs) and their characteristics can be defined as follows: A GOP can be decoded regardless of whether previous pictures have been decoded. An open GOP is a group of pictures in which, if decoding starts from the first intra picture of the open GOP, pictures in the output order prior to the first intra picture may not be correctly decoded. In other words, pictures in an open GOP may reference pictures belonging to a previous GOP (in inter prediction). A specific NAL unit type, the CRA NAL unit type, can be used for coded slices, so that an HEVC decoder can recognize the intra picture that starts an open GOP. A closed GOP is a group of pictures in which all pictures can be correctly decoded if decoding starts from the first intra picture of the closed GOP. In other words, pictures in a closed GOP do not reference pictures of a previous GOP. In H.264 / AVC and HEVC, a closed GOP can start with an IDR picture. In HEVC, a closed GOP can also start from a BLA_W_RADL or BLA_N_LP picture. An open GOP coding structure is potentially more efficient in compression compared to a closed GOP coding structure due to greater flexibility in reference picture selection.

[0091] A decoded picture buffer (DPB) may be used in an encoder and / or decoder. There are two reasons for buffering decoded pictures: for reference in inter-prediction and for reordering decoded pictures into output order. Because H.264 / AVC and HEVC provide a lot of flexibility for both reference picture marking and output order reordering, separate buffers for reference picture buffering and output picture buffering may waste memory resources. Therefore, the DPB can include a unified decoded picture buffering process for reference pictures and output order reordering. When a decoded picture is no longer used as a reference and is no longer needed for output, the decoded picture can be removed from the DPB.

[0092] In many coding modes of H.264 / AVC and HEVC, reference pictures for inter prediction are indicated by an index into a reference picture list. The index can be coded using variable length coding, whereby a smaller index typically results in a shorter value for the corresponding syntax element. In H.264 / AVC and HEVC, two reference picture lists (Reference Picture List 0 and Reference Picture List 1) are generated for each bi-predictive (B) slice, and one reference picture list (Reference Picture List 0) is formed for each inter-coded (P) slice.

[0093] Many coding standards, including H.264 / AVC and HEVC, may have a decoding process for deriving a reference picture index into a reference picture list, which can be used to indicate which of multiple reference pictures is used for inter-prediction for a particular block. The reference picture index may be coded into the bitstream by the encoder in some inter-coding modes, or may be derived (by the encoder and decoder) using neighboring blocks, for example, in some other inter-coding modes.

[0094] The motion parameter types or motion information may include, but are not limited to, one or more of the following types: - an indication of the prediction type (e.g., intra-prediction, uni-prediction, bi-prediction) and / or the number of reference pictures; - an indication of prediction direction such as inter (aka temporal) prediction, inter-layer prediction, inter-view prediction, view synthesis prediction (VSP), and inter-component prediction (which may be indicated per reference picture and / or per prediction type; in some embodiments, inter-view prediction and view synthesis prediction together may be considered as one prediction direction); and / or - an indication of the reference picture type, such as short-term and / or long-term and / or inter-layer reference picture (which may be indicated for example per reference picture), - a reference index into the reference picture list and / or other identifier of the reference picture (It can be indicated for example per reference picture, its type can depend on the prediction direction and / or the reference picture type, and it may be accompanied by other relevant information such as the reference picture list to which the reference index applies, etc.), - horizontal motion vector component (which can be indicated, for example, per prediction block or per reference index, etc.), - a vertical motion vector component (which can be indicated, for example, per prediction block or per reference index, etc.), one or more parameters, such as a picture order count difference and / or a relative camera distance between a picture containing or associated with a motion parameter and its reference picture, that can be used for scaling horizontal and / or vertical motion vector components in one or more motion vector prediction processes (the one or more parameters can be indicated, for example, for each reference picture or for each reference index, etc.); - the coordinates of the block to which the motion parameters and / or motion information apply, e.g. the coordinates of the top left sample of the block in luma sample units; - The extent (e.g. width and height) of the block to which the motion parameters and / or motion information apply.

[0095] Compared to previous video coding standards, the versatile video codec (H.266 / VVC) introduces several new coding tools, such as: Intra prediction - 67 intra modes with wide-angle mode expansion - Block size and mode dependent 4-tap interpolation filter - Position-dependent intra-prediction combining (PDPC) - Cross-Component Linear Model Intra Prediction (CCLM) - Multi-reference line intra prediction - Intra-subpartition - Weighted intra prediction using matrix multiplication Inter-picture prediction - Block motion copy with spatial, temporal, history-based, and pairwise average merge candidates - Affine motion inter-prediction - Sub-block based temporal motion vector prediction - Adaptive motion vector resolution - 8x8 block-based motion compression for temporal motion estimation - High precision (1 / 16 pel) motion vector storage and motion compensation using an 8-tap interpolation filter for the luma component and a 4-tap interpolation filter for the chroma component - Triangular partition - Combination of intra- and inter-prediction - Merge by MVD (MMVD) - Symmetric MVD coding - Bidirectional Optical Flow - Decoder-side motion vector refinement - Bi-prediction using Cu level weights Transformation, quantization, and coefficient coding - Multiple linear transform choices using DCT2, DCT7, and DCT8 - Secondary transformation of the low frequency zone - Sub-block transform of inter prediction residuals - Dependent quantization with max QP increased from 51 to 63 - Transform coefficient coding with sign data hiding - Transform-skip residual coding Entropy coding - Arithmetic coding engine with adaptive double window probability updates In-loop filter - In-Loop Reshaping - Deblocking filter with powerful longer filters - Sample adaptive offset - Adaptive Loop Filter Screen content coding - Current picture reference by reference area restriction 360-degree video coding - Horizontal wraparound motion compensation High-level syntax and parallel processing - Reference picture management using direct reference picture list signaling - Tile groups with rectangular shape tile groups

[0096] Partitioning in VVC is performed similarly to HEVC, i.e., each picture is divided into coding tree units (CTUs). A picture can also be divided into slices, tiles, bricks, and subpictures. CTUs can be split into smaller CUs using a quadtree structure. Each CU can be split using a quadtree and nested multi-type trees, including 3-split and 2-split. However, there are specific rules for inferring partitioning at picture boundaries, and redundant split patterns are not allowed in nested multi-type partitioning.

[0097] In the new coding tools listed above, a cross-component linear model (CCLM) prediction mode is used in VVC to reduce cross-component redundancy, where chroma samples are predicted based on the reconstructed luma samples of the same CU by using a linear model as follows: pred C (i,j)=α·rec L '(i,j)+β (Eq. 1a)

[0098] where pred C (i,j) represents the predicted chroma sample in a CU, and rec L '(i,j) represents the downsampled reconstructed luma sample of the same CU.

[0099] Alternatively, the following formula

number

[0100] The CCLM parameters (α and β) are derived using up to four adjacent chroma samples and their corresponding downsampled luma samples. Assuming the current chroma block dimensions are W × H, let W' and H' be - When LM mode is applied, W'=W, H'=H - When LM-A mode is applied, W'=W+H - When LM-L mode is applied, H'=H+W is set as

[0101] In this specification, LM-A mode refers to linear model_above, where only the above template (i.e., sample values ​​from adjacent positions above the CU) are used to calculate the linear model coefficients. To obtain more samples, the above template is extended to (W+H). And then, LM-L mode refers to linear model_left, where only the left template (i.e., sample values ​​from adjacent positions to the left of the CU) is used to calculate the linear model coefficients. To obtain more samples, the left template is extended to (H+W). For non-square blocks, the above template is extended to W+W, and the left template is extended to H+H.

[0102] The top adjacent positions are represented as S[0,-1]...S[W'-1,-1], and the left adjacent positions are represented as S[-1,0]...S[-1,H'-1]. Then, the four samples are - When LM mode is applied and both the upper and left adjacent samples are available, S[W' / 4,-1], S[3*W' / 4,-1], S[-1,H' / 4], S[-1,3*H' / 4], - If LM-A mode is applied or only upper adjacent samples are available, S[W' / 8,-1], S[3*W' / 8,-1], S[5*W' / 8,-1], S[7*W' / 8,-1], - S[-1,H' / 8], S[-1,3*H' / 8], S[-1,5*H' / 8], S[-1,7*H' / 8] if LM-L mode is applied or only the left adjacent sample is available is selected to be.

[0103] The four adjacent luma samples at the selected position are downsampled and compared four times to find two smaller values ​​x0A and x1A and two larger values ​​x0B and x1B. Their corresponding chroma sample values ​​are represented as y0A, y1A, y0B, and y1B. Then, xA, xB, yA, and yB are X a =(x 0 A +x 1 A +1)>>1;X b =(x 0 B +x 1 B +1)>>1;Y a =(y 0 A +y 1 A +1)>>1;Y b =(y 0 B +y 1 B +1)>>1 (Formula 2) is derived as:

[0104] Finally, the linear model parameters α and β are obtained by the following equations:

number

[0105] FIG. 5 shows an example of the locations of the left and top samples as well as the samples of the current block involved in CCLM mode.

[0106] The division operation to calculate the parameter α is performed using a lookup table. To reduce the memory required to store the table, the diff value (the difference between the maximum and minimum values) and the parameter α are expressed in exponential notation. For example, the diff is approximated with a 4-bit significant part and an exponent. As a result, the 1 / diff table is reduced to 16 elements for 16 values ​​of the significant part as follows: DivTable[ ]={0,7,6,5,5,4,4,3,3,2,2,1,1,1,1,0} (Formula 5)

[0107] This provides the benefit of reducing both the computational complexity as well as the memory size required to store the necessary tables.

[0108] To align the chroma sample locations of a 4:2:0 video sequence, two types of downsampling filters are applied to the luma samples to achieve a 2:1 downsampling ratio in both the horizontal and vertical directions. The choice of downsampling filter is specified by the SPS level flag. The two downsampling filters are as follows, corresponding to "Type-0" and "Type-2" content, respectively:

number

[0109] Note that if the upper reference line is at a CTU boundary, only one luma line (a common line buffer in intra prediction) is used to produce the downsampled luma samples.

[0110] This parameter calculation is performed as part of the decoding process, and not simply as an encoder search operation, and as a result, no syntax is used to communicate the α and β values ​​to the decoder.

[0111] In chroma intra-mode coding, a total of eight intra modes are allowed for chroma intra-mode coding. These modes include five conventional intra modes and three cross-component linear model modes (CCLM, LM_A, and LM_L). The chroma mode signaling and derivation process is shown in Table 1. Chroma mode coding directly depends on the intra-prediction mode of the corresponding luma block. In an I slice, since separate block partitioning structures are enabled for the luma component and the chroma component, one chroma block may correspond to multiple luma blocks. Therefore, in chroma DM mode, the intra-prediction mode of the corresponding luma block that covers the center position of the current chroma block is directly inherited. [Table 1]

[0112] As shown in Table 2, a single binarization table is used regardless of the value of sps_cclm_enabled_flag. [Table 2]

[0113] In Table 2, the first bin indicates whether it is normal mode (0) or LM mode (1). If the first bin is LM mode, the next bin indicates whether it is LM_CHROMA (0). If the next bin is not LM_CHROMA, the next bin indicates whether it is LM_L (0) or LM_A (1). In this case, if sps_cclm_enabled_flag is 0, the first bin of the corresponding intra_chroma_pred_mode binarization table can be discarded before entropy coding. Or, in other words, the first bin is inferred to be 0 and therefore not coded. This single binarization table is used both when sps_cclm_enabled_flag is 0 and when it is 1. The first two bins of Table 2 are context coded with their own context model, and the remaining bins are bypass coded.

[0114] In addition, to reduce luma-chroma latency in the dual tree, when the 64x64 luma coding tree node is partitioned using Not Split (intra subpartition (ISP) is not used for 64x64 CUs) or QT, the chroma CUs in the 32x32 / 32x16 chroma coding tree node are allowed to use CCLM in the following manner: - If a 32x32 chroma node is not split or partitioned, all chroma CUs in the 32x32 node can use CCLM. - If a 32x32 chroma node is partitioned with horizontal BT and the 32x16 child node does not split or uses vertical BT split, all chroma CUs within the 32x16 chroma node can use CCLM.

[0115] In all other luma and chroma coding tree split conditions, CCLM is not allowed for chroma CU.

[0116] Multi-Model LM (MMLM) The CCLM included in VVC is extended by adding three multi-model LM (MMLM) modes. In each MMLM mode, reconstructed neighboring samples are classified into two classes using a threshold that is the average of the reconstructed neighboring samples of luma. A linear model for each class is derived using the least mean square (LMS) method. In the CCLM mode, the LMS method is also used to derive the linear model. Figures 6a and 6b show two luma-chroma models obtained for 17 luma (Y) thresholds in the sample domain and spatial domain, respectively. Each luma-chroma model has its own linear model parameters α and β. As can be seen in Figure 6b, each luma-chroma model corresponds to a spatial segmentation of the content (i.e., corresponding to different objects or textures in the scene).

[0117] Convolutional Cross-Component Model (CCCM) An improved version of cross-component prediction, known as CCCM, uses a 2D filter kernel to derive a luma-chroma model. The filter coefficients are derived at the decoder side using the reconstructed input data and a set of chroma samples. For filter coefficient derivation, a co-located reference sample area (consisting of reconstructed luma and chroma samples) is defined for both luma and chroma, as shown in FIG. 7, with the commonly used 4:2:0 chroma downsampling applied. The reference sample area for a given block can be, for example, six lines above and to the left, as shown in FIG. 7, and any number of reference lines (which can be realized by both the encoder and the decoder) can be used. In general, the reference samples can include any chroma and luma samples reconstructed by both the encoder and the decoder. After the reference samples are determined, the filter coefficients can be derived using, for example, different types of linear regression tools, such as ordinary least squares estimation, orthogonal matching pursuit, optimized orthogonal matching pursuit, ridge regression, or least absolute shrinkage and selection operators.

[0118] The dimensions of the filter kernel can be, for example, 1x3 (1D vertical), 3x1 (1D horizontal), 3x3, 7x7, or any dimension, and can be formed as a cross or diamond (as shown in Figure 8), or any given shape (by selecting only a subset of all possible kernel locations). When referring to samples within the filter kernel, the following notation is used: North (top), East (right), South (bottom), West (left), and center, as indicated in Figure 8 using the letters N, E, S, W, C.

[0119] The overall method of reconstructing chroma samples using a convolution between a filter kernel obtained at the decoder side and a set of input data is referred to herein as the Convolutional Cross-Component Model (CCCM). To perform the CCCM operation, the following steps can be applied: 1) Define a co-position reference area for the luma and chroma components. 2) Downsample the luma samples to match the chroma grid (optional). 3) Scan the luma and chroma samples of the reference area and collect available statistical data (such as autocorrelation matrix and cross-correlation vector) based on the filter shape. 4) Solve for the filter coefficients by minimizing the squared error (or any other metric) based on available statistical data (such as the autocorrelation matrix and cross-correlation vectors). 5) Compute the predicted chroma block by convolving the downsampled luma samples with a filter kernel.

[0120] We define the (possibly downsampled) luma samples as a 2D array Y(x,y) indexed using horizontal x and vertical y coordinates. We further define the co-located chroma samples as a 2D array C(x,y), and the filter kernel (i.e., coefficients) as a 3x3 array F(i,j). At the sample level, we define the convolution between Y and F as

number

[0121] If other data terms are used, such as nonlinear square root terms, the additional convolution is

number

number

number

[0122] Multiple Reference Line (MRL) intra prediction Multiple Reference Line (MRL) intra prediction uses more reference lines for intra prediction. Figure 9 shows an example of four reference lines, where samples from segments A and F are padded with the nearest samples from segments B and E, respectively, rather than being obtained from reconstructed neighboring samples. HEVC intra picture prediction uses the nearest reference line (i.e., reference line 0). In MRL, two additional lines (reference line 1 and reference line 3) are used.

[0123] The index (mrl_idx) of the selected reference line is signaled and used to generate the intra prediction. For reference line idx greater than 0, include only additional reference line modes in the MPM list and signal only the mpm index without the remaining modes. After the reference line index is signaled, intra prediction modes and planar modes are excluded from the intra prediction modes if a non-zero reference line index is signaled.

[0124] MRL is disabled on the first line of a block in a CTU to prevent the use of extended reference samples outside the current CTU line. Furthermore, PDPC is disabled if additional lines are used. In MRL mode, the derivation of DC values ​​in DC intra prediction modes for non-zero reference line indexes is consistent with the derivation for reference line index 0. MRL requires the storage of three adjacent luma reference lines with the CTU to generate predictions. The CCLM tool also requires three adjacent luma reference lines for the downsampling filter. The definition of MRL, which uses the same three lines, is aligned with CCLM to reduce decoder memory requirements.

[0125] Intra-Subpartition (ISP) The intra subpartition (ISP) divides a luma intra-prediction block vertically or horizontally into two or four subpartitions depending on the block size. For example, the minimum block size for ISP is 4x8 (or 8x4). If the block size is larger than 4x8 (or 8x4), the corresponding block is divided into four subpartitions. It has been confirmed that Mx128 (M ≤ 64) and 128xN (N ≤ 64) ISP blocks can create potential problems with 64x64 VDPUs. For example, an Mx128 CU in the single-tree case has an Mx128 luma TB and two corresponding M / 2x64 chroma TBs. When a CU uses ISP, the luma TB is divided into four Mx32 TBs (only horizontal splits are possible), each of which is smaller than a 64x64 block. However, in the current design of ISP, chroma blocks are not divided. Therefore, both chroma components have a size larger than a 32x32 block. Similarly, a similar situation can occur with a 128xN CU that uses ISP. Therefore, these two cases are problematic for a 64x64 decoder pipeline. For this reason, the CU size that can use ISP is limited to a maximum of 64x64. All subpartitions must have at least 16 samples.

[0126] Matrix Weighted Intra Prediction (MIP) The matrix-weighted intra prediction (MIP) method is a new intra prediction technique added to VVC. To predict samples of a rectangular block of width W and height H, matrix-weighted intra prediction (MIP) takes as input one line of H reconstructed adjacent boundary samples to the left of the block and one line of W reconstructed adjacent boundary samples above the block. If reconstructed samples are not available, they are generated in the same way as conventional intra prediction. The generation of the prediction signal is based on the following three steps: averaging, matrix-vector multiplication, and linear interpolation, as shown in Figure 10.

[0127] Decoder-side Intra-mode Derivation (DIMD) When DIMD is applied, two intra-mode predictors are derived from the reconstructed neighboring samples, and these two predictors are combined with the planar mode predictor using weights derived from the gradients as described in JVET-O0449. The division operations in the weight derivation are performed using the same look-up table (LUT)-based integerization scheme used by CCLM. For example, the division operations in the orientation calculation are Orient=G y / G x is calculated by the following LUT-based scheme: x=Floor(Log2(Gx)) normDiff=((Gx<<4)>>x)&15 x+=(3+(normDiff !=0)?1:0) Orient=(Gy * (DivSigTable[normDiff]|8)+(1<<(x-1)))>>x where: DivSigTable

[16] ={0,7,6,5,5,4,4,3,3,2,2,1,1,1,1,0} is.

[0128] The derived intra modes are included in the primary list of intra most probable modes (MPMs), so the DIMD process is performed before the MPM list is constructed. The primary derived intra modes of a DIMD block are stored with the block and are used to build the MPM lists of neighboring blocks.

[0129] Fusion for Template-Based Intra-Mode Derivation (TIMD) For each intra-prediction mode of the MPM, the SATD between the predicted sample of the template and the reconstructed sample is calculated. First, the two intra-prediction modes with the smallest SATD are selected as the TIMD mode. These two TIMD modes are fused using weights after applying the PDPC process, and such weighted intra-prediction is used to code the current CU. Position-dependent intra-prediction combining (PDPC) is included in the derivation of the TIMD mode.

[0130] The costs of the two selected modes are compared to a threshold and a cost factor of 2 is applied in the test as follows: costMode2<2*costMode1

[0131] If this condition is true, fusion is applied, otherwise only mode1 is used.

[0132] The weights of modes are calculated from their SATD costs as follows: weight1=costMode2 / (costMode1+costMode2) weight2=1-weight1

[0133] The division operation is performed using the same look-up table (LUT) based integerization scheme used by CCLM.

[0134] Low Frequency Non-Separable Transform (LFNST) In VVC, LFNST is applied between the forward linear transform and quantization (at the encoder), and between the inverse quantization and the inverse linear transform (at the decoder side), as shown in Figure 11. In LFNST, a 4x4 non-separable transform or an 8x8 non-separable transform is applied depending on the block size. For example, 4x4 LFNST is applied to small blocks (i.e., min(width, height)<8), and 8x8 LFNST is applied to larger blocks (i.e., min(width, height)>4).

[0135] The application of the non-separable transform used in LFNST is explained as follows using the example input: To apply a 4x4 LFNST, a 4x4 input block X

number

number

number

[0136] The non-separable transformation is

number

number

number

[0137] Reduced non-separable transformation LFNST (Low Frequency Non-Separable Transformation) is based on a direct matrix multiplication method for applying non-separable transformation, and as a result, it is implemented in a single pass without a large number of iterations. However, the dimension of the non-separable transformation matrix needs to be reduced in order to minimize the computational complexity and the memory space for storing the transformation coefficients. Therefore, the reduced non-separable transformation (or RST) method is used in LFNST. The main concept of the reduced non-separable transformation is to map an N (N is generally equal to 64 in 8×8 NSST) -dimensional vector to an R -dimensional vector in a different space, where N / R (R < N) is the reduction coefficient. Therefore, instead of an N×N matrix, the RST matrix becomes an R×N matrix

Number

[0138] The inverse transform matrix of RT is the transpose of its forward transform. In 8x8 LFNST, a reduction factor of 4 is applied, reducing the traditional 8x8 non-separable transform matrix size (64x64 direct matrix) to a 16x48 direct matrix. Therefore, a 48x16 inverse RST matrix is ​​used at the decoder side to generate the core (primary) transform coefficients of the 8x8 upper-left region. If a 16x48 matrix is ​​applied instead of a 16x64 with the same transform set configuration, each of them takes 48 input data from three 4x4 blocks in the upper-left 8x8 block, excluding the lower-right 4x4 block. With the help of dimensionality reduction, the memory usage for storing all LFNST matrices is reduced from 10KB to 8KB with a reasonable performance degradation. To reduce complexity, LFNST is restricted to be applicable only when all coefficients outside the first coefficient subgroup are insignificant. Therefore, when LFNST is applied, all primary-only transform coefficients must be zero. This allows LFNST index signaling to be adjusted by the last significant position, thus avoiding the extra coefficient scan required in current LFNST designs to check significant coefficients only at certain positions. LFNST's worst-case processing (in terms of multiplications per pixel) limits non-separable transforms of 4x4 and 8x8 blocks to 8x16 and 8x48 transforms, respectively. In those cases, when LFNST is applied, the last significant scan position must be less than 8, and for other sizes, it must be less than 16. For blocks of shape 4xN and Nx4 with N>8, the proposed restrictions mean that LFNST is now applied only once, and that LFNST is only applied to the top-left 4x4 region. Since all first-order-only coefficients are zero when LFNST is applied, the number of operations required for the first-order transform in such cases is reduced. From the encoder's perspective, coefficient quantization is significantly simplified when LFNST transforms are tested. Rate-distortion optimized quantization must be performed on at most the first 16 coefficients (in scan order), with the remaining coefficients being forced to zero.

[0139] LFNST conversion selection There are four transform sets in total for LFNST, with two non-separable transform matrices (kernels) used per transform set. The mapping from intra prediction modes to transform sets is predefined as shown in the table below. If one of the three CCLM modes (INTRA_LT_CCLM, INTRA_T_CCLM, or INTRA_L_CCLM) is used for the current block (81<=predModeIntra<=83), transform set 0 is selected for the current chroma block. For each transform set, the selected non-separable secondary transform candidate is further specified by an explicitly signaled LFNST index. The index is signaled in the bitstream once per intra CU after the transform coefficients. [Table 3]

[0140] LFNST index signaling and interaction with other tools Because LFNST is restricted to be applicable only when all coefficients outside the first coefficient subgroup are insignificant, LFNST index coding depends on the position of the last significant coefficient. In addition, the LFNST index is context coded, but is independent of the intra prediction mode; only the first bin is context coded. Furthermore, LFNST applies to intra CUs in both intra slices and inter slices, and to both luma and chroma. When dual trees are enabled, the LFNST indexes for luma and chroma are signaled separately. In inter slices (where dual trees are disabled), a single LFNST index is signaled and used for both luma and chroma.

[0141] Considering that CUs larger than 64x64 are implicitly split (TU tiling) due to the existing maximum transform size limitation (64x64), LFNST index lookup can increase data buffering by a factor of four for a certain number of decode pipeline stages. Therefore, the maximum size allowed for LFNST is limited to 64x64. Note that LFNST is only enabled for DCT2. LFNST index signaling is placed before MTS index signaling.

[0142] Using a scaling matrix for perceptual quantization is not clear because it is not clear that the scaling matrix specified for the primary matrix can be useful for the LFNST coefficients. Therefore, using a scaling matrix for the LFNST coefficients is not allowed. In single-tree partition mode, chroma LFNST is not applied.

[0143] The convolutional cross-component model (CCCM) uses spatial luma samples to model the relationship between luma and chroma for cross-component prediction. Even though this method can model the cross-channel relationship of samples, it is still suboptimal when the spatial samples of the luma and chroma channels are not sufficiently correlated.

[0144] Next, an improved method for achieving cross-component prediction is introduced that provides enhanced modeling of cross-channel relationships of samples.

[0145] 12 shows a method according to one embodiment, which includes receiving an image block unit of a frame (1200), the image block unit including samples in color channels, the color channels including at least one chrominance channel and one luminance channel; reconstructing samples of the luminance channel of the image block unit (1202); determining a reference area for predicting target samples of at least one color channel of the image block unit (1204), the reference area including one or more of reference samples in adjacent blocks in a current color channel / frame, reference samples adjacent to a co-located block in a reference color channel / frame, and / or reference samples inside a co-located block in a reference color channel / frame; determining gradient values ​​and / or position values ​​of spatial samples in the reference area to be used in a cross-component prediction model (1206); and predicting the target samples of at least one color channel of the image block unit using the cross-component prediction model based on at least the gradient values ​​and / or the position values ​​of spatial samples in the reference area (1208).

[0146] Therefore, in this method, gradient values ​​and / or position values ​​of spatial samples within the reference area are determined to be used in cross-component prediction models such as cross-component linear models (CCLM) and convolutional cross-component models (CCCM), thereby enhancing the correlation between the reference samples and the target samples.

[0147] According to one embodiment, the method includes using said gradient values ​​and / or said position values ​​of spatial samples in a cross-component prediction model together with or instead of spatial sample values.

[0148] According to one embodiment, the method includes using a combination of the gradient values ​​and the position values ​​of spatial samples in a cross-component prediction model, together with or instead of the spatial sample values.

[0149] Thus, a CCCM model can consist of one or more of gradient and / or position values ​​calculated from spatial samples, which may be used in conjunction with or in place of one or more of the spatial samples used in the CCCM model.

[0150] According to one embodiment, the method includes indicating the use of gradient values ​​and / or position values ​​of spatial samples when signaling within or along a bitstream containing predicted image block units.

[0151] Thus, the use of the gradient values ​​and / or the position values ​​may be signaled within or along the bitstream containing the encoded image data. Alternatively, the use of the gradient values ​​and / or the position values ​​may be predefined, for example in a standard specification. Further alternatively, the use of the gradient values ​​and / or the position values ​​may be estimated at the decoder side.

[0152] In the following, various embodiments are described primarily in the context of improving the performance of a convolutional cross-component model (CCCM). The CCCM method is given as an example of the present invention, and it should be understood that the proposed method and embodiments may be used in any other method with a similar concept, such as local illumination compensation (LIC) and other intra- or inter-prediction methods. It is further noted that cross-channel prediction may be performed from one chroma channel to another (e.g., from Cb to Cr, or vice versa), or from a chroma channel to luma (Cb / Cr to Y), or vice versa.

[0153] Various embodiments can be implemented with various types of filter kernels of M-tap filters, such as those shown in FIG. 8, which illustrates a 3-tap vertical filter (1×3 (1D vertical)), a 3-tap horizontal filter (3×1 (1D horizontal)), a 5-tap cross filter, and a 25-tap diamond filter, using the notation North (N, up), East (E, right), South (S, down), West (W, left), and Center (C).

[0154] Gradient Value According to one embodiment, the method includes predicting the target samples of at least one color channel of an image block unit using the following cross-component prediction model: pred=P0·C+P1·N+P2·S+P3·W+P4·E+…+P n ·grad hor +P n+1 ·grad ver +P n+2 ·grad diag +…+P M-1 β where pred is the predicted chroma sample, P is the model coefficient for each tap of the M-tap filter, (C,N,S,W,E) are example spatial samples at locations (center, north, south, west, east), and grad hor , grad ver , grad diag are the directional gradient values ​​in the horizontal, vertical, and diagonal directions, respectively, and β is an offset or bias value.

[0155] Herein, the gradient value may be calculated in different ways, for example, the gradient value may be calculated for each direction. grad hor =WE grad ver =NS grad diag1 =NW-SE grad diag2 =NE-SW Here, NW, SW, NE, and SE indicate samples at northwest, southwest, northeast, and southeast locations, respectively.

[0156] The directional gradient can also be calculated using more than two samples. grad hor =(2×W+NW+SW)-(2×E+NE+SE) grad ver =(2×N+NW+NE)-(2×S+SW+SE) grad diag1 =(2×NW+N+W)-(2×SE+S+E) grad diag2 =(2×NE+N+E)-(2×SW+S+W)

[0157] In the above example, a weight value of 2 is given to the center of the sample for each direction. However, different weight values ​​may be used, for example, for each sample. The values ​​of the weight parameters used may be fixed or signaled for each block (e.g., as a weight index). Alternatively, the weight parameters may be determined at the encoder and decoder sides based on certain factors such as the block size, the availability of reference samples, the coding mode of one or more of the neighboring blocks, the coding mode of the reference block in the reference channel, etc.

[0158] According to one embodiment, the method includes using the absolute value of the gradient value instead of the signed value.

[0159] According to one embodiment, the method includes using the sum or average of gradient values ​​in different directions as parameters of a cross-component prediction model.

[0160] For example, the average, weighted average, or sum of horizontal and vertical gradients, or the sum or average of different diagonal gradients may be used herein, for example, as follows: grad1=abs(grad hor )+abs(grad ver ) grad2=abs(grad diag1 )+abs(graddiag2 ) grad3=abs(grad hor )+abs(grad ver )+abs(grad diag1 )+abs(grad diag2 ) Here, abs() indicates the absolute value of the gradient.

[0161] Another example using the average gradient is grad1=(abs(grad hor )+abs(grad ver )) / 2 grad2=(abs(grad diag1 )+abs(grad diag2 )) / 2 grad3=(abs(grad hor )+abs(grad ver )+abs(grad diag1 )+abs(grad diag2 )) / 4.

[0162] According to one embodiment, the method includes using both absolute and signed versions of gradient values ​​in the cross-component prediction model. For example, the absolute value of the horizontal gradient and the signed version of the vertical gradient may be used in the model.

[0163] According to one embodiment, the method includes using one or more of the gradient values ​​in a nonlinear function, and using the output of the nonlinear function as a parameter of the cross-component prediction model. For example, a square root or a power of two of one or more gradients may be used. In another example, a sum, weighted sum, or average of a nonlinear function using one or more gradient information may be used as a parameter of a filter. In yet another example, one or more outputs of the nonlinear function using one or more of the gradient information as inputs may be combined with one or more outputs of the nonlinear function using one or more of the spatial samples as inputs. The nonlinear functions used for each input type may be the same or different. For example, for spatial samples, the nonlinear function may be a square root function, while for gradient information, it may be a power of N function, or vice versa. In another example, one or more of the gradient information may be combined with one or more of the spatial samples, and the combined version is then used as an input to the nonlinear function.

[0164] According to one embodiment, the use of gradient information in the filter is determined based on a consistency measure, which can be calculated, for example, by applying a cross-component model to a particular area of ​​a reference sample and determining how the presence of gradient information affects the prediction accuracy.

[0165] According to one embodiment, the input type of the nonlinear function used in the cross-component model may be determined based on the gradient value. For example, one or more sums, averages, or weighted averages of the gradient information may be calculated using the reference area samples, and if the value is above / below a certain threshold, the input to the nonlinear function may be one or more of the gradient information or one or more of the spatial samples.

[0166] According to one embodiment, when chroma subsampling is enabled, the gradient values ​​are obtained with downsampled luma samples or using the full luma grid.

[0167] According to one embodiment, the method includes combining luma downsampling and a gradient filter into a single filtering operation.

[0168] According to one embodiment, the method includes using higher order derivatives of the gradient values ​​as parameters of a cross-component prediction model.

[0169] In other words, a first order gradient is obtained, and then a second order (or even higher order) gradient is obtained, for example, iteratively. A higher order gradient can be obtained, for example, using one iteration for the horizontal and vertical directions, resulting in the following second order gradient: grad hor2 =(WW-W)-(WE)=WW-2W+E grad ver2 =(NN-N)-(NS)=NN-2N+S

[0170] Or, in some cases, grad hor2 =(WE)-(E-EE)=E-2E+EE grad ver2 =(NS)-(S-SS)=N-2S+SS.

[0171] The horizontal and vertical cubic gradients are grad hor3 =(WW-W)-(WE)-(WE)+(E-EE)=WW-3W+3E-EE grad ver3 =(NN-N)-(NS)-(NS)+(S-SS)=NN-3N+3S-SS It can be defined as:

[0172] Note that higher order versions follow binomial coefficients. Higher order gradients can be calculated in any direction, i.e. horizontally, vertically, or diagonally.

[0173] According to one embodiment, the oblique gradient can be obtained using any angle instead of a fixed 45 degree oblique.

[0174] According to one embodiment, the gradient direction may correspond to one or more directions of angular intra prediction modes. For example, angular intra prediction modes from one or more of the neighboring blocks, such as the co-located luma block, may be used. In another example, the angular intra prediction method for determining the gradient direction may be obtained using texture analysis methods such as DIMD and TIMD methods applied to some or all of the reconstructed reference samples and / or some or all of the reconstructed samples in the reference channel.

[0175] Position Value According to one embodiment, the method includes predicting the target samples of at least one color channel of an image block unit using the following cross-component prediction model: pred=P0·C+P1·N+P2·S+P3·W+P4·E+…+P n X+P n+1 Y+…+P M-1 β where pred is the predicted chroma sample, P is the model coefficient for each tap in the model, (C,N,S,W,E) are example spatial samples at locations (center, north, south, west, east), X and Y are the horizontal and vertical coordinates of the center sample, respectively, and β is an offset or bias value.

[0176] Herein, the position values ​​may be calculated in different ways: for example, the coordinates of the sample relative to the top left corner of the block, or the coordinates of the sample relative to the top left corner of the reference area used to derive the filter parameters, or the coordinates of the sample relative to the top left corner of the picture, or the coordinates of the sample relative to the top left corner of the CTU / tile / slice / subpicture may be used.

[0177] According to one embodiment, the method includes using the horizontal and / or vertical location of the center sample.

[0178] According to one embodiment, the method includes using horizontal and / or vertical locations of spatially adjacent samples in addition to or instead of location information of the central sample.

[0179] Thus, location information (horizontal and / or vertical locations) of spatially neighboring samples, e.g., north, south, west, and east, may be used together with or instead of the location information of the center tap sample. The use of location information in the filter may be determined based on the availability of reference samples from neighboring blocks, the block size (height and width), the coding mode of one or more of the neighboring blocks, the coding mode of the reference block in the reference channel, etc.

[0180] According to one embodiment, the use of location information in the filter is determined based on the magnitude or absolute value of the model coefficients associated with the sample coordinate.

[0181] According to one embodiment, the use of location information in the filter is determined based on a consistency measure, which can be calculated, for example, by applying a cross-component model to a particular area of ​​a reference sample and determining how the presence of location information affects the prediction accuracy.

[0182] According to one embodiment, the method includes using one or more of the position values ​​in a non-linear function and using the output of the non-linear function as parameters of a cross-component prediction model.

[0183] For example, a polynomial version of the location information may be used. For example, the CCCM may be an M-tap filter such as: pred=P0·C+P1·N+P2·S+P3·W+P4·E+…+P mX n +P m+1 Y n +P m+2 X n-1 +P m+3 Y n-1 +…+P k X+P k+1 Y+…+P M-1 β

[0184] According to one embodiment, a filter is selected from a set of two or more candidate filters, at least one of which includes location information and at least one other of which does not. For example, a first filter can be determined to include model coefficients associated with at least a horizontal gradient, a vertical gradient, an x-coordinate, and a y-coordinate, and a second filter can be determined to include model coefficients associated with at least a horizontal gradient, a vertical gradient, a first diagonal gradient, and a second diagonal gradient. In this example, if at least one of the model coefficients associated with the coordinates has an absolute value greater than a threshold, the first filter can be selected; otherwise, the second filter can be selected.

[0185] As an example of using both one or more of the location information and one or more of the gradient information together with or instead of one or more of the spatial sample values ​​in the predictive model, the CCCM may be implemented as an M-tap filter, for example, as follows: pred=P0·C+P1·N+P2·S+P3·W+P4·E+…+P n ·grad hor +P n+1 ·grad ver +P n+2 ·grad diag +…+P k X+P k+1 Y+…+P M-1 β

[0186] An alternative implementation of the M-tap filter may be as follows. pred=P0·C+P1·N+P2·S+P3·W+P4·E+…+P n ·gradhor +P n+1 ·grad ver +P n+2 ·grad diag +…+P k X 2 +P k+1 Y 2 +P k+2 X Y + P k+3 X+P k+4 Y+…+P M-1 β

[0187] According to one embodiment, the method includes using a rate-distortion optimization (RDO) algorithm to determine a combination of a filter and the gradient values ​​and / or the position values ​​of spatial samples to be used together with or instead of spatial sample values ​​in a cross-component prediction model.

[0188] Therefore, different filters with different combinations of the above embodiments may be defined, and the best performing version may be determined based on a rate-distortion optimization (RDO) algorithm on the encoder side and / or decoder side. If the selection is made on the encoder side, an index or flag of the selected version may be signaled to the decoder in the bitstream.

[0189] According to one embodiment, the use of a combination of gradient and location information in a filter is determined based on a consistency measure, which can be calculated, for example, by applying a cross-component model to a particular area of ​​a reference sample and determining how the presence of the combination of gradient and location information affects the prediction accuracy.

[0190] According to one embodiment, the final prediction of a block can be obtained by combining two or more of the different versions of the above-mentioned embodiments. For example, two models can be derived. The first model can be derived from the left sample side of the reference sample using one or more of location and / or gradient information, and the second model can be derived from the above-mentioned reference sample using one or more of location and gradient information. The location and gradient information in the first model and the second model can be the same or different, or can have one or more of the same location and / or gradient information.

[0191] The spatial component can be considered to represent the low-frequency subband of the luma-chroma model, while the gradient component corresponds to the high-frequency subband. Hence, different training areas can be considered for the two different subbands.

[0192] According to one embodiment, coefficients for the spatial components can be obtained using a larger training area, and then the obtained spatial coefficients are applied over a smaller training area and the resulting prediction is subtracted from the smaller training area, after which the gradient coefficients are solved.

[0193] According to one embodiment, the spatial components can be obtained using only the left or top side of the training area. The obtained spatial coefficients are then applied across the entire training area and the resulting prediction is subtracted from the entire training area before solving for the gradient coefficients.

[0194] According to one embodiment, the gradient components can be obtained using only the left or top side of the training area: the spatial coefficients are obtained over the entire training area and subtracted from the other training areas, after which the gradient coefficients are calculated.

[0195] According to one embodiment, the spatial components can use further subsampled luma and chroma grids when larger training areas are used, so that low and high frequency components can be separated.

[0196] According to one embodiment, higher order gradients can be thought of as partitions of high frequency subbands, and therefore further partitioning of the training area can be applied to gradients of different orders.

[0197] According to one embodiment, when there are two or more cross-component models for predicting a block, the classification parameters may be determined based on one or more of the gradient and / or location information. For example, different models may be derived and applied for samples with gradients greater than or less than a certain threshold. In another example, different models may be derived and applied depending on the magnitude of the gradient in each direction.

[0198] According to one embodiment, the transform selection, e.g., LFNST, applied to a block predicted by one or more of the described embodiments may be determined based on one or more of the gradient values. For example, the LFNST transform type may be selected based on one or more magnitudes and directions of the gradient information used in the prediction model.

[0199] According to one aspect, an apparatus includes: means for receiving an image block unit of a frame, the image block unit including samples in color channels, the color channels including at least one chrominance channel and one luminance channel; means for reconstructing samples of the luminance channel of the image block unit; means for determining a reference area for predicting target samples of at least one color channel of the image block unit, the reference area including one or more of reference samples in adjacent blocks in a current color channel / frame, reference samples adjacent to a co-located block in a reference color channel / frame, and / or reference samples within the co-located block in a reference color channel / frame; means for determining gradient values ​​and / or position values ​​of spatial samples in the reference area to be used in a cross-component prediction model; and means for predicting the target samples of at least one color channel of the image block unit using the cross-component prediction model based on at least the gradient values ​​and / or the position values ​​of spatial samples in the reference area.

[0200] According to one embodiment, the apparatus comprises means for using said gradient values ​​and / or said position values ​​of spatial samples together with or instead of spatial sample values ​​in a cross-component prediction model.

[0201] According to one embodiment, the apparatus includes means for using a combination of said gradient values ​​and said position values ​​of spatial samples in a cross-component prediction model, together with or instead of the spatial sample values.

[0202] According to one embodiment, the apparatus includes means for indicating the use of gradient values ​​and / or position values ​​of spatial samples when signaling within or along a bitstream including predicted image block units.

[0203] According to one embodiment, the device is configured to use the following cross-component prediction model: pred=P0·C+P1·N+P2·S+P3·W+P4·E+…+P n ·grad hor +P n+1 ·grad ver +P n+2 ·grad diag +…+P M-1 β where pred is a predicted chroma sample, P is a model coefficient for each tap of an M-tap filter, (C,N,S,W,E) are example spatial samples at locations (center, north, south, west, east), and grad hor , grad ver , grad diag are the directional gradient values ​​in the horizontal, vertical, and diagonal directions, respectively, and β is the offset value.

[0204] According to one embodiment, the apparatus comprises means for using the sum or average of gradient values ​​in different directions as parameters of a cross-component prediction model.

[0205] According to one embodiment, the apparatus includes means for using one or more of the gradient values ​​in a non-linear function and using the output of the non-linear function as a parameter of a cross-component prediction model.

[0206] According to one embodiment, the apparatus includes means for using higher order derivatives of the gradient values ​​as parameters of a cross-component prediction model.

[0207] According to one embodiment, the device is configured to use the following cross-component prediction model: pred=P0·C+P1·N+P2·S+P3·W+P4·E+…+P n X+P n+1 Y+…+P M-1 β where pred is a predicted chroma sample, P is a model coefficient for each tap of an M-tap filter, (C,N,S,W,E) are example spatial samples at locations (center, north, south, west, east), X and Y are the horizontal and vertical coordinates of the center sample, respectively, and β is an offset value.

[0208] According to one embodiment, the device comprises means for using the horizontal and / or vertical location of the central sample.

[0209] According to one embodiment, the apparatus comprises means for using the horizontal and / or vertical locations of spatially adjacent samples together with or instead of the location information of the central sample.

[0210] According to one embodiment, the apparatus includes means for using one or more of the position values ​​in a non-linear function and using the output of the non-linear function as parameters of a cross-component prediction model.

[0211] According to one embodiment, the apparatus includes means for determining, using a rate-distortion optimization (RDO) algorithm, a combination of a filter and the gradient values ​​and / or the position values ​​of spatial samples to be used together with or instead of spatial sample values ​​in a cross-component prediction model.

[0212] In a further aspect, an apparatus is provided, the apparatus including at least one processor and at least one memory, the at least one memory having stored therein code that, when executed by the at least one processor, causes the apparatus to at least receive image block units of a frame, the image block units including samples in color channels, the color channels including at least one chrominance channel and one luminance channel; reconstruct samples of the luminance channel of the image block units; and determine a reference area for predicting target samples of the at least one color channel of the image block units. determining that the reference area includes one or more of reference samples in adjacent blocks in a current color channel / frame, reference samples adjacent to a co-located block in a reference color channel / frame, and / or reference samples inside the co-located block in a reference color channel / frame; determining gradient values ​​and / or position values ​​of spatial samples in the reference area to be used in a cross-component prediction model; and predicting the target samples of at least one color channel of the image block unit using the cross-component prediction model based on at least the gradient values ​​and / or the position values ​​of the spatial samples in the reference area.

[0213] According to one embodiment, the apparatus includes code that causes the apparatus to use the gradient values ​​and / or the position values ​​of spatial samples in a cross-component prediction model, together with or in place of spatial sample values.

[0214] According to one embodiment, the apparatus includes code that causes the apparatus to use a combination of the gradient values ​​and the position values ​​of spatial samples in a cross-component prediction model, along with or in place of the spatial sample values.

[0215] According to one embodiment, the apparatus includes code that causes the apparatus to indicate use of gradient values ​​and / or position values ​​of spatial samples when signaling within or along a bitstream that includes a predicted image block unit.

[0216] According to one embodiment, the apparatus includes code for causing the apparatus to predict the target samples of at least one color channel of an image block unit using the following cross-component prediction model: pred=P0·C+P1·N+P2·S+P3·W+P4·E+…+P n ·grad hor +P n+1 ·grad ver +P n+2 ·grad diag +…+P M-1 β where pred is the predicted chroma sample, P is the model coefficient for each tap of the M-tap filter, (C,N,S,W,E) are example spatial samples at locations (center, north, south, west, east), and grad hor , grad ver , grad diag are the directional gradient values ​​in the horizontal, vertical, and diagonal directions, respectively, and β is the offset value.

[0217] According to one embodiment, the apparatus includes code that causes the apparatus to use the sum or average of gradient values ​​in different directions as parameters for a cross-component prediction model.

[0218] According to one embodiment, the apparatus includes code for causing the apparatus to use one or more of the gradient values ​​in a non-linear function and use the output of the non-linear function as a parameter of a cross-component prediction model.

[0219] According to one embodiment, the apparatus includes code that causes the apparatus to use higher order derivatives of the gradient values ​​as parameters of a cross-component prediction model.

[0220] According to one embodiment, the apparatus includes code for causing the apparatus to predict the target samples of at least one color channel of an image block unit using the following cross-component prediction model: pred=P0·C+P1·N+P2·S+P3·W+P4·E+…+P n X+P n+1 Y+…+P M-1 β where pred is the predicted chroma sample, P is the model coefficient for each tap of the M-tap filter, (C,N,S,W,E) are example spatial samples at locations (center, north, south, west, east), X and Y are the horizontal and vertical coordinates of the center sample, respectively, and β is an offset value.

[0221] According to one embodiment, the device includes code that causes the device to use the horizontal and / or vertical location of the center sample.

[0222] According to one embodiment, the device includes code that causes the device to use the horizontal and / or vertical locations of spatially adjacent samples in addition to or in place of the location information of the central sample.

[0223] According to one embodiment, the apparatus includes code for causing the apparatus to use one or more of the position values ​​in a non-linear function and use the output of the non-linear function as a parameter of a cross-component prediction model.

[0224] According to one embodiment, the apparatus includes code that causes the apparatus to use a rate-distortion optimization (RDO) algorithm to determine a combination of a filter and the gradient values ​​and / or the position values ​​of spatial samples to be used together with or in place of spatial sample values ​​in a cross-component prediction model.

[0225] Such an apparatus may, for example, include the functional units disclosed in any of Figures 1, 2, 4a and 4b for implementing the embodiments.

[0226] Such an apparatus further includes code stored in the at least one memory, which when executed by the at least one processor causes the apparatus to perform one or more of the embodiments disclosed herein.

[0227] FIG. 13 is a graphical representation of an exemplary multimedia communication system in which various embodiments can be implemented. A data source 1510 provides a source signal in analog, uncompressed digital, or compressed digital format, or any combination of these formats. An encoder 1520 can include or involve preprocessing, such as data format conversion and / or filtering, of the source signal. The encoder 1520 encodes the source signal into a coded media bitstream. Note that the bitstream to be decoded can be received directly or indirectly from a remote device located within virtually any type of network. Additionally, the bitstream can be received from local hardware or software. An encoder 1520 may be capable of encoding two or more media types, such as audio and video, or two or more encoders 1520 may be required to code source signals of different media types. An encoder 1520 may also be capable of obtaining synthetically created input, such as graphics and text, and generating a coded bitstream of the synthetic media. In the following, only the processing of one coded media bitstream of one media type is considered for simplicity of explanation. It should be noted, however, that a real-time broadcast service typically includes several streams (typically at least one audio, video, and text subtitled stream). It should also be noted that while a system may include many encoders, the diagram shows only one encoder 1520 for ease of explanation without loss of generality. It should be further appreciated that while the text and examples included herein may specifically describe an encoding process, those skilled in the art will understand that the same concepts and principles apply to the corresponding decoding process, and vice versa.

[0228] The coded media bitstreams may be transferred to storage 1530. Storage 1530 may include any type of mass memory for storing coded media bitstreams. The format of the coded media bitstreams in storage 1530 may be a basic self-contained bitstream format, or one or more coded media bitstreams may be encapsulated in a container file, or the coded media bitstreams may be encapsulated in a segment format suitable for DASH (or a similar streaming system) and stored as a sequence of segments. If one or more media bitstreams are encapsulated in a container file, a file generator (not shown) may be used to store the one or more media bitstreams in a file and create file format metadata, which may also be stored in the file. The encoder 1520 or storage 1530 may include the file generator, or the file generator may be operably attached to either the encoder 1520 or storage 1530. Some systems operate “live,” i.e., omitting storage and transferring the coded media bitstreams from the encoder 1520 directly to the sender 1540. The coded media bitstreams may then be forwarded to a sender 1540, also referred to as a server, as needed. The format used for transmission may be a basic self-contained bitstream format, a packet stream format, a segment format suitable for DASH (or a similar streaming system), or one or more coded media bitstreams may be encapsulated in a container file. The encoder 1520, storage 1530, and server 1540 may reside on the same physical device or may be included in separate devices.The encoder 1520 and server 1540 may operate on live real-time content, in which case the coded media bitstream is generally not permanently stored but rather buffered for short periods in the content encoder 1520 and / or server 1540 to smooth out processing delays, transmission delays, and fluctuations in the coded media bitrate.

[0229] The server 1540 sends the coded media bitstream using a communication protocol stack. The stack may include, but is not limited to, one or more of Real-time Transport Protocol (RTP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Transmission Control Protocol (TCP), and Internet Protocol (IP). If the communication protocol stack is packet-oriented, the server 1540 encapsulates the coded media bitstream into packets. For example, if RTP is used, the server 1540 encapsulates the coded media bitstream into RTP packets according to an RTP payload format. Generally, each media type has its own RTP payload format. Note again that while the system may include more than one server 1540, for simplicity, the following description considers only one server 1540.

[0230] If the media content is encapsulated in a container file for storage 1530 or for inputting data to sender 1540, sender 1540 may include or be operatively attached to a "send file parser" (not shown). In particular, if the container file is not transmitted as such, but rather at least one of the contained coded media bitstreams is encapsulated for transport via a communication protocol, the send file parser locates the appropriate portion of the coded media bitstream to be carried via the communication protocol. The send file parser may also assist in creating the correct format for the communication protocol, such as packet headers and payloads. The multimedia container file may include encapsulation instructions, such as hint tracks in ISOBMFF, for encapsulating at least one of the contained media bitstreams based on the communication protocol.

[0231] The server 1540 may or may not be connected to the gateway 1550 through a communication network, which may be, for example, a CDN, the Internet, and / or a combination of one or more access networks. The gateway may additionally or alternatively be referred to as a middlebox. In DASH, the gateway may be an edge server (of a CDN) or a web proxy. Note that a system may generally include any number of gateways, etc., but for simplicity, the following description considers only one gateway 1550. The gateway 1550 may perform various types of functions, such as converting packet streams according to one communication protocol stack to another, merging and forking data streams, and manipulating data streams according to downlink and / or receiver capabilities, such as controlling the bit rate of a forwarding stream according to current downlink network conditions. The gateway 1550 may be a server entity in various embodiments.

[0232] The system typically includes one or more receivers 1560 capable of receiving, demodulating, and deencapsulating the transmitted signals into coded media bitstreams. The coded media bitstreams may be transferred to recording storage 1570. The recording storage 1570 may include any type of mass memory for storing the coded media bitstreams. Alternatively, or additionally, the recording storage 1570 may include computational memory, such as random access memory. The format of the coded media bitstreams in the recording storage 1570 may be a basic self-contained bitstream format, or one or more coded media bitstreams may be encapsulated in a container file. When there are multiple coded media bitstreams, such as related audio and video streams, container files are typically used, and the receiver 1560 includes or is attached to a container file generator that generates the container file from the input streams. Some systems operate "live," i.e., omitting the recording storage 1570 and transferring the coded media bitstreams from the receiver 1560 directly to the decoder 1580. In some systems, only the most recent portion of the recorded stream, for example, only the most recent 10 minute excerpt of the recorded stream, is maintained in recording storage 1570, while previous recording data is discarded from recording storage 1570.

[0233] The coded media bitstreams may be transferred from the recording storage 1570 to the decoder 1580. If there are many coded media bitstreams, such as audio and video streams, that are related to each other and encapsulated in a container file, or if a single media bitstream is encapsulated in a container file, for example for easier access, a file parser (not shown) is used to decapsulate each coded media bitstream from the container file. The recording storage 1570 or the decoder 1580 may include the file parser, or the file parser may be attached to either the recording storage 1570 or the decoder 1580. It should also be noted that while a system may include many decoders, only one decoder 1570 is discussed here for ease of explanation and without loss of generality.

[0234] The coded media bitstream may be further processed by a decoder 1570, the output of which is one or more uncompressed media streams. Finally, a renderer 1590 may play the uncompressed media streams, for example, over loudspeakers or a display. The receiver 1560, recording storage 1570, decoder 1570, and renderer 1590 may reside on the same physical device or may be included in separate devices.

[0235] The sender 1540 and / or gateway 1550 can be configured to perform switching between different representations, for example, switching between different viewports of 360-degree video content, view switching, bitrate adaptation, and / or fast startup, and / or the sender 1540 and / or gateway 1550 can be configured to select the representation to be transmitted. Switching between different representations can occur for many reasons, such as in response to a request from the receiver 1560 or to general conditions such as the throughput of the network over which the bitstream is carried. In other words, the receiver 1560 can initiate the switch between representations. A request from the receiver can be, for example, a request for a segment or subsegment with a different representation than before, a request for a change in the transmitted scalability layer and / or sublayer, or a change to a rendering device with different capabilities compared to before. A request for a segment can be an HTTP GET request. A request for a subsegment can be an HTTP GET request with a byte range. Additionally or alternatively, bitrate adjustment or bitrate adaptation can be used, for example, to provide so-called fast start-up in streaming services, where the bitrate of the transmitted stream is lower than the channel bitrate after the start of streaming or random access in order to start playback immediately and to achieve a buffer occupancy level that tolerates occasional packet delays and / or retransmissions. Bitrate adaptation can include multiple representation or layer-up switching and representation or layer-down switching operations performed in various orders.

[0236] The decoder 1580 can be configured to perform switching between different representations, e.g., for switching between different viewports of 360-degree video content, view switching, bitrate adaptation, and / or fast start-up, and / or the decoder 1580 can be configured to select a transmitted representation. Switching between different representations can be done for many reasons, such as to achieve faster decoding operations or to adapt the transmitted bitstream, e.g., with respect to bitrate, to prevailing conditions such as the throughput of the network over which the bitstream is carried. For example, faster decoding operations may be required if a device including the decoder 1580 is multitasking and uses computing resources for purposes other than decoding the video bitstream. In another example, faster decoding operations may be required when content is played at a pace faster than normal playback speed, e.g., at twice or three times the conventional real-time playback rate.

[0237] While some embodiments have been described above with reference to and / or using HEVC and / or VVC terminology, it should be understood that the embodiments may be implemented using any video encoder and / or video decoder as well.

[0238] In the above, where example embodiments are described with reference to an encoder, it should be understood that the resulting bitstream and decoder may have corresponding elements therein. Similarly, where example embodiments are described with reference to a decoder, it should be understood that the encoder may have a structure and / or computer program for generating a bitstream that is decoded by the decoder. For example, some embodiments have been described with reference to generating predictive blocks as part of encoding. Embodiments may similarly be realized by generating predictive blocks as part of decoding, with the difference that coding parameters such as horizontal and vertical offsets are decoded from the bitstream rather than determined by the encoder.

[0239] The above-described embodiments of the present invention describe the codec in terms of separate encoder and decoder devices to aid in understanding the processes involved. However, it will be understood that the devices, structures, and operations may be implemented as a single encoder-decoder device / structure / operation. Furthermore, it is possible that the coder and decoder may share some or all common elements.

[0240] Although the above examples describe embodiments of the present invention operating within a codec within an electronic device, it will be appreciated that the present invention as defined in the claims may be implemented as part of any video codec. Thus, for example, embodiments of the present invention may be implemented in a video codec that is capable of performing video coding over a fixed or wired communication path.

[0241] Thus, the user equipment may include a video codec such as those described in the embodiments of the present invention above. It should be understood that the term user equipment is intended to encompass any suitable type of wireless user equipment, such as a mobile phone, a portable data processing device, or a mobile web browser.

[0242] Additionally, elements of a public land mobile network (PLMN) may include video codecs such as those mentioned above.

[0243] In general, various embodiments of the present invention may be implemented in hardware or special purpose circuits, software, logic, or any combination thereof. For example, certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the invention is not limited thereto. While various aspects of the present invention may be illustrated and described using block diagrams, flow charts, or other graphical representations, it will be appreciated that these blocks, apparatus, systems, techniques, or methods described herein may be implemented in, by way of non-limiting example, hardware, software, firmware, special purpose circuits or logic, general purpose hardware or controller or other computing device, or combinations thereof.

[0244]

[0013] Embodiments of the present invention may be implemented by computer software executable by a data processor of a portable device, such as within a processor entity, or by hardware, or by a combination of software and hardware. Furthermore, in this regard, it should be noted that any block of logic flow as in the figures may represent program steps, or interconnected logic circuits, blocks, and functions, or combinations of program steps and logic circuits, blocks, and functions. Software may be stored on physical media, such as memory chips or memory blocks implemented within a processor, magnetic media, such as hard disks or floppy disks, and optical media, such as DVDs and CDs, and data variants thereof.

[0245] The memory may be of any type suitable for the local technology environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed and removable memory, etc. The data processor may be of any type suitable for the local technology environment and may include, by way of non-limiting examples, one or more of a general purpose computer, a special purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture.

[0246] Embodiments of the present invention can be practiced in a variety of components, such as integrated circuit modules. The design of integrated circuits is generally a highly automated process. Complex and powerful software tools are available to convert logic-level designs into semiconductor circuit designs ready to be etched and formed on semiconductor substrates.

[0247] Programs offered by Synopsys, Inc. of Mountain View, Calif., and Cadence Design of San Jose, Calif., use well-established design rules and a library of pre-stored design modules to automatically route conductors and place components on semiconductor chips. After the design of a semiconductor circuit is complete, the resulting design in a standard electronic format (e.g., Opus, GDSII, etc.) can be sent to a semiconductor manufacturing facility or "fab" for fabrication.

[0248] The foregoing description has provided a complete and informative description of exemplary embodiments of the present invention, by way of illustrative and non-limiting example. However, various changes and modifications will become apparent to those skilled in the art in light of the foregoing description, when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of this invention will still fall within the scope of the present invention.

Claims

1. means for receiving an image block unit of a frame, the image block unit including samples in color channels, the color channels including at least one chrominance channel and one luminance channel; means for reconstructing samples of the luminance channel of the image block unit; a means for determining a reference area for predicting a target sample of at least one color channel of the image block unit, the reference area including one or more of reference samples in adjacent blocks in a current color channel / frame, reference samples adjacent to a co-located block in a reference color channel / frame, and / or reference samples inside the co-located block in the reference color channel / frame; means for determining gradient and / or position values ​​of spatial samples within said reference area for use in a cross-component prediction model; means for predicting the target samples of at least one color channel of the image block unit using the cross-component prediction model based on at least the gradient values ​​and / or the position values ​​of the spatial samples within the reference area; An apparatus comprising:

2. means for using the gradient values ​​and / or the position values ​​of the spatial samples in the cross-component prediction model together with or instead of the spatial sample values; The apparatus of claim 1 , comprising:

3. means for using a combination of the gradient value and the position value of the spatial sample in the cross-component prediction model together with or in place of the spatial sample value; 3. The apparatus of claim 1 or 2, comprising:

4. means for indicating the use of the gradient values ​​and / or the position values ​​of the spatial samples when signaling within or along a bitstream containing a predicted image block unit; The apparatus according to any one of claims 1 to 3, comprising:

5. The following cross-component prediction model 1000 sq. ft 0 ・THIS 1 ・13 2 ・WATCH 3 ・WHAT 4 ・H...Z n ・ツツツ hor 4. 10 n+1 ・ツツツ ver 4. 10 n+2 ・ツツツ diag Z...NO M-1 ・β means for predicting the target sample of at least one color channel of the image block unit using Including, where pred is the predicted chroma sample, P is the model coefficient for each tap of the M-tap filter, (C,N,S,W,E) are example spatial samples at locations (center, north, south, west, east), and grad hor , grad ver , grad diag 5. The apparatus of claim 1, wherein β is a directional gradient value in the horizontal, vertical, and diagonal directions, respectively, and β is an offset or bias value.

6. means for using the sum or average of the gradient values ​​in different directions as parameters of the cross-component prediction model; The apparatus according to any one of claims 1 to 5, comprising:

7. means for using one or more of the gradient values ​​in a non-linear function and using the output of the non-linear function as parameters of the cross-component prediction model; The apparatus according to any one of claims 1 to 6, comprising:

8. means for using higher order derivatives of said gradient values ​​as said parameters of said cross-component prediction model; The apparatus of claim 7 , comprising:

9. The following cross-component prediction model 1000 sq. ft 0 ・THIS 1 ・13 2 ・WATCH 3 ・WHAT 4 ・H...Z n ・SIZ n+1 ・1Z...Z M-1 ・β means for predicting the target sample of at least one color channel of the image block unit using Including, 9. The apparatus of claim 1, wherein pred is a predicted chroma sample, P is a model coefficient for each tap of an M-tap filter, (C, N, S, W, E) are example spatial samples at locations (center, north, south, west, east), X and Y are the horizontal and vertical coordinates of the center sample, respectively, and β is an offset or bias value.

10. Means for using the horizontal and / or vertical location of the center sample The apparatus according to any one of claims 1 to 9, comprising:

11. means for using the horizontal and / or vertical locations of spatially adjacent samples together with or instead of the location information of said central sample; The apparatus of claim 10, comprising:

12. means for using one or more of the position values ​​in a non-linear function and using the output of the non-linear function as the parameters of the cross-component prediction model; 12. The apparatus according to claim 1, comprising:

13. means for determining, using a rate-distortion optimization (RDO) algorithm, a combination of a filter and the gradient values ​​and / or the position values ​​of the spatial samples to be used together with or instead of the spatial sample values ​​in the cross-component prediction model; 13. The apparatus according to any one of claims 1 to 12, comprising:

14. receiving an image block unit of a frame, the image block unit including samples in color channels, the color channels including at least one chrominance channel and one luminance channel; reconstructing the luma channel samples of the image block unit; determining a reference area for predicting a target sample of at least one color channel of the image block unit, the reference area including one or more of reference samples in adjacent blocks in a current color channel / frame, reference samples adjacent to a co-located block in a reference color channel / frame, and / or reference samples inside the co-located block in the reference color channel / frame; determining gradient and / or position values ​​of spatial samples within said reference area for use in a cross-component prediction model; predicting the target samples of at least one color channel of the image block unit using the cross-component prediction model based on at least the gradient values ​​and / or the position values ​​of the spatial samples within the reference area; A method comprising:

15. using the gradient values ​​and / or the position values ​​of the spatial samples in the cross-component prediction model together with or instead of the spatial sample values; 15. The method of claim 14, comprising:

Images