Apparatus, method and computer program for video encoding and decoding

Abstract

A method comprising: receiving an image block unit of a frame, the image block unit comprising samples in a color channel, the color channel comprising at least one chroma channel and one luma channel; reconstructing a sample of the luminance channel of an image block unit; determining a reference region for predicting samples of at least one color channel of the image block unit, wherein the reference region comprises one or more of reference samples in neighboring blocks in the current color channel / frame, adjacent to a co-located block in the reference color channel / frame, and / or within a co-located block in the reference color channel / frame; determining a filter coefficient of the filter based on a reference sample and a form of the filter used for the prediction; and predicting samples of at least one color channel of the image block unit by performing a convolution between the samples of the luminance channel and the filter coefficients.

Description

Technical Field

[0001] The present invention relates to apparatus, methods and computer programs for video encoding and decoding. Background Art

[0002] In video coding, video and image samples are often encoded using a color representation, such as YUV or YCbCr consisting of one luma and two chroma channels. In these cases, the luma channel (which primarily represents the lighting of the scene) is usually encoded at a certain resolution, while the chroma channels (which typically represent the differences between certain color components) are usually encoded at a second resolution lower than the luma signal. The purpose of this differential representation is to decorrelate the color components and enable more efficient compression of the data.

[0003] In the Universal Video Coding (VVC / H.266) standard, the cross-component linear model (CCLM) is used as a linear model for predicting samples in chroma channels (e.g., Cb and Cr). The model parameters are derived based on reconstructed samples in the neighborhood of chroma blocks, co-located neighboring samples in luma blocks, and reconstructed samples within co-located luma blocks.

[0004] The purpose of this model is to find the correlation of samples between two or more channels. However, the first-order linear model of the CCLM method is not sufficient to model the spatial correlation between luma and chroma samples (such as texture or surface), so the performance is suboptimal. Summary of the invention

[0005] Now, in order to at least alleviate the above problems, an enhancement method for achieving better correlation between luma and chroma channels is introduced in this paper.

[0006] The protection scope of various embodiments of the present invention is defined by the independent claims. Embodiments and features described in this specification that do not belong to the scope of the independent claims (if any) should be interpreted as examples that help understand various embodiments of the present invention.

[0007] The method according to the first aspect includes: receiving an image block unit of a frame, the image block unit including samples in a color channel, the color channel including at least one chrominance channel and one luminance channel; reconstructing samples of the luminance channel of the image block unit; determining a reference area for predicting samples of at least one color channel of the image block unit, wherein the reference area includes: one or more reference samples of reference samples in adjacent blocks in the current color channel / frame, adjacent to a co-located block in a reference color channel / frame, and / or within a co-located block in a reference color channel / frame; determining filter coefficients of a filter based on the reference samples and the morphology of the filter used for the prediction; and predicting samples of at least one color channel of the image block unit by performing convolution between samples of the luminance channel and the filter coefficients.

[0008] The apparatus according to the second aspect includes: a component for receiving an image block unit of a frame, the image block unit including samples in color channels including at least one chrominance channel and one luminance channel; a component for reconstructing samples of the luminance channel of the image block unit; a component for determining a reference area for predicting samples of at least one color channel of the image block unit, wherein the reference area includes: one or more reference samples of reference samples in neighboring blocks in a current color channel / frame, adjacent to a co-located block in a reference color channel / frame, and / or within a co-located block in a reference color channel / frame; a component for determining filter coefficients of a filter based on the reference samples and the morphology of the filter used for the prediction; and a component for predicting samples of at least one color channel of the image block unit by performing convolution between samples of the luminance channel and the filter coefficients.

[0009] According to one embodiment, the device comprises means for downsampling the samples of the luma channel to correspond to the size of the samples of the chroma samples before determining the filter coefficients.

[0010] According to one embodiment, the means for determining the filter coefficients comprises: means for obtaining statistical data for linear regression from reference samples based on the morphology of the filter; and means for obtaining the filter coefficients by applying a linear regression method to the obtained statistical data.

[0011] According to one embodiment, the statistical data include an autocorrelation matrix and a cross-correlation vector.

[0012] According to one embodiment, the filter has the form of a one-dimensional (1D-) vector of filter coefficients or a two-dimensional (2D-) array of filter coefficients.

[0013] According to one embodiment, the device comprises means for applying auxiliary parameters external to the morphology of the filter to said convolution.

[0014] According to one embodiment, the device includes: a component for inputting the following items to the filter: a luma sample co-located with the output chroma sample, at least one luma sample above the co-located sample, at least one luma sample below the co-located sample, at least one luma sample to the left of the co-located sample, at least one luma sample to the right of the co-located sample, and a bias term.

[0015] According to one embodiment, the apparatus comprises means for inputting to the filter a term whose value is determined by applying a non-linear function to the values ​​of the co-located samples.

[0016] According to one embodiment, the apparatus comprises: means for storing filter coefficients in a memory; and means for receiving an indication from an encoder to reuse at least a subset of previously stored filter coefficients to predict samples of at least one color channel of a current image block unit.

[0017] According to one embodiment, the device includes: a component for determining samples outside a reference area; a component for obtaining statistical data for linear regression from samples outside the reference area based on the morphology of a filter; a component for obtaining filter coefficients by applying a linear regression method to the obtained statistical data; and a component for storing the filter coefficients in a memory.

[0018] According to one embodiment, the apparatus comprises means for replacing any missing downsampled luma samples with zero-valued samples.

[0019] According to one embodiment, the apparatus comprises: means for determining samples that do not belong to any object or texture within the image block unit; and means for removing the samples from a reference area.

[0020] The apparatus and the computer-readable storage medium storing code as described above are therefore arranged to perform the above-described method and one or more embodiments related thereto. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] For a better understanding of the present invention, reference will now be made, by way of example, to the accompanying drawings, in which:

[0022] Figure 1 An electronic device using an embodiment of the present invention is schematically shown;

[0023] Figure 2 A user equipment suitable for adopting an embodiment of the present invention is schematically shown;

[0024] Figure 3 Also schematically illustrated are electronic devices employing embodiments of the present invention connected using wireless and wired network connections;

[0025] Figure 4a and 4b Schematically shows an encoder and a decoder suitable for implementing an embodiment of the present invention;

[0026] Figure 5 The locations of samples used to derive parameters for the cross-component linear model are shown;

[0027] Figure 6a and 6b An example of classifying brightness samples into two categories in the sample domain and the spatial domain is shown accordingly;

[0028] Figure 7 An example of four reference lines adjacent to a prediction block is shown;

[0029] Figure 8 The matrix-weighted intra prediction process is shown;

[0030] Fig. 9 A flowchart of a method for predicting samples in at least one color channel according to an embodiment of the present invention is shown;

[0031] Fig.10 shows an example of a co-located reference sample region consisting of reconstructed luma and chroma samples defined for both luma and chroma in accordance with an embodiment of the present invention;

[0032] Fig.11 shows various examples of filter kernel sizes according to embodiments of the present invention; and

[0033] Fig.12 A schematic diagram of an example multimedia communication system is shown in which various embodiments may be implemented. DETAILED DESCRIPTION

[0034] Suitable means and possible mechanisms for predicting chrominance samples are described in further detail below. In this regard, reference is first made to Figure 1 and Figure 2 ,in Figure 1 A block diagram of a video encoding system according to an example embodiment is shown as a schematic block diagram of an exemplary apparatus or electronic device 50, which may incorporate a codec according to an embodiment of the present invention. Figure 2 A layout of an apparatus according to an example embodiment is shown. Figure 1 and Figure 2 The elements of are explained next.

[0035] The electronic device 50 may be, for example, a mobile terminal or user equipment of a wireless communication system. However, it will be appreciated that embodiments of the present invention may be implemented in any electronic device or apparatus that may require encoding and decoding or encoding or decoding video images.

[0036] Device 50 can include a housing 30 for combining and protecting the device. Device 50 can also include a display 32 in the form of a liquid crystal display. In other embodiments of the present invention, the display can be any suitable display technology suitable for displaying images or videos. Device 50 can also include a keyboard 34. In other embodiments of the present invention, any suitable data or user interface mechanism can be adopted. For example, the user interface can be implemented as a virtual keyboard or as a data input system as a touch display part.

[0037] The device may include a microphone 36 or any suitable audio input, which may be a digital or analog signal input. The device 50 may also include an audio output device, which in embodiments of the present invention may be any of the following: headphones 38, speakers, or analog audio or digital audio output connections. The device 50 may also include a battery (or in other embodiments of the present invention, the device may be powered by any suitable mobile energy device, such as a solar cell, a fuel cell, or a clockwork generator). The device may also include a camera capable of recording or capturing images and / or video. The device 50 may also include an infrared port for short-range line-of-sight communication with other devices. In other embodiments, the device 50 may also include any suitable short-range communication solution, such as, for example, a Bluetooth wireless connection or a USB / Firewire wired connection.

[0038] The apparatus 50 may include a controller 56, a processor or processor circuitry for controlling the apparatus 50. The controller 56 may be connected to a memory 58, which in embodiments of the invention may store data in the form of image and audio data and / or may also store instructions for implementation on the controller 56. The controller 56 may also be connected to a codec circuitry 54, which may be adapted to perform encoding and decoding of audio and / or video data, or to assist in encoding and decoding performed by the controller.

[0039] The apparatus 50 may also include a card reader 48 and a smart card 46, such as a UICC and a UICC reader, for providing user information and adapted to provide authentication information for authenticating and authorizing the user on the network.

[0040] The device 50 may include a radio interface circuit system 52, which is connected to the radio interface of the controller and is suitable for generating wireless communication signals, such as for communicating with a cellular communication network, a wireless communication system or a wireless local area network. The device 50 may also include an antenna 44 connected to the radio interface circuit system 52 for transmitting radio frequency signals generated at the radio interface circuit system 52 to (multiple) other devices, and for receiving radio frequency signals from (multiple) other devices.

[0041] The device 50 may include a camera capable of recording or detecting individual frames, which are then passed to the codec 54 or controller for processing. The device may receive video image data from another device for processing before transmission and / or storage. The device 50 may also receive images for encoding / decoding via a wireless or wired connection. The structural elements of the device 50 described above represent examples of components for performing corresponding functions.

[0042] about Figure 3 , an example of a system in which embodiments of the present invention may be utilized is shown. System 10 includes a plurality of communication devices that may communicate via one or more networks. System 10 may include any combination of wired or wireless networks, including but not limited to wireless cellular telephone networks (such as GSM, UMTS, CDMA networks, etc.), wireless local area networks (WLANs) (such as defined by any of the IEEE 802.x standards), Bluetooth personal area networks, Ethernet local area networks, token ring local area networks, wide area networks, and the Internet.

[0043] System 10 may include both wired and wireless communication devices and / or apparatuses 50 suitable for implementing embodiments of the present invention.

[0044] For example, Figure 3 The system shown in shows a representation of the mobile phone network 11 and the Internet 28. Connectivity to the Internet 28 may include, but is not limited to, long-range wireless connections, short-range wireless connections, and various wired connections, including but not limited to telephone lines, cable lines, power lines, and similar communication paths.

[0045] The example communication devices shown in system 10 may include, but are not limited to, electronic devices or apparatuses 50, a combination of a personal digital assistant (PDA) and mobile phone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, and a laptop computer 22. Apparatus 50 may be stationary or mobile when carried by a moving individual. Apparatus 50 may also be located in a vehicle, including, but not limited to, a car, a truck, a taxi, a bus, a train, a boat, an airplane, a bicycle, a motorcycle, or any similar suitable vehicle.

[0046] Embodiments may also be implemented in a set-top box; i.e., a digital television receiver, which may or may not have a display or wireless capabilities, a tablet or (notebook) personal computer (PC) with a combination of hardware or software or encoder / decoder implementations, various operating systems, as well as chipsets, processors, DSPs and / or embedded systems that provide hardware / software based encoding.

[0047] Some or additional devices may send and receive calls and messages and communicate with service providers via a wireless connection 25 with a base station 24. The base station 24 may be connected to a network server 26 to allow communication between the mobile phone network 11 and the Internet 28. The system may include additional communication devices and various types of communication devices.

[0048] The communication device can use various transmission technologies to communicate, including but not limited to code division multiple access (CDMA), global system for mobile communications (GSM), universal mobile telecommunications system (UMTS), time division multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-Internet protocol (TCP-IP), short message service (SMS), multimedia message service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.11 and any similar wireless communication technology. The communication device involved in implementing various embodiments of the present invention can use various media to communicate, including but not limited to radio, infrared, laser, cable connection and any suitable connection.

[0049] In telecommunications and data networks, a channel can refer to a physical channel or a logical channel. A physical channel can refer to a physical transmission medium, such as a wire, while a logical channel can refer to a logical connection on a multiplexed medium, capable of carrying several logical channels. A channel can be used to carry an information signal (e.g., a bit stream) from one or more transmitters (or senders) to one or more receivers.

[0050] The MPEG-2 transport stream (TS) specified in ISO / IEC 13818-1 or also in ITU-T Recommendation H.222.0 is a format for carrying audio, video and other media as well as program metadata or other metadata in a multiplexed stream. A packet identifier (PID) is used to identify elementary streams (also called packetized elementary streams) within a TS. Therefore, a logical channel within an MPEG-2 TS can be considered to correspond to a specific PID value.

[0051] Available media file format standards include ISO Base Media File Format (ISO / IEC 14496-12, which may be abbreviated as ISOBMFF) and a file format for NAL unit structured video (ISO / IEC 14496-15), which is derived from ISOBMFF.

[0052] A video codec consists of an encoder that converts the input video into a compressed representation suitable for storage / transmission and a decoder that can decompress the compressed video representation back into a viewable form. A video encoder and / or a video decoder can also be separate from each other, i.e., they do not need to form a codec. Typically, an encoder discards some information from the original video sequence in order to represent the video in a more compact form (i.e., at a lower bitrate).

[0053] A typical hybrid video encoder (e.g., many encoder implementations of ITU-T H.263 and H.264) encodes video information in two stages. First, the pixel values ​​in a certain image region (or "block") are predicted by motion compensation means (looking up and indicating an area in a previously encoded video frame that closely corresponds to the block being encoded) or by spatial means (using the pixel values ​​surrounding the block to be encoded in a specified manner). Second, the prediction error (i.e., the difference between the predicted pixel block and the original pixel block) is encoded. This is typically accomplished by transforming the difference in pixel values ​​using a specified transform (e.g., a discrete cosine transform (DCT) or a variant thereof), quantizing the coefficients, and entropy encoding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (image quality) and the size of the resulting encoded video representation (file size or transmission bit rate).

[0054] In temporal prediction, the prediction source is a previously decoded image (also known as a reference image). In intra-block copy (IBC; also known as intra-block copy prediction), prediction is applied similarly to temporal prediction, but the reference picture is the current picture, and in the prediction process, only previously decoded samples can be referenced. Inter-layer or inter-view prediction can be applied similarly to temporal prediction, but the reference picture is a decoded picture from another scalable layer or from another view, respectively. In some cases, inter-frame prediction may refer only to temporal prediction, while in other cases, inter-frame prediction may collectively refer to temporal prediction and any of intra-block copy, inter-layer prediction, and inter-view prediction, provided that they are performed in the same or similar process as temporal prediction. Inter-frame prediction or temporal prediction may sometimes be referred to as motion compensation or motion compensated prediction.

[0055] Motion compensation can be performed with full sample or sub-sample accuracy. In the case of full sample accurate motion compensation, the motion can be represented as a motion vector with integer values ​​for horizontal and vertical displacements, and the motion compensation process uses those displacements to effectively copy samples from a reference picture. In the case of sub-sample accurate motion compensation, the motion vector is represented by a fractional or decimal value for the horizontal and vertical components of the motion vector. In the case where the motion vector refers to a non-integer position in the reference picture, a sub-sample interpolation process is typically called to calculate predicted sample values ​​based on the reference sample and the selected sub-sample position. The sub-sample interpolation process typically consists of a horizontal filter compensating for the horizontal offset relative to the full sample position, followed by a vertical filter compensating for the vertical offset relative to the full sample position. However, in some environments, vertical processing can also be done before horizontal processing.

[0056] Inter prediction (also known as temporal prediction, motion compensation or motion compensated prediction) reduces temporal redundancy. In inter prediction, the prediction source is a previously decoded picture. Intra prediction exploits the fact that adjacent pixels within the same picture may be correlated. Intra prediction can be performed in the spatial or transform domain, i.e., sample values ​​or transform coefficients can be predicted. Intra prediction is typically used for intra-frame coding, where inter prediction is not applied.

[0057] One result of the encoding process is a set of coded parameters, such as motion vectors and quantized transform coefficients. Many parameters can be entropy coded more efficiently if they are first predicted from spatially or temporally neighboring parameters. For example, a motion vector can be predicted from spatially neighboring motion vectors, and only the difference with respect to the motion vector predictor can be coded. Prediction of coding parameters and intra-frame prediction can be collectively referred to as intra-picture prediction.

[0058] Figure 4a and 4b An encoder and decoder suitable for employing embodiments of the present invention are shown. A video codec includes an encoder that converts an input video into a compressed representation suitable for storage / transmission and a decoder that can decompress the compressed video representation back into a viewable form. Typically, an encoder discards and / or loses certain information in the original video sequence in order to represent the video in a more compact form (i.e., at a lower bit rate). Figure 4a An example of the encoding process is shown. Figure 4a The image to be encoded (I n ); The predicted representation of the image block (P' n ); prediction error signal (D n ); reconstructed prediction error signal (D' n ); Preliminary reconstructed image (I' n ); Final reconstructed image (R' n ); Transform (T) and Inverse Transform (T -1), quantization (Q) and inverse quantization (Q -1 ); Entropy coding (E); Reference frame memory (RFM); Inter-frame prediction (P inter ); intra prediction (P intra ); mode selection (MS) and filtering (F).

[0059] An example of the decoding process is shown in Figure 4b is shown in . Figure 4b The predicted representation of the image block (P' n ); reconstructed prediction error signal (D' n ); Preliminary reconstructed image (I' n ); Final reconstructed image (R' n ); inverse transform (T -1 ); inverse quantization (Q -1 ); Entropy decoding (E -1 ); reference frame memory (RFM); prediction (inter or intra) (P); and filtering (F).

[0060] Many hybrid video encoders encode video information in two stages. First, the pixel values ​​in a particular image region (or "block") are predicted, for example, by motion compensation means (looking up and indicating an area in a previously encoded video frame that closely corresponds to the block being encoded) or by spatial means (using the pixel values ​​surrounding the block to be encoded in a specified manner). Second, the prediction error (i.e., the difference between the predicted pixel block and the original pixel block) is encoded. This is typically done by transforming the difference in pixel values ​​using a specified transform (e.g., the discrete cosine transform (DCT) or a variant thereof), quantizing the coefficients, and entropy encoding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (image quality) and the size of the resulting encoded video representation (file size or transmission bit rate). By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and the size of the resulting encoded video representation (file size or transmission bit rate). Video codecs may also provide a transform skip mode, which the encoder may choose to use. In transform skip mode, the prediction error is encoded in the sample domain, for example by deriving sample-wise difference values ​​with respect to certain neighboring samples and encoding the sample-wise difference values ​​using an entropy encoder.

[0061] Entropy coding / decoding can be performed in a variety of ways. For example, context-based coding / decoding can be applied, where both the encoder and the decoder modify the context state of the coding parameters based on the coding parameters of the previous coding / decoding. Context-based coding can be, for example, context-adaptive binary arithmetic coding (CABAC) or context-based variable length coding (CAVLC) or any similar entropy coding. Entropy coding / decoding can be performed alternatively or additionally using a variable length coding scheme, such as Huffman coding / decoding or Exp-Golomb coding / decoding. Decoding coding parameters from an entropy-coded bitstream or codeword can be referred to as parsing.

[0062] The phrase along the bitstream (e.g., indicating along the bitstream) may be defined to refer to out-of-band transmission, signaling, or storage in a manner that out-of-band data is associated with the bitstream. The phrase decoding along the bitstream or similar terms may refer to decoding the referred out-of-band data associated with the bitstream (which may be obtained from the out-of-band transmission, signaling, or storage). For example, indicating along the bitstream may refer to metadata in a container file that encapsulates the bitstream.

[0063] The H.264 / AVC standard was developed by the Joint Video Team (JVT) of the Video Coding Experts Group (VCEG) of the Telecommunication Standardization Sector of the International Telecommunication Union (ITU-T) and the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard was published by two parent standardization organizations and is known as ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10, also known as MPEG-4 Part 10 Advanced Video Coding (AVC). There are multiple versions of the H.264 / AVC standard, with new extensions or features integrated into the specification. These extensions include Scalable Video Coding (SVC) and Multi-view Video Coding (MVC).

[0064] Version 1 of the High Efficiency Video Coding (H.265 / HEVC aka HEVC) standard was developed by the Joint Collaboration Team - Video Coding (JCT-VC) of VCEG and MPEG. The standard was published by two parent standardization organizations and is known as ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2, also known as MPEG-H Part 2 High Efficiency Video Coding (HEVC). Subsequent versions of H.265 / HEVC include Scalable, Multi-view, Fidelity Range, 3D and Screen Content Coding extensions, which may be abbreviated as SHVC, MV-HEVC, REXT, 3D-HEVC and SCC, respectively.

[0065] Versatile Video Coding (VVC) (MPEG-I Part 3), also known as ITU-T H.266, is a video compression standard developed by the Joint Video Experts Group (JVET) of the Moving Picture Experts Group (MPEG) (formally known as ISO / IEC JTC1 SC29 WG11) and the Video Coding Experts Group (VCEG) of the International Telecommunication Union (ITU) as the successor to HEVC / H.265.

[0066] Some key definitions, bitstream and coding structures and concepts of H.264 / AVC and HEVC will be described in this section as examples of video encoders, decoders, encoding methods, decoding methods and bitstream structures, in which embodiments can be implemented. Some key definitions, bitstream and coding structures and concepts of H.264 / AVC are the same as those in HEVC - therefore, they will be described together below. Aspects of the present invention are not limited to H.264 / AVC or HEVC, but a description is given of one possible basis for which the present invention can be partially or fully implemented.

[0067] Similar to many earlier video coding standards, the bitstream syntax and semantics and the decoding process for error-free bitstreams are specified in H.264 / AVC and HEVC. The encoding process is not specified, but the encoder must produce a conforming bitstream. The conformance of the bitstream and the decoder can be verified by a hypothetical reference decoder (HRD). These standards contain coding tools that help cope with transmission errors and losses, but the use of these tools in encoding is optional, and the decoding process for erroneous bitstreams is not specified.

[0068] The basic unit for the input of an H.264 / AVC or HEVC encoder and the output of an H.264 / AVC or HEVC decoder, respectively, is a picture. A picture given as input to an encoder may also be referred to as a source picture, while a picture decoded by a decoder may be referred to as a decoded picture.

[0069] The source picture and the decoded picture each include one or more sample arrays, such as one of the following sets of sample arrays:

[0070] - Luminance (Y) only (monochrome).

[0071] - Luma and two chrominances (YCbCr or YCgCo).

[0072] - Green, Blue and Red (GBR, also known as RGB).

[0073] - An array representing otherwise unspecified monochromatic or tristimulus color samples (e.g., YZX, also known as XYZ).

[0074] In H.264 / AVC and HEVC, a picture can be either a frame or a field. A frame consists of a matrix of luma samples and possibly corresponding chroma samples. A field is a set of alternating sample rows of a frame and can be used as encoder input when the source signal is interlaced. The chroma sample array may not be present (so monochrome sampling may be used) or may be subsampled compared to the luma sample array. The chroma format can be summarized as follows:

[0075] - In monochrome sampling, there is only one sample array, which can nominally be considered as the brightness array.

[0076] - In 4:2:0 sampling, each of the two chroma arrays has half the height and half the width of the luma array.

[0077] - In 4:2:2 sampling, each of the two chroma arrays has the same height and half the width of the luma array.

[0078] - In 4:4:4 sampling, when separate color planes are not used, each of the two chroma arrays has the same height and width as the luma array.

[0079] In H.264 / AVC and HEVC, sample arrays can be encoded into the bitstream as separate color planes, and the separately encoded color planes can be decoded from the bitstream accordingly. When separate color planes are used, each of them is processed separately (by the encoder and / or decoder) as a picture with monochrome sampling.

[0080] Partitioning can be defined as dividing a set into subsets such that every element of the set is in exactly one of the subsets.

[0081] When describing the operation of HEVC encoding and / or decoding, the following terms may be used. A coding block may be defined as an NxN sample block for a certain value of N, such that the division of a coding tree block into coding blocks is a partition. A coding tree block (CTB) may be defined as an NxN sample block for a certain value of N, such that the division of a component into coding tree blocks is a partition. A coding tree unit (CTU) may be defined as a coding tree block of luma samples of a picture with three sample arrays, two corresponding coding tree blocks of chroma samples, or a coding tree block of samples of a monochrome picture, or a picture encoded using three separate color planes and a syntax structure used to encode samples. A coding unit (CU) may be defined as a coding block of luma samples of a picture with three sample arrays, two corresponding coding blocks of chroma samples, or a coding block of samples of a monochrome picture, or a picture encoded using three separate color planes and a syntax structure used to encode samples. A CU with a maximum allowed size may be referred to as an LCU (maximum coding unit) or a coding tree unit (CTU), and a video picture is divided into non-overlapping LCUs.

[0082] A CU consists of one or more prediction units (PUs) defining a prediction process for samples within the CU and one or more transform units (TUs) defining a prediction error coding process for samples within the CU. Typically, a CU consists of a square block of samples whose size can be selected from a predefined set of possible CU sizes. Each PU and TU can be further split into smaller PUs and TUs to increase the granularity of the prediction and prediction error coding processes accordingly. Each PU has prediction information associated with it that defines which prediction will be applied for pixels within the PU (e.g., motion vector information for inter-prediction PUs and intra-prediction directionality information for intra-prediction PUs).

[0083] Each TU may be associated with information describing the prediction error decoding process for samples within the TU (including, for example, DCT coefficient information). Whether prediction error coding is applied for each CU is typically signaled at the CU level. In the absence of a prediction error residual associated with a CU, it may be considered that there is no TU for the CU. The partitioning of an image into CUs and the partitioning of a CU into PUs and TUs is typically signaled in the bitstream to allow a decoder to reproduce the expected structure of these units.

[0084] In HEVC, a picture can be divided into tiles, which are rectangular and contain an integer number of LCUs. In HEVC, the division of tiles forms a regular grid where the height and width of the tiles differ from each other by at most one LCU. In HEVC, a slice is defined as an integer number of coding tree units contained in an independent slice segment and all subsequent dependent slice segments (if any) before the next independent slice segment (if any) in the same access unit. In HEVC, a slice segment is defined as an integer number of coding tree units that are ordered consecutively in a tile scan and contained in a single NAL unit. The division of each picture into slice segments is a partitioning. In HEVC, an independent slice segment is defined as a slice segment for which the values ​​of the syntax elements of the slice segment header are not inferred from the values ​​for the previous slice segment, while a dependent slice segment is defined as a slice segment for which the values ​​of some syntax elements of the slice segment header are inferred from the values ​​of the previous independent slice segment in decoding order. In HEVC, a slice header is defined as a slice segment header of an independent slice segment, which is an independent slice segment before the current slice segment or the current dependent slice segment, and a slice segment header is defined as a part of a coded slice segment, which contains data units related to the first or all coding tree units represented in the slice segment. If slices are not used, the CU is scanned in the raster scan order of LCUs within a slice or within a picture. Within an LCU, the CU has a specific scanning order.

[0085] The decoder reconstructs the output video by applying a prediction component similar to the encoder to form a predicted representation of pixel blocks (using motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (the inverse operation of prediction error encoding, recovering the quantized prediction error signal in the spatial pixel domain). After applying the prediction and prediction error decoding components, the decoder adds the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) may also apply additional filtering components to improve the quality of the output video before passing it for display and / or storing it as a prediction reference for upcoming frames in the video sequence.

[0086] The filtering may, for example, include one or more of: deblocking, sample adaptive offset (SAO) and / or adaptive loop filtering (ALF). H.264 / AVC includes deblocking, while HEVC includes deblocking and SAO.

[0087] In a typical video codec, motion information is indicated using a motion vector associated with each motion compensated image block (such as a prediction unit). Each of these motion vectors represents the displacement of an image block in a picture to be encoded (on the encoder side) or decoded (on the decoder side) and a prediction source block in one of the previously encoded or decoded pictures. In order to efficiently represent motion vectors, those motion vectors are usually differentially encoded relative to block-specific predicted motion vectors. In a typical video codec, predicted motion vectors are created in a predefined manner, such as calculating the median of the encoded or decoded motion vectors of adjacent blocks. Another way to create motion vector predictions is to generate a list of candidate predictions from adjacent blocks and / or co-located blocks in a temporal reference picture and signal the selected candidate as a motion vector predictor. In addition to predicting motion vector values, it is also possible to predict which reference pictures are used for motion compensated predictions, and this prediction information can be represented, for example, by a reference index of a previously encoded / decoded picture. The reference index is usually predicted based on adjacent blocks and / or co-located blocks in a temporal reference picture. In addition, typical high-efficiency video codecs employ an additional motion information encoding / decoding mechanism, commonly referred to as merging / merging mode, in which all motion field information (including motion vectors and corresponding reference picture indices for each available reference picture list) is predicted and used without any modification / correction. Similarly, predicting the motion field information is performed using the motion field information of neighboring blocks and / or co-located blocks in the temporal reference picture, and the used motion field information is signaled in the motion field candidate list list populated with the motion field information of the available neighboring / co-located blocks.

[0088] In a typical video codec, the motion compensated prediction residuals are first transformed using a transform kernel (such as DCT) and then encoded. The reason for this is that there is usually still some correlation between the residuals, and the transform can help reduce this correlation and provide more efficient coding in many cases.

[0089] Video coding standards and specifications may allow the encoder to divide a coded picture into coded slices, etc. Intra-picture prediction is typically disabled in slice boundaries. Therefore, slices can be viewed as a way to divide a coded picture into independently decodable segments. In H.264 / AVC and HEVC, intra-picture prediction can be disabled in slice boundaries. Therefore, slices can be viewed as a way to divide a coded picture into independently decodable segments, and therefore slices are typically viewed as basic units for transmission. In many cases, the encoder can indicate in the bitstream which types of intra-picture prediction are turned off in slice boundaries, and the decoder operation takes this information into account when, for example, determining which prediction sources are available. For example, if adjacent CUs are located in different slices, samples from adjacent CUs may be considered unavailable for intra prediction.

[0090] The basic unit for the output of the H.264 / AVC or HEVC encoder and the input of the H.264 / AVC or HEVC decoder is the network abstraction layer (NAL) unit accordingly. In order to be transmitted over a packet-oriented network or stored in a structured file, the NAL unit can be encapsulated into a packet or similar structure. The bitstream format has been specified in H.264 / AVC and HEVC for a transmission or storage environment that does not provide a frame structure. The bitstream format separates NAL units from each other by appending a start code in front of each NAL unit. In order to avoid erroneous detection of NAL unit boundaries, the encoder runs a byte-oriented start code simulation prevention algorithm that adds simulation prevention bytes to the NAL unit payload if the start code would otherwise appear. In order to enable direct gateway operation between packet-oriented and stream-oriented systems, start code simulation prevention can always be performed regardless of whether the byte stream format is used. The NAL unit can be defined as a grammatical structure containing an indication of the data type to be followed and bytes containing data in the form of RBSP, interspersed with simulation prevention bytes if necessary. A Raw Byte Sequence Payload (RBSP) may be defined as a syntax structure containing an integer number of bytes encapsulated in a NAL unit. An RBSP is either empty or has the form of a data bit string containing a syntax element, followed by an RBSP stop bit, and followed by zero or more subsequent bits equal to 0.

[0091] The NAL unit consists of a header and a payload. In H.264 / AVC and HEVC, the NAL unit header indicates the type of the NAL unit.

[0092] In HEVC, a two-byte NAL unit header is used for all specified NAL unit types. The NAL unit header contains one reserved bit, a six-bit NAL unit type indication, a three-bit nuh_temporal_id_plus1 indication for the temporal level (which may need to be greater than or equal to 1), and a six-bit nuh_layer_id syntax element. The temporal_id_plus1 syntax element can be regarded as a temporal identifier for the NAL unit, and the zero-based TemporalId variable can be derived as follows: TemporalId = temporary_id_plus1–1. The abbreviation TID can be used interchangeably with the TemporalId variable. TemporalId equal to 0 corresponds to the lowest temporal level. The value of temporal_id_plus1 needs to be non-zero to avoid start code simulation involving two NAL unit header bytes. The bitstream created by excluding all VCL NAL units with TemporalId greater than or equal to the selected value and including all other VCL NAL units still conforms to the requirements. Therefore, a picture with TemporalId equal to tid_value does not use any picture with TemporalId greater than tid_value as inter prediction reference. A sublayer or temporal sublayer may be defined as a temporal scalable layer (or temporal layer TL) of a temporal scalable bitstream, consisting of VCL NAL units with a specific value of the TemporalId variable and associated non-VCL NAL units. nuh_layer_id may be understood as a scalable layer identifier.

[0093] NAL units can be classified into video coding layer (VCL) NAL units and non-VCL NAL units. VCL NAL units are usually coded slice NAL units. In HEVC, VCL NAL units contain syntax elements representing one or more CUs.

[0094] A non-VCL NAL unit may be, for example, one of the following types: a sequence parameter set, a picture parameter set, a supplemental enhancement information (SEI) NAL unit, an access unit delimiter, an end-of-sequence NAL unit, an end-of-bitstream NAL unit, or a filler data NAL unit. A parameter set may be necessary for reconstruction of a decoded picture, while many other non-VCL NAL units are not necessary for reconstruction of decoded sample values.

[0095] Parameters that remain unchanged in a coded video sequence may be included in a sequence parameter set. In addition to parameters that may be required by the decoding process, a sequence parameter set may optionally contain video usability information (VUI), which includes parameters that may be important for buffering, picture output timing, rendering, and resource reservation. In HEVC, a sequence parameter set RBSP includes parameters that may be referenced by one or more picture parameter set RBSPs or one or more SEI NAL units containing a cache period SEI message. A picture parameter set contains parameters that may remain unchanged in several coded pictures. A picture parameter set RBSP may include parameters that may be referenced by a coded slice NAL unit of one or more coded pictures.

[0096] In HEVC, a video parameter set (VPS) can be defined as a syntax structure containing syntax elements applicable to zero or many complete coded video sequences, which are determined by the contents of syntax elements found in the SPS, which are referenced by syntax elements found in each slice segment header.

[0097] A video parameter set RBSP may include parameters that may be referenced by one or more sequence parameter set RBSPs.

[0098] The relationship and hierarchy between video parameter sets (VPS), sequence parameter sets (SPS) and picture parameter sets (PPS) can be described as follows. The VPS is located at a level above the SPS in the parameter set hierarchy and in the context of scalability and / or 3D video. The VPS may include parameters common to all slices in all (scalability or view) layers in the entire coded video sequence. The SPS includes parameters common to all slices in a specific (scalability or view) layer in the entire coded video sequence and may be shared by multiple (scalability or view) layers. The PPS includes parameters common to all slices in a specific layer representation (a representation of a scalability or view layer in an access unit) and may be shared by all slices in multiple layer representations.

[0099] The VPS may provide information about dependencies of layers in the bitstream, as well as a lot of other information applicable to all slices in all (scalability or view) layers in the entire coded video sequence. The VPS may be considered to consist of two parts, a base VPS and a VPS extension, where the VPS extension may optionally be present.

[0100] Out-of-band transmission, signaling or storage may additionally or alternatively be used for purposes other than tolerance of transmission errors, such as ease of access or session negotiation. For example, a sample entry for a track in a file conforming to the ISO base media file format may include a parameter set, while the coded data in the bitstream is stored elsewhere in the file or in another file. The phrase "along the bitstream" (e.g., indicating along the bitstream) or "along a coding unit of a bitstream" (e.g., indicating along a coding tile) may be used in the claims and the described embodiments to refer to out-of-band transmission, signaling or storage in a manner in which the out-of-band data is correspondingly associated with the bitstream or coding unit. The phrase "decoding along the bitstream or along a coding unit of a bitstream" or similar phrases may refer to correspondingly decoding the referred out-of-band data associated with the bitstream or coding unit (which may be obtained from out-of-band transmission, signaling or storage).

[0101] A SEI NAL unit may contain one or more SEI messages that are not required for decoding of output pictures but may facilitate related processes such as picture output timing, rendering, error detection, error concealment, and resource conservation.

[0102] A coded picture is an encoded representation of a picture.

[0103] In HEVC, a coded picture can be defined as a coded representation of a picture containing all coding tree units of the picture. In HEVC, an access unit (AU) can be defined as a set of NAL units that are associated with each other according to a specified classification rule, are consecutive in decoding order, and contain at most one picture with any particular nuh_layer_id value. In addition to VCL NAL units containing coded pictures, an access unit may also contain non-VCL NAL units. The specified classification rule may, for example, associate pictures with the same output time or picture output count value into the same access unit.

[0104] A bitstream can be defined as a sequence of bits in the form of a stream of NAL units or a stream of bytes, which forms a representation of a coded image and associated data, thereby forming one or more coded video sequences. A first bitstream may be followed by a second bitstream in the same logical channel, such as in the same file or in the same connection of a communication protocol. An elementary stream (in the context of video coding) can be defined as a sequence of one or more bitstreams. The end of the first bitstream can be indicated by a specific NAL unit, which can be called the end of bitstream (EOB) NAL unit and which is the last NAL unit of the bitstream. In HEVC and its current draft extensions, the EOB NAL unit is required to have nuh_layer_id equal to 0.

[0105] In H.264 / AVC, a coded video sequence is defined as a sequence of consecutive access units in decoding order from an IDR access unit (inclusive) to the next IDR access unit (exclusive) or to the end of the bitstream, whichever occurs first.

[0106] In HEVC, a coded video sequence (CVS) may be defined, for example, as a sequence of access units that include, in decoding order, an IRAP access unit with NoRaslOutputFlag equal to 1, followed by zero or more access units with NoRaslOutputFlag equal to 1 that are not IRAP access units, including all subsequent access units up to but not including any subsequent access units that are IRAP access units with NoRaslOutputFlag equal to 1. An IRAP access unit may be defined as an access unit in which a base layer picture is an IRAP picture. For each IDR picture, each BLA picture, and each IRAP picture (which is the first picture in that particular layer in the bitstream in decoding order), the value of NoRaslOutputFlag is equal to 1, which is the first IRAP picture in decoding order after the ending NAL unit of the sequence with the same nuh_layer_id value. There may be methods for providing the HandleCraAsBlaFlag value to the decoder from an external entity (such as a player or receiver) that can control the decoder. For example, when a player seeks a new position in the bitstream or listens to the radio and starts decoding, and then starts decoding from a CRA picture, HandleCraAsBlaFlag can be set to 1. When HandleCraAsBlaFlag is equal to 1 for a CRA picture, the CRA picture will be processed and decoded like a BLA picture.

[0107] In HEVC, a coded video sequence may additionally or alternatively (to the above specification) be designated as ending when a specific NAL unit, which may be referred to as an End of Sequence (EOS) NAL unit, appears in the bitstream and has nuh_layer_id equal to 0.

[0108] A group of pictures (GOP) and its characteristics can be defined as follows. A GOP can be decoded regardless of whether any previous pictures are decoded. An open GOP is a group of pictures where, when decoding starts from the initial intra picture of an open GOP, pictures preceding the initial intra picture in the output order may not be correctly decoded. In other words, pictures of an open GOP may reference pictures belonging to the previous GOP (in inter prediction). The HEVC decoder can identify intra pictures starting from an open GOP because a specific NAL unit type CRA NAL unit type can be used for its coded fragments. A closed GOP is a group of pictures where all pictures can be correctly decoded when decoding starts from the initial intra picture of a closed GOP. In other words, no picture in a closed GOP references any picture in the previous GOP. In H.264 / AVC and HEVC, a closed GOP can start with an IDR picture. In HEVC, a closed GOP can also start with a BLA_W_RADL or BLA_N_LP picture. Compared to the closed GOP coding structure, the open GOP coding structure may be more efficient in compression because there is more flexibility in selecting reference pictures.

[0109] The decoded picture buffer (DPB) can be used for the encoder and / or decoder. There are two reasons for caching decoded pictures, for reference in inter-frame prediction and for reordering decoded pictures to output order. Since H.264 / AVC and HEVC provide great flexibility for both reference picture marking and output reordering, separate buffers for reference picture caching and output picture caching may waste memory resources. Therefore, the DPB can include a unified decoded picture caching process for reference pictures and output reordering. When a decoded picture is no longer used as a reference and does not need to be output, the decoded picture can be removed from the DPB.

[0110] In many coding modes of H.264 / AVC and HEVC, reference pictures used for inter prediction are indicated using indices to reference pairs of picture lists. The index can be encoded using variable length coding, which typically results in smaller indices having shorter values ​​for the corresponding syntax elements. In H.264 / AVC and HEVC, two reference picture lists (reference picture list 0 and reference picture list 1) are generated for each bidirectionally predicted (B) slice, and one reference picture list (reference picture list 0) is formed for each inter-coded (P) slice.

[0111] Many coding standards, including H.264 / AVC and HEVC, may have a decoding process to derive a reference picture index to a reference picture list, which can be used to indicate which of multiple reference pictures is used for inter prediction of a particular block. The reference picture index may be encoded into the bitstream by the encoder in certain inter coding modes or derived (by the encoder and decoder) using, for example, neighboring blocks in certain other inter coding modes.

[0112] The motion parameter type or motion information may include but is not limited to one or more of the following types:

[0113] - an indication of the prediction type (e.g., intra prediction, uni-prediction, bi-prediction) and / or the number of reference pictures;

[0114] - indication of prediction direction, such as inter (aka temporal) prediction, inter-layer prediction, inter-view prediction, view synthesis prediction (VSP) and inter-component prediction (which may be indicated per reference picture and / or per prediction type, where in some embodiments inter-view and view synthesis prediction may be jointly considered as one prediction direction) and / or

[0115] - an indication of the reference picture type, such as a short-term reference picture and / or a long-term reference picture and / or an inter-layer reference picture (may be indicated by, for example, a reference picture)

[0116] - a reference index to a reference picture list and / or any other identifier of a reference picture (which may, for example, be indicated per reference picture and its type may depend on the prediction direction and / or the reference picture type and it may be accompanied by other relevant information such as the reference picture list to which the reference index applies, etc.);

[0117] - horizontal motion vector component (which may be indicated, for example, by prediction block or by reference index, etc.);

[0118] - vertical motion vector component (which may be indicated, for example, by prediction block or by reference index, etc.);

[0119] - one or more parameters, such as a picture order count difference and / or a relative camera separation between a picture and its reference pictures comprising or associated with motion parameters, which may be used for scaling of horizontal motion vector components and / or vertical motion vector components in one or more motion vector prediction processes (wherein the one or more parameters may be indicated, for example, per reference picture or per reference index, etc.);

[0120] - the coordinates of the block to which the motion parameters and / or motion information apply, e.g. the coordinates of the top left sample of the block in units of luma samples;

[0121] - The extent (e.g., width and height) of the block to which the motion parameters and / or motion information apply. Compared to previous video coding standards, the Versatile Video Codec (H.266 / VVC) introduces a variety of new coding tools, such as the following:

[0122] Intra-frame prediction

[0123] – 67 Intra-frame modes with wide-angle mode expansion

[0124] – Block size and mode dependent 4-tap interpolation filter

[0125] – Position-dependent intra prediction combination (PDPC)

[0126] – Cross-Component Linear Model Intra Prediction (CCLM)

[0127] –Multiple reference line intra prediction

[0128] – Intra-frame sub-partitioning

[0129] – Weighted intra prediction with matrix multiplication

[0130] Inter-picture prediction

[0131] – Block motion replication with spatial, temporal, history-based and pairwise average merge candidates

[0132] – Affine motion inter-frame prediction

[0133] – Sub-block based temporal motion vector prediction

[0134] – Adaptive motion vector resolution

[0135] – 8x8 block-based motion compression for temporal motion prediction

[0136] – High-precision (1 / 16 pixel) motion vector storage and motion compensation with 8-tap interpolation filters for luma components and 4-tap interpolation filters for chroma components

[0137] – Triangular Partition

[0138] – Combined intra and inter prediction

[0139] – Merged with MVD (MMVD)

[0140] – Symmetrical MVD encoding

[0141] – Bidirectional optical flow

[0142] – Decoder side motion vector refinement

[0143] – Bidirectional prediction with CU level weights

[0144] Transform, quantization and coefficient coding

[0145] – Multiple major transform choices with DCT2, DST7 and DCT8

[0146] – Secondary transform for low frequency region

[0147] – Sub-block transform for inter-frame prediction residual

[0148] – Dependent quantization with max QP increased from 51 to 63

[0149] – Transform coefficient coding with sign data hiding

[0150] – Transform skip residual coding

[0151] Entropy coding

[0152] – Arithmetic coding engine with adaptive dual-window probability update

[0153] Loop filter

[0154] – Loop Reshaping

[0155] – Deblocking filter with strong and longer filters

[0156] – Sample Adaptive Offset

[0157] – Adaptive loop filter

[0158] Screen content encoding:

[0159] – Current picture reference with reference area restrictions

[0160] 360-degree video encoding

[0161] – Horizontal surround motion compensation

[0162] High-level syntax and parallel processing

[0163] – Reference picture management with direct reference picture list signaling

[0164] – Tile groups with rectangular tile groups

[0165] The partitioning in VVC is similar to that in HEVC, that is, each picture is divided into coding tree units (CTUs). Pictures can also be divided into slices, tiles, blocks, and sub-pictures. CTUs can be divided into smaller CUs using a quadtree structure. Each CU can be divided using quadtrees and nested multi-type trees (including ternary trees and binary trees). However, there are specific rules for inferring partitions in picture boundaries, and redundant partitioning patterns are not allowed in nested multi-type partitions.

[0166] Among the new coding tools listed above, the cross-component linear model (CCLM) prediction mode is used in VVC to reduce cross-component redundancy. Among them, chroma samples are predicted based on the reconstructed luma samples of the same CU using a linear model, as follows:

[0167] pred C (i, j) = α·rec L ′(i, j)+β (Equation 1a)

[0168] where pred C (i, j) represents the predicted chroma sample in the CU and rec L ′(i, j) represents the downsampled reconstructed luma samples of the same CU.

[0169] Alternatively, the following equation can be used for CCLM:

[0170]

[0171] The >> operation represents a right shift of value k.

[0172] The CCLM parameters (α and β) are derived from up to four adjacent chroma samples and their corresponding downsampled luma samples. Assuming the current chroma block size is W×H, W' and H' are set to

[0173] - When LM mode is applied, W'=W, H'=H;

[0174] - When LM-A mode is applied, W'=W+H;

[0175] - When LM-L mode is applied, H'=H+W;

[0176] Among them, LM-A mode refers to linear model_above, in which only the upper template (i.e., the sample values ​​from the adjacent position above the CU) is used to calculate the linear model coefficients. In order to obtain more samples, the upper template is expanded to (W+H). LM-L mode in turn refers to linear model_left, in which only the left template (i.e., the sample values ​​from the adjacent position on the left side of the CU) is used to calculate the linear model coefficients. In order to obtain more samples, the left template is expanded to (H+W). For non-square blocks, the above template is expanded to W+W and the left template is expanded to H+H.

[0177] The upper adjacent position is represented as S[0,-1]...S[W'-1,-1], and the left adjacent position is represented as S[-1,0]...S[-1,H'-1]. Then, four samples are selected as

[0178] - when LM mode is applied and both the upper and left neighboring samples are available, S[W' / 4, -1], S[3*W' / 4, -1], S[-1, H' / 4], S[-1, 3*H' / 4];

[0179] - when LM-A mode is applied or only upper adjacent samples are available, S[W' / 8,-1], S[3*W' / 8,-1], S[5*W' / 8,-1], S[7*W' / 8,-1];

[0180] - when LM-L mode is used or only left-neighboring samples are available, S[-1,H' / 8], S[-1,3*H' / 8], S[-1,5*H' / 8], S[-1,7*H' / 8];

[0181] The four adjacent luma samples at the selected position are downsampled and compared four times to find two smaller values: x0A and x1A, and two larger values: x0B and x1B. Their corresponding chroma sample values ​​are denoted as y0A, y1A, y0B, and y1B. Then xA, xB, yA, and yB are derived as follows:

[0182] X a =(x 0 A +x 1 A +1)>>1;X b =(x 0 B +x 1 B +1)>>1;Y a =(y 0 A +y 1 A +1)>>1;Y b =(y 0 B +y 1 B +1)>>1(Equation 2)

[0183] Finally, the linear model parameters α and β are obtained according to the following equations:

[0184]

[0185] β=Y b -a·X b (Equation 4)

[0186] Figure 5 An example of the positions of the left and upper samples and samples of the current block involved in CCLM mode is shown.

[0187] The division operation to calculate the parameter α is implemented by a lookup table. In order to reduce the memory required to store the table, the diff value (the difference between the maximum and minimum values) and the parameter α are represented by exponential notation. For example, diff is approximated with a 4-bit significant part and an exponent. Therefore, the table for 1 / diff is reduced to 16 elements for 16 significant part values, as follows:

[0188] DivTable[] = {0,7,6,5,5,4,4,3,3,2,2,1,1,1,1,0} (Equation 5)

[0189] This reduces both the computational complexity and the memory size required to store the required tables.

[0190] To match the chroma sample positions for 4:2:0 video sequences, two types of downsampling filters are applied to luma samples to achieve a 2 to 1 downsampling ratio in both horizontal and vertical directions. The choice of downsampling filter is specified by the SPS level flag. The two downsampling filters are as follows, corresponding to "type 0" and "type 2" content, respectively:

[0191]

[0192] It is worth noting that when the upper reference line is at a CTU boundary, only one luma line (common line buffer in intra prediction) is used to make the downsampled luma samples.

[0193] This parameter calculation is performed as part of the decoding process, not just as an encoder search operation. Therefore, no syntax is used to communicate the α and β values ​​to the decoder.

[0194] For chroma intra mode coding, a total of 8 intra modes are allowed for chroma intra mode coding. Those modes include five traditional intra modes and three cross-component linear model modes (CCLM, LM_A, and LM_L). The chroma mode signaling and derivation process are shown in Table 1. Chroma mode coding depends directly on the intra prediction mode of the corresponding luminance block. Since the separate block partition structure for luminance and chrominance components is enabled in I slices, one chroma block may correspond to multiple luminance blocks. Therefore, for the chroma DM mode, the intra prediction mode of the corresponding luminance block covering the center position of the current chroma block is directly inherited.

[0195]

[0196] Table 1.

[0197] Regardless of the value of sps_cclm_enabled_flag, a single binarization table is used, as shown in Table 2.

[0198] Value of intra_chroma_pred_mode Binary String 4 00 0 0100 1 0101 2 0110 3 0111 5 10 6 110 7 111

[0199] Table 2.

[0200] In Table 2, the first bin indicates whether it is normal (0) or LM mode (1). If it is LM mode, the next bin indicates whether it is LM_CHROMA (0). If it is not LM_CHROMA, the next 1 bin indicates whether it is LM_L (0) or LM_A (1). For this case, when sps_cclm_enabled_flag is 0, the first bin of the bin table corresponding to intra_chroma_pred_mode can be discarded before entropy coding. Or in other words, the first bin is inferred to be 0 and therefore not encoded. This single bin table is used for the cases where sps_cclm_enabled_flag is equal to 0 and 1. The first two bins in Table 2 are context coded using their own context model, and the remaining bins are bypass coded.

[0201] In addition, to reduce luma-chroma latency in dual trees, when a 64x64 luma CT node is partitioned using no split (and intra sub-partitioning (ISP) is not used for 64x64 CU) or QT, the chroma CU in a 32x32 / 32x16 chroma CT node can be allowed to use CCLM in the following manner:

[0202] - If a 32x32 chroma node is not split or partitioned by QT, all chroma CUs in the 32x32 node can use CCLM

[0203] - If a 32x32 chroma node is partitioned using horizontal BT, and the 32x16 child node is not partitioned or uses vertical BT partitioning, all chroma CUs in the 32x16 chroma node can use CCLM.

[0204] Under all other luma and chroma coding tree partitioning conditions, CCLM is not allowed for chroma CUs.

[0205] Multi-model LM (MMLM)

[0206] The CCLM included in VVC is extended by adding three multi-model LM (MMLM) modes. In each MMLM mode, the reconstructed neighboring samples are classified into two classes using a threshold value, which is the average value of the neighboring samples of the brightness reconstruction. The linear model of each class is derived using the least mean square (LMS) method. For the CCLM mode, the LMS method is also used to derive the linear model. Figure 6a and 6bTwo luminance-to-chrominance models obtained for a luminance (Y) threshold 17 in the sample domain and the spatial domain are shown accordingly. Each luminance-to-chrominance model has its own linear model parameters α and β. Figure 6b As can be seen in , each luminance-to-chrominance model corresponds to a spatial segmentation of the content (i.e., they correspond to different objects or textures in the scene).

[0207] Multiple Reference Line (MRL) Intra Prediction

[0208] Multiple reference line (MRL) intra prediction uses more reference lines for intra prediction. Figure 7 In , an example of 4 reference lines is depicted, where the samples of segments A and F are not taken from the reconstructed neighboring samples, but are filled with the closest samples from segments B and E, respectively. HEVC intra picture prediction uses the nearest reference line (i.e., reference line 0). In MRL, 2 additional lines (reference line 1 and reference line 3) are used.

[0209] The index of the selected reference line (mrl_idx) is signaled and used to generate the intra predictor. For reference line idx greater than 0, only additional reference line modes are included in the MPM list, and only the mpm index is signaled without signaling the remaining modes. The reference line index is signaled before the intra prediction mode, and in case a non-zero reference line index is signaled, the planar mode is excluded from the intra prediction mode.

[0210] MRL is disabled for the first row of blocks within a CTU to prevent the use of extended reference samples outside the current CTU row. In addition, PDPC is disabled when additional rows are used. For MRL mode, the derivation of the DC value in the DC intra prediction mode for non-zero reference row index is consistent with the derivation of reference row index 0. MRL requires storage of 3 adjacent luma reference rows with a CTU to generate the prediction. The CCLM tool also requires 3 adjacent luma reference rows for its downsampling filter. MLR uses the same 3 line definition as CCLM to reduce the storage requirements of the decoder.

[0211] Intra-frame sub-partitioning (ISP)

[0212] Intra subpartitioning (ISP) divides the luma intra prediction block into 2 or 4 subpartitions vertically or horizontally, depending on the block size. For example, the minimum block size for ISP is 4x8 (or 8x4). If the block size is larger than 4x8 (or 8x4), the corresponding block will be divided by 4 subpartitions. It has been noted that M×128 (M≤64) and 128×N (N≤64) ISP blocks may cause potential problems for 64×64 VDPUs. For example, an M×128 CU in a single tree case has an M×128 luma TB and two corresponding M / 2×64 chroma TBs. If the CU uses ISP, the luma TB will be divided into four M×32 TBs (only horizontal partitioning is possible), each of which is smaller than a 64×64 block. However, in the current ISP design, the chroma blocks are not divided. Therefore, the two chroma components will have a size larger than a 32×32 block. Similarly, a similar situation will occur for a 128×N CU using ISP. Therefore, both cases are a problem for a 64×64 decoder pipeline. Therefore, the CU size that can use ISP is limited to a maximum of 64×64. All sub-partitions meet the condition of having at least 16 samples.

[0213] Matrix Weighted Intra Prediction (MIP)

[0214] The matrix weighted intra prediction (MIP) method is a newly added intra prediction technique in VVC. To predict the samples of a rectangular block of width W and height H, the matrix weighted intra prediction (MIP) takes as input a row of H reconstructed adjacent boundary samples on the left side of the block and a row of W reconstructed adjacent boundary samples above the block. If the reconstructed samples are not available, they are generated as in traditional intra prediction. The generation of the prediction signal is based on the following three steps, which are averaging, matrix-vector multiplication, and linear interpolation, as Figure 8 shown.

[0215] Inter-frame prediction in VVC

[0216] The merge list may include the following candidates:

[0217] a) Spatial MVP from spatially neighboring CUs b) Temporal MVP from co-located CUs c) History-based MVP from FIFO table d) Pairwise average MVP (using candidates already in the list) e) Zero MV.

[0218] Merge mode wide motion vector difference (MMVD) is to signal the MVD and resolution index after signaling the merge candidate. In symmetric MVD, the motion information of list 1 is derived from the motion information of list 0 in case of bi-prediction.

[0219] In affine prediction, several motion vectors are indicated / signaled for different corners of a block, which are used to derive the motion vector of the sub-block. In affine merging, the affine motion information of a block is generated based on the normal or affine motion information of neighboring blocks.

[0220] In sub-block based temporal motion vector prediction, the motion vectors of sub-blocks of the current block are predicted from appropriate sub-blocks in the reference frame, which are indicated by the motion vectors of spatially neighboring blocks (if available).

[0221] In Adaptive Motion Vector Resolution (AMVR), the precision of MVD is signaled for each CU.

[0222] In bidirectional prediction with CU level weights, the index indicates the weight value for the weighted average of the two prediction blocks. Bidirectional optical flow (BDOF) refines the motion vector in the case of bidirectional prediction. BDOF uses the motion vector sent with the signal to generate two prediction blocks. Then motion refinement is calculated to minimize the error between the two prediction blocks using their gradient values. The final prediction block is refined using motion refinement and gradient values.

[0223] Therefore, it can be concluded that the purpose of the linear model of the CCLM method used in VVC / H.266 is to find the correlation of samples between two or more chrominance channels (e.g., Cb and Cr) by deriving model parameters based on reconstructed samples in the neighborhood of the chrominance block, co-located adjacent samples in the luminance block, and reconstructed samples within the co-located luminance block.

[0224] However, the first-order linear model of CCLM is not sufficient to model spatial correlations between luma and chroma samples, such as texture or surface.

[0225] We now introduce an improved method for achieving cross-component prediction that is more suitable for modeling the spatial correlation between luma and chroma samples.

[0226] According to one aspect, a method Fig. 9 is shown in which the method comprises:

[0227] An image block unit of a frame is received (900), the image block unit comprising samples in color channels, the color channels comprising at least one chrominance channel and one luminance channel; samples of the luminance channel of the image block unit are reconstructed (902); a reference area for predicting samples of at least one color channel of the image block unit is determined (904), wherein the reference area comprises: one or more reference samples of reference samples in neighboring blocks in a current color channel / frame, adjacent to a co-located block in a reference color channel / frame, and / or within a co-located block in a reference color channel / frame; filter coefficients of a filter are determined (906) based on the reference samples and a morphology of the filter used for the prediction; and samples of at least one color channel of the image block unit are predicted (908) by performing a convolution between the samples of the luminance channel and the filter coefficients.

[0228] Therefore, the method uses the convolution between the filter kernel derived from the input data set and the decoder side to predict the chrominance samples. The form of the filter and the number of coefficients are agreed upon by the encoder and the decoder. The input data of the convolution can be specified in a variety of ways, and can include, for example, the configuration of the luminance sample co-located with the chrominance sample, the adjacent co-located luminance sample or any other luminance sample. For the filter coefficient, it is derived that the co-located reference sample area including the reconstructed luminance and chrominance samples is defined for both luminance and chrominance prediction, wherein the reference sample can contain any chrominance and luminance sample reconstructed by both the encoder and the decoder. Once the reference sample is determined, the filter coefficient is derived based on the form of the reference sample and the filter kernel. Finally, the predicted chrominance block can be calculated by convolving the luminance sample with the filter kernel.

[0229] Thus, the approach enables the use of 2D filter kernels, thereby enhancing the modeling of spatial correlations between luma and chroma samples, such as texture or surface.

[0230] According to one embodiment, the method comprises, before determining the filter coefficients, downsampling the samples of the luma channel to correspond to the size of the samples of the chroma samples.

[0231] Fig.10 An example of a co-located reference sample region comprising reconstructed luma and chroma samples defined for both luma and chroma is shown, wherein the commonly used 4:2:0 chroma downsampling has been applied. For the purpose of determining the filter coefficients for predicting the chroma samples, at least the corresponding reference luma samples may be downsampled to the size of the chroma samples. For example, the reference sample region for a given block may be as follows: Fig.10 The left and top six rows are shown in , but it should be noted that any number of reference lines that can be implemented by both the encoder and decoder can be used.

[0232] According to one embodiment, determining the filter coefficients includes: obtaining statistical data for linear regression from reference samples based on the morphology of the filter; and obtaining the filter coefficients by applying a linear regression method to the obtained statistical data.

[0233] Once the reference samples are determined, the filter coefficients can be derived by scanning the luminance and chrominance samples of the reference area, collecting available statistics from the reference samples based on the filter morphology, and solving for the filter coefficients by applying some linear regression method to the obtained statistics. Here, different types of linear regression tools (such as ordinary least squares estimation, orthogonal matching pursuit, optimized orthogonal matching pursuit, ridge regression, or least absolute shrinkage and selection operators) can be used.

[0234] According to one embodiment, the statistics include an autocorrelation matrix and a cross-correlation vector. Thus, based on the reference samples and the morphology of the filter kernel, the autocorrelation matrix and the cross-correlation vector can be defined, and then the filter coefficients can be solved, for example, by minimizing the squared error (or any other metric) based on the autocorrelation matrix and the cross-correlation vector.

[0235] According to one embodiment, the filter has the form of a one-dimensional (1D-) vector of filter coefficients or a two-dimensional (2D-) array of filter coefficients.

[0236] Fig.11 Various examples of filter kernel dimensions are shown, which can be, for example, 1x3 (1D vertical), 3x1 (1D horizontal), 3x3, 7x7, or any dimension. The filter kernel can be shaped as a cross or diamond, such as Fig.11 , or by selecting only a subset of all possible kernel positions to shape into any given shape. When referring to samples within the filter kernel, the following notation is used: North (top), East (right), South (bottom), West (left), and Center, as shown in Fig.11 As shown, the letters N, E, S, W, C are used.

[0237] We define the (possibly downsampled) luma samples as a 2D array Y(x,y) indexed using the horizontal x-coordinate and the vertical y-coordinate. We also define the co-located chroma samples as a 2D array C(x,y), and the filter kernel (i.e., coefficients) as a 3x3 array F(i,j). At the sample level, the convolution between Y and F can be defined as:

[0238]

[0239] According to one embodiment, the method comprises applying auxiliary parameters external to the morphology of the filter to said convolution.

[0240] Other parameters or data items external to the filter kernel may be applied as additional terms in the linear combination. These terms may be, for example, bias terms, spatial coordinates, recursively reconstructed chrominance samples, or nonlinear terms such as the square root of co-located luma samples or the quadratic power of co-located luma sample values. In general, the nonlinear terms may be any nonlinear function applied to any reconstructed samples or other available data. More than one nonlinear term may be used to be able to capture different kinds of nonlinear dependencies between luma and chroma sample values.

[0241] The introduction of parameters or data terms outside the filter kernel, especially non-linear terms, even further enhances the modeling of spatial correlations between luma and chroma samples.

[0242] When using the auxiliary parameters (such as the nonlinear square root term), the above convolution becomes:

[0243]

[0244] in are filter coefficients that are outside the 2D filter kernel, but they are obtained as part of the linear equation used to solve the 2D filter coefficients as described above.

[0245] Another example of using the auxiliary parameters is to add a bias term to a convolution:

[0246]

[0247] As a further example, luma samples squared may be used instead of or in addition to the square root term, as follows:

[0248]

[0249] F and Both (i.e. CCCM coefficients) are obtained using reconstructed samples near the chrominance block. However, in cases where the immediate area cannot describe the cross-component relationship between luma and chroma, CCCM coefficients derived from more distant samples may need to be considered.

[0250] According to one embodiment, the method comprises storing filter coefficients in a memory; and receiving an indication from an encoder to reuse at least a subset of previously stored filter coefficients for predicting samples of at least one color channel of a current image block.

[0251] Therefore, the encoder and decoder can store CCCM coefficients in memory, allowing all or some subset of CCCM coefficients obtained so far to be tested at the encoder. Once a suitable set of CCCM coefficients is found, the index identifying these coefficients is signaled to the decoder. This reuse of earlier CCCM coefficients can be referred to as history-based CCCM.

[0252] According to one embodiment, the method includes: determining samples outside a reference area; obtaining statistical data for linear regression from the samples outside the reference area based on the morphology of the filter; obtaining filter coefficients by applying a linear regression method to the obtained statistical data; and storing the filter coefficients in a memory.

[0253] To further increase the count of filter coefficient sets, the encoder and decoder can also include coefficients from non-CCCM blocks in memory by scanning the luminance and chrominance samples of the reference area, and collecting available statistics from the reference samples based on the filter morphology, and solving the filter coefficients by applying some linear regression method to the obtained statistics. This can be applied to all or some subset of non-CCCM chrominance blocks. The resulting CCCM coefficients are stored in memory.

[0254] Again, the autocorrelation matrix and cross-correlation vector can be defined based on the reference samples and the shape of the filter kernel, and then the filter coefficients can be solved, for example, by minimizing the squared error (or any other metric) based on the autocorrelation matrix and cross-correlation vector. In addition, before determining the filter coefficients, the samples outside the reference region can be downsampled to correspond to the sample size of the chroma samples.

[0255] To further improve the diversity of the obtained historical CCCM coefficients, the encoder and decoder can also perform the above steps of downsampling, obtaining statistics, and obtaining filter coefficients on the CCCM block. The obtained CCCM coefficients can be used to cover earlier CCCM coefficients or stored as additional CCCM coefficients.

[0256] By combining the locally derived CCCM coefficients with the history-based CCCM, a more refined chrominance prediction can be obtained. Let us call the history-based CCCM coefficients F h and The merging weight w is defined as 0<w<1.0, and the weighted CCCM coefficient becomes F w =wF h +(1-w)F and This weighting can also be applied after convolution of the reconstructed chroma samples if the historically based CCCM filter structure does not match the locally derived CCCM filter structure. The best w is searched at the encoder side and signaled to the decoder. Multiple historically based and / or locally derived CCCM coefficients can be obtained by searching and signaling multiple w i weights are combined.

[0257] When calculating the model parameters or filter coefficients F and When , different choices may be made depending on whether operating in single-tree mode (with three color components represented by a single coding unit) or in dual-tree mode (with separate coding units for luma and separate coding units for two chroma components).

[0258] In dual-tree mode, where luma samples are typically available to cover the spatial region of a complete coding tree unit consisting of multiple coding units, luma samples below (S) and to the right (E) of a chroma coding unit may be available. In this case, the luma samples may be advantageously used for both filter coefficient derivation and the actual sample prediction process, which maps luma sample values ​​to chroma sample values ​​using a convolution operation.

[0259] However, in the single-tree case, such luma samples are not available and may not be used by model creation or sample prediction operations. The luma samples of the current coding unit can be used for CCCM model creation, for example, if the south (S) sample below the center (C) sample is included in the filter kernel, then when collecting statistics for the boundary samples directly above the current coding unit, the south sample can be obtained from within the current coding unit. However, considering the implementation of parallelism, this may not be ideal because such an operation requires that the luma samples of the current coding unit are available when the CCCM model is generated for the chroma component; therefore, luma sample decoding is prevented from being performed in parallel with CCCM model creation. To avoid this, the use of luma samples within the current coding unit may be allowed for dual-tree operations, but not for single-tree operations. Alternatively, if the luma samples within the current coding unit are used for CCCM model creation, this may be signaled in the video bitstream to allow the decoder to configure itself to process luma and chroma in parallel or sequentially.

[0260] According to one embodiment, the method comprises replacing any missing downsampled luma samples with zero-valued samples.

[0261] During the CCCM model creation and CCCM sample prediction process, some of the possible downsampled luma samples required by the convolution kernel may not be available. For example, some samples may be outside the picture boundary or belong to an area that has not yet been decoded. In these cases, these samples may be generated in different ways. For example, the unavailable samples may be replaced by the closest available samples. This process is often called padding. During the CCCM model creation phase, depending on the method chosen for deriving the CCCM coefficients, it is sometimes beneficial to use zero-valued samples for unavailable samples instead of using traditional padding methods. This is especially beneficial when using methods based on autocorrelation matrices for coefficient estimation, because zero-valued inputs do not introduce updates to such matrices, thereby ignoring the potential negative impact of duplicating samples at the reference region boundaries.

[0262] There may be additional bitstream signaling to update the CCCM model derived from the decoded samples. For example, an update term to be added to the center luma sample position coefficient F(1,1) may be signaled, and the decoder may perform additional adjustments to another coefficient based on the decoded update term.

[0263] Different measures can be taken to balance the computational requirements of different types of coding units. For example, the number and position of reference samples used in the CCCM model creation process can be determined based on the size of the coding unit, the number and spatial position of coefficients in the CCCM model can be adjusted based on the size of the coding unit, or the number and spatial position of coefficients in the CCCM model can be adjusted based on the number of CCCM models required within a coding unit.

[0264] Similarly, different measures may be taken to balance the memory and memory bandwidth requirements of the CCCM process. For example, in some environments, it may be beneficial to limit the reference lines used above the coding tree unit. In this case, different numbers of reference lines may be used for CCCM model creation depending on whether the sample block to be predicted is located at or near the top boundary of the coding tree unit or far away from the boundary. As an example, the number of reference lines required above the coding tree unit can be limited to three, it can also be limited to two, and it can also be limited to one row of samples.

[0265] According to one embodiment, the method comprises: determining samples that do not belong to any object or texture within the image block unit; and removing the samples from a reference area.

[0266] An outlier removal process can be applied to the reference area samples of luminance and chrominance to determine samples that do not belong to objects or textures within the block. Such a process can be completed based on determining certain statistics or characteristics from the co-located luminance samples within the block. For example, one or more of the luminance samples of the co-located block with minimum and maximum intensity values ​​can be used as a metric for removing reference area luminance and possibly corresponding chrominance samples from the CCCM parameter derivation process. Alternatively, the minimum and maximum values ​​can be used to clip reference area samples that exceed the intensity range of samples within the block. In another approach, clustering and / or classification methods can be used for the co-located luminance samples in the block to determine the characteristics of the samples, and thus reference area samples that match the characteristics of the determined co-located luminance samples are then used for CCCM parameter derivation.

[0267] Instead of one CCCM model, multiple CCCM models may be derived for a single block of chroma samples. In such cases, different kinds of conditions may be used to select the CCCM model that is applicable to each chroma sample. For example, if two CCCM models exist for a chroma block, the average luminance value of the reference samples may be used as a threshold, and if the co-located luminance sample is less than the threshold, the first CCCM model may be selected for predicting the chroma sample, otherwise the second CCCM model may be used for the chroma sample. In addition, different conditions may be used to enable a block to have multiple CCCM models, for example, multiple models may be allowed only if the chroma block has at least a specified number of samples or at least a specified number of reference samples are used for the block. This selection may be beneficial because small blocks may not benefit significantly from multiple CCCM models, but there are typically additional signaling and computational requirements associated with multiple models, making those benefits relatively small for smaller blocks compared to larger blocks.

[0268] In the following, embodiments are disclosed relating to various implementation options for a decoder to derive and / or apply various parameters related to a prediction model.

[0269] According to one embodiment, a set of chroma samples is predicted using a filtering operation that takes as input at least a luma sample co-located with the chroma sample, another luma sample at a different location, and a term derived from the luma sample or multiple luma samples using a non-linear operation; wherein the filtering operation outputs predicted chroma samples.

[0270] According to one embodiment, any number of CCCM filters may be applied when predicting chroma samples. For example, chroma samples may be divided into two or more classes, where each class uses its own CCCM coefficients, just like in multi-model CCLM.

[0271] According to one embodiment, the nonlinear term of the convolution is defined as a nonlinear function of any reconstructed or otherwise available data, such as reconstructed luma or chroma samples.

[0272] The non-linear function used in the non-linear term may be, for example, a square root or any exponential function, a sine function, a (linear or non-linear) combination of multiple non-linear functions or any arbitrary function that can be calculated at the decoder.

[0273] According to one embodiment, the final prediction of a block is achieved by combining two or more CCCM predictions with different non-linear terms. For example, a prediction can be achieved using a CCCM model without non-linear terms and one or more predictions with the same or different non-linear terms, where the final prediction is then calculated by a weighted average of the predictions. The weight values ​​for such operations can be fixed, can be signaled into the bitstream, or can be determined based on certain factors (such as block size, CCCM coefficients in different models, neighboring block prediction modes, etc.).

[0274] According to one embodiment, the decision to use nonlinear terms in the CCCM model is based on a texture analysis mechanism of reference samples and / or luma samples within a block. For example, the average or DC value of the reference area and co-located luma samples can be used as a factor for deciding to use nonlinear terms. If the average values ​​differ too much from each other (e.g., based on a certain threshold), the nonlinear terms are not used, and vice versa. In another example, the slope of the linear relationship between the reference luma and chroma samples is used to determine the use of nonlinear terms in the CCCM. In such a case, some reference luma and chroma samples are used for a linear model (e.g., linear regression); if the slope of the linear model is less than a threshold or greater than a threshold, the nonlinear terms are used in the CCCM process.

[0275] According to one embodiment, a binary choice of the use of the non-linear term is signaled by the encoder to the decoder.

[0276] According to one embodiment, the choice of the selected non-linear function is signaled by the encoder to the decoder at any given accuracy.

[0277] According to one embodiment, the shape, size, width, height or other data that determines the filter kernel selection is signaled by the encoder to the decoder using any given precision. The signaling can be done at different levels, such as CU, PU, ​​TU, CTU, slice, tile, sub-picture, picture or sequence level.

[0278] As described above, the filter kernel size can be 1D or 2D, ranging from 1x1 to any size that can be achieved at the decoder. In addition, the shape of the filter kernel can be, for example, rows or columns, crosses or diamonds, or any shape that can be achieved under the size of the filter kernel.

[0279] According to one embodiment, the filter morphology may cover only a subset of elements covered by the size of the filter kernel.

[0280] According to one embodiment, the shape and / or size of the filter kernel may be adaptively determined based on certain factors, such as block size, availability of reference samples, position of the block in a frame, coding modes of neighboring blocks, etc.

[0281] According to one embodiment, the filter consists of center (co-site), north (up), south (down), west (left), and east (right) brightness samples, a bias term derived from the center brightness sample, and a non-linear term.

[0282] According to one embodiment, the filter consists of center (co-site), south (bottom), west (left), and east (right) brightness samples, a bias term derived from the center brightness sample, and a non-linear term.

[0283] According to one embodiment, the filter consists of center (co-site), north (up), south (down), west (left), and east (right) brightness samples, a bias term, and the square root of the center brightness sample.

[0284] According to one embodiment, the filter consists of center (co-site), north (up), south (down), west (left), and east (right) brightness samples, a bias term, and the squared center brightness sample.

[0285] According to one embodiment, the filter consists of center (co-site), north (up), south (down), west (left), and east (right) luma samples, a bias term, and the squared center luma sample, which is scaled to have the same dynamic range as the luma samples.

[0286] According to one embodiment, a binary choice of usage of CCCM based on history is signaled by the encoder to the decoder.

[0287] According to one embodiment, the index (or position) of the selected history-based CCCM coefficient is signaled by the encoder to the decoder with any given precision.

[0288] According to one embodiment, a binary choice of use of CCCM based on weighted history is signaled by the encoder to the decoder.

[0289] According to one embodiment, the weights for the history-based CCCM are signaled by the encoder to the decoder using any given precision.

[0290] It should be noted that all the above options regarding the history-based CCCM (including the weighting variables) can be signaled by the encoder to the decoder separately for each chroma component.

[0291] According to one embodiment, the history-based CCCM coefficients are calculated (and / or updated) at the CU level or CTU level.

[0292] According to one embodiment, the correct CCCM coefficients from multiple coefficient sets of the history-based CCCM are determined at the decoder side. This can be done, for example, by using the available coefficients from the history and predicting one or more of the reference region samples and calculating the cost (e.g., distortion or difference) relative to the reconstructed reference samples. The coefficient set that results in the minimum cost is then selected and used to predict the samples within the block.

[0293] According to one embodiment, the historical weighted CCCM can also be determined based on a similar method, i.e., by predicting one or more of the reference sample regions with different historical CCCM coefficients and different weight values, and the (multiple) best performance coefficients and (multiple) weight sets that provide the minimum cost are used to predict the samples of the block.

[0294] According to one embodiment, fixed luma downsampling may be skipped and instead the full luma grid is used to derive CCCM coefficients, thereby obtaining an implicit downsampling filter in the process for the case of downsampled chroma.

[0295] According to one embodiment, a binary choice to skip fixed luma downsampling is signaled by the encoder to the decoder.

[0296] According to one embodiment, the luma samples used in the CCCM model creation are low pass filtered.

[0297] According to one embodiment, the CCCM coefficients include additional data items located outside the filter kernel, such as already reconstructed (ie causal) chrominance samples or spatial coordinates.

[0298] According to one embodiment, the choice of additional data items is signaled by the encoder to the decoder at any given precision.

[0299] According to one embodiment, recursive prediction is used to include the causal chrominance neighborhood into the prediction.

[0300] According to one embodiment, a binary choice to use recursive prediction is signaled by the encoder to the decoder.

[0301] According to one embodiment, the spatial coordinates of the reconstructed samples may be used as additional data items in the filter. This allows the chrominance samples to be modeled using a surface model, such as a planar or quadratic model.

[0302] According to one embodiment, a binary choice of the use of spatial coordinates for filtering is signaled by the encoder to the decoder.

[0303] According to one embodiment, the selected surface model is signaled by the encoder to the decoder at any given accuracy.

[0304] According to one embodiment, one or more CCCM modes are applied as filtering and / or adjustment operations to non-CCCM predictions. For example, CCCM filtering or adjustment can be applied to blocks predicted by angle prediction modes or existing CCLM modes. The parameters of the CCCM filter can be determined by using the re-predicted adjacent samples and the non-CCCM predictions used to predict the samples in the block. Then, the re-predicted adjacent samples and the reconstructed adjacent samples are used to derive CCCM filter parameters.

[0305] According to one embodiment, the final prediction of the block is achieved by combining one or more non-CCCM predictions (e.g., angular intra prediction CCLM) and one or more CCCM predictions. The weights of such different prediction combinations can be fixed or signaled in the bitstream, or can be determined based on the prediction modes of neighboring blocks and / or the prediction mode of the co-located luminance block.

[0306] According to one embodiment, the CCCM coefficients of the block are derived using reference samples above the block, reference samples to the left of the block, or reference samples above and to the left of the block. The selection of said reference side samples for parameter derivation may be signaled in the bitstream or may be decided at the decoder side, for example by using a texture analysis mechanism on the reference samples.

[0307] According to one embodiment, the shape, size, width, height or other data determining the selection of reference samples for CCCM model creation is signaled by the encoder to the decoder.

[0308] According to one embodiment, the type of nonlinear term in the CCCM model is determined based on a texture analysis mechanism of reference samples and / or luminance samples within a block.

[0309] According to one embodiment, the reference channel can be luma, chroma, R, G, B or any other color space format. For example, CCCM can use reconstructed chroma Cb samples to predict chroma Cr samples and vice versa.

[0310] According to one embodiment, the CCCM scheme may also be used to predict samples other than chroma samples. For example, instead of using luma and chroma samples, an embodiment may use intra and inter samples to further adjust inter samples to match intra neighbors or vice versa.

[0311] According to one aspect, an apparatus includes: a component for receiving an image block unit of a frame, the image block unit including samples in color channels including at least one chrominance channel and one luminance channel; a component for reconstructing samples of the luminance channel of the image block unit; a component for determining a reference area for predicting samples of at least one color channel of the image block unit, wherein the reference area includes: one or more reference samples of reference samples in neighboring blocks in a current color channel / frame, adjacent to a co-located block in a reference color channel / frame, and / or within a co-located block in a reference color channel / frame; a component for determining filter coefficients of a filter based on the reference samples and a morphology of the filter used for the prediction; and a component for predicting samples of at least one color channel of the image block unit by performing convolution between samples of the luminance channel and the filter coefficients.

[0312] According to one embodiment, the device comprises means for downsampling the samples of the luma channel to correspond to the size of the samples of the chroma samples before determining the filter coefficients.

[0313] According to one embodiment, the means for determining the filter coefficients comprises: means for obtaining statistical data for linear regression from reference samples based on the morphology of the filter; and means for obtaining the filter coefficients by applying a linear regression method to the obtained statistical data.

[0314] According to one embodiment, the statistical data include an autocorrelation matrix and a cross-correlation vector.

[0315] According to one embodiment, the filter has the form of a one-dimensional (1D-) vector of filter coefficients or a two-dimensional (2D-) array of filter coefficients.

[0316] According to one embodiment, the device comprises means for applying auxiliary parameters external to the morphology of the filter to said convolution.

[0317] According to one embodiment, the apparatus comprises means for inputting the following items to the filter: a luma sample co-located with the output chroma sample, at least one luma sample above the co-located sample, at least one luma sample below the co-located sample, at least one luma sample to the left of the co-located sample, at least one luma sample to the right of the co-located sample, and a bias term.

[0318] According to one embodiment, the apparatus comprises means for inputting to the filter a value determined by applying a non-linear function to the values ​​of the co-located samples.

[0319] According to one embodiment, the apparatus comprises: means for storing filter coefficients in a memory; and means for receiving an indication from an encoder to reuse at least a subset of previously stored filter coefficients to predict samples of at least one color channel of a current image block unit.

[0320] According to one embodiment, the device includes: a component for determining samples outside a reference area; a component for obtaining statistical data for linear regression from samples outside the reference area based on the morphology of a filter; a component for obtaining filter coefficients by applying a linear regression method to the obtained statistical data; and a component for storing the filter coefficients in a memory.

[0321] According to one embodiment, the apparatus comprises means for replacing any missing downsampled luma samples with zero-valued samples.

[0322] According to one embodiment, the apparatus comprises: means for determining samples that do not belong to any object or texture within the image block unit; and means for removing the samples from a reference area.

[0323] As another aspect, a device is provided, comprising: at least one processor and at least one memory, the at least one memory having code stored thereon, which, when executed by the at least one processor, causes the device to at least perform: receiving an image block unit of a frame, the image block unit comprising samples in a color channel, the color channel comprising at least one chrominance channel and one luminance channel; reconstructing samples of the luminance channel of the image block unit; determining a reference area for predicting samples of at least one color channel of the image block unit, wherein the reference area comprises: one or more reference samples of reference samples in neighboring blocks in a current color channel / frame, adjacent to a co-located block in a reference color channel / frame, and / or within a co-located block in a reference color channel / frame; determining filter coefficients of a filter based on the reference samples and a morphology of the filter used for the prediction; and predicting samples of at least one color channel of the image block unit by performing convolution between samples of the luminance channel and the filter coefficients.

[0324] Such means may include, for example Figure 1 , 2 , 4a and 4b for implementing the functional units disclosed in any one of the embodiments.

[0325] Such an apparatus also includes code stored in the at least one memory, which, when executed by the at least one processor, causes the apparatus to perform one or more embodiments disclosed herein.

[0326] Fig.1215 is a graphical representation of an example multimedia communication system in which various embodiments can be implemented. A data source 1510 provides a source signal in an analog, uncompressed digital or compressed digital format or any combination of these formats. An encoder 1520 may include or be connected to preprocessing, such as data format conversion and / or filtering of the source signal. Encoder 1520 encodes the source signal into a coded media bitstream. It should be noted that the bitstream to be decoded can be received directly or indirectly from a remote device located in almost any type of network. In addition, the bitstream can be received from local hardware or software. Encoder 1520 may be able to encode more than one media type, such as audio and video, or may require more than one encoder 1520 to encode different media types of source signals. Encoder 1520 may also obtain synthetically generated inputs, such as graphics and text, or it may be able to produce a coded bitstream of synthetic media. In the following, a coded media bitstream that only processes one media type is considered to be a simplified description. However, it should be noted that a real-time broadcast service generally includes multiple streams (generally at least one audio, video and text subtitle stream). It should also be noted that the system may include many encoders, but only one encoder 1520 is represented in the figure to simplify the description without lack of generality. It should also be understood that although the text and examples contained herein may specifically describe the encoding process, those skilled in the art will understand that the same concepts and principles also apply to the corresponding decoding process, and vice versa.

[0327] The coded media bitstream may be transmitted to the memory 1530. The memory 1530 may include any type of mass storage to store the coded media bitstream. The format of the coded media bitstream in the memory 1530 may be a basic self-contained bitstream format, or one or more coded media bitstreams may be encapsulated into a container file, or the coded media bitstream may be encapsulated into a segment format suitable for DASH (or a similar streaming system) and stored as a segment sequence. If one or more media bitstreams are encapsulated in a container file, a file generator (not shown in the figure) may be used to store one or more media bitstreams in a file and create file format metadata, which may also be stored in the file. The encoder 1520 or the memory 1530 may include a file generator, or the file generator may be operably connected to the encoder 1520 or the memory 1530. Some systems operate in "real time", i.e., omitting storage and transmitting the coded media bitstream directly from the encoder 1520 to the transmitter 1540. The coded media bitstream may then be transmitted to the transmitter 1540 (also referred to as a server) as needed. The format used in the transmission can be an elementary self-contained bitstream format, a packet stream format, a Segment format suitable for DASH (or similar streaming systems), or one or more coded media bitstreams can be encapsulated into a container file. The encoder 1520, storage 1530, and server 1540 can reside in the same physical device or can be included in separate devices. The encoder 1520 and server 1540 can operate live real-time content, in which case the coded media bitstreams are typically not stored permanently, but are cached for a short period of time in the content encoder 1520 and / or server 1540 to smooth out processing delays, transmission delays, and variations in the coded media bitrate.

[0328] The server 1540 sends the encoded media stream using a communication protocol stack. The protocol stack may include, but is not limited to, one or more of a real-time transport protocol (RTP), a user datagram protocol (UDP), a hypertext transfer protocol (HTTP), a transmission control protocol (TCP), and an Internet protocol (IP). When the communication protocol stack is a packet-oriented protocol stack, the server 1540 encapsulates the encoded media stream into packets. For example, when RTP is used, the server 1540 encapsulates the encoded media stream into RTP packets according to the RTP payload format. Typically, each media type has a dedicated RTP payload format. It should be noted again that the system may include more than one server 1540, but for simplicity, the following description only considers one server 1540.

[0329] If the media content is encapsulated in a container file for storage 1530 or for inputting data to a transmitter 1540, the transmitter 1540 may include or be operably attached to a "send file parser" (not shown in the figure). Specifically, if the container file is not sent in this way, but at least one of the contained coded media bitstreams is encapsulated for transmission via a communication protocol, the send file parser locates the appropriate portion of the coded media bitstream to be transmitted via the communication protocol. The send file parser can also help create the correct format for the communication protocol, such as packet headers and payloads. The multimedia container file may contain encapsulation instructions, such as a hint track in ISOBMFF, for encapsulating at least one of the contained media bitstreams on the communication protocol.

[0330] Server 1540 may or may not be connected to gateway 1550 via a communication network, which may be, for example, a CDN, the Internet, and / or a combination of one or more access networks. A gateway may also or alternatively be referred to as a middlebox. For DASH, a gateway may be an edge server (of a CDN) or a web proxy. It should be noted that the system may generally include any number of gateways, etc., but for simplicity, the following description only considers one gateway 1550. Gateway 1550 may perform different types of functions, such as converting a packet stream according to one communication protocol stack to another communication protocol stack, merging and forking data streams, and manipulating data streams according to downlink and / or receiver capabilities, such as controlling the bit rate of the forwarded stream according to current downlink network conditions. Gateway 1550 may be a server entity in various embodiments.

[0331] The system includes one or more receivers 1560, which are generally capable of receiving, demodulating and decapsulating the transmitted signal into a coded media bitstream. The coded media bitstream can be transmitted to a recording memory 1570. The recording memory 1570 can include any type of large capacity memory to store the coded media bitstream. The recording memory 1570 can alternatively or additionally include a computing memory, such as a random access memory. The format of the coded media bitstream in the recording memory 1570 can be a basic self-contained bitstream format, or one or more coded media bitstreams can be encapsulated into a container file. If there are multiple coded media bitstreams (such as audio streams and video streams) associated with each other, the container file is generally used, and the receiver 1560 includes or is attached to a container file generator, which generates a container file from the input stream. Some systems operate "real time", that is, the recording memory 1570 is omitted and the coded media bitstream is transmitted directly from the receiver 1560 to the decoder 1580. In some systems, only the most recent portion of the recorded stream (eg, the most recent 10 minute excerpt of the recorded stream) is retained in the recording storage 1570, while any earlier recorded data is discarded from the recording storage 1570.

[0332] The coded media bitstream may be transferred from the record storage 1570 to the decoder 1580. If there are many coded media bitstreams, such as audio and video streams, associated with each other and encapsulated in a container file, or a single media bitstream is encapsulated in a container file, for example, for ease of access, a file parser (not shown in the figure) is used to decapsulate each coded media bitstream from the container file. The record storage 1570 or the decoder 1580 may include the file parser, or the file parser is attached to the record storage 1570 or the decoder 1580. It should also be noted that the system may include many decoders, but only one decoder 1570 is discussed here to simplify the description without lack of generality.

[0333] The coded media bitstream may be further processed by a decoder 1570, the output of which is one or more uncompressed media streams. Finally, a renderer 1590 may reproduce the uncompressed media streams using, for example, a speaker or a display. The receiver 1560, the recording storage 1570, the decoder 1570, and the renderer 1590 may be located in the same physical device or may be included in separate devices.

[0334] The transmitter 1540 and / or the gateway 1550 may be configured to perform switching between different representations, for example, switching between different viewports of the 360-degree video content, view switching, bitrate adaptation and / or fast start, and / or the transmitter 1540 and / or the gateway 1550 may be configured to select (multiple) representations sent. The switching between different representations may be for a variety of reasons, such as in response to a request from the receiver 1560 or the current conditions (such as throughput) of the network through which the bitstream is transmitted. In other words, the receiver 1560 may initiate switching between representations. The request from the receiver may be, for example, a request for a segment or sub-segment from a different representation than before, a request for a change in the scalability layer and / or sub-layer sent, or a change in a rendering device with different capabilities than before. The request for the segment may be an HTTP GET request. The request for the sub-segment may be an HTTP GET request with a byte range. Additionally or alternatively, bitrate adjustment or bitrate adaptation may be used, for example, to provide so-called fast start in streaming services, where after starting or randomly accessing streaming media, the bitrate of the stream sent is lower than the channel bitrate in order to start playback immediately and achieve a buffer occupancy level that can tolerate occasional packet delays and / or retransmissions. Bitrate adaptation may include multiple representation or layer up-switching and representation or layer down-switching operations in various orders.

[0335] Decoder 1580 can be configured to perform switching between different representations, for example, for switching between different viewports of 360-degree video content, view switching, bit rate adaptation and / or fast start, and / or decoder 1580 can be configured to select (multiple) sent representations. Switching between different representations can occur for a variety of reasons, such as to achieve faster decoding operations or to adapt the sent bitstream (e.g., in terms of bitrate) to the current conditions (such as throughput) of the network through which the bitstream is transmitted. For example, if a device including decoder 1580 is multitasking and uses computing resources for purposes other than decoding video bitstreams, faster decoding operations may be required. In another example, when content is played at a speed faster than normal playback speed, faster decoding operations may be required, for example, two or three times faster than the traditional real-time playback rate.

[0336] In the above, some embodiments have been described with reference to and / or using the terminology of HEVC and / or VVC. It should be understood that the embodiments may similarly be implemented using any video encoder and / or video decoder.

[0337] In the above, where example embodiments have been described with reference to an encoder, it is to be understood that there may be corresponding elements in the resulting bitstream and decoder. Similarly, where example embodiments have been described with reference to a decoder, it is to be understood that the encoder may have a structure and / or computer program for generating a bitstream to be decoded by the decoder. For example, some embodiments have been described in relation to generating prediction blocks as part of encoding. Embodiments may be similarly implemented by generating prediction blocks as part of decoding, with the difference that encoding parameters (such as horizontal offset and vertical offset) are decoded from the bitstream rather than determined by the encoder.

[0338] The embodiments of the present invention described above describe the codec in the form of separate encoder and decoder devices to help understand the processes involved. However, it should be understood that the devices, structures and operations can be implemented as a single encoder-decoder device / structure / operation. In addition, the encoder and decoder can share some or all common elements.

[0339] Although the above examples describe embodiments of the present invention operating in a codec within an electronic device, it should be understood that the present invention defined in the claims can be implemented as part of any video codec. Thus, for example, embodiments of the present invention can be implemented in a video codec that can implement video encoding via a fixed or wired communication path.

[0340] Thus, the user equipment may comprise a video codec such as those described in the above embodiments of the invention.It should be understood that the term user equipment is intended to cover any suitable type of wireless user equipment, such as a mobile phone, a portable data processing device or a portable web browser.

[0341] Furthermore, elements of a public land mobile network (PLMN) may also include a video codec as described above.

[0342] In general, various embodiments of the present invention may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, although the present invention is not limited thereto. Although various aspects of the present invention may be illustrated and described as block diagrams, flow charts, or using some other graphical representations, it is well understood that, as non-limiting examples, the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0343] Embodiments of the present invention may be implemented by computer software executable by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. In this regard, it should also be noted that any box of the logic flow shown in the figure may represent program steps, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. The software may be stored on physical media such as memory chips or memory blocks implemented within a processor, magnetic media (such as hard disks or floppy disks), and optical media (such as DVDs and their data variants CDs).

[0344] The memory may be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The data processor may be of any type suitable for the local technical environment and may include, as non-limiting examples, one or more of the following: a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture.

[0345] Embodiments of the present invention may be practiced in various components, such as integrated circuit modules. The design of integrated circuits is generally a highly automated process. Complex and powerful software tools are available to convert a logic level design into a semiconductor circuit design that is ready to be etched and formed on a semiconductor substrate.

[0346] Programs such as those offered by Synopsys Inc., Mountain View, Calif., and Cadence Design Inc., San Jose, Calif., use well-established design rules and a library of pre-stored design blocks to automatically route conductors and position components on a semiconductor chip. Once the design for a semiconductor circuit has been completed, the final design in a standardized electronic format (e.g., Opus, GDSII, etc.) can be sent to a semiconductor manufacturing plant or "fab" for fabrication.

[0347] The foregoing description has provided a complete and informative description of exemplary embodiments of the present invention by way of exemplary and non-limiting examples. However, various modifications and adaptations may become apparent to those skilled in the relevant arts in view of the foregoing description when read in conjunction with the accompanying drawings and the appended claims. Nevertheless, all such and similar modifications to the teachings of the present invention will still fall within the scope of the present invention.

Claims

1. A device comprising: means for receiving an image block unit of a frame, the image block unit comprising samples in color channels, the color channels comprising at least one chrominance channel and one luma channel; means for reconstructing samples of said luminance channel of said image block unit; A means for determining a reference region of samples for predicting at least one color channel of the image block unit, wherein the reference region comprises: one or more reference samples in a neighboring block in a current color channel / frame, adjacent to a co-located block in a reference color channel / frame, and / or within the co-located block in a reference color channel / frame; means for determining filter coefficients of said filter based on said reference samples and a morphology of the filter used for said prediction; as well as Means for predicting samples of at least one color channel of the image block unit by performing a convolution between the samples of the luma channel and the filter coefficients.

2. The device according to claim 1, comprising: Means for downsampling the samples of the luma channel to correspond to a size of the samples of the chroma samples prior to determining the filter coefficients.

3. The apparatus according to claim 1 or 2, wherein the means for determining the filter coefficients comprises: means for obtaining statistics for linear regression from said reference samples based on said morphology of said filter; as well as means for obtaining said filter coefficients by applying a linear regression method to said obtained statistical data. The apparatus of claim 3 , wherein the statistical data comprises an autocorrelation matrix and a cross-correlation vector.

5. An apparatus according to any preceding claim, wherein the form of the filter is: a one-dimensional (1D-) vector of the filter coefficients or a two-dimensional (2D-) array of the filter coefficients.

6. Apparatus according to any preceding claim, comprising: Means for applying auxiliary parameters external to the morphology of the filter to the convolution.

7. Apparatus according to any preceding claim, comprising: Components for inputting the following items to the filter: a luma sample co-located with an output chroma sample, at least one luma sample above the co-located sample, at least one luma sample below the co-located sample, at least one luma sample to the left of the co-located sample, at least one luma sample to the right of the co-located sample, and a bias term.

8. The device according to claim 7, comprising: Means for inputting to said filter a value determined by applying a non-linear function to said values ​​of said co-located samples.

9. An apparatus according to any preceding claim, comprising: means for storing said filter coefficients in a memory; as well as Means for receiving an indication from an encoder to reuse at least a subset of previously stored filter coefficients to predict samples of at least one color channel of the current image block unit.

10. An apparatus according to any preceding claim, comprising: means for determining a sample outside of said reference region; means for obtaining statistics for linear regression from said samples outside said reference region based on said morphology of said filter; means for obtaining said filter coefficients by applying a linear regression method to said obtained statistical data; as well as Means for storing said filter coefficients in said memory.

11. The device according to any one of claims 2 to 10, comprising: Means for replacing any missing downsampled luma samples with zero-valued samples.

12. Apparatus according to any preceding claim, comprising: means for determining samples that do not belong to any object or texture within said image block unit; as well as Means for removing said sample from said reference area.

13. A method comprising: receiving an image block unit of a frame, the image block unit comprising samples in a color channel, the color channel comprising at least one chroma channel and one luma channel; Reconstructing samples of the brightness channel of the image block unit; Determine a reference area for predicting samples of at least one color channel of the image block unit, wherein the reference area includes: one or more reference samples in a neighboring block in a current color channel / frame, adjacent to a co-located block in a reference color channel / frame, and / or within the co-located block in a reference color channel / frame; determining filter coefficients of the filter based on the reference samples and the shape of the filter used for the prediction; and The samples of at least one color channel of the image block unit are predicted by performing convolution between the samples of the luma channel and the filter coefficients.

14. The method according to claim 13, comprising: Prior to determining the filter coefficients, the samples of the luma channel are downsampled to correspond to a size of the samples of the chroma samples.

15. The method of claim 13 or 14, wherein determining the filter coefficients comprises: obtaining statistics for linear regression from the reference samples based on the morphology of the filter; as well as The filter coefficients are obtained by applying a linear regression method to the obtained statistical data.

Images