Apparatus, method and computer program for video encoding and decoding

Abstract

A method comprising: receiving an image block unit of a frame, the image block unit comprising samples in color channels, the color channels comprising at least one chroma channel and one luma channel; reconstructing a sample of the luminance channel of an image block unit; determining a reference region for predicting a target sample of at least one color channel of the image block unit, the reference region comprising one or more reference samples in an adjacent block in a current color channel / frame, and one or more reference samples adjacent to a co-localization block in a reference color channel / frame, and / or one or more reference samples within a co-localization block in a reference color channel / frame; determining a weight for predicting the target sample based on a ratio between the normalized luminance histogram in the reference region and the normalized luminance histogram of the co-localization target sample; determining a filter coefficient of the filter for the prediction based on the weight, the reference sample and the shape of the filter; and predicting samples of at least one color channel of the image block unit based on the samples of the luminance channel and the filter coefficients.

Description

Technical Field

[0001] The present invention relates to apparatus, methods and computer programs for video encoding and decoding. Background Art

[0002] In video coding, video and image samples are often encoded using a color representation (such as YUV or YCbCr) consisting of one luma (or luminance) channel and two chroma (or chrominance) channels. In these cases, the luma channel, which primarily represents the scene lighting, is usually encoded at a certain resolution, while the chroma channels, which typically represent the differences between specific color components, are usually encoded at a second resolution lower than the luma signaling. The purpose of this differential representation is to decorrelate the color components and enable more efficient data compression.

[0003] In the Versatile Video Coding (VVC / H.266) standard, a cross-component linear model (CCLM) or a convolutional cross-component model (CCCM) is used as a linear model for predicting samples in chrominance channels (e.g., Cb and Cr). The model parameters are based on reconstructed samples near chrominance blocks, co-located neighboring samples in luma blocks, and reconstructed samples within co-located luma blocks.

[0004] Both CCLM and CCCM exploit the correlation between chrominance and luma channels and obtain a set of prediction parameters based on a reference sample set, usually by least squares minimization. In the prediction step, the model parameters are used together with another set of luma samples to obtain chrominance predictions. Ideally, the two luma sample sets would have similar intensity distributions so that the model parameters fit both. However, the intensity distributions of the reference luma samples and the co-located luma samples of the target chroma samples may be very different, resulting in poor fit of the model parameters. Summary of the invention

[0005] Now, in order to at least alleviate the above mentioned problems, this paper introduces an enhanced method for achieving cross-component prediction which reduces the prediction error between luma and chroma samples.

[0006] The scope of protection sought by various embodiments of the present invention is set out in the independent claims. Embodiments and features described in this specification that do not fall within the scope of the independent claims, if any, are to be construed as examples that aid in understanding the various embodiments of the present invention.

[0007] The method according to the first aspect includes receiving an image block unit of a frame, the image block unit including multiple samples in color channels, the color channels including at least one chrominance channel and one luminance channel; reconstructing samples of the luminance channel of the image block unit; determining a reference area for predicting target samples of at least one color channel of the image block unit, wherein the reference area includes one or more reference samples in adjacent blocks in the current color channel / frame, one or more reference samples adjacent to a co-located block in a reference color channel / frame, and / or one or more reference samples within a co-located block in a reference color channel / frame; determining a weight for predicting the target sample based on a ratio between a normalized luminance histogram in the reference area and a normalized luminance histogram of the co-located target sample; determining a filter coefficient of a filter for prediction based on the weight, the one or more reference samples and a shape of the filter; and predicting samples of at least one color channel of the image block unit based on the samples of the luminance channel and the filter coefficient.

[0008] The apparatus according to the second aspect includes a component for receiving an image block unit of a frame, the image block unit including a plurality of samples in a color channel, the color channel including at least one chrominance channel and one luminance channel; a component for reconstructing samples of a luminance channel of the image block unit; a component for determining a reference area for predicting a target sample of at least one color channel of the image block unit, wherein the reference area includes one or more reference samples in a neighboring block in a current color channel / frame, one or more reference samples neighboring a co-located block in a reference color channel / frame, and / or one or more reference samples within a co-located block in a reference color channel / frame; a component for determining a weight for predicting the target sample based on a ratio between a normalized luminance histogram in the reference area and a normalized luminance histogram of the co-located target sample; a component for determining filter coefficients of a filter for prediction based on the weights, one or more reference samples and a shape of the filter; and a component for predicting samples of at least one color channel of the image block unit based on samples of the luminance channel and the filter coefficients.

[0009] According to an embodiment, the means for determining the weight comprises means for calculating a ratio between a normalized brightness histogram in the reference region and a normalized brightness histogram of a sample of the co-located target sample.

[0010] According to an embodiment, the means for determining the weights comprises means for deriving the weights based on a temporal histogram, each entry of the temporal histogram corresponding to a single brightness intensity value.

[0011] According to an embodiment, the means for determining the weights comprises means for determining the same weights for a plurality of adjacent target samples.

[0012] According to an embodiment, the means for determining the weights comprises means for deriving the weights based on a strip histogram, wherein a plurality of brightness intensity values ​​are grouped into a single entry.

[0013] According to an embodiment, the apparatus comprises means for receiving signaling from an encoder to apply weights to a subset of images or a subset of image blocks.

[0014] According to an embodiment, the apparatus comprises means for deriving weights using a normalized histogram of co-located luma samples, down-sampled luma samples or original resolution luma samples.

[0015] According to an embodiment, the device comprises means for pre-processing the luma samples by a linear or non-linear filter before the weight derivation.

[0016] According to an embodiment, the apparatus comprises means for computing the normalized histogram and / or the weights by division-free integer operations using a properly configured lookup table for obtaining an exponential representation of the divisor.

[0017] As described above, the apparatus and the computer-readable storage medium having code stored thereon are thus arranged to perform one or more of the above-described methods and embodiments related thereto. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] For a better understanding of the present invention, reference will now be made by way of example to the accompanying drawings, in which:

[0019] Figure 1 An electronic device using an embodiment of the present invention is schematically shown;

[0020] Figure 2 Schematically illustrates a user equipment suitable for using an embodiment of the present invention;

[0021] Figure 3 Further schematically illustrated are electronic devices using embodiments of the present invention connected using wireless and wired network connections;

[0022] Figure 4a and Figure 4b Schematically shows an encoder and a decoder suitable for implementing an embodiment of the present invention;

[0023] Figure 5 The sample locations used to derive the cross-component linear model (CCLM) parameters are illustrated;

[0024] Figure 6a and Figure 6b Examples of classifying brightness samples into two categories, sample domain and spatial domain, are shown respectively;

[0025] Figure 7An example of a co-located reference sample region consisting of reconstructed luma and chroma samples defined for both luma and chroma of a convolutional cross-component model (CCCM) is shown;

[0026] Figure 8 Various examples of CCCM filter kernel sizes are shown;

[0027] Fig. 9 An example of four reference lines adjacent to a prediction block is illustrated;

[0028] Fig.10 illustrates the matrix-weighted intra prediction process;

[0029] Fig.11 A flowchart of a method for predicting samples in at least one color channel according to an embodiment of the present invention is shown; and

[0030] Fig.12 A schematic diagram of an example multimedia communication system is shown in which various embodiments may be implemented. DETAILED DESCRIPTION

[0031] Suitable devices and possible mechanisms for predicting chrominance samples are described in more detail below. In this regard, reference is first made to Figure 1 and Figure 2 ,in Figure 1 A block diagram of a video encoding system according to an example embodiment is shown as a schematic block diagram of an example apparatus or electronic device 50 which may incorporate a codec according to an embodiment of the invention. Figure 2 The layout of the device according to the exemplary embodiment is shown. Figure 1 and Figure 2 components.

[0032] The electronic device 50 may be, for example, a mobile terminal or user equipment of a wireless communication system. However, it should be understood that the embodiments of the present invention may be implemented in any electronic device or device that may need to encode and decode video images or needs encoding or decoding.

[0033] The device 50 may include a housing 30 for incorporating and protecting the device. The device 50 may further include a display 32 in the form of a liquid crystal display. In other embodiments of the present invention, the display may be any suitable display technology suitable for displaying images or videos. The device 50 may further include a keyboard 34. In other embodiments of the present invention, any suitable data or user interface mechanism may be employed. For example, as part of a touch-sensitive display, the user interface may be implemented as a virtual keyboard or data input system.

[0034] The device may include a microphone 36 or any suitable audio input, which may be a digital or analog signaling input. The device 50 may further include an audio output device, which in embodiments of the present invention may be any of the following: headphones 38, speakers, or analog audio or digital audio output connections. The device 50 may also include a battery (or in other embodiments of the present invention, the device may be powered by any suitable mobile energy device, such as a solar cell, a fuel cell, or a clockwork generator). The device may further include a camera capable of recording or capturing images and / or videos. The device 50 may further include an infrared port for short-range line-of-sight communication with other devices. In other embodiments, the device 50 may further include any suitable short-range communication solution, such as a Bluetooth wireless connection or a USB / FireWire wired connection.

[0035] The apparatus 50 may include a controller 56, a processor or processor circuit for controlling the apparatus 50. The controller 56 may be connected to a memory 58 which, in embodiments of the invention, may store data in the form of both image and audio data and / or may also store instructions for implementation on the controller 56. The controller 56 may further be connected to a codec circuit 54 which is adapted to implement encoding and decoding of audio and / or video data or to assist in encoding and decoding performed by the controller.

[0036] The apparatus 50 may further include a card reader 48 and a smart card 46, such as a UICC and a UICC reader for providing user information and adapted to provide authentication information to authenticate and authorize the user at the network.

[0037] The device 50 may include a radio interface circuit 52 connected to the controller and adapted to generate wireless communication signaling, for example, for communicating with a cellular communication network, a wireless communication system or a wireless local area network. The device 50 may further include an antenna 44 connected to the radio interface circuit 52 for transmitting the radio frequency signaling generated by the radio interface circuit 52 to other devices and for receiving the radio frequency signaling from other devices.

[0038] The apparatus 50 may include a camera capable of recording or detecting a single frame, which is then passed to the codec 54 or controller for processing. The apparatus may receive video image data from another device for processing before transmission and / or storage. The apparatus 50 may also receive images wirelessly or via a wired connection for encoding / decoding. The structural elements of the apparatus 50 described above represent examples of components for performing corresponding functions.

[0039] about Figure 3An example of a system in which embodiments of the present invention may be utilized is shown. System 10 includes multiple communication devices capable of communicating over one or more networks. System 10 may include any combination of wired or wireless networks, including but not limited to wireless cellular telephone networks (such as GSM, UMTS, CDMA networks, etc.), wireless local area networks (such as wireless local area networks (WLANs) defined by any IEEE 802.x standards), Bluetooth personal area networks, Ethernet local area networks, token ring local area networks, wide area networks, and the Internet.

[0040] System 10 may include both wired and wireless communication devices and / or apparatus 50 suitable for implementing embodiments of the present invention.

[0041] For example, Figure 3 The illustrated system shows a representation of the mobile telephone network 11 and the Internet 28. Connections to the Internet 28 may include, but are not limited to, long-range wireless connections, short-range wireless connections, and various wired connections including, but not limited to, telephone lines, cable lines, power lines, and similar communications pathways.

[0042] The example communication devices shown in system 10 may include, but are not limited to, electronic devices or apparatuses 50, a combination of a personal digital assistant (PDA) and mobile phone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, a notebook computer 22. Apparatus 50 may be fixed or mobile when carried by a mobile individual. Apparatus 50 may also be located in a vehicle, including, but not limited to, a car, a truck, a taxi, a bus, a train, a boat, an airplane, a bicycle, a motorcycle, or any similar suitable mode of transportation.

[0043] Embodiments may also be implemented in set-top boxes (i.e., digital television receivers) that may or may not have display or wireless capabilities; in tablet computers or (laptop) personal computers (PCs) with hardware or software or a combination of encoder / decoder implementations, in various operating systems, and in chipsets, processors, DSPs, and / or embedded systems providing hardware / software based encoding.

[0044] Some or additional devices may send and receive calls and messages and communicate with service providers via a wireless connection 25 to a base station 24. The base station 24 may be connected to a network server 26 that allows communication between the mobile telephone network 11 and the Internet 28. The system may include additional communication devices and various types of communication devices.

[0045] The communication device may communicate using various transmission technologies, including but not limited to code division multiple access (CDMA), global system for mobile communications (GSM), universal mobile telecommunications system (UMTS), time division multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-Internet protocol (TCP-IP), short message service (SMS), multimedia message service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.11 and any similar wireless communication technology. The communication device involved in implementing various embodiments of the present invention may communicate using various media, including but not limited to radio, infrared, laser, cable connection and any suitable connection.

[0046] In telecommunications and data networks, a channel can refer to a physical channel or a logical channel. A physical channel can refer to a physical transmission medium, such as a wire, while a logical channel can refer to a logical connection through a multiplexed medium capable of transmitting multiple logical channels. A channel can be used to transmit information signaling, such as a bit stream, from one or more senders (or transmitters) to one or more receivers.

[0047] The MPEG-2 Transport Stream (TS), specified in ISO / IEC 13818-1 or equivalently in ITU-T Recommendation H.222.0, is a format for carrying audio, video and other media as well as program metadata or other metadata in a multiplexed transport stream. A packet identifier (PID) is used to identify an elementary stream (also called a packetized elementary stream) in a TS. Therefore, a logical channel in an MPEG-2 TS can be considered to correspond to a specific PID value.

[0048] Available media file format standards include ISO Base Media File Format (ISO / IEC 14496-12, which may be abbreviated as ISOBMFF) and a file format for NAL unit structured video derived from ISOBMFF (ISO / IEC 14496-15).

[0049] A video codec consists of an encoder that transforms the input video into a compressed representation suitable for storage / transmission and a decoder that can decompress the compressed video representation back into a visual form. The video encoder and / or video decoder can also be separated from each other, i.e., they do not need to form a codec. Typically, the encoder discards some information in the original video sequence in order to represent the video in a more compact form (i.e., at a lower bit rate).

[0050] Typical hybrid video encoders, such as multiple encoder implementations of ITU-T H.263 and H.264, encode video information in two stages. First, pixel values ​​in certain picture areas are predicted, for example, by a motion compensation component (finding and indicating an area in a previously encoded video frame that closely corresponds to the block being encoded) or by a spatial component (using the pixel values ​​around the block to be encoded in a specified manner). Second, the prediction error (i.e., the difference between the predicted pixel block and the original pixel block) is encoded. This is usually accomplished by transforming the difference in pixel values ​​using a specified transform (e.g., a discrete cosine transform (DCT) or a variant thereof), quantizing the coefficients, and entropy encoding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and the size of the resulting encoded video representation (file size or transmission bit rate).

[0051] In temporal prediction, the source of the prediction is a previously decoded picture (also called a reference picture). In intra block copy (IBC; also called intra block copy prediction), the application of prediction is similar to temporal prediction but the reference picture is the current picture, and only previously decoded samples can be referenced during the prediction process. The application of inter-layer prediction or inter-view prediction can be similar to temporal prediction, but the reference picture is a decoded picture from another scalable layer or another view, respectively. In some cases, inter prediction may refer only to temporal prediction, while in other cases, inter prediction may be collectively referred to as temporal prediction and any intra block copy, inter-layer prediction, and inter-view prediction, provided that they are performed in the same or similar process as temporal prediction. Inter prediction or temporal prediction may sometimes be referred to as motion compensation or motion compensated prediction.

[0052] Motion compensation can be performed with full sample or sub-sample accuracy. In the case of full sample accurate motion compensation, the motion can be represented as a motion vector with integer values ​​for horizontal and vertical displacements, and the motion compensation process uses these displacements to effectively copy samples from a reference image. In the case of sub-sample accurate motion compensation, the motion vector is represented by fractional or decimal values ​​for the horizontal and vertical components of the motion vector. In the case where the motion vector refers to a non-integer position in the reference image, a sub-sample interpolation process is typically invoked to calculate predicted sample values ​​based on reference samples and selected sub-sample positions. The sub-sample interpolation process typically consists of horizontal filtering to compensate for horizontal offsets relative to the full sample positions, followed by vertical filtering to compensate for vertical offsets relative to the full sample positions. However, in some environments, vertical processing can also be done before horizontal processing.

[0053] Inter prediction (which may also be referred to as temporal prediction, motion compensation or motion compensated prediction) reduces temporal redundancy. In inter prediction, the prediction source is a previously decoded picture. Intra prediction exploits the fact that adjacent pixels within the same picture may be correlated. Intra prediction can be performed in the spatial or transform domain, i.e., sample values ​​or transform coefficients can be predicted. Intra prediction is typically utilized in intra coding, where inter prediction is not applied.

[0054] One result of the encoding process is a set of coded parameters, such as motion vectors and quantized transform coefficients. If multiple parameters are first predicted from spatially or temporally adjacent parameters, the parameters can be more efficiently entropy coded. For example, a motion vector can be predicted from spatially adjacent motion vectors, and only the difference with respect to the motion vector predictor can be encoded. Prediction of coding parameters and intra-frame prediction can be collectively referred to as intra-picture prediction.

[0055] Figure 4a and Figure 4b An encoder and decoder suitable for use with embodiments of the present invention are shown. A video codec consists of an encoder that transforms an input video into a compressed representation suitable for storage / transmission and a decoder that can decompress the compressed video representation back into a visual form. Typically, an encoder discards and / or loses some information in the original video sequence in order to represent the video in a more compact form (i.e., at a lower bit rate). Figure 4a An example of the encoding process is shown in . Figure 4a The image to be encoded (I n ); The predicted representation of the image block (P' n ); prediction error signaling (D n ); reconstructed prediction error signal (D' n ); Preliminary reconstructed image (I' n ); Final reconstructed image (R' n ); Transform (T) and Inverse Transform (T -1 ), quantization (Q) and dequantization (Q -1 ); Entropy coding (E); Reference frame memory (RFM); Inter-frame prediction (P inter ); intra prediction (P intra ); mode selection (MS) and filtering (F).

[0056] Figure 4b An example of the decoding process is shown in . Figure 4b The image block (P' n ) prediction representation; reconstructed prediction error signaling (D' n ); Preliminary reconstructed image (I' n ); Final reconstructed image (R' n ); inverse transform (T -1 ); Dequantization (Q-1 ); Entropy decoding (E -1 ); reference frame memory (RFM); prediction (inter or intra) (P); and filtering (F).

[0057] A number of hybrid video encoders encode video information in two stages. First, the pixel values ​​in a particular picture area (or "block") are predicted, for example, by a motion compensation device (finding and indicating an area in one of the previously encoded video frames that closely corresponds to the block being encoded) or a spatial device (using pixel values ​​around the block that are encoded in a particular way). Second, the prediction error (i.e., the difference between the predicted pixel block and the original pixel block) is encoded. This is typically achieved by transforming the difference in pixel values ​​using a particular transform (e.g., discrete cosine transform (DCT) or a variant thereof), quantizing coefficients, and entropy encoding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and the size of the resulting encoded video representation (file size or transmission bit rate). The video codec may further provide a transform skip mode, which the encoder may choose to use. In transform skip mode, the prediction error is encoded in the sample domain, for example by deriving a sample-wise difference relative to certain adjacent samples and encoding the sample-wise difference using an entropy encoder.

[0058] Entropy coding / decoding can be performed in a variety of ways. For example, context-based coding / decoding can be applied, in which both the encoder and the decoder modify the context state of the coding parameters based on the coding parameters of the previous coding / decoding. Context-based coding can be, for example, context-adaptive binary arithmetic coding (CABAC) or context-based variable length coding (CAVLC) or any similar entropy coding. Entropy coding / decoding is alternatively or additionally performed using a variable length coding scheme, such as Huffman coding / decoding or Exp-Golomb coding / decoding. Decoding the coding parameters of the bitstream or codeword from the entropy coding can be referred to as parsing.

[0059] Phrases along the bitstream (e.g., indication along the bitstream) may be defined to refer to out-of-band transmission, signaling, or storage in a manner that out-of-band data is associated with the bitstream. Similar phrases such as decoding along the bitstream may refer to decoding out-of-band data associated with the bitstream (which may be obtained from the out-of-band transmission, signaling, or storage). For example, indication along the bitstream may refer to metadata in a container file that encapsulates the bitstream.

[0060] The H.264 / AVC standard was developed by the Joint Video Team (JVT) of the Video Coding Experts Group (VCEG) of the Telecommunication Standardization Sector of the International Telecommunication Union (ITU-T) and the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard was published by the two parent standardization organizations and is known as ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10, also known as MPEG-4 Part 10 Advanced Video Coding (AVC). There are multiple versions of the H.264 / AVC standard with new extensions or features integrated into the specification. These extensions include Scalable Video Coding (SVC) and Multi-view Video Coding (MVC).

[0061] Version 1 of the High Efficiency Video Coding (H.265 / HEVC also known as HEVC) standard was developed by the Joint Collaboration Team - Video Coding (JCT-VC) of VCEG and MPEG. The standard is published by the two parent standardization organizations and is known as ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2, also known as MPEG-H Part 2 High Efficiency Video Coding (HEVC). Subsequent versions of H.265 / HEVC include Scalable, Multi-view, Fidelity Range, 3D, and Screen Content Coding extensions, which can be abbreviated as SHVC, MV-HEVC, REXT, 3D-HEVC, and SCC, respectively.

[0062] Versatile Video Coding (VVC) (MPEG-I Part 3), also known as ITU-T H.266, is a video compression standard developed by the Joint Video Experts Team (JVET) of the Moving Picture Experts Group (MPEG), (formally ISO / IEC JTC1 SC29 WG11) and the Video Coding Experts Group (VCEG) of the International Telecommunication Union (ITU), and is the successor to HEVC / H.265.

[0063] In this section, some key definitions, bitstream and coding structures, and concepts of H.264 / AVC and HEVC are described as examples of video encoders, decoders, encoding methods, decoding methods, and bitstream structures, where embodiments may be implemented. Some key definitions, bitstream and coding structures and concepts of H.264 / AVC are the same as those in HEVC, so they are described jointly below. Aspects of the present invention are not limited to H.264 / AVC or HEVC, but are described for one possible basis on which the present invention may be partially or fully implemented.

[0064] Similar to several earlier video coding standards, bitstream syntax and semantics as well as a decoding process for error-free bitstreams are specified in H.264 / AVC and HEVC. The encoding process is not specified, but the encoder must generate a conforming bitstream. Bitstream and decoder conformance can be verified using a hypothetical reference decoder (HRD). The standard contains coding tools that help deal with transmission errors and losses, but the use of the coding tools is optional and no decoding process is specified for erroneous bitstreams.

[0065] The basic unit for input to an H.264 / AVC or HEVC encoder and the basic unit for output of an H.264 / AVC or HEVC decoder are pictures, respectively. A picture given as input to an encoder may also be referred to as a source picture, and a picture decoded by a decoder may be referred to as a decoded picture.

[0066] The source picture and the decoded picture each consist of one or more sample arrays, such as one of the following sample array sets:

[0067] - Luminance (Y) only (monochrome).

[0068] - Luma and two chrominances (YCbCr or YCgCo).

[0069] - Green, Blue and Red (GBR, also known as RGB).

[0070] - Arrays representing other unspecified monochromatic or tristimulus color sampling (e.g., YZX, also called XYZ).

[0071] In H.264 / AVC and HEVC, a picture can be either a frame or a field. A frame consists of a matrix of luma samples and possibly a corresponding matrix of chroma samples. A field is a set of alternating sample rows of a frame and can be used as encoder input when the source signaling is interlaced. The chroma sample array may not be present (so monochrome sampling may be used) or the chroma sample array may be subsampled compared to the luma sample array. The chroma formats can be summarized as follows:

[0072] - In monochrome sampling, there is only one sample array, which can nominally be thought of as a brightness array.

[0073] - In 4:2:0 sampling, each of the two chroma arrays has half the height and half the width of the luma array.

[0074] - In 4:2:2 sampling, each of the two chroma arrays has the same height and half the width of the luma array.

[0075] - In 4:4:4 sampling, when separate color planes are not used, each of the two chroma arrays has the same height and width as the luma array.

[0076] In H.264 / AVC and HEVC, sample arrays can be encoded into the bitstream as separate color planes, and the separately encoded color planes in the bitstream can be decoded separately. When separate color planes are in use, each of them is processed separately (by the encoder and / or decoder) as a monochrome sampled picture.

[0077] Partitioning can be defined as dividing a set into subsets such that each element of the set is in exactly one of these subsets.

[0078] When describing the operation of HEVC encoding and / or decoding, the following terms may be used. A coding block may be defined as a block of N x N samples for some values ​​of N, such that the partitioning of a coding tree block to a coding block is partitioned. A coding tree block (CTB) may be defined as a block of N×N samples for some values ​​of N, such that the partitioning of a component to a coding tree block is partitioned. A coding tree unit (CTU) may be defined as a coding tree block of luma samples, two corresponding coding tree blocks of chroma samples of a picture with three sample arrays, or a coding tree block of samples of a monochrome picture, or a coding tree block of samples of a picture encoded using three separate color planes and a syntax structure for encoding samples. A coding unit (CU) may be defined as a coding block of luma samples, two corresponding coding blocks of chroma samples of a picture with three sample arrays, or a coding block of samples of a monochrome picture, or a coding block of samples of a picture encoded using three separate color planes and a syntax structure for encoding samples. A CU with the maximum allowed size is often referred to as an LCU (maximum coding unit) or a CTU (coding tree unit), and a video picture is divided into non-overlapping LCUs.

[0079] A CU consists of one or more prediction units (PUs) that define a prediction process for samples within the CU and one or more transform units (TUs) that define a prediction error coding process for samples in the CU. Typically, a CU consists of square blocks of samples, whose size can be selected from a predefined set of possible CU sizes. Each PU and TU can be further split into smaller PUs and TUs to increase the granularity of the prediction and prediction error coding processes, respectively. Each PU has prediction information associated with it that defines what type of prediction is applied to the pixels within the PU (e.g., motion vector information for inter-prediction PUs and intra-prediction directionality information for intra-prediction PUs).

[0080] Each TU may be associated with information about the prediction error decoding process for the samples within the TU (including, for example, DCT coefficient information). Whether prediction error coding is applied to each CU is typically signaled at the CU level. In the absence of a prediction error residual associated with a CU, it may be considered that there are no TUs for the CU. The partitioning of an image into CUs and the partitioning of a CU into PUs and TUs is typically signaled in the bitstream, allowing a decoder to reproduce the expected structure of these units.

[0081] In HEVC, pictures can be partitioned into tiles, which are rectangular and contain an integer number of LCUs. In HEVC, the partitioning of tiles forms a regular grid, where the height and width of the tiles differ from each other by at most one LCU. In HEVC, a slice is defined as an integer number of coding tree units contained in an independent segment and all subsequent dependent segments (if any) before the next independent segment (if any) within the same access unit. In HEVC, a segment is defined as an integer number of coding tree units that are ordered consecutively in a tile scan and contained in a single NAL unit. The division of each picture into segments is a partition. In HEVC, an independent segment is defined as a segment for which the values ​​of the syntax elements of the segment header of the segment are not inferred from the values ​​of the previous segment, and a dependent segment is defined as a segment for which the values ​​of certain syntax elements of the segment header of the segment are inferred from the values ​​of the previous independent segment in decoding order. In HEVC, a slice header is defined as a slice header of an independent slice that is either the current slice or an independent slice before the current dependent slice, and a slice header is defined as the part of a coded slice that contains data elements about the first or all coding tree units represented in the slice. If tiles are not used, CUs are scanned in the raster scan order of LCUs within tiles or within a picture. Within an LCU, CUs have a specific scanning order.

[0082] The decoder reconstructs the output video by applying a prediction component similar to the encoder to form a predicted representation of pixel blocks (using motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (the inverse operation of prediction error encoding to recover the quantized prediction error signaling in the spatial pixel domain). After applying the prediction and prediction error decoding components, the decoder adds the prediction and prediction error signaling (pixel values) to form the output video frame. The decoder (and encoder) may also apply additional filtering means to improve the quality of the output video before passing it to a display and / or storing it as a prediction reference for upcoming frames in the video sequence.

[0083] The filtering may, for example, include one or more of: deblocking, sample adaptive offset (SAO) and / or adaptive loop filtering (ALF).H.264 / AVC includes deblocking, while HEVC includes both deblocking and SAO.

[0084] In a typical video codec, motion information is indicated using a motion vector (e.g., prediction unit) associated with each motion compensated image block. Each of these motion vectors represents the displacement of an image block in a picture to be encoded (on the encoder side) or decoded (on the decoder side) and a prediction source block in one of the previously encoded or decoded pictures. In order to efficiently represent motion vectors, these motion vectors are usually differentially encoded for block-specific predicted motion vectors. In a typical video codec, predicted motion vectors are created in a predetermined manner, for example by calculating the median of the encoded or decoded motion vectors of neighboring blocks. Another way to create a motion vector prediction is to generate a list of candidate predictions from neighboring blocks and / or co-located blocks in a temporal reference picture, and signal the selected candidate as a motion vector predictor. In addition to predicting the motion vector value, it is also possible to predict which (multiple) reference pictures are used for motion compensated prediction, and this prediction information can be represented, for example, by a reference index of a previously encoded / decoded picture. The reference index is usually predicted based on neighboring blocks and / or co-located blocks in a temporal reference picture. Furthermore, typical high efficiency video codecs employ an additional motion information encoding / decoding mechanism, often referred to as merge / merge mode, where all motion field information including motion vectors and corresponding reference picture indexes for each available reference picture list is predicted and used without any modification / correction. Similarly, predicting the motion field information is performed using motion field information of neighboring blocks and / or co-located blocks in temporal reference pictures, and the used motion field information is signaled in a motion field candidate list populated with the motion field information of the available neighboring / co-located blocks.

[0085] In a typical video codec, the motion compensated prediction residuals are first transformed using a transform kernel (such as discrete cosine transform) and then encoded. The reason for this is that there is usually still some correlation between the residuals, and in many cases, the transform can help reduce this correlation and provide more efficient coding.

[0086] Video coding standards and specifications may allow the encoder to divide a coded picture into coded slices, etc. Intra-picture prediction is typically disabled on slice boundaries. Therefore, slicing can be viewed as a way to split a coded picture into independently decodable slices. In H.264 / AVC and HEVC, intra-picture prediction may be disabled on slice boundaries. Therefore, slicing may be viewed as a way to split a coded picture into independently decodable slices, and thus slices are typically viewed as basic units for transmission. In a number of cases, the encoder may indicate in the bitstream which types of intra-picture prediction are turned off on slice boundaries, and the decoder operation may take this information into account, for example, when inferring which prediction sources are available. For example, if adjacent CUs are in different slices, samples from adjacent CUs may be considered unavailable for intra prediction.

[0087] The basic unit for the output of the H.264 / AVC or HEVC encoder and the input of the H.264 / AVC or HEVC decoder is the network abstraction layer (NAL) unit. In order to be transmitted over a packet-oriented network or stored in a structured file, the NAL unit can be encapsulated into a packet or similar structure. In H.264 / AVC and HEVC, a byte stream format is specified for a transmission or storage environment that does not provide a frame structure. The byte stream format separates NAL units from each other by appending a start code before each NAL unit. In order to avoid erroneous detection of NAL unit boundaries, the encoder runs a byte-oriented start code emulation prevention algorithm, which adds an emulation prevention byte to the NAL unit payload if a start code appears otherwise. In order to implement simple gateway operations between packet-oriented and stream-oriented systems, start code emulation prevention can always be performed regardless of whether the byte stream format is in use. The NAL unit can be defined as a syntax structure containing an indication of the type of data to be followed, and bytes containing data in the form of RBSP, in which emulation prevention bytes are interspersed when necessary. The original byte sequence payload (RBSP) can be defined as a syntax structure containing an integer number of bytes encapsulated in a NAL unit. The RBSP is either empty or has the form of a data bit string containing a syntax element followed by a RBSP stop bit and by zero or more subsequent bits equal to zero.

[0088] The NAL unit consists of a header and a payload. In H.264 / AVC and HEVC, the NAL unit header indicates the type of the NAL unit.

[0089] In HEVC, a two-byte NAL unit header is used for all specified NAL unit types. The NAL unit header contains 1 reserved bit, a 6-bit NAL unit type indication, a 3-bit nuh_temporal_id_plus1 indication for the temporal level (which may need to be greater than or equal to 1), and a 6-bit nuh_layer_id syntax element. The temporal_id_plus1 syntax element can be considered as a temporal identifier for the NAL unit, and the zero-based TemporalId variable can be derived as follows: TemporalId = temporal_id_plus1-1. The abbreviation TID can be used interchangeably with the TemporalId variable. A TemporalId equal to 0 corresponds to the lowest temporal level. In order to avoid start code emulation involving two NAL unit header bytes, the value of temporal_id_plus1 needs to be non-zero. The bitstream created by excluding all VCL NAL units with a TemporalId greater than or equal to the selected value and including all other VCL NAL units still conforms. Thus, a picture with TemporalId equal to tid_value does not use any picture with TemporalId greater than tid_value as inter prediction reference. A sublayer or temporal sublayer may be defined as a temporal scalable layer (or temporal layer, TL) of a temporal scalable bitstream, which consists of VCL NAL units and associated non-VCL NAL units with a specific value of the TemporalId variable. nuh_layer_id may be understood as a scalability layer identifier.

[0090] NAL units can be classified into video coding layer (VCL) NAL units and non-VCL NAL units. VCL NAL units are usually coded slice NAL units. In HEVC, VCL NAL units contain syntax elements representing one or more CUs.

[0091] A non-VCL NAL unit may be, for example, one of the following types: a sequence parameter set, a picture parameter set, a supplemental enhancement information (SEI) NAL unit, an access unit delimiter, an end-of-sequence NAL unit, an end-of-bitstream NAL unit, or a filler data NAL unit. Reconstruction of a decoded image may require a parameter set, while reconstruction of decoded sample values ​​does not require multiple other non-VCL NAL units.

[0092] Parameters that remain unchanged through the encoded video sequence may be included in a sequence parameter set. In addition to parameters that may be required for the decoding process, the sequence parameter set may optionally contain video usability information (VUI), which includes parameters that may be important for buffering, image output timing, rendering, and resource reservation. In HEVC, the sequence parameter set RBSP includes parameters that can be referenced by one or more picture parameter set RBSPs, or one or more SEI NAL units containing a buffering period SEI message. The picture parameter set contains parameters that may remain unchanged in several coded pictures. The picture parameter set RBSP may include parameters that can be referenced by the coded slice NAL units of one or more coded pictures.

[0093] In HEVC, a video parameter set (VPS) can be defined as a syntax structure containing syntax elements that apply to zero or more complete coded video sequences as determined by the contents of syntax elements found in the SPS referenced by syntax elements found in the PPS referenced by syntax elements found in each segment header.

[0094] A video parameter set RBSP may include parameters that can be referenced by one or more sequence parameter set RBSPs.

[0095] The relationship and hierarchy between video parameter sets (VPS), sequence parameter sets (SPS), and picture parameter sets (PPS) can be described as follows. In the parameter set hierarchy and in the context of scalability and / or 3D video, VPS is one level higher than SPS. VPS may include parameters that are common to all slices across all (scalability or view) layers in the entire coded video sequence. SPS includes parameters that are common to all slices in a specific (scalability or view) layer in the entire coded video sequence, and may be shared by multiple (scalability or view) layers. PPS includes parameters that are common to all slices in a specific layer representation (a representation of a scalability or view layer in an access unit), and may be shared by all slices in multiple layer representations.

[0096] The VPS can provide information about the dependencies of layers in the bitstream, as well as a number of other information applicable to all slices of all (scalability or view) layers in the entire coded video sequence. The VPS can be viewed as consisting of two parts, a base VPS and a VPS extension, where the VPS extension can be optionally present.

[0097] Out-of-band transmission, signaling or storage may additionally or alternatively be used for other purposes beyond tolerance of transmission errors, such as ease of access or session negotiation. For example, sample entries for a track in a file conforming to the ISO base media file format may include parameter sets, while the coded data in the bitstream is stored elsewhere in the file or in another file. The phrase "along the bitstream" (e.g., indicating along the bitstream) or "along the coding unit of the bitstream" (e.g., indicating along the coding tile) may be used in the claims and described embodiments to refer to out-of-band transmission, signaling or storage in a manner in which the out-of-band data is associated with the bitstream or coding unit, respectively. The phrase "decoding along the bitstream" or "decoding along the coding unit of the bitstream" and the like may refer to decoding the referenced out-of-band data (which may be obtained from out-of-band transmission, signaling or storage) associated with the bitstream or coding unit, respectively.

[0098] An SEI NAL unit may contain one or more SEI messages that are not required for decoding of the output picture, but may assist in related processes such as picture output timing, presentation, error detection, error concealment, and resource reservation.

[0099] A coded picture is an encoded representation of a picture.

[0100] In HEVC, a coded picture can be defined as the coded representation of a picture containing all coding tree units of the picture. In HEVC, an access unit (AU) can be defined as a set of NAL units that are associated with each other according to a specified classification rule, are consecutive in decoding order, and contain at most one picture with any particular nuh_layer_id value. In addition to VCL NAL units containing coded pictures, access units can also contain non-VCL NAL units. The specified classification rule can, for example, associate pictures with the same output time or picture output count value to the same access unit.

[0101] A bitstream can be defined as a sequence of bits in the form of a NAL unit stream or a byte stream, which forms a representation of a coded picture and associated data forming one or more coded video sequences. A first bitstream may follow a second bitstream in the same logical channel (such as in the same file or in the same connection of a communication protocol). An elementary stream (in the case of video coding) may be defined as a sequence of one or more bitstreams. The end of the first bitstream may be indicated by a specific NAL unit, which may be referred to as a bitstream (EOB) NAL end unit, and which is the last NAL unit of the bitstream. In HEVC and its current draft extensions, the EOB NAL unit is required to have nuh_layer_id equal to 0.

[0102] In H.264 / AVC, a coded video sequence is defined as a sequence of consecutive access units in decoding order from an IDR access unit (inclusive) to the next IDR access unit (exclusive) or the end of the bitstream, whichever comes first.

[0103] In HEVC, for example, a coded video sequence (CVS) may be defined as a sequence of access units that, in decoding order, includes an IRAP access unit with NoRaslOutputFlag equal to 1, zero or more access units following a non-IRAP access unit with NoRaslOutputFlag equal to 1, including all subsequent access units up to but not including any subsequent access unit that is an IRAP access unit with NoRaslOutputFlag equal to 1. An IRAP access unit may be defined as an access unit whose base layer picture is an IRAP picture. For each IDR picture, each BLA picture, and each IRAP picture (the first picture in that particular layer in the bitstream in decoding order), the value of NoRaslOutputFlag is equal to 1, the IRAP picture being the first IRAP picture in decoding order after the end NAL unit of the sequence having the same nuh_layer_id value. There may be a component that provides the value of HandleCraAsBlaFlag to the decoder from an external entity that can control the decoder, such as a player or receiver. HandleCraAsBlaFlag may be set to 1, for example, by a player that seeks to a new position in the bitstream or tunes into a broadcast and starts decoding, and then starts decoding from a CRA picture. When HandleCraAsBlaFlag is equal to 1 for a CRA picture, the CRA picture is processed and decoded as if it were a BLA picture.

[0104] In HEVC, a coded video sequence may additionally or alternatively (to the above description) be designated as ending when a specific NAL unit, which may be referred to as an end-of-sequence (EOS) NAL unit, appears in the bitstream and has nuh_layer_id equal to 0.

[0105] A group of pictures (GOP) and its characteristics can be defined as follows. A GOP can be decoded regardless of whether the previous picture is decoded. An open GOP is a group of pictures in which the pictures before the initial intra-prediction picture in the output order may not be correctly decoded when decoding starts from the initial intra-prediction picture of the open GOP. In other words, the pictures of the open GOP can refer to pictures belonging to the previous GOP (in inter-frame prediction). Since a specific NAL unit type (CRA NAL unit type) is available for its coded fragments, the HEVC decoder can identify the intra-prediction picture that starts the open GOP. A closed GOP is a group of pictures in which all pictures can be correctly decoded when decoding starts from the initial intra-picture of the closed GOP. In other words, no picture in the closed GOP refers to any picture in the previous GOP. In H.264 / AVC and HEVC, a closed GOP may start with an IDR picture. In HEVC, a closed GOP may also start with a BLA_W_RADL or BLA_N_LP picture. An open GOP coding structure may be more efficient in compression than a closed GOP coding structure due to greater flexibility in selecting reference pictures.

[0106] The decoded picture buffer (DPB) can be used in the encoder and / or decoder. The decoded pictures are buffered for two reasons, for reference for inter-frame prediction and for reordering the decoded pictures to output order. Since H.264 / AVC and HEVC provide great flexibility for reference picture marking and output reordering, separate buffers for reference picture buffering and output picture buffering may waste storage resources. Therefore, the DPB may include a unified decoded picture buffering process for reference pictures and output reordering. When the DPB is no longer used as a reference, the decoded picture can be deleted from the DPB without output.

[0107] In multiple coding modes of H.264 / AVC and HEVC, reference pictures used for inter prediction are indicated by indices to reference picture lists. The index can be encoded with variable length coding, which generally makes the index with a smaller value for the corresponding syntax element smaller. In H.264 / AVC and HEVC, two reference picture lists (reference picture list 0 and reference picture list 1) are generated for each bidirectionally predicted (B) slice, and one reference picture list (reference picture list 0) is formed for each inter-coded (P) slice.

[0108] Multiple coding standards including H.264 / AVC and HEVC may have a decoding process to derive a reference picture index of a reference picture list, which may be used to indicate which of multiple reference pictures to use for inter prediction of a particular block. The reference picture index may be encoded into the bitstream by the encoder in some inter coding modes, or the reference picture index may be derived (by the encoder and decoder) using neighboring blocks in some other inter coding modes, for example.

[0109] The motion parameter type or motion information may include but is not limited to one or more of the following types:

[0110] - an indication of the prediction type (e.g. intra prediction, unidirectional prediction, bidirectional prediction) and / or the number of reference pictures;

[0111] - an indication of a prediction direction, such as inter (also called temporal) prediction, inter-layer prediction, inter-view prediction, view synthesis prediction (VSP) and inter-component prediction (which may be indicated per reference picture and / or prediction type, and where in some embodiments inter-view prediction and view synthesis prediction may be considered together as one prediction direction), and / or

[0112] - an indication of the reference picture type, such as a short-term reference picture and / or a long-term reference picture and / or an inter-layer reference picture (which may be indicated per reference picture, for example)

[0113] - a reference index to a reference picture list and / or any other identifier of a reference picture (which may be indicated e.g. per reference picture and its type may depend on the prediction direction and / or the reference picture type and may be accompanied by other relevant information such as the reference picture list or similar information where the reference index is applicable);

[0114] - horizontal motion vector component (which can be indicated, for example, per prediction block or per reference index, etc.);

[0115] - vertical motion vector component (which can be indicated, for example, per prediction block or per reference index, etc.);

[0116] - one or more parameters, such as a picture order count difference and / or a relative camera separation between a picture containing or associated with a motion parameter and its reference pictures, which parameters may be used to scale a horizontal motion vector component and / or a vertical motion vector component in one or more motion vector prediction processes (wherein the one or more parameters may be indicated, for example, per reference picture or per reference index, etc.);

[0117] - the coordinates of the block to which the motion parameters and / or motion information apply, e.g. the coordinates of the top left corner sample of the block in luma sample units;

[0118] - The extent (eg width and height) of the block to which the motion parameters and / or motion information apply.

[0119] Compared with previous video coding standards, the General Video Codec (H.266 / VVC) introduces a variety of new coding tools, such as the following:

[0120] Intra-frame prediction

[0121] -67 Intra-frame modes with wide-angle mode expansion

[0122] - Block size and mode dependent 4-tap interpolation filter

[0123] -Position-dependent intra prediction combination (PDPC)

[0124] -Cross-Component Linear Model Intra Prediction (CCLM)

[0125] -Multiple reference line intra prediction

[0126] - Intra-frame sub-partition

[0127] -Weighted intra prediction with matrix multiplication

[0128] Inter-picture prediction

[0129] - Block motion replication with spatial, temporal, history-based and pairwise average merge candidates

[0130] -Affine motion inter-frame prediction

[0131] - Sub-block based temporal motion vector prediction

[0132] - Adaptive motion vector resolution

[0133] - 8×8 block-based motion compression for temporal motion prediction

[0134] - High-precision (1 / 16 pixel) motion vector storage and motion compensation, 8-tap interpolation filter for luminance component and 4-tap interpolation filter for chrominance component

[0135] -Triangular partition

[0136] - Combined intra prediction and inter prediction

[0137] -Merge with Motion Vector Difference (MVD)

[0138] -Symmetrical MVD encoding

[0139] - Bidirectional optical flow

[0140] -Decoder side motion vector refinement

[0141] -CU-level bidirectional weighted prediction

[0142] Transform, quantization and coefficient coding

[0143] - Multiple main transform options, including DCT2, DST7 and DCT8

[0144] - Secondary transform for low frequency region

[0145] -Sub-block transform of inter prediction residual

[0146] -Quantization dependent, max QP increased from 51 to 63

[0147] - Transform coefficient coding with signed data hiding

[0148] -Transform skips remaining encoding

[0149] Entropy coding

[0150] -Arithmetic coding engine with adaptive dual-window probability update

[0151] Loop filter

[0152] - Reshape within the ring

[0153] - Enhanced long filter deblocking filter

[0154] - Sample Adaptive Offset

[0155] - Adaptive loop filter

[0156] Screen content encoding:

[0157] - Current picture reference with limited reference area

[0158] 360-degree video encoding

[0159] -Horizontal surround motion compensation

[0160] Advanced syntax and parallel processing

[0161] - Reference picture management with direct reference picture list signaling

[0162] - Tile groups with rectangular tile groups

[0163] The partitioning implementation in VVC is similar to that in HEVC, that is, each picture is divided into coding tree units (CTUs). Pictures can also be divided into slices, tiles, bricks, and sub-pictures. CTUs can be split into smaller CUs using a quadtree structure. Each CU can be partitioned using quadtrees and nested multi-type trees, including ternary and binary splits. However, there are specific rules to infer partitions at picture boundaries, and redundant splitting patterns are not allowed in nested multi-type partitions.

[0164] Among the new coding tools listed above, the cross-component linear model (CCLM) prediction mode is used in VVC to reduce cross-component redundancy. The chrominance samples are predicted based on the reconstructed luma samples of the same CU using the following linear model:

[0165] pred C (i, j) = α·rec L ′(i, j)+β Equation (1a)

[0166] where pred C (i, j) represents the predicted chroma samples in the CU, and rec L ′(i, j) represents the downsampled reconstructed luma samples of the same CU.

[0167] Alternatively, the following equation can be used for CCLM:

[0168]

[0169] The >> operation represents a bit shift to the right by k values.

[0170] The CCLM parameters (α and β) are derived from at most four adjacent chroma samples and their corresponding downsampled luma samples. Assuming the current chroma block size is W×H, W' and H' are set to

[0171] - When LM mode is applied, W'=W, H'=H;

[0172] - When LM-A mode is applied, W'=W+H;

[0173] - When LM-L mode is applied, H'=H+W;

[0174] In this article, LM-A mode refers to Linear Model_Top, in which only the upper template (i.e., the sample values ​​from the adjacent position above the CU) is used to calculate the linear model coefficients. In order to obtain more samples, the upper template is expanded to (W+H). LM-L mode refers to Linear Model_Left, in which only the left template (i.e., the sample values ​​from the adjacent position on the left side of the CU) is used to calculate the linear model coefficients. In order to obtain more samples, the left template is expanded to (H+W). For non-square blocks, the above template is expanded to W+W, and the left template is expanded to H+H.

[0175] The upper adjacent position is represented as S[0,1]…S[W'-1,-1], and the left adjacent position is represented as S[-1,0]…S[-1,H'-1]. Then four samples are selected as

[0176] - When LM mode is applied and both upper and left neighboring samples are available, S[W' / 4,1], S[3*W' / 4,1], S[1,3*H' / 4];

[0177] - When LM-A mode is applied or only upper neighboring samples are available, S[W' / 8,1], S[3*W' / 8,1], S[5*W' / 8,1];

[0178] - When LM-L mode is applied or only left neighbor samples are available, S[-1, H' / 8], S[-1, 3*H' / 8], S[-1, 5*H' / 8], S[-1, 7*H' / 8];

[0179] The four adjacent luma samples at the selected position are downsampled and compared four times to find two smaller values: x0A and x1A, and two larger values: x0B and x1B. The corresponding chroma sample values ​​are denoted as y0A, y1A, y0B, and y1B. Then xA, xB, yA, and yB are derived as follows:

[0180] X a =(x 0 A +x 1 A +1)>>1;X b =(x 0 B +x 1 B +1)>>1;Y a =(y 0 A +y 1 A +1)>>1;

[0181] Y b =(y 0 B +y 1 B +1)>>1 Equation (2)

[0182] Finally, the linear model parameters α and β are obtained according to the following equations:

[0183]

[0184] β=Y b -α·X b Equation (4)

[0185] Figure 5 Examples of the locations of the left and upper samples are shown along with the samples of the current block involved in CCLM mode.

[0186] The division operation to calculate the parameter α is implemented using a lookup table. To reduce the memory required to store the table, the diff value (the difference between the maximum and minimum values) and the parameter α are expressed in exponential notation. For example, diff is approximated as a 4-bit significant part and an exponent. Therefore, the table for 1 / diff is simplified to 16 elements, corresponding to the 16 values ​​of the significant part, as shown below:

[0187] DivTable[] = {0,7,6,5,5,4,4,3,3,2,2,1,1,1,1,0} Equation (5)

[0188] This provides the benefit of both reducing the complexity of the calculations and reducing the memory size required to store the required tables.

[0189] To match the chroma sample positions of a 4:2:0 video sequence, two types of downsampling filters are applied to the luma samples to achieve a 2:1 downsampling ratio in both the horizontal and vertical directions. The choice of downsampling filter is specified by the SPS level flag. The two downsampling filters are as follows, corresponding to "type 0" and "type 2" content, respectively:

[0190]

[0191] Note that when the upper reference line is at a CTU boundary, only one luma line (common line buffer in intra prediction) is used for downsampling luma samples.

[0192] The parameter calculation is performed as part of the decoding process and not just as an encoder search operation. Therefore, no syntax is used to communicate the α and β values ​​to the decoder.

[0193] For chroma intra mode coding, a total of 8 intra modes are allowed for chroma intra mode coding. These modes include five traditional intra modes and three cross-component linear model modes (CCLM mode, LM_A mode, and LM_L mode). The chroma mode signaling and derivation process are shown in Table 1. Chroma mode coding depends directly on the intra prediction mode of the corresponding luminance block. Since the independent block partition structure of luminance and chrominance components is enabled in the I segment, a chroma block can correspond to multiple luminance blocks. Therefore, for the chroma DM mode, the intra prediction mode of the corresponding luminance block covering the center position of the current chroma block is directly inherited.

[0194]

[0195]

[0196] Table 1

[0197] Regardless of the value of sps_cclm_enabled_flag, a single binary table is used, as shown in Table 2.

[0198]

[0199] Table 2

[0200] In Table 2, the first bin indicates whether it is normal mode (0) or LM mode (1). If it is LM mode, the next bin indicates whether it is LM_CHROMA (0). If it is not LM_CHROMA, the next bin indicates whether it is LM_L (0) or LM_A (1). For this case, when sps_cclm_enabled_flag is 0, the first bin of the binary table corresponding to intra_chroma_pred_mode can be discarded before entropy coding. Or, in other words, the first bin is inferred to be 0 and therefore not encoded. This single binary table is used for both cases where sps_cclm_enabled_flag is equal to 0 and equal to 1. The first two bins in Table 2 are context coded with their own context model, and the remaining bins are bypass coded.

[0201] Additionally, to reduce luma-chroma latency in dual trees, when a 64x64 luma coding tree node is split with NotSplit (and the 64x64 CU does not use intra sub-partitioning (ISP)) or QT, the chroma CU in a 32x32 / 32x16 chroma coding tree node is allowed to use CCLM as follows:

[0202] - If a 32x32 chroma node is not split or is split by a partitioned QT, all chroma CUs in the 32x32 node can use CCLM

[0203] - If a 32x32 chroma node is split by horizontal BT and the 32x16 child node is not split or uses vertical BT split, all chroma CUs in the 32x16 chroma node can use CCLM.

[0204] Under all other luma and chroma coding tree split conditions, CCLM is not allowed for chroma CUs.

[0205] Multi-model LM (MMLM)

[0206] The CCLM included in VVC is extended by adding three multi-mode LM (MMLM) modes. In each MMLM mode, the reconstructed neighboring samples are divided into two classes using a threshold value, which is the average value of the brightness reconstructed neighboring samples. The least mean square (LMS) method is used to derive the linear model for each class. For the CCLM mode, the LMS method is also used to derive the linear model. Figure 6a and Figure 6bTwo luminance-chrominance models obtained for a luminance (Y) threshold of 17 in the sampling domain and the spatial domain are illustrated respectively. Each luminance-chrominance model has its own linear model parameters α and β. Figure 6b As can be seen in , each luminance-chrominance model corresponds to a spatial segmentation of the content (i.e., they correspond to different objects or textures in the scene).

[0207] Convolutional Cross-Component Model (CCCM)

[0208] An improved version of cross-component prediction, called CCCM, uses a 2D filter kernel to derive a luma-chroma model. The filter coefficients are derived at the decoder side using the reconstructed input data set and the chroma samples. For the derivation of the filter coefficients, a co-located reference sample region (consisting of the reconstructed luma and chroma samples) is defined for both luma and chroma, as Figure 7 , where the commonly used 4:2:0 chroma downsampling is applied. The reference sample area for a given block can be, for example, Figure 7 Six lines are shown above and to the left, but any number of reference lines may be used (as may be implemented by the encoder and decoder). In general, the reference samples may contain any chrominance and luma samples that have been reconstructed by both the encoder and decoder. Once the reference samples are determined, the filter coefficients may be derived, for example, using different types of linear regression tools such as ordinary least squares estimation, orthogonal matching pursuit, optimized orthogonal matching pursuit, ridge regression, or least absolute shrinkage and selection operators.

[0209] The filter kernels can be of size, for example, 1×3 (vertical 1D), 3×1 (horizontal 1D), 3×3, 7×7, or any size, and can be (by selecting only a subset of all possible kernel positions) cross or diamond shaped (e.g. Figure 8 When referring to samples in the filter kernel, the following notation is used: North (upper), East (right), South (lower), West (left), and Center, as illustrated in FIG. 4 using the letters N, E, S, W, C.

[0210] The general approach of reconstructing chrominance samples using convolution between a filter kernel obtained at the decoder side and an input data set is referred to herein as a convolutional cross-component model (CCCM). The following steps may be applied to perform a CCCM operation:

[0211] 1) Define co-located reference regions on luma and chroma components.

[0212] 2) Downsample the luma samples to match the chroma grid (optional).

[0213] 3) Scan the luminance and chrominance samples of the reference region and collect available statistics (such as autocorrelation matrix and cross-correlation vector) based on the filter shape.

[0214] 4) Solve for the filter coefficients by minimizing the squared error (or any other metric) based on available statistics (e.g., autocorrelation matrix and cross-correlation vector).

[0215] 5) Compute the predicted chroma block by convolving the downsampled luma samples with the filter kernel.

[0216] Let us define the (possibly downsampled) luma samples as a 2D array Y(x,y) indexed using a horizontal x-coordinate and a vertical y-coordinate. Let us also define the co-located chroma samples as a 2D matrix C(x,y) and the filter kernel (i.e. coefficients) as a 3×3 matrix F(i,j). At the sample level, we define the convolution between Y and F as:

[0217]

[0218] When other data terms are used (such as nonlinear square root terms), the additional convolution becomes,

[0219]

[0220] in, The filter coefficients that lie outside the 2D filter kernel have been obtained as part of the system of linear equations used to solve the 2D filter coefficients in step 4 above. Similarly, we can add a bias term to the convolution,

[0221]

[0222] Multiple Reference Line (MRL) Intra Prediction

[0223] Multiple reference line (MRL) intra prediction uses more reference lines for intra prediction. Fig. 9 In Figure 1, an example of 4 reference lines is depicted, where the samples of fragments A and F are not taken from the reconstructed neighboring samples, but are filled with the closest samples from fragments B and E, respectively. HEVC intra-image prediction uses the nearest reference line (i.e., reference line 0). In MRL, two additional lines are used (reference line 1 and reference line 3).

[0224] The index of the selected reference line (mrl_idx) is signaled and used to generate intra prediction values. For reference line idx greater than 0, only additional reference line modes are included in the MPM list, and only the MPM index is signaled, not the remaining modes. The reference line index is signaled before the intra prediction mode, and if a non-zero reference line index is signaled, planar mode is excluded from the intra prediction mode.

[0225] To prevent the use of extended reference samples outside the current CTU row, MRL is disabled for the first row of blocks within a CTU. In addition, PDPC is disabled when additional lines are used. For MRL mode, the derivation of the DC value in DC intra prediction mode for non-zero reference line index is consistent with the derivation for reference line index 0. MRL requires storage of 3 adjacent luma reference lines and CTUs to generate predictions. The CCLM tool also requires 3 adjacent luma reference lines for its downsampling filter. MLR uses the same 3-row definition consistent with CCLM to reduce storage requirements for the decoder.

[0226] Intra-frame sub-partition

[0227] Intra sub-partitioning (ISP) divides the luma intra prediction block vertically or horizontally into 2 or 4 sub-partitions depending on the block size. For example, the minimum block size of ISP is 4x8 (or 8x4). If the block size is larger than 4x8 (or 8×4), the corresponding block is divided into 4 sub-partitions. It has been noted that M×128 (M≤64) and 128×N (N≤64) ISP modules may cause potential problems for 64×64 VDPU. For example, an M×128 CU in a single tree structure has an M×128 luma TB and two corresponding M / 2×64 chroma TBs. If the CU uses ISP, the luma TB will be divided into four M×32 TBs (only horizontal partitioning is possible), each of which is smaller than a 64×64 block. However, in the current ISP design, the chroma blocks are not divided. Therefore, both chroma components have a size larger than a 32×32 block. Similarly, using ISP, a 128×N CU can also create a similar situation. Therefore, these two cases are problems for the 64×64 decoder pipeline. For this reason, the CU size that can use ISP can be limited to a maximum of 64×64. All sub-partitions meet the condition of having at least 16 samples.

[0228] Matrix Weighted Intra Prediction (MIP)

[0229] The matrix weighted intra prediction (MIP) method is a new intra prediction technique added to VVC. In order to predict the samples of a rectangular block with a width of W and a height of H, the matrix weighted intra prediction (MIP) takes a row of H reconstructed adjacent boundary samples on the left side of the block and a row of W reconstructed adjacent boundary samples above the block as input. If the reconstructed samples are not available, the reconstructed samples are generated according to the traditional intra prediction mode. The generation of prediction signaling is based on the following three steps, where Fig.10 Shown are averaging, matrix-vector multiplication, and linear interpolation.

[0230] Inter-frame prediction in VVC

[0231] The merge list may include the following candidates:

[0232] a) Spatial MVP from spatially neighboring CUs

[0233] b) Temporary MVP from the same CU

[0234] c) Based on historical MVP in FIFO table

[0235] d) Pairwise average MVP (using candidates already in the list)

[0236] e) Zero MV.

[0237] Merge Mode Wide Motion Vector Difference (MMVD) means that MVD and resolution index signaling are sent after merge candidate signaling. In symmetric MVD, the motion information of list 1 is derived from the motion information of list 0 in case of bi-directional prediction.

[0238] In affine prediction, several motion vectors are indicated / signaled for different corners of a block for deriving the motion vector of a sub-block. In affine merging, the affine motion information of a block is generated based on the normal or affine motion information of neighboring blocks.

[0239] In sub-block based temporal motion vector prediction, the motion vector of a sub-block of the current block is predicted from an appropriate sub-block in a reference frame, which is indicated by the motion vector of a spatially neighboring block (if available).

[0240] In Adaptive Motion Vector Resolution (AMVR), the MVD precision is sent for each CU.

[0241] In CU-level bidirectional weighted prediction, the index represents the weight value of the weighted average of the two prediction blocks. Bidirectional optical flow (BDOF) refines the motion vector in the case of bidirectional prediction. BDOF generates two prediction blocks using the motion vector signaled. Then, motion refinement is calculated using the gradient values ​​of the two prediction blocks to minimize the error between the two prediction blocks. The final prediction block is refined using motion refinement and gradient values.

[0242] Therefore, cross-component intra prediction tools (such as CCLM and CCCM) predict samples in chrominance channels (e.g., Cb and Cr) using linear models based on luma samples. Both use least squares minimization to derive model parameters for luma-chrominance mapping. Such models can have any number of inputs, such as co-located luma samples, a 1D set of neighboring luma samples, or a 2D set of neighboring luma samples. All models can also be augmented with additional inputs, such as biases and / or nonlinear terms (e.g., quadratic functions of luma samples). The luma-chrominance model parameters θ can be expressed in matrix notation as:

[0243] θ=(X T X) -1 (XT y)

[0244] Where (X T X) corresponds to the autocorrelation matrix, and X T y is the cross-correlation vector with X containing the regressors (or inputs), i.e., the luma samples co-located and downsampled on the reference region, and y contains the corresponding chroma samples on the reference region. Thus, the actual model maps luma samples to chroma samples with a certain error bound e,

[0245] y=Xθ+e

[0246] We can further represent the entries of vector e as,

[0247] e i =y i -X i θ

[0248] Where i represents the row index in the entire matrix equation, and the model parameters θ are selected to minimize the sum of squared errors The prediction of the target chrominance block can be expressed as,

[0249]

[0250] where hat symbols are used to denote luminance, chrominance, and corresponding errors belonging to a sample set different from the one used in the derivation of the model parameters.

[0251] However, in the above parameter derivation, X and There may be significant differences in the luminance intensity distributions between X and X. Therefore, prediction errors for luminance intensities that perform poorly in the distribution of X usually have larger magnitudes, and vice versa. The model thus becomes overfit to the most likely luminance intensity in X, and may underfit other luminance intensities. However, the target chrominance patches may have significantly different luminance intensity distributions compared to the reference set, resulting in poor predictions for the chrominance patches.

[0252] An improved method for implementing cross-component prediction is now presented, which reduces the prediction error between luma and chroma samples.

[0253] According to one aspect of the method Fig.11, wherein the method includes receiving (1100) an image block unit of a frame, the image block unit including a plurality of samples in color channels, the color channels including at least one chrominance channel and one luminance channel; reconstructing (1102) samples of the luminance channel of the image block unit; determining (1104) a reference region for predicting target samples of at least one color channel of the image block unit, wherein the reference region includes one or more reference samples in adjacent blocks in a current color channel / frame, one or more reference samples adjacent to a co-located block in a reference color channel / frame, and / or one or more reference samples within a co-located block in a reference color channel / frame; determining (1106) a weight for predicting the target sample based on a ratio between a normalized luminance histogram in the reference region and a normalized luminance histogram of the co-located target sample; determining (1108) filter coefficients of the filter used for prediction based on the weights, the one or more reference samples and a shape of the filter; and predicting (1110) samples of at least one color channel of the image block unit based on the samples of the luminance channel and the filter coefficients.

[0254] Therefore, the method predicts chrominance samples using cross-component prediction between the input data set and the filter kernel derived on the decoder side. The shape of the filter and the number of coefficients are agreed upon by both the encoder and the decoder. The input data for cross-component prediction can be specified in a wide variety of ways and can include, for example, luma samples co-located with chroma samples, adjacent co-located luma samples, or any other configuration of luma samples. For the derivation of filter coefficients, a co-located reference sample region comprising reconstructed luma and chroma samples is defined for prediction of both luma and chroma, where the reference samples can contain any chroma and luma samples that have been reconstructed by both the encoder and the decoder.

[0255] In order to find better matching model parameters, the same The brightness sample distribution of is very close to the brightness sample distribution of. Therefore, weighted least squares is minimized, where the error term e i are appropriately weighted during parameter derivation so that the overall effect mimics that of using Derivation of the parameters of the target distribution.

[0256] The filter coefficients are derived based on the weights, reference samples and the shape of the filter kernel. Finally, the predicted chroma block can be calculated by applying the luma samples to the filter kernel.

[0257] According to an embodiment, determining the weight comprises calculating a ratio between a normalized brightness histogram in the reference region and a normalized brightness histogram of a sample co-located with the target sample.

[0258] Therefore, the weight can be derived from the ratio of the normalized intensity counts between the reference luminance block and the target luminance block. The normalized intensity counts correspond to a normalized histogram representing the probability of each intensity value. For example, if the probability of luminance value 543 in the reference region is twice that of the target block, then when X i，0 = = 543, we should i Assign a weight of 0.5, where X i，0 are the brightness samples used to obtain the normalized histogram (here the first column of X) and the double equal sign indicates logical equality.

[0259] The general weighting scheme for all i can be obtained by deriving a weight vector w, whose entries are defined as:

[0260]

[0261] where 1 is added to avoid division by 0, and where p represents the normalized brightness histogram of the reference sample, defined as,

[0262]

[0263] N defines the number of luma reference samples, and 1023 illustrates the case of 10-bit content.

[0264] Similarly, represents the normalized luminance histogram of co-located luminance samples on the target chrominance block, where Normalization is performed using the total number of co-located luminance samples corresponding to a given block. Note that N and The division of , because ignoring the normalization operation is equivalent to w i Multiply Therefore, it is the ratios between the different entries in w that are meaningful, not their overall scale.

[0265] The weighted model parameters can be derived by multiplying each row of the model by the corresponding weight. In matrix notation, the weighted model parameters become,

[0266] θ=(X T WX) -1 (X T Wy)

[0267] where W is composed of a single entry w i Nx1 column vector.

[0268] According to an embodiment, the method comprises determining the same weight for a plurality of adjacent target samples.

[0269] Therefore, a band weighting scheme can be used instead of the instantaneous sample-level weighting scheme as follows,

[0270]

[0271] M is, for example, 8 or 16. In many practical scenarios, especially for natural images and videos, the band weighting scheme gives more robust results compared to the temporal weighting.

[0272] According to an embodiment, the method comprises deriving weights based on a temporal histogram, each entry of the temporal histogram corresponding to a single brightness intensity value.

[0273] According to an embodiment, a method comprises deriving weights based on a strip histogram, wherein a plurality of brightness intensity values ​​are grouped into a single entry.

[0274] Therefore, the method enables mitigation of possible overfitting or underfitting due to distribution deviation by using weighted least squares minimization, where the weights of the sample-level or band-level granularity are derived from the ratio between two sets of normalized brightness histograms corresponding to the reference region and the target block region.

[0275] According to an embodiment, the weighted model parameter derivation scheme may be used with any cross-component prediction scheme that exploits the correlation between reference luma and reference chroma samples.

[0276] According to an embodiment, the method comprises receiving signaling from an encoder to apply weights to a subset of an image or an image block.

[0277] Therefore, weighted model parameter derivation may be activated only on specific frames, slices, CTUs or PU sets, using (on / off) signaling from the encoder to the decoder, for example using entropy coding or fixed-length coding.

[0278] According to an embodiment, weighted model parameter derivation may be used in conjunction with a multi-model CCLM or a multi-model CCCM.

[0279] In the following, embodiments are disclosed related to various implementation options for a decoder to derive and / or apply various parameters related to a prediction model.

[0280] Depending on the embodiment, the weights may be calculated using floating point or fixed point arithmetic.

[0281] Depending on the embodiment, the weights may be applied using floating point or fixed point multiplication.

[0282] According to an embodiment, the normalized histogram and / or weights may be calculated using integer arithmetic without division using an appropriately configured lookup table for obtaining the exponential representation of the divisor.

[0283] Depending on the embodiment, the actual model may have any number of inputs (ie, filter taps), and these inputs may correspond to any available spatial luma neighbors, bias terms, non-linear terms, or any other terms available at the decoder.

[0284] According to an embodiment, the weights may be obtained using a normalized histogram of co-located luma samples, down-sampled luma samples, or original resolution luma samples.

[0285] According to an embodiment, the histogram may or may not be normalized before calculating the weight w.

[0286] According to an embodiment, the luma samples may be pre-processed by some components (such as linear or non-linear filters) before weight derivation.

[0287] According to an embodiment, the number of bands used in the band weighting may be signaled from the encoder to the decoder in a sequence, frame, slice, CTU, or PU manner.

[0288] According to an embodiment, the use of band weighting may be signaled from the encoder to the decoder in a sequence, frame, slice, CTU or PU manner.

[0289] Depending on the embodiment, the weight w may be scaled by any factor deemed necessary due to the numerical stability or accuracy of the chosen digital representation and precision requirements.

[0290] Depending on the embodiment, the weights w may be inferred from neighboring blocks, or may be taken from a sequence, picture, slice or CTU level lookup table based on some decoder side parameterization such as block size, color components or reference sample properties.

[0291] According to one aspect, an apparatus includes a component for receiving an image block unit of a frame, the image block unit including multiple samples in color channels, the color channels including at least one chrominance channel and one luminance channel; a component for reconstructing samples of the luminance channel of the image block unit; a component for determining a reference area for predicting a target sample of at least one color channel of the image block unit, wherein the reference area includes one or more reference samples in a neighboring block in a current color channel / frame, one or more reference samples neighboring a co-located block in a reference color channel / frame, and / or one or more reference samples within a co-located block in a reference color channel / frame; a component for determining a weight for predicting the target sample based on a ratio between a normalized luminance histogram in the reference area and a normalized luminance histogram of the co-located target sample; a component for determining filter coefficients of a filter for the prediction based on the weights, one or more reference samples, and a shape of the filter; and a component for predicting samples of at least one color channel of the image block unit based on the samples of the luminance channel and the filter coefficients.

[0292] According to an embodiment, said means for determining the weight comprises means for calculating a ratio between a normalized brightness histogram in a reference region and a normalized brightness histogram of a sample co-located with said target sample.

[0293] According to an embodiment, said means for determining the weights comprise means for deriving the weights based on a temporal histogram, each entry of the temporal histogram corresponding to a single brightness intensity value.

[0294] According to an embodiment, said means for determining weights comprises means for determining the same weights for a plurality of adjacent target samples.

[0295] According to an embodiment, said means for determining the weights comprises means for deriving the weights based on a strip histogram, wherein a plurality of brightness intensity values ​​are grouped into a single entry.

[0296] According to an embodiment, the apparatus comprises means for receiving signaling from an encoder to apply weights to a subset of images or a subset of image blocks.

[0297] According to an embodiment, the apparatus comprises means for deriving weights using a normalized histogram of co-located luma samples, down-sampled luma samples or original resolution luma samples.

[0298] According to an embodiment, the device comprises means for pre-processing the luma samples by a linear or non-linear filter before the weight derivation.

[0299] According to an embodiment, the apparatus comprises means for computing the normalized histogram and / or the weights by division-free integer operations using a properly configured lookup table for obtaining an exponential representation of the divisor.

[0300] As another aspect, a device is provided, comprising: at least one processor and at least one memory, wherein the at least one memory stores codes, and when the at least one processor executes the codes, the device is caused to at least perform the following steps: receiving an image block unit of a frame, the image block unit comprising a plurality of samples in a color channel, the color channel comprising at least one chroma channel and one luminance channel; reconstructing samples of a luminance channel of the image block unit; determining a reference area for predicting a target sample of at least one color channel of the image block unit, wherein the reference area comprises one or more reference samples in an adjacent block in a current color channel / frame, one or more reference samples adjacent to a co-located block in a reference color channel / frame, and / or one or more reference samples within a co-located block in a reference color channel / frame; determining a weight for predicting the target sample based on a ratio between a normalized luminance histogram in the reference area and a normalized luminance histogram of the co-located target sample; determining a filter coefficient of a filter for prediction based on the weight, the one or more reference samples, and a shape of the filter; and predicting samples of at least one color channel of the image block unit based on the samples of the luminance channel and the filter coefficient.

[0301] According to an embodiment, the apparatus comprises code which, when executed by the at least one processor, causes the apparatus to calculate a ratio between a normalized brightness histogram in the reference region and a normalized brightness histogram of a sample co-located with the target sample.

[0302] According to an embodiment, the apparatus comprises code which, when executed by the at least one processor, causes the apparatus to derive weights based on a temporal histogram, each entry of the temporal histogram corresponding to a single brightness intensity value.

[0303] According to an embodiment, the apparatus comprises code which, when executed by the at least one processor, causes the apparatus to determine a same weight for a plurality of adjacent target samples.

[0304] According to an embodiment, the apparatus comprises code which, when executed by the at least one processor, causes the apparatus to derive weights based on a strip histogram, wherein a plurality of brightness intensity values ​​are grouped into a single entry.

[0305] According to an embodiment, the apparatus comprises code which, when executed by the at least one processor, causes the apparatus to receive signaling from an encoder to apply weights to a subset of images or a subset of image blocks.

[0306] According to an embodiment, the apparatus comprises code which, when executed by the at least one processor, causes the device to derive weights using a normalized histogram of co-located luma samples, down-sampled luma samples, or original resolution luma samples.

[0307] According to an embodiment, the apparatus comprises code which, when executed by the at least one processor, causes the apparatus to pre-process the luma samples by a linear or non-linear filter before the weight derivation.

[0308] According to an embodiment, the apparatus comprises code which, when executed by the at least one processor, causes the apparatus to calculate the normalized histogram and / or weights by division-free integer operations using a properly set lookup table for obtaining an exponential representation of a divisor.

[0309] These devices may include functional units for implementing the embodiments (e.g., any Figure 1 , Figure 2 , Figure 4a and Figure 4b public).

[0310] Such an apparatus also includes code stored in at least one memory, which, when executed by at least one processor, causes the apparatus to perform one or more embodiments disclosed herein.

[0311] Fig.12 15 is a graphical representation of an example multimedia communication system in which various embodiments can be implemented. A data source 1510 provides source signaling in an analog, uncompressed digital or compressed digital format or any combination of these formats. An encoder 1520 may include preprocessing such as data format conversion and / or filtering of source signaling or be connected to the preprocessing. Encoder 1520 encodes source signaling into a coded media bitstream. It should be noted that the bitstream to be decoded can be received directly or indirectly from a remote device located in almost any type of network. Additionally, the bitstream can be received from local hardware or software. Encoder 1520 may be able to encode more than one media type, such as audio and video, or may require more than one encoder 1520 to encode different media types of source signaling. Encoder 1520 may also obtain synthetically generated inputs, such as graphics and text, or it may be able to generate a coded bitstream of synthetic media. In the following, only the processing of a coded media bitstream of a media type is considered to simplify the description. However, it should be noted that a real-time broadcast service generally includes multiple streams (generally at least one audio, video and text subtitle stream). It should also be noted that the system may include multiple encoders, but only one encoder 1520 is shown in the figure to simplify the description without lack of generality. It should be further understood that although the text and examples contained herein may specifically describe the encoding process, those skilled in the art will understand that the same concepts and principles also apply to the corresponding decoding process, and vice versa.

[0312] The coded media bitstream may be transmitted to a storage device 1530. The storage device 1530 may include any type of mass storage for storing coded media bitstreams. The format of the coded media bitstream in the storage device 1530 may be a basic self-contained bitstream format, or one or more coded media bitstreams may be encapsulated into a container file. If one or more media bitstreams are encapsulated in a container file, a file generator (not shown in the figure) may be used to store one or more media bitstreams in a file and create file format metadata that may also be stored in the file. The encoder 1520 or the storage device 1530 may include a file generator, or the file generator may be operably attached to the encoder 1520 or the storage device 1530. Some systems operate "live", i.e., storage is omitted and the coded media bitstream is transmitted directly from the encoder 1520 to the transmitter 1540. Based on the need, the coded media bitstream may then be transmitted to the transmitter 1540 (also referred to as a server). The format used in the transmission may be a basic self-contained bitstream format, a packet stream format, a segment format suitable for DASH (or a similar streaming system), or one or more coded media bitstreams may be encapsulated into a container file. The encoder 1520, storage device 1530, and server 1540 may be located in the same physical device, or they may be included in separate devices. The encoder 1520 and server 1540 may operate with live real-time content, in which case the coded media bitstream is typically not stored permanently, but is cached for a short period of time in the content encoder 1520 and / or server 1540 to smooth out processing delays, transmission delays, and variations in the coded media bitrate.

[0313] The server 1540 uses a communication protocol stack to send the coded media bitstream. The stack may include, but is not limited to, one or more of a real-time transport protocol (RTP), a user datagram protocol (UDP), a hypertext transfer protocol (HTTP), a transmission control protocol (TCP), and an Internet protocol (IP). When the communication protocol stack is packet-oriented, the server 1540 encapsulates the coded media bitstream into packets. For example, when RTP is used, the server 1540 encapsulates the coded media bitstream into RTP packets according to the RTP payload format. Typically, each media type has a dedicated RTP payload format. It should be noted again that the system may include multiple servers 1540, but for simplicity, the following description only considers one server 1540.

[0314] If the media content is encapsulated in a container file for storage device 1530 or for inputting data to transmitter 1540, transmitter 1540 may include or be operably attached to a "send file parser" (not shown in the figure). In particular, if the container file is not sent as such, but at least one of the contained coded media bitstreams is encapsulated for transmission via a communication protocol, the send file parser locates the appropriate portion of the coded media bitstream to be transmitted via the communication protocol. The send file parser can also help create a correctly formatted communication protocol, such as packet headers and payloads. The multimedia container file may contain encapsulation instructions, such as a hint track in ISOBMFF, for encapsulating at least one of the contained media bitstreams on the communication protocol.

[0315] Server 1540 may or may not be connected to gateway 1550 via a communication network, which may be, for example, a CDN, the Internet, and / or a combination of one or more access networks. A gateway may also or alternatively be referred to as a middlebox. For DASH, a gateway may be an edge server (of a CDN) or a network proxy. Note that a system may typically include any number of gateways, etc., but for simplicity, the following description only considers one gateway 1550. Gateway 1550 may perform different types of functions, such as converting a packet stream from one communication protocol stack to another communication protocol stack, merging and diverting data streams, and processing data streams according to downlink and / or receiver capabilities, such as controlling the bit rate of a forwarded stream according to a dominant downlink network condition. In various embodiments, gateway 1550 may be a server entity.

[0316] The system includes one or more receivers 1560, which are generally capable of receiving, demodulating and decapsulating the transmitted signaling into a coded media bitstream. The coded media bitstream may be transmitted to a recording storage device 1570. The recording storage device 1570 may include any type of large capacity storage for storing the coded media bitstream. The recording storage device 1570 may alternatively or additionally include a computing memory, such as a random access memory. The format of the coded media bitstream in the recording storage device 1570 may be a basic self-contained bitstream format, or one or more coded media bitstreams may be encapsulated into a container file. If there are multiple coded media bitstreams associated with each other (such as audio streams and video streams), a container file is generally used, and the receiver 1560 includes a container file generator that generates a container file from an input stream or is attached to the container file generator. Some systems operate "live", i.e., the recording storage device 1570 is omitted and the coded media bitstream is transmitted directly from the receiver 1560 to the decoder 1580. In some systems, only the most recent portion of the recorded stream, such as an excerpt of the most recent 10 minutes of recorded stream, is retained in the recording storage 1570 , while any earlier recorded data is discarded from the recording storage 1570 .

[0317] The coded media bitstream may be transferred from the recording storage device 1570 to the decoder 1580. If there are multiple coded media bitstreams such as audio streams and video streams that are associated with each other and encapsulated into a container file, or a single media bitstream is encapsulated in a container file, for example, for easier access, a file parser (not shown in the figure) is used to decapsulate each coded media bitstream from the container file. The recording storage device 1570 or the decoder 1580 may include the file parser, or the file parser is attached to the recording storage device 1570 or the decoder 1580. It should also be noted that the system may include multiple decoders, but only one decoder 1580 is discussed here to simplify the description without lack of generality.

[0318] The coded media bitstream may be further processed by a decoder 1580, the output of which is one or more uncompressed media streams. Finally, a renderer 1590 may reproduce the uncompressed media streams, for example, using a speaker or display. The receiver 1560, the recording storage 1570, the decoder 1580, and the renderer 1590 may be located in the same physical device, or they may be included in separate devices.

[0319] The sender 1540 and / or the gateway 1550 may be configured to perform switching between different representations, for example, for switching between different viewports of 360-degree video content, view switching, bitrate adaptation, and / or fast start, and / or the sender 1540 and / or the gateway 1550 may be configured to select (multiple) representations for transmission. Switching between different representations may occur for a variety of reasons, such as in response to a request from the receiver 1560 or a prevailing condition of the network transmitting the bitstream, such as throughput. In other words, the receiver 1560 may initiate switching between representations. The request from the receiver may be, for example, a request for a segment or sub-segment from a different representation than previously, a request for a change in the scalability layer and / or sub-layer of the transmission, or a change in a rendering device having different capabilities compared to the previous scalability layer and / or sub-layer. The request for the data segment may be an HTTP GET request. The request for the sub-segment may be an HTTP GET request with a byte range. Additionally or alternatively, bitrate adjustment or bitrate adaptation may be used, for example, in streaming services to provide so-called fast start, where the bitrate of the transport stream is lower than the channel bitrate after starting or randomly accessing the stream, so as to start playback immediately and reach a buffer occupancy level that tolerates occasional packet delays and / or retransmissions. Bitrate adaptation may include multiple representation or layer up-switching and representation or layer down-switching operations occurring in various orders.

[0320] The decoder 1580 may be configured to perform switching between different representations, such as for switching between different viewports of 360-degree video content, view switching, bitrate adaptation, and / or fast start, and / or the decoder 1580 may be configured to select (multiple) representations for transmission. Switching between different representations may occur for a variety of reasons, such as to achieve faster decoding operations or to adapt the transmitted bitstream (e.g., in terms of bitrate) to the prevailing conditions of the network transmitting the bitstream, such as throughput. For example, if a device including the decoder 1580 is multitasking and uses computing resources for purposes other than decoding a video bitstream, faster decoding operations may be required. In another example, faster decoding operations may be required when the content is played at a faster speed than normal playback speed, such as two or three times faster than conventional real-time playback speed.

[0321] In the above, some embodiments have been described with reference to and / or using the terminology of HEVC and / or VVC. It should be understood that the embodiments can be similarly implemented with any video encoder and / or video decoder.

[0322] In the above, where example embodiments have been described with reference to an encoder, it is to be understood that the generated bitstream and decoder may have corresponding elements therein. Similarly, where example embodiments have been described with reference to a decoder, it is to be understood that the encoder may have a structure and / or computer program for generating a bitstream to be decoded by a decoder. For example, some embodiments have been described in relation to generating a prediction block as part of encoding. Embodiments may similarly be implemented by generating a prediction block as part of decoding, with the difference that encoding parameters (such as horizontal offset and vertical offset) are decoded from the bitstream rather than determined by the encoder.

[0323] The embodiments of the present invention described above describe the codec in terms of separate encoder and decoder devices to help understand the processes involved. However, it should be understood that the device, structure and operation can be implemented as a single encoder-decoder device / structure / operation. In addition, the encoder and decoder can share some or all common elements.

[0324] Although the above examples describe embodiments of the present invention operating within a codec within an electronic device, it should be understood that the present invention defined in the claims can be implemented as part of any video codec. Thus, for example, embodiments of the present invention can be implemented in a video codec that can implement video encoding via a fixed or wired communication path.

[0325] Hence, the user equipment may comprise a video codec such as described above in embodiments of the invention.It should be understood that the term user equipment is intended to cover any suitable type of wireless user equipment, such as a mobile phone, a portable data processing device or a portable web browser.

[0326] Furthermore, elements of a public land mobile network (PLMN) may also include a video codec as described above.

[0327] In general, various embodiments of the present invention may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Although various aspects of the present invention may be shown and described as block diagrams, flow charts, or using some other graphical representations, it is well understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof, as non-limiting embodiments.

[0328] Embodiments of the present invention may be implemented by computer software executable by a data processor of a mobile device (such as in a processor entity), or by hardware, or by a combination of software and hardware. Further in this regard, it should be noted that any box of the logic flow as in the accompanying drawings may represent program steps or interconnected logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on physical media such as memory chips, memory blocks implemented within a processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and their data variants CDs.

[0329] The memory may be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based storage devices, magnetic storage devices and systems, optical storage devices and systems, fixed memory, and removable memory. The data processor may be of any type suitable for the local technical environment and may include one or more general-purpose computers, special-purpose computers, microprocessors, digital signaling processors (DSPs), and processors based on multi-core processor architectures, as non-limiting examples.

[0330] Embodiments of the present invention may be practiced in various components such as integrated circuit modules. The design of integrated circuits is generally a highly automated process. Complex and powerful software tools are available to convert a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

[0331] Programs such as those provided by Synopsys, Inc. of Mountain View, Calif., and Cadence Design, Inc. of San Jose, Calif., use well-established design rules and a library of pre-stored design modules to automatically route conductors and position components on a semiconductor chip. After the design of a semiconductor circuit has been completed, the final design in a normalized electronic format (e.g., Opus, GDSII, etc.) can be transferred to a semiconductor manufacturing facility or "fab" for fabrication.

[0332] The above description has provided a comprehensive and informative description of exemplary embodiments of the present invention by way of exemplary and non-limiting examples. However, when read in conjunction with the accompanying drawings and the appended claims, various modifications and adaptations may become apparent to those skilled in the relevant art in view of the foregoing description. However, all such and similar modifications of this teaching of the present invention will still fall within the scope of the present invention.

Claims

1. A device comprising means for receiving an image block unit of a frame, the image block unit comprising a plurality of samples in color channels, the color channels comprising at least one chrominance channel and one luma channel; means for reconstructing samples of said luminance channel of said image block unit; means for determining a reference region for predicting target samples of at least one color channel of the image block unit, wherein the reference region comprises: one or more reference samples in a neighboring block in a current color channel / frame, one or more reference samples neighboring a co-located block in a reference color channel / frame, and / or one or more reference samples within the co-located block in a reference color channel / frame; means for determining a weight for predicting the target sample based on a ratio between a normalized brightness histogram in the reference region and a normalized brightness histogram co-located with the target sample; means for determining filter coefficients of said filter for said prediction based on said weights, said one or more reference samples and a shape of the filter; and Means for predicting samples of at least one color channel of the image block unit based on the samples of the luma channel and the filter coefficients.

2. The apparatus according to claim 1, wherein the means for determining the weight comprises Means for calculating said ratio between said normalized brightness histogram in said reference region and a normalized brightness histogram of said sample co-located with said target sample.

3. The apparatus according to claim 1 or 2, wherein the means for determining the weight comprises Means for deriving said weights based on a momentary histogram, each entry of said momentary histogram corresponding to a single luminance intensity value.

4. The apparatus according to claim 1 or 2, wherein the means for determining the weight comprises Means for determining said same weight for a plurality of adjacent target samples.

5. The apparatus according to claim 4, wherein the means for determining the weight comprises Means for deriving said weights based on a band histogram wherein a plurality of luminance intensity values ​​are grouped into a single entry.

6. A device according to any preceding claim, comprising Means for receiving signaling from an encoder to apply said weights to a subset of images or a subset of image blocks.

7. A device according to any preceding claim, comprising Means for deriving said weights using a normalized histogram of co-located luma samples, downsampled luma samples, or original resolution luma samples.

8. A device according to any preceding claim, comprising Means for pre-processing said luma samples through a linear or non-linear filter prior to weight derivation.

9. A device according to any preceding claim, comprising Means for computing said normalized histogram and / or weights by division-free integer arithmetic using a suitably configured lookup table for obtaining an exponential representation of said divisor.

10. A method comprising receiving an image block unit of a frame, the image block unit comprising a plurality of samples in a color channel, the color channel comprising at least one chroma channel and one luma channel; Reconstructing samples of the brightness channel of the image block unit; Determine a reference region for predicting a target sample of at least one color channel of the image block unit, wherein the reference region comprises one or more reference samples in a neighboring block in a current color channel / frame, one or more reference samples neighboring a co-located block in a reference color channel / frame, and / or one or more reference samples within the co-located block in a reference color channel / frame; determining a weight for predicting the target sample based on a ratio between a normalized brightness histogram in the reference region and a normalized brightness histogram co-located with the target sample; determining filter coefficients of the filter for the prediction based on the weights, the one or more reference samples, and a shape of the filter; as well as Samples of at least one color channel of the image block unit are predicted based on the samples of the luma channel and the filter coefficients.

11. The method according to claim 10, comprising: The ratio between the normalized brightness histogram in the reference region and the normalized brightness histogram of the sample co-located with the target sample is calculated.

12. The method according to claim 10 or 11, comprising: The weights are derived based on a momentary histogram, each entry of which corresponds to a single brightness intensity value.

13. The method according to claim 10 or 12, comprising: The same weight is determined for a plurality of adjacent target samples.

14. The method according to claim 13, comprising: The weights are derived based on a band histogram, where multiple brightness intensity values ​​are grouped into a single entry.

15. The method according to any one of claims 10 to 14, comprising Signaling is received from an encoder to apply the weights to a subset of images or a subset of image blocks.

Images