Apparatus, method and computer program for video encoding and decoding

Abstract

A method comprising: receiving an image block unit of a frame, the image block unit comprising samples in a first chroma channel, a second chroma channel, and one luma channel; defining a co-located reference area on the luminance component of the luminance channel, the chrominance component of the first chrominance channel and the chrominance component of the second chrominance channel; performing motion-compensated prediction on the luma component, the first chroma component, and the second chroma component using the reference frame and the motion vector; obtaining a first mapping function and a second mapping function using co-located luma prediction and chroma prediction, the first mapping function mapping the luma component into the first chroma component, and the second mapping function mapping the luma component into the second chroma component; reconstructing a brightness residual error; and obtaining a first chroma prediction by using the first function and the luminance residual, and obtaining a second chroma prediction by using the second function and the luminance residual.

Description

Technical Field

[0001] The present invention relates to apparatus, methods and computer programs for video encoding and decoding. Background Art

[0002] In video coding, video and image samples are typically coded using a color representation such as YUV or YCbCr consisting of one luma and two chroma channels. In these cases, the luma channel, which primarily represents the illumination of the scene, is typically coded at a certain resolution, while the chroma channels, which generally represent the differences between certain color components, are typically coded at a second resolution lower than the resolution of the luma signal. The intent of this differential representation is to decorrelate the color components and enable more efficient compression of the data.

[0003] In the Universal Video Coding (VVC / H.266) standard, a cross-component linear model (CCLM) is used as a linear model for predicting samples in chroma channels (e.g., Cb and Cr). Model parameters are derived based on reconstructed samples in the neighborhood of the chroma block, co-located neighboring samples in the luma block, and reconstructed samples within the co-located luma block. An improved version of cross-component prediction, called CCCM, uses a two-dimensional (2D) filter kernel to derive a luma-to-chroma model. The filter coefficients are derived on the decoder side using the input data and a reconstructed set of chroma samples.

[0004] Inter-coding in modern video coding standards such as H.266 uses single-tree motion compensated prediction: regions from one or more reference frames are copied to the target frame using sub-pixel interpolation. To take into account changes in illumination and / or color differences between the reference and target frames, additional tools such as weighted prediction (WP), bidirectional prediction with CU-level weights (BCW), combined inter- and intra-frame prediction (CIIP), or local illumination compensation (LIC) are used. Single-tree coding signals a set of partitioning information per block, and the partitioning is shared between luma and chroma components. Due to the small amount of chroma-specific coding tools, inter-coding usually implicitly sets a higher priority for luma blocks than for chroma. Even if chroma-specific tools such as CCCM or CCLM are used, they will not be very effective because chroma blocks in reference frames are usually of higher quality than intra-predicted cross-component blocks. In order to benefit from cross-component tools in inter prediction, there needs to be a way to model chroma using mainly co-located motion-compensated luma and chroma. Summary of the invention

[0005] Now to at least alleviate the above problems, this paper introduces an enhanced method for achieving better cross-component residual models for prediction.

[0006] The scope of protection sought by various embodiments of the present invention is set out by the independent claims. Embodiments and features described in this specification that do not belong to the scope of the independent claims (if any) are to be construed as examples useful for understanding various embodiments of the present invention.

[0007] According to some embodiments, a method for directly mapping luma residuals into chroma predictions is provided. Specifically, the mapping can be applied to single-tree inter-coded blocks that share the same partitioning between luma blocks and chroma blocks. An intuitive way to perform the mapping after decoding the luma residuals of a single-tree inter block is also disclosed. The method can also be extended to dual-tree blocks or any other partitioning scheme. Additional methods for improving the robustness of the mapping process are also provided. Alternative configurations such as chroma-to-chroma prediction are also described.

[0008] According to a first aspect, a method includes: receiving an image block unit of a frame, the image block unit including samples in a first chroma channel, a second chroma channel and a luma channel; defining a co-located reference area on a luma component of the luma channel, a chroma component of the first chroma channel and a chroma component of the second chroma channel; performing motion-compensated prediction on the luma component, the first chroma component and the second chroma component using a reference frame and a motion vector; obtaining a first mapping function and a second mapping function using the co-located luma prediction and chroma prediction, the first mapping function mapping the luma component to the first chroma component and the second mapping function mapping the luma component to the second chroma component; reconstructing a luma residual; and obtaining a first chroma prediction by using the first function and the luma residual, and obtaining a second chroma prediction by using the second function and the luma residual.

[0009] According to a second aspect, an apparatus includes: a component for receiving an image block unit of a frame, the image block unit including samples in a first chroma channel, a second chroma channel and a luma channel; a component for defining a co-located reference area on a luma component of the luma channel, a chroma component of the first chroma channel and a chroma component of the second chroma channel; a component for performing motion-compensated prediction on the luma component, the first chroma component and the second chroma component using a reference frame and a motion vector; a component for obtaining a first mapping function and a second mapping function using the co-located luma prediction and chroma prediction, the first mapping function mapping the luma component to the first chroma component and the second mapping function mapping the luma component to the second chroma component; a component for reconstructing a luma residual; and a component for obtaining a first chroma prediction by using the first function and the luma residual and a second chroma prediction by using the second function and the luma residual.

[0010] According to an embodiment, the apparatus comprises means for downsampling the co-located luma component to match a sub-sampled chroma grid.

[0011] According to an embodiment, the apparatus comprises means for filtering the co-located luminance component using a one-dimensional filter or a two-dimensional filter.

[0012] According to an embodiment, the apparatus comprises means for mapping a luma component to a first chroma component or a second chroma component or a first chroma component to a second chroma component using a first mapping function or a second mapping function.

[0013] According to an embodiment, the apparatus comprises means for mapping the first chrominance component to the second chrominance component using the first mapping function or the second mapping function.

[0014] According to an embodiment, the apparatus comprises means for signaling usage of a cross-component residual model (CCRM) from an encoder to a decoder at a transform unit, prediction unit, coding unit, coding tree unit, slice, frame or sequence level.

[0015] According to an embodiment, the apparatus comprises means for cropping or spatially filtering the luma residual before applying the cross-component model.

[0016] According to an embodiment, the apparatus comprises means for determining, for the first chroma component and the second chroma component respectively, the type and amount of filtering to be applied to the luma residual.

[0017] According to an embodiment, the apparatus comprises means for determining the type and amount of filtering to be applied to the luma residual based on the output of the first mapping function or the output of the second mapping function.

[0018] According to an embodiment, the apparatus comprises means for applying the type and amount of filtering differently to the luma residual at a sub-block or sample level.

[0019] According to an embodiment, the apparatus comprises means for using different mapping functions in block prediction based on one or more thresholds derived on luma, luma residual or chroma.

[0020] According to an embodiment, the apparatus comprises means for mapping chrominance components to luma components.

[0021] According to an embodiment, the apparatus comprises means for mapping a first chrominance component to a second chrominance component.

[0022] According to an embodiment, the apparatus comprises means for upsampling an input to the mapping function to match the brightness grid.

[0023] According to an embodiment, the apparatus comprises means for upsampling the output of the mapping function to match the brightness grid.

[0024] According to an embodiment, the apparatus comprises means for mapping a first chrominance component to a second chrominance component.

[0025] As described above, the apparatus and the computer-readable storage medium having code stored thereon are thus arranged to perform the above-described method and one or more embodiments related thereto. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] For a better understanding of the present invention, reference will now be made by way of example to the accompanying drawings, in which:

[0027] Figure 1 An electronic device using an embodiment of the present invention is schematically shown;

[0028] Figure 2 Schematically illustrates a user equipment suitable for adopting an embodiment of the present invention;

[0029] Figure 3 Further schematically illustrated are electronic devices employing embodiments of the present invention using wireless and wired network connections;

[0030] Figure 4a and Figure 4b Schematically shows an encoder and a decoder suitable for implementing an embodiment of the present invention;

[0031] Figure 5 The locations of samples used to derive parameters for a cross-component linear model are shown;

[0032] Figure 6a and Figure 6b Examples of classifying brightness samples into two categories in the sample domain and the spatial domain are shown respectively;

[0033] Figure 7 An example of four reference rows adjacent to a prediction block is shown;

[0034] Figure 8 The matrix-weighted intra prediction process is shown;

[0035] Fig. 9 A flowchart of a method for predicting samples in at least one color channel according to an embodiment of the present invention is shown;

[0036] Fig.10 shows an example of a co-located reference sample region consisting of reconstructed luma and chroma samples defined for both luma and chroma according to an embodiment of the present invention;

[0037] Fig.11 Various examples of filter kernel sizes according to one approach are shown;

[0038] Fig.12shows top and left neighboring blocks used in combined inter and intra prediction (CIIP) weight derivation according to one approach;

[0039] Fig.13 A schematic diagram of an example multimedia communication system is shown in which various embodiments may be implemented. DETAILED DESCRIPTION

[0040] Suitable devices and possible mechanisms for predicting chrominance samples are described in more detail below. Figure 1 and Figure 2 ,in Figure 1 A block diagram of a video decoding system according to an example embodiment is shown as a schematic block diagram of an exemplary apparatus or electronic device 50 incorporating a codec according to an embodiment of the invention. Figure 2 The layout of the device according to the exemplary embodiment is shown. Figure 1 and Figure 2 components.

[0041] The electronic device 50 may be, for example, a mobile terminal or user equipment of a wireless communication system. However, it should be understood that the embodiments of the present invention may be implemented in any electronic device or apparatus that may need to encode and decode or encode or decode video images.

[0042] The device 50 may include a housing 30 for incorporating and protecting the device. The device 50 may also include a display 32 in the form of a liquid crystal display. In other embodiments of the present invention, the display may be any suitable display technology suitable for displaying images or videos. The device 50 may also include a keypad 34. In other embodiments of the present invention, any suitable data interface or user interface mechanism may be employed. For example, the user interface may be implemented as a virtual keyboard or data entry system as part of a touch-sensitive display.

[0043] The device may include a microphone 36 or any suitable audio input that may be a digital or analog signal input. The device 50 may also include an audio output device, which in embodiments of the present invention may be any of the following: headphones 38, speakers, or analog audio or digital audio output connections. The device 50 may also include a battery (or in other embodiments of the present invention, the device may be powered by any suitable mobile energy device, such as a solar cell, a fuel cell, or a clockwork generator). The device may also include a camera capable of recording or capturing images and / or video. The device 50 may also include an infrared port for short-range line-of-sight communication with other devices. In other embodiments, the device 50 may also include any suitable short-range communication solution, such as a Bluetooth wireless connection or a USB / Firewire wired connection.

[0044] The apparatus 50 may include a controller 56, a processor or processor circuit for controlling the apparatus 50. The controller 56 may be connected to a memory 58 which, in embodiments of the invention, may store data in the form of image and audio data and / or may also store instructions for implementation on the controller 56. The controller 56 may also be connected to a codec circuit 54 which is suitable for performing encoding and decoding of audio and / or video data or assisting in encoding and decoding performed by the controller.

[0045] The apparatus 50 may also include a card reader 48 and a smart card 46, such as a UICC and a UICC reader, for providing user information and suitable for providing authentication information for authentication and authorization of the user at the network.

[0046] The device 50 may include a radio interface circuit 52 connected to the controller and adapted to generate wireless communication signals, for example for communicating with a cellular communication network, a wireless communication system or a wireless local area network. The device 50 may also include an antenna 44 connected to the radio interface circuit 52 for transmitting radio frequency signals generated at the radio interface circuit 52 to other devices and for receiving radio frequency signals from other devices.

[0047] The apparatus 50 may include a camera capable of recording or detecting individual frames, which are then passed to a codec 54 or controller for processing. The apparatus may receive video image data for processing from another device before transmission and / or storage. The apparatus 50 may also receive images for encoding / decoding wirelessly or through a wired connection. The structural elements of the apparatus 50 described above represent examples of components for performing corresponding functions.

[0048] about Figure 3 , showing an example of a system in which embodiments of the present invention may be utilized. System 10 includes a plurality of communication devices that may communicate over one or more networks. System 10 may include any combination of wired or wireless networks, including but not limited to wireless cellular telephone networks (such as GSM, UMTS, CDMA networks, etc.), wireless local area networks (WLANs) such as defined by any of the IEEE 802.x standards, Bluetooth personal area networks, Ethernet local area networks, token ring local area networks, wide area networks, and the Internet.

[0049] System 10 may include wired and wireless communication devices and / or apparatuses 50 suitable for implementing embodiments of the present invention.

[0050] For example, Figure 3The illustrated system shows a representation of the mobile telephone network 11 and the Internet 28. Connectivity to the Internet 28 may include, but is not limited to, long range wireless connections, short range wireless connections, and various wired connections including, but not limited to, telephone lines, cable lines, power lines, and similar communications pathways.

[0051] The example communication devices shown in system 10 may include, but are not limited to, electronic devices or apparatuses 50, a combination of a personal digital assistant (PDA) and mobile phone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, a notebook computer 22. Apparatus 50 may be stationary or mobile when carried by an individual who is moving. Apparatus 50 may also be located in a mode of transportation, including, but not limited to, a car, a truck, a taxi, a bus, a train, a ship, an airplane, a bicycle, a motorcycle, or any similar suitable mode of transportation.

[0052] Embodiments may also be implemented in set-top boxes, i.e., digital TV receivers, which may or may not have a display or wireless capabilities; in tablet or (laptop) personal computers (PCs), which have hardware or software or a combination of encoder / decoder implementations; in various operating systems, and in chipsets, processors, DSPs and / or embedded systems that provide hardware / software based encoding.

[0053] Some or additional devices may send and receive calls and messages and communicate with a service provider via a wireless connection 25 to a base station 24. The base station 24 may be connected to a network server 26 that allows communication between the mobile phone network 11 and the Internet 28. The system may include additional communication devices and various types of communication devices.

[0054] The communication device may communicate using various transmission technologies, including but not limited to code division multiple access (CDMA), global system for mobile communications (GSM), universal mobile telecommunications system (UMTS), time division multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-internet protocol (TCP-IP), short message service (SMS), multimedia messaging service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.11 and any similar wireless communication technology. The communication device involved in implementing various embodiments of the present invention may communicate using various media, including but not limited to radio, infrared, laser, cable connection and any suitable connection.

[0055] In telecommunications and data networks, a channel can refer to a physical channel or a logical channel. A physical channel can refer to a physical transmission medium such as a wire, while a logical channel can refer to a logical connection on a multiplexed medium capable of transmitting several logical channels. A channel can be used to transmit an information signal (e.g., a bit stream) from one or more senders (or transmitters) to one or more receivers.

[0056] The MPEG-2 Transport Stream (TS), specified in ISO / IEC 13818-1 or equivalently in ITU-T Recommendation H.222.0, is a format for carrying audio, video and other media, as well as program metadata or other metadata in a multiplexed stream. A packet identifier (PID) is used to identify an elementary stream (also called a packetized elementary stream) within a TS. Therefore, a logical channel within an MPEG-2 TS can be considered to correspond to a specific PID value.

[0057] Available media file format standards include ISO Base Media File Format (ISO / IEC 14496-12, which may be abbreviated as ISOBMFF) and a file format for NAL unit structured video (ISO / IEC 14496-15), which is derived from ISOBMFF.

[0058] A video codec includes an encoder that transforms an input video into a compressed representation suitable for storage / transmission and a decoder that can decompress the compressed video representation back into a visual form. The video encoder and / or video decoder may also be separate from each other, i.e., not required to form a codec. Typically, an encoder discards some information in the original video data in order to represent the video in a more compact form (i.e., a lower bit rate).

[0059] A typical hybrid video encoder (e.g., many encoder implementations of ITU-T H.263 and H.264) decodes video information in two stages. First, the first pixel value in a particular picture area (or "block") is predicted, for example, by a motion compensation component (finding and indicating an area in one of the previously encoded video frames that closely corresponds to the block being encoded) or by a spatial component (using pixel values ​​surrounding the block that are encoded in a specified manner). Second, the prediction error is encoded, that is, the difference between the predicted pixel block and the original pixel block. This is usually done by transforming the difference in pixel values ​​using a specified transform (e.g., a discrete cosine transform (DCT) or a variant thereof), quantizing the coefficients and entropy encoding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and the size of the resulting encoded video representation (file size or transmission bit rate).

[0060] In temporal prediction, the prediction source is a previously decoded picture (i.e., a reference picture). In intra block copy (IBC, also known as intra block copy prediction), prediction is applied similarly to temporal prediction, but the reference picture is the current picture, and only previously decoded samples can be referenced in the prediction process. Inter-layer or inter-view prediction can be applied similarly to temporal prediction, but the reference picture is a decoded picture from another scalable layer or from another view, respectively. In some cases, inter-frame prediction may refer to only temporal prediction, while in other cases, inter-frame prediction may collectively refer to temporal prediction and any of intra-frame block copy, inter-layer prediction, and inter-view prediction, provided that they are performed in the same or similar process as temporal prediction. Inter-frame prediction or temporal prediction may sometimes be referred to as motion compensation or motion-compensated prediction.

[0061] Motion compensation can be performed with full sample or sub-sample accuracy. In the case of full sample accurate motion compensation, the motion can be represented as a motion vector with integer values ​​for horizontal and vertical displacements, and the motion compensation process effectively copies samples from a reference picture using those displacements. In the case of sub-sample accurate motion compensation, the motion vector is represented by a fractional or decimal value for the horizontal and vertical components of the motion vector. In the case where the motion vector references a non-integer position in the reference picture, a sub-sample interpolation process is typically called to calculate predicted sample values ​​based on the reference sample and the selected sub-sample position. The sub-sample interpolation process typically includes horizontal filtering, which compensates for horizontal offsets relative to the full sample position, followed by vertical filtering, which compensates for vertical offsets relative to the full sample position. However, vertical processing can also be done before horizontal processing in some environments.

[0062] Inter prediction (also known as temporal prediction, motion compensation or motion-compensated prediction) reduces temporal redundancy. In inter prediction, the prediction source is a previously decoded picture. Intra prediction exploits the fact that adjacent pixels within the same picture may be correlated. Intra prediction can be performed in the spatial domain or in the transform domain, i.e., sample values ​​or transform coefficients can be predicted. Intra prediction is typically used for intra coding, where inter prediction is not applied.

[0063] One result of the encoding process is a set of coded parameters, such as motion vectors and quantized transform coefficients. Many parameters can be entropy coded more efficiently if they are first predicted from spatially or temporally neighboring parameters. For example, a motion vector can be predicted from spatially neighboring motion vectors, and only the difference with respect to the motion vector predictor can be coded. The prediction of coding parameters and intra-frame prediction can be collectively referred to as intra-picture prediction.

[0064] Figure 4a and Figure 4bAn encoder and decoder suitable for use with embodiments of the present invention are shown. A video codec consists of an encoder that transforms an input video into a compressed representation suitable for storage / transmission and a decoder that can decompress the compressed video representation back into a visual form. Typically, an encoder discards and / or loses some information in the original video sequence in order to represent the video in a more compact form (i.e., at a lower bit rate). Figure 4a An example of the encoding process is shown. Figure 4a The image to be encoded (I n ); The predicted representation of the image block (P' n ); prediction error signal (D n ); Reconstruct the prediction error signal (D' n ); Preliminary reconstructed image (I' n ); Final reconstructed image (R' n ); Transform (T) and Inverse Transform (T -1 ), quantization (Q) and inverse quantization (Q -1 ); Entropy coding (E); Reference frame memory (RFM); Inter-frame prediction (P inter ); intra prediction (P intra ); mode selection (MS) and filtering (F).

[0065] Figure 4b An example of the decoding process is shown in . Figure 4b The predicted representation (P') of the image block is shown. n ); Reconstruct the prediction error signal (D' n ); Preliminary reconstructed image (I' n ); Final reconstructed image (R' n ); inverse transform (T -1 ); inverse quantization (Q -1 ); Entropy decoding (E -1 ); reference frame memory (RFM); prediction (inter or intra) (P); and filtering (F).

[0066] Many hybrid video encoders encode video information in two stages. First, the first pixel values ​​in a specific picture area (or "block") are predicted, for example by motion compensation components (finding and indicating the area in one of the previously encoded video frames that closely corresponds to the block being encoded) or by spatial components (using pixel values ​​around the block that are encoded in a specified way). In the first stage, predictive coding can be applied, such as so-called sample prediction and / or so-called syntax prediction.

[0067] In sample prediction, pixel or sample values ​​in a particular picture area or "block" are predicted. These pixel or sample values ​​may be predicted using, for example, one or more of motion compensation or intra prediction mechanisms.

[0068] The motion compensation mechanism (which may also be referred to as inter-prediction, temporal prediction, or motion-compensated temporal prediction or motion-compensated prediction or MCP) involves finding and indicating an area in one of the previously encoded video frames that closely corresponds to the block being decoded. Inter-prediction can reduce temporal redundancy.

[0069] Intra-frame prediction (where pixel or sample values ​​can be predicted by a spatial mechanism) involves finding and indicating spatial region relationships. Intra-frame prediction exploits the fact that neighboring pixels within the same picture may be correlated. Intra-frame prediction can be performed in the spatial domain or in the transform domain, i.e., it is possible to predict sample values ​​or transform coefficients. Intra-frame prediction is typically used for intra-frame coding, where inter-frame prediction is not applied.

[0070] In syntax prediction, which may also be referred to as parameter prediction, syntax elements and / or syntax element values ​​and / or variables derived from syntax elements are predicted from earlier coded syntax elements (de) and / or earlier derived variables. A non-limiting example of syntax prediction is provided below.

[0071] In motion vector prediction, motion vectors, for example, for inter-frame and / or inter-view prediction, can be encoded relative to block-specific prediction motion vectors. In many video codecs, the prediction motion vector is created in a predefined way, for example by calculating the median of the encoded or decoded motion vectors of neighboring blocks. Another way to create motion vector predictions, sometimes called advanced motion vector prediction (AMVP), is to generate a list of candidate predictions from neighboring blocks and / or co-located blocks in temporal reference pictures and signal the selected candidate as a motion vector predictor. In addition to predicting motion vector values, reference indexes of previously encoded / decoded pictures can be predicted. The reference index is typically predicted from neighboring blocks and / or co-located blocks in temporal reference pictures. Differential encoding of motion vectors is typically disabled across slice boundaries.

[0072] Block partitioning may be predicted, for example from CTU to CU and down to PU.

[0073] In filter parameter prediction, filter parameters, for example, used for sample adaptive offset, may be predicted.

[0074] Prediction methods that use image information from previously coded images may also be referred to as inter-frame prediction methods, which may also be referred to as temporal prediction and motion compensation. Prediction methods that use image information within the same image may also be referred to as intra-frame prediction methods.

[0075] Secondly, the prediction error is encoded, that is, the difference between the predicted pixel block and the original pixel block. This is usually done by transforming the difference in pixel values ​​using a specified transform (e.g., discrete cosine transform (DCT) or its variant), quantizing the coefficients and entropy encoding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and the size of the resulting encoded video representation (file size or transmission bit rate). The video codec can also provide a transform skip mode that the encoder can choose to use. In transform skip mode, the prediction error is encoded in the sample domain, for example by deriving sample-by-sample differences relative to certain adjacent samples and encoding the sample-by-sample differences with an entropy encoder.

[0076] Entropy coding / decoding can be performed in a variety of ways. For example, context-based coding / decoding can be applied, in which both the encoder and the decoder modify the context state of the coding parameters based on the coding parameters of the previous coding / decoding. Context-based coding can be, for example, context-adaptive binary arithmetic coding (CABAC) or context-based variable length coding (CAVLC) or any similar entropy coding. Entropy coding / decoding can be performed alternatively or additionally using a variable length coding scheme (e.g., Huffman coding / decoding or exponential Golomb coding / decoding). The decoding of the coding parameters of the bitstream or codeword from the entropy coding can be referred to as parsing.

[0077] The phrase along the bitstream (e.g., indication along the bitstream) may be defined to refer to out-of-band transmission, signaling, or storage in a manner that out-of-band data is associated with the bitstream. Decoding along the bitstream or similar phrases may refer to decoding the referenced out-of-band data associated with the bitstream (which may be obtained from the out-of-band transmission, signaling, or storage). For example, indication along the bitstream may refer to metadata in a container file that encapsulates the bitstream.

[0078] The H.264 / AVC standard is developed by the Video Coding Experts Group (VCEG) of the Telecommunication Standardization Sector of the International Telecommunication Union (ITU-T) and the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard is issued by two parent standardization organizations, and it is called ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10, also known as MPEG-4 Part 10 Advanced Video Coding (AVC). There are already multiple versions of the H.264 / AVC standard, integrating new extensions or features into the specification. These extensions include scalable video coding (SVC) and multi-view video coding (MVC).

[0079] Version 1 of the High Efficiency Video Coding (H.265 / HEVC, also known as HEVC) standard was developed by the Joint Collaboration Team on Video Coding (JCT-VC) of VCEG and MPEG. The standard is published by two parent standardization organizations and is referred to as ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2, also known as MPEG-H Part 2 High Efficiency Video Coding (HEVC). Subsequent versions of H.265 / HEVC include scalable, multi-view, fidelity range, three-dimensional, and screen content coding extensions, which may be abbreviated as SHVC, MV-HEVC, REXT, 3D-HEVC, and SCC, respectively.

[0080] Versatile Video Coding (VVC) (MPEG-I Part 3), also known as ITU-T H.266, is a video compression standard developed by the Joint Video Experts Group (JVET) (formally ISO / IEC JTC1SC29 WG11) of the Moving Picture Experts Group (MPEG) and the Video Coding Experts Group (VCEG) of the International Telecommunication Union (ITU) as the successor to HEVC / H.265.

[0081] This section describes some key definitions, bitstream and coding structures and concepts of H.264 / AVC and HEVC, taking video encoders, decoders, encoding methods, decoding methods and bitstream structures as examples, in which the embodiments can be implemented. Some key definitions, bitstream and coding structures and concepts of H.264 / AVC are the same as those in HEVC, so they are described jointly below. Aspects of the present invention are not limited to H.264 / AVC or HEVC, but are described for one possible basis on which the present invention can be partially or fully implemented.

[0082] Similar to many earlier video coding standards, bitstream syntax and semantics as well as the decoding process for error-free bitstreams are specified in H.264 / AVC and HEVC. The encoding process is not specified, but the encoder must generate a conforming bitstream. Bitstream and decoder conformance can be verified with a hypothetical reference decoder (HRD). The standards contain coding tools that help cope with transmission errors and losses, but the use of the tools in encoding is optional, and there is no specified decoding process for erroneous bitstreams.

[0083] The basic unit for input to an H.264 / AVC or HEVC encoder and output to an H.264 / AVC or HEVC decoder is a picture. A picture given as input to an encoder may also be referred to as a source picture, and a picture decoded by a decoder may be referred to as a decoded picture.

[0084] The source picture and the decoded picture each include one or more sample arrays, such as one of the following sample array sets: - Luminance (Y) only (monochrome). - Luma and two chrominances (YCbCr or YCgCo). - Green, Blue and Red (GBR, also known as RGB). - Arrays representing other unspecified monochromatic or tristimulus sampling (e.g., YZX, also called XYZ).

[0085] In H.264 / AVC and HEVC, a picture can be a frame or a field. A frame consists of a matrix of luma samples and possibly corresponding chroma samples. A field is a collection of alternating sample rows of a frame and can be used as encoder input when the source signal is interlaced. The chroma sample array may not be present (and therefore monochrome sampling may be used), or the chroma sample array may be subsampled when compared to the luma sample array. The chroma format can be summarized as follows: - In monochrome sampling, there is only one sample array, which can be nominally considered as a brightness array. - In 4:2:0 sampling, each of the two chroma arrays has half the height and half the width of the luma array. - In 4:2:2 sampling, each of the two chroma arrays has the same height and half the width of the luma array. - In 4:4:4 sampling, when separate color planes are not used, each of the two chroma arrays has the same height and width as the luma array.

[0086] In H.264 / AVC and HEVC, sample arrays can be encoded into the bitstream as separate color planes, and the separately encoded color planes can be decoded from the bitstream separately. When separate color planes are used, each of them is processed separately (by the encoder and / or decoder) as a picture with monochrome sampling.

[0087] Partitioning can be defined as dividing a set into subsets such that every element in the set is in exactly one of the subsets.

[0088] When describing the operation of HEVC encoding and / or decoding, the following terms may be used. A coding block may be defined as an N×N block of samples for some value of N, such that the division of a coding tree block to a coding block is a partition. A coding tree block (CTB) may be defined as an N×N block of samples for some value of N, such that the division of a component to a coding tree block is a partition. A coding tree unit (CTU) may be defined as a coding tree block of luma samples, two corresponding coding tree blocks of chroma samples of a picture with three sample arrays, or a coding tree block of samples of a monochrome picture or a picture encoded using three separate color planes and a syntax structure for encoding samples. A coding unit (CU) may be defined as a coding block of luma samples, two corresponding coding blocks of chroma samples of a picture with three sample arrays, or a coding block of samples of a monochrome picture or a picture encoded using three separate color planes and a syntax structure for encoding samples. A CU with the maximum allowed size may be named an LCU (maximum coding unit) or a coding tree unit (CTU), and a video picture is divided into non-overlapping LCUs.

[0089] A CU consists of one or more prediction units (PUs), which define the prediction process for samples within the CU, and one or more transform units (TUs), which define the prediction error coding process for samples in the CU. Typically, a CU consists of a square block of samples with a size that can be selected from a predefined set of possible CU sizes. Each PU and TU can be further split into smaller PUs and TUs to increase the granularity of the prediction and prediction error coding processes, respectively. Each PU has prediction information associated with it, defining what type of prediction will be applied for the pixels within the PU (e.g., motion vector information for inter-prediction PUs and intra-prediction directionality information for intra-prediction PUs).

[0090] Each TU may be associated with information describing the prediction error decoding process for the samples within the TU, including, for example, DCT coefficient information. It is usually signaled at the CU level whether prediction error coding is applied for each CU. If there is no prediction error residual associated with a CU, then it may be considered that there is no TU for the CU. The partitioning of an image into CUs and the partitioning of a CU into PUs and TUs is usually signaled in the bitstream, allowing a decoder to reproduce the expected structure of these units.

[0091] In HEVC, pictures can be partitioned in tiles, which are rectangular and contain an integer number of LCUs. In HEVC, partitioning into tiles forms a regular grid, where the height and width of the tiles differ from each other by one LCU at the maximum value. In HEVC, a slice is defined as an integer number of coding tree units contained in an independent slice segment and all subsequent dependent slice segments (if any) before the next independent slice segment (if any) within the same access unit. In HEVC, a slice segment is defined as an integer number of coding tree units that are ordered consecutively in a tile scan and contained in a single NAL unit. The division of each picture into slice segments is a partition. In HEVC, an independent slice segment is defined as a slice segment whose values ​​of the syntax elements of the slice segment header are not inferred from the values ​​of the previous slice segment; and a dependent slice segment is defined as a slice segment whose values ​​of some syntax elements of the slice segment header are inferred from the values ​​of the previous independent slice segment in the decoding order. In HEVC, a slice header is defined as a slice segment header of an independent slice segment, which is the current slice segment or the independent slice segment preceding the current dependent slice segment, and a slice segment header is defined as a part of a coded slice segment containing data elements related to the first or all coding tree units represented in the slice segment. If tiles are not used, CUs are scanned in the raster scan order of LCUs within tiles or within pictures. Within an LCU, CUs have a specific scanning order.

[0092] The decoder reconstructs the output video by applying a prediction component similar to the encoder to form a predicted representation of a pixel block (using motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (the inverse operation of prediction error encoding to restore the quantized prediction error signal in the spatial pixel domain). After applying the prediction and prediction error decoding components, the decoder sums the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) may also apply additional filtering components to improve the quality of the output video before passing it for display and / or storing it as a prediction reference for an upcoming frame in a video sequence.

[0093] The filtering may, for example, include one or more of: deblocking, sample adaptive offset (SAO), and / or adaptive loop filtering (ALF).H.264 / AVC includes deblocking, while HEVC includes both deblocking and SAO.

[0094] In a typical video codec, motion information is indicated by a motion vector associated with each motion compensated image block (such as a prediction unit). Each of these motion vectors represents the displacement of an image block in a picture to be encoded (on the encoder side) or decoded (on the decoder side) from a prediction source block in one of the previously encoded or decoded pictures. In order to efficiently represent motion vectors, these motion vectors are usually encoded differentially relative to the block-specific predicted motion vector. In a typical video codec, the predicted motion vector is created in a predefined manner, such as calculating the median of the encoded or decoded motion vectors of adjacent blocks. Another way to create a motion vector prediction is to generate a list of candidate predictions from adjacent blocks and / or co-located blocks in a temporal reference picture and signal the selected candidate as a motion vector predictor. In addition to predicting the motion vector value, it is possible to predict which reference picture(s) are used for motion compensated prediction, and this prediction information can be represented, for example, by a reference index of a previously encoded / decoded picture. The reference index is usually predicted from adjacent blocks and / or co-located blocks in a temporal reference picture. Furthermore, typical high efficiency video codecs employ an additional motion information encoding / decoding mechanism, generally referred to as merge / merge mode, in which all motion field information including motion vectors and corresponding reference picture indexes for each available reference picture list is predicted and used without any modification / correction. Similarly, the prediction of the motion field information is performed using the motion field information of neighboring blocks and / or co-located blocks in the temporal reference pictures, and the used motion field information is signaled in the list of motion field candidate lists populated with the motion field information of the available neighboring / co-located blocks.

[0095] In a typical video codec, the prediction residual after motion compensation is first transformed with a transform kernel (such as DCT) and then encoded. The reason for this is that in many cases there is still some correlation between the residual and the transform, and this helps to reduce this correlation and provide more efficient coding.

[0096] Video coding standards and specifications may allow the encoder to divide a coded picture into coded slices or the like. Typically, intra-picture prediction is disabled across slice boundaries. Therefore, a slice can be viewed as a way to split a coded picture into independently decodable slices. In H.264 / AVC and HEVC, intra-picture prediction is disabled across slice boundaries. Therefore, a slice can be viewed as a way to split a coded picture into independently decodable slices, and therefore a slice is typically viewed as a basic unit for transmission. In many cases, the encoder can indicate in the bitstream which types of intra-picture prediction are turned off across slice boundaries, and the decoder operation, for example, takes this information into account when inferring which prediction sources are available. For example, if neighboring CUs reside in different slices, samples from neighboring CUs may be considered unavailable for intra prediction.

[0097] The basic unit for the output of an H.264 / AVC or HEVC encoder and the input of an H.264 / AVC or HEVC decoder is the Network Abstraction Layer (NAL) unit, respectively. For transmission over a packet-oriented network or storage in a structured file, the NAL unit can be encapsulated into a packet or similar structure. A byte stream format has been specified in H.264 / AVC and HEVC for transmission or storage environments that do not provide a framing structure. The byte stream format separates NAL units from each other by appending a start code to the front of each NAL unit. To avoid erroneous detection of NAL unit boundaries, the encoder runs a byte-oriented start code simulation prevention algorithm that adds simulation prevention bytes to the NAL unit payload when a start code appears in other cases. In order to achieve direct gateway operation between packet-oriented and stream-oriented systems, start code simulation prevention can always be performed regardless of whether the byte stream format is in use. The NAL unit can be defined as a syntax structure and a byte that contains an indication of the type of data to follow, and the byte contains data in the form of an RBSP, interspersed with simulation prevention bytes when necessary. A Raw Byte Sequence Payload (RBSP) may be defined as a syntax structure containing an integer number of bytes encapsulated in a NAL unit. An RBSP is either empty or has the form of a data bit string containing syntax elements, followed by an RBSP stop bit, followed by a number of subsequent bits that are zero or equal to 0.

[0098] The NAL unit consists of a header and a payload. In H.264 / AVC and HEVC, the NAL unit header indicates the type of the NAL unit.

[0099] In HEVC, a two-byte NAL unit header is used for all specified NAL unit types. The NAL unit header contains a reserved bit, a 6-bit NAL unit type indication, a three-bit nuh_temporal_id_plus1 indication for the temporal level (which may need to be greater than or equal to 1), and a six-bit nuh_layer_id syntax element. The temporal_id_plus1 syntax element can be regarded as a temporal identifier for the NAL unit, and the zero-based TemporalId variable can be derived as follows: TemporalId = temporal_id_plus1-1. The abbreviation TID can be used interchangeably with the TemporalId variable. TemporalId equal to 0 corresponds to the lowest temporal level. The value of temporal_id_plus1 needs to be non-zero in order to avoid start code simulation involving two NAL unit header bytes. The bitstream is generated by excluding all VCL NAL units with TemporalId greater than or equal to the selected value and including all other VCL NAL units that are consistent. Therefore, a picture with TemporalId equal to tid_value does not use any picture with TemporalId greater than tid_value as an inter prediction reference. A sublayer or temporal sublayer may be defined as a temporal scalable layer (or temporal layer TL) of a temporally scalable bitstream, which consists of VCL NAL units and associated non-VCL NAL units with a specific value of the TemporalId variable. nuh_layer_id may be understood as a scalability layer identifier.

[0100] NAL units can be classified into video coding layer (VCL) NAL units and non-VCL NAL units. VCL NAL units are usually coded slice NAL units. In HEVC, VCL NAL units contain syntax elements representing one or more CUs.

[0101] A non-VCL NAL unit may be, for example, one of the following types: a sequence parameter set, a picture parameter set, a supplemental enhancement information (SEI) NAL unit, an access unit delimiter, an end-of-sequence NAL unit, an end-of-bitstream NAL unit, or a filler data NAL unit. A parameter set may be required to reconstruct a decoded picture, while many of the other non-VCL NAL units are not necessary for reconstructing decoded sample values.

[0102] Parameters that remain unchanged through the encoded video sequence can be included in a sequence parameter set. In addition to the parameters that may be required for the decoding process, the sequence parameter set optionally includes video usability information (VUI), which includes parameters important for buffering, picture output timing, rendering, and resource reservation. In HEVC, the sequence parameter set RBSP includes parameters that can be referenced by one or more picture parameter set RBSPs or one or more SEI NAL units containing a buffering period SEI message. The picture parameter set contains such parameters that may remain unchanged in several coded pictures. The picture parameter set RBSP may include parameters that can be referenced by the coded slice NAL units of one or more coded pictures.

[0103] In HEVC, a video parameter set (VPS) can be defined as a syntax structure containing syntax elements that apply to zero or more complete coded video sequences, as determined by the content of the syntax elements found in the PPS, which are referenced by syntax elements found in the PPS, which are referenced by syntax elements found in each slice segment header.

[0104] A video parameter set RBSP may include parameters referenced by one or more sequence parameter set RBSPs.

[0105] The relationship and hierarchy between video parameter sets (VPS), sequence parameter sets (SPS) and picture parameter sets (PPS) can be described as follows. The VPS resides at a level above the SPS in the parameter set hierarchy and in the context of scalability and / or 3D video. The VPS may include parameters common to all slices across all (scalability or view) layers in a complete coded video sequence. The SPS includes parameters common to all slices in a specific (scalability or view) layer in a complete coded video sequence and may be shared by multiple (scalability or view) layers. The PPS includes parameters common to all slices in a specific layer representation (a representation of a scalability or view layer in an access unit) and may be shared by all slices in multiple layer representations.

[0106] The VPS may provide information about dependencies of layers in the bitstream, as well as a lot of other information that may apply to all slices across all (scalability or view) layers in a complete coded video sequence. The VPS may be viewed as comprising two parts: a base VPS and a VPS extension, where the VPS extension is optionally present.

[0107] Out-of-band transmission, signaling or storage may additionally or alternatively be used for other purposes besides tolerating transmission errors, such as ease of access or session negotiation. For example, a sample entry for a track in a file conforming to the ISO base media file format may include a parameter set, while the coded data in the bitstream is stored elsewhere in the file or in another file. The phrase along the bitstream (e.g., indicated along the bitstream) or along a decoding unit of the bitstream (e.g., indicated along a coding tile) may be used in the claims and described embodiments to refer to out-of-band transmission, signaling or storage in a manner such that the out-of-band data is associated with the bitstream or decoding unit, respectively. The phrase decoding along the bitstream or along a decoding unit of the bitstream or the like may refer to decoding the involved out-of-band data (which may be obtained from the out-of-band transmission, signaling or storage) associated with the bitstream or decoding unit, respectively.

[0108] A SEI NAL unit may contain one or more SEI messages that are not required for decoding of output pictures but may assist with related processes such as picture output timing, rendering, error detection, error concealment, and resource conservation.

[0109] A coded picture is an encoded representation of a picture.

[0110] In HEVC, a coded picture can be defined as a coded representation of a picture that contains all coding tree units of the picture. In HEVC, an access unit (AU) can be defined as a set of NAL units that are associated with each other according to a specified classification rule, are consecutive in decoding order, and contain at most one picture with any specific value of nuh_layer_id. In addition to VCL NAL units containing coded pictures, access units can also contain non-VCL NAL units. The specified classification rule can, for example, associate pictures with the same output time or picture output count value into the same access unit.

[0111] A bitstream can be defined as a sequence of bits in the form of a NAL unit stream or a byte stream, which forms a representation of a coded picture and associated data forming one or more coded video sequences. The first bitstream may be followed by a second bitstream in the same logical channel, for example in the same file or in the same connection of a communication protocol. An elementary stream (in the context of video coding) can be defined as a sequence of one or more bitstreams. The end of the first bitstream can be indicated by a specific NAL unit, which may be referred to as an end-of-bitstream (EOB) NAL unit and which is the last NAL unit of the bitstream. In HEVC and its current draft extensions, the EOB NAL unit is required to have a nuh_layer_id equal to 0.

[0112] In H.264 / AVC, a coded video sequence is defined as a sequence of consecutive access units in decoding order from an IDR access unit (inclusive) to the next IDR access unit (exclusive) or to the end of the bitstream, whichever occurs first.

[0113] In HEVC, a coded video sequence (CVS) may be defined, for example, as a sequence of access units that, in decoding order, contains an IRAP access unit with NoRaslOutputFlag equal to 1, followed by zero or more access units that are not IRAP access units with NoRaslOutputFlag equal to 1, including all subsequent access units up to but not including any subsequent access unit of an IRAP access unit with NoRaslOutputFlag equal to 1. An IRAP access unit may be defined as an access unit in which a base layer picture is an IRAP picture. The value of NoRaslOutputFlag is equal to 1 for each IDR picture, each BLA picture, and each IPAP picture, wherein the IRAP picture is the first picture in a particular layer in the bitstream in decoding order, and is the first IRAP picture that follows the end of a sequence of NAL units with the same nuh_layer_id value in decoding order. There is a component that provides the value of HandleCraAsBlaFlag to the decoder from an external entity that can control the decoder, such as a player or receiver. HandleCraAsBlaFlag may be set to 1, for example, by a player that seeks to a new position in the bitstream or tunes to a broadcast and starts decoding and then starts decoding from a CRA picture. When HandleCraAsBlaFlag is equal to 1 for a CRA picture, the CRA picture is processed and decoded as if it were a BLA picture.

[0114] In HEVC, a coded video sequence may additionally or alternatively (described above) be designated as ending when a specific NAL unit, which may be referred to as an end of sequence (EOS) NAL unit, appears in the bitstream with nuh_layer_id equal to 0.

[0115] A group of pictures (GOP) and its characteristics can be defined as follows. A GOP can be decoded regardless of whether any previous pictures are decoded. An open GOP is a group of pictures in which the pictures before the initial intra picture in the output order may not be correctly decoded when decoding starts from the initial intra picture of the open GOP. In other words, the pictures of the open GOP may refer to pictures belonging to the previous GOP (in inter prediction). The HEVC decoder is able to identify the intra picture that starts the open GOP because a specific NAL unit type (CRA NAL unit type) is available for its coded slices. A closed GOP is a group of pictures in which all pictures can be correctly decoded when decoding starts from the initial intra picture of the closed GOP. In other words, no picture in the closed GOP refers to any picture in the previous GOP. In H.264 / AVC and HEVC, a closed GOP can start from an IDR picture. In HEVC, a closed GOP can also start from a BLA_W_RADL or BLA_N_LP picture. Compared to the closed GOP coding structure, the open GOP coding structure is potentially more efficient in compression due to the greater flexibility in the selection of reference pictures.

[0116] The decoded picture buffer (DPB) can be used in the encoder and / or in the decoder. There are two reasons for buffering decoded pictures, for reference in inter-frame prediction and for reordering decoded pictures to output order. Because H.264 / AVC and HEVC provide a lot of flexibility for both reference picture marking and output reordering, separate buffers for reference picture buffering and output picture buffering may waste memory resources. Therefore, the DPB can include a unified decoded picture buffering process for reference pictures and output reordering. When a decoded picture is no longer used as a reference and is not needed for output, it can be removed from the DPB.

[0117] In many coding modes of H.264 / AVC and HEVC, reference pictures used for inter prediction are indicated by indices pointing to reference picture lists. The indices may be encoded using variable length coding, which generally results in smaller indices having smaller values ​​for the corresponding syntax elements. In H.264 / AVC and HEVC, two reference picture lists (reference picture list 0 and reference picture list 1) are generated for each bidirectionally predicted (B) slice, and one reference picture list (reference picture list 0) is formed for each inter-coded (P) slice.

[0118] Many coding standards (including H.264 / AVC and HEVC) may have a decoding process to derive a reference picture index for a reference picture list, which may be used to indicate which of multiple reference pictures is used for inter prediction of a particular block. The reference picture index may be encoded into the bitstream by the encoder in some inter coding modes, or it may be derived (by the encoder and decoder) using neighboring blocks in some other inter coding modes, for example.

[0119] The motion parameter type or motion information may include but is not limited to one or more of the following types: - an indication of the prediction type (e.g., intra prediction, unidirectional prediction, bidirectional prediction) and / or the number of reference pictures; - an indication of a prediction direction, such as inter (i.e., temporal) prediction, inter-layer prediction, inter-view prediction, view synthesis prediction (VSP), and inter-component prediction (which may be indicated per reference picture and / or per prediction type, and where inter-view and view synthesis prediction may be considered together as one prediction direction in some embodiments), and / or - an indication of the reference picture type, such as a short-term reference picture and / or a long-term reference picture and / or an inter-layer reference picture (which may be indicated, for example, according to the reference picture). - a reference index to a reference picture list and / or any other identifier of a reference picture (which may be indicated, for example, per reference picture, and its type may depend on the prediction direction and / or the reference picture type, and which may be accompanied by other relevant information, such as the reference picture list or the like to which the reference index applies); - a horizontal motion vector component (which may be indicated, for example, per prediction block or per reference index or similar); - a vertical motion vector component (which may be indicated, for example, per prediction block or per reference index or similar); - one or more parameters, such as a picture order count difference and / or a relative camera spacing between a picture containing or associated with motion parameters and its reference pictures, which may be used to scale a horizontal motion vector component and / or a vertical motion vector component in one or more motion vector prediction processes (wherein the one or more parameters may be indicated, for example, per reference picture or per reference index or the like); - the coordinates of the block to which the motion parameters and / or motion information apply, e.g. the coordinates of the top left sample of the block in luma sample units; - The extent (eg width and height) of the block to which the motion parameters and / or motion information apply.

[0120] Compared to previous video coding standards, the General Video Codec (H.266 / VVC) introduces multiple new coding tools, such as the following: Intra-frame prediction -67 Intra-frame modes with wide-angle mode expansion - Block size and mode dependent 4-tap interpolation filter -Position Dependent Intra Prediction Combination (PDPC) -Cross-Component Linear Model Intra Prediction (CCLM) -Multiple reference lines intra prediction - Intra-frame sub-division -Weighted intra prediction with matrix multiplication Inter-picture prediction - Block motion replication with spatial, temporal, history-based and pairwise average merge candidates -Affine motion inter-frame prediction - Sub-block based temporal motion vector prediction - Adaptive motion vector resolution - 8×8 block-based motion compression for temporal motion prediction - High-precision (1 / 16 pixel) motion vector storage and motion compensation with 8-tap interpolation filtering for luma components and 4-tap interpolation filtering for chroma components -Triangulation - Combined intra and inter prediction -Merge with MVD (MMVD) -Symmetrical MVD encoding - Bidirectional optical flow -Decoder side motion vector refinement - Bidirectional prediction with CU level weights Transform, quantization and coefficient coding - Multiple main transform options with DCT2, DST7 and DCT8 - Secondary transformation for low frequency region -Sub-block transform for inter prediction residual - Dependent quantization with max QP increased from 51 to 63 - Transform coefficient coding with sign data hiding -Transform skip residual coding Entropy coding -Arithmetic coding engine with adaptive dual-window probability update In-loop filter -Intra-annular plastic surgery - Deblocking filter with strong longer filters - Sample Adaptive Offset - Adaptive loop filter Screen content encoding: - Current picture reference with reference area restrictions 360-degree video encoding -Horizontal surround motion compensation Advanced syntax and parallel processing - Reference picture management with direct reference picture list signaling - Tile groups with rectangular tile groups

[0121] Partitioning in VVC is performed similarly to HEVC, i.e., each picture is divided into coding tree units (CTUs). Pictures can also be divided into slices, tiles, bricks, and sub-pictures. CTUs can be split into smaller CUs using a quadtree structure. Each CU can be partitioned using quadtrees and nested multi-type trees, including ternary and binary splits. However, there are specific rules for inferring partitions in picture boundaries, and redundant split patterns are not allowed in nested multi-type partitions.

[0122] Among the new coding tools listed above, the cross-component linear model (CCLM) prediction mode is used in VVC to reduce cross-component redundancy. Among them, the chroma samples are predicted based on the reconstructed luminance samples of the same CU through the following linear model: pred C (i,j)=α·rec L ′(i,j)+β (Equation 1a)

[0123] where pred C (i, j) represents the predicted chroma sample in the CU, and rec L ′(i,j) represents the downsampled reconstructed luma samples of the same CU.

[0124] Alternatively, the following equation can be used for CCLM: pred C (i,j)=α·rec′ L (i,j) >>k+β (Equation 1b) The >> operation represents a bit shift right by value k.

[0125] The CCLM parameters (α and β) are derived using up to four adjacent chroma samples and their corresponding downsampled luma samples. Assuming the current chroma block size is W×H, W' and H' are set to -W'=W, H'=H, when LM mode is applied; -W'=W+H, when LM-A mode is applied; -H'=H+W, when LM-L mode is applied;

[0126] In this article, LM-A mode refers to linear model_above, in which only the upper template (i.e., the sample values ​​from the neighboring position above the CU) is used to calculate the linear model coefficients. In order to obtain more samples, the upper template is expanded to (W+H). LM-L mode refers to linear model_left, in which only the left template (i.e., the sample values ​​from the neighboring position on the left side of the CU) is used to calculate the linear model coefficients. In order to obtain more samples, the left template is expanded to (H+W). For non-square blocks, the upper template is expanded to W+W, and the left template is expanded to H+H.

[0127] The upper neighboring position is represented as S[0,-1]…S[W'-1,-1], and the left neighboring position is represented as S[-1,0]…S[-1,H'-1]. Then four samples are selected as -S[W' / 4,-1],S[3*W' / 4,-1],S[-1,H' / 4],S[-1,3*H' / 4], when LM mode is applied and both upper and left neighboring samples are available; -S[W' / 8,-1],S[3*W' / 8,-1],S[5*W' / 8,-1],S[7*W' / 8,-1], when LM-A mode is applied or only upper neighbor samples are available; -S[-1,H' / 8],S[-1,3*H' / 8],S[-1,5*H' / 8],S[-1,7*H' / 8], when LM-L mode is applied or only left neighbor samples are available;

[0128] The four neighboring luma samples at the selected position are downsampled and compared four times to find two smaller values: x0A and x1A, and two larger values: x0B and x1B. Their corresponding chroma sample values ​​are denoted as y0A, y1A, y0B, and y1B. Subsequently, xA, xB, yA, and yB are derived as: X a =(x 0 A +x 1 A +1)>>1;X b =(x 0 B +x 1 B +1)>>1;Y a =(y 0 A +y 1 A +1)>>1;Y b =(y 0 B +y 1B +1)>>1(Equation 2)

[0129] Finally, the linear model parameters α and β are obtained according to the following equations: β=Y b -α·X b (Equation 4)

[0130] Figure 5 An example of the positions of the left and above samples and the samples of the current block involved in the CCLM mode is shown.

[0131] The division operation for calculating the parameter α is implemented using a lookup table. To reduce the memory required to store the table, the diff value (the difference between the maximum and minimum values) and the parameter α are represented by exponential notation. For example, diff is approximated using a 4-bit significant part and an exponent. Therefore, for 16 values ​​of the significant part, the table for 1 / diff is reduced to 16 elements, as follows: DivTable[] = {0,7,6,5,5,4,4,3,3,2,2,1,1,1,1,0} (Equation 5)

[0132] This provides the benefit of reducing computational complexity as well as the memory size required to store the required tables.

[0133] In addition to the upper template and the left template being used to calculate the linear model coefficients together, they can also be alternatively used in 2 other LM modes, called LM_A mode and LM_L mode.

[0134] In LM_A mode, only the upper template is used to calculate the linear model coefficients. In order to obtain more samples, the upper template is expanded to (W+H). In LM_L mode, only the left template is used to calculate the linear model coefficients. In order to obtain more samples, the left template is expanded to (H+W).

[0135] For non-square blocks, expand the upper template to W+W and the left template to H+H.

[0136] To match the chroma sample positions for 4:2:0 video sequences, two types of downsampling filters are applied to the luma samples to achieve a 2:1 downsampling ratio in both the horizontal and vertical directions. The choice of downsampling filter is specified by the SPS level flag. The two downsampling filters are as follows, corresponding to "type 0" and "type 2" content, respectively:

[0137] Note that when the upper reference line is at a CTU boundary, only one luma line (common line buffer in intra prediction) is used for downsampling luma samples.

[0138] This parameter calculation is performed as part of the decoding process, not just an encoder search operation. As a result, no syntax is used to convey the α and β values ​​to the decoder.

[0139] For chroma intra mode coding, a total of 8 intra modes are allowed for chroma intra mode coding. These modes include five traditional intra modes and three cross-component linear model modes (CCLM, LM_A, and LM_L). The chroma mode signaling and derivation process are shown in Table 1. Chroma mode coding depends directly on the intra prediction mode of the corresponding luminance block. Since separate block partition structures for luminance and chrominance components are enabled in the I stripe, one chroma block can correspond to multiple luminance blocks. Therefore, for the chroma DM mode, the intra prediction mode of the corresponding luminance block covering the center position of the current chroma block is directly inherited. Table 1

[0140] Regardless of the value of sps_cclm_enabled_flag, a single binarization table is used, as shown in Table 2. Table 2

[0141] In Table 2, the first binary bit indicates whether it is normal (0) or LM mode (1). If it is LM mode, then the next binary bit indicates whether it is LM_CHROMA (0). If it is not LM_CHROMA, then the next binary bit indicates whether it is LM_L (0) or LM_A (1). For this case, when sps_cclm_enabled_flag is 0, the first binary bit of the binarization table for the corresponding intra_chroma_pred_mode can be discarded before entropy coding. Or, in other words, it is inferred that the first binary bit is 0 and therefore not encoded. This single binary table is used for both cases where sps_cclm_enabled_flag is equal to 0 and 1. The first two binary bits in Table 2 are context encoded using their own context model, and the remaining binary bits are bypass encoded.

[0142] Additionally, to reduce luma-chroma latency in dual trees, when a 64x64 luma coding tree node is partitioned with unsplit (and intra sub-partitioning (ISP) is not used for 64x64 CUs) or QT, chroma CUs in 32x32 / 32x16 chroma coding tree nodes are allowed to use CCLM in the following manner: - If a 32x32 chroma node is not split or partitioned QT split, then all chroma CUs in the 32x32 node can use CCLM. - If a 32x32 chroma node is partitioned using horizontal BT, and the 32x16 child node is not split or uses vertical BT splitting, then all chroma CUs in the 32x16 chroma node can use CCLM.

[0143] Under all other luma and chroma coding tree split conditions, CCLM is not allowed for chroma CUs.

[0144] Multi-Model LM (MMLM)

[0145] The CCLM included in VVC is extended by adding three multi-model LM (MMLM) modes. In each MMLM mode, the reconstructed neighboring samples are classified into two categories using a threshold value that is the average value of the brightness reconstruction neighboring samples. The linear model of each class is derived using the least mean square (LMS) method. For the CCLM mode, the LMS method is also used to derive the linear model. Figure 6a and Figure 6b Two luminance-to-chrominance models obtained for a luminance (Y) threshold of 17 in the sample domain and the spatial domain are shown, respectively. Each luminance-to-chrominance model has its own linear model parameters α and β. Figure 6b As can be seen in , each luminance-to-chrominance model corresponds to a spatial fragment of the content (i.e., they correspond to different objects or textures in the scene).

[0146] Convolutional Cross-Component Model (CCCM)

[0147] An improved version of cross-component prediction called CCCM uses a 2D filter kernel to derive a luma-to-chroma model. The filter coefficients are derived at the decoder side using the input data and a reconstructed set of chroma samples. For filter coefficient derivation, Fig.10 As shown in , a co-located reference sample area (composed of reconstructed luma and chroma samples) is defined for both luma and chroma, where the commonly used 4:2:0 chroma downsampling has been applied. The reference sample area for a given block can be, for example, six lines above and to the left of the given block, as Fig.10 As shown, however, any number of reference rows (which can be implemented by both the encoder and decoder) can be used. Typically, the reference samples can contain any chrominance and luma samples that have been reconstructed by both the encoder and decoder. Once the reference samples are determined, the filter coefficients can be derived using, for example, different types of linear regression tools such as ordinary least squares estimation, orthogonal matching pursuit, optimized orthogonal matching pursuit, ridge regression, or least absolute shrinkage and selection operators.

[0148] Fig.11Various examples are shown in which the filter kernel can be sized, for example, 1x3 (1D vertical), 3x1 (1D horizontal), 3x3, 7x7, or any size, and can be shaped (by selecting only a subset of all possible kernel positions) as a cross or diamond (e.g. Fig.11 When referring to samples within the filter kernel, the following notation is used: North (above), East (right), South (below), West (left), and Center, as shown in Fig.11 The letters N, E, S, W, and C are used.

[0149] Here, the overall approach of reconstructing chroma samples using convolution between the filter kernel obtained at the decoder side and the input data set is called convolutional cross-component model (CCCM). The following steps can be applied to perform CCCM operation: 1. Define co-located reference regions on luma and chroma components. 2. Downsample luma samples to match the chroma grid (optional). 3. Scan the luminance and chrominance samples of the reference region and collect available statistics (such as autocorrelation matrix and cross-correlation vector) based on the filter shape. 4. Solve for the filter coefficients by minimizing the squared error (or any other metric) based on available statistics (such as the autocorrelation matrix and cross-correlation vector). 5. Compute the predicted chroma block by convolving the downsampled luma samples with the filter kernel.

[0150] The potentially downsampled luma samples can be defined as a 2D array Y(x,y) indexed using horizontal x-coordinates and vertical y-coordinates. Let us also define the co-located chroma samples as a 2D array C(x,y) and the filter kernel (i.e., coefficients) as a 3x3 array F(i,j). At the sample level, the convolution between Y and F can be defined as

[0151] When additional data terms such as nonlinear square root terms are used, the additional convolution becomes in, As part of solving the linear equations for the 2D filter coefficients in step 4 above, the filter coefficients outside the 2D filter kernel are also obtained.

[0152] Similarly, the bias term can be added to the convolution using

[0153] Use the reconstructed samples near the chrominance block to obtain F and (ie, CCCM coefficients.) However, in cases where the immediate neighborhood cannot describe the cross-component relationship between luma and chroma, it may be necessary to consider CCCM coefficients derived from more distant samples.

[0154] Multiple Reference Line (MRL) Intra Prediction

[0155] Multiple reference line (MRL) intra prediction uses more reference lines for intra prediction. Figure 7 In Figure 1, an example of 4 reference lines is depicted, where the samples of fragments A and F are not taken from reconstructed neighboring samples, but are filled with the nearest samples from fragments B and E, respectively. HEVC intra prediction uses the nearest reference line (i.e., reference line 0). In MRL, 2 additional lines are used (reference line 1 and reference line 3).

[0156] The index of the selected reference row (mrl_idx) is signaled and used to generate the intra predictor. For reference row idx greater than 0, only additional reference row modes are included in the MPM list, and only the mpm index is signaled without the remaining modes. The reference row index is signaled before the intra prediction mode, and in case a non-zero reference row index is signaled, the planar mode is excluded from the intra prediction mode.

[0157] MRL is disabled for the first row of blocks inside a CTU to prevent the use of extended reference samples outside the current CTU row. In addition, PDPC is disabled when additional rows are used. For MRL mode, the derivation of DC values ​​in DC intra prediction mode for non-zero reference row indices is aligned with the derivation of DC values ​​for reference row index 0. MRL requires 3 neighboring luma reference rows to be stored with the CTU to generate the prediction. The CCLM tool also requires 3 neighboring luma reference rows for its downsampling filter. The definition of MLR using the same 3 rows is aligned to CCLM to reduce the storage requirements of the decoder.

[0158] Intra-frame sub-partitioning (ISP)

[0159] Intra-frame sub-partitioning (ISP) divides the luma intra prediction block vertically or horizontally into 2 or 4 sub-partitions according to the block size. For example, the minimum block size for ISP is 4x8 (or 8x4). If the block size is larger than 4x8 (or 8x4), the corresponding block is divided into 4 sub-partitions. It has been noted that M×128 (where M≤64) and 128×N (where N≤64) ISP blocks can generate potential problems with 64×64 VDPU. For example, an M×128 CU in a single tree case has an M×128 luma TB and two corresponding M / 2×64 chroma TBs. If the CU uses ISP, the luma TB will be divided into four M×32 TBs (only horizontal splitting is possible), each of which is smaller than a 64×64 block. However, in the current design of ISP, the chroma blocks are not divided. Therefore, the two chroma components will have a size larger than a 32×32 block. Similarly, a similar situation can be created using 128×N CU using ISP. Therefore, these two cases are problems for the 64×64 decoder pipeline. For this reason, the CU size that can use ISP is limited to a maximum of 64×64. All sub-partitions meet the condition of having at least 16 samples.

[0160] Matrix Weighted Intra Prediction (MIP)

[0161] The matrix weighted intra prediction (MIP) method is an intra prediction technique newly added to VVC. For predicting samples of a rectangular block of width W and height H, the matrix weighted intra prediction (MIP) takes as input a row of H reconstructed neighboring boundary samples on the left side of the block and a row of W reconstructed neighboring boundary samples above the block. If the reconstructed samples are not available, they are generated as done in conventional intra prediction. The generation of the prediction signal is based on the following three steps, which are averaging, matrix-vector multiplication, and linear interpolation, as Figure 8 shown.

[0162] Inter-frame prediction in VVC

[0163] The merge list can include the following candidates: 1) Spatial MVP from spatially adjacent CUs 2) Time MVP from Co-located CU 3) History-based MVP from FIFO table 4) Pairwise average MVP (using candidates already in the list) 5) Zero MV.

[0164] Merge mode wide motion vector difference (MMVD) refers to the MVD and resolution index signaled after signaling merge sequence.

[0165] In symmetric MVD, in case of bi-directional prediction, the motion information of list-1 is derived from the motion information of list-0.

[0166] In affine prediction, several motion vectors are indicated / signaled for different corners of a block, which are used to derive the motion vectors of the sub-blocks. In affine merge prediction, the affine motion information of a block is generated based on the normal or affine motion information of neighboring blocks.

[0167] In sub-block based temporal motion vector prediction, the motion vector of a sub-block of the current block is predicted from the appropriate sub-block in the reference frame indicated by the motion vector of the spatially neighboring block (if available).

[0168] In Adaptive Motion Vector Resolution (AMVR), the precision of MVD is signaled for each CU.

[0169] In bidirectional prediction with CU-level weight, the index indicates the weight value of the weighted average of two prediction blocks.

[0170] Bidirectional optical flow (BDOF) refines the motion vector in case of bidirectional prediction. BDOF generates two prediction blocks using the signaled motion vector. Then, motion refinement is calculated to minimize the error between the two prediction blocks using its gradient value. The final prediction block is refined using the motion refinement and gradient value.

[0171] Bidirectional prediction (BCW) and weighted prediction (WP) with CU level weights

[0172] In HEVC, a bidirectional prediction signal is generated by averaging two prediction signals obtained from two different reference pictures and / or using two different motion vectors. In VVC, the bidirectional prediction mode is extended beyond simple averaging to allow a weighted average of the two prediction signals. P bi-pred =((8-w)*P0+w*P1+4)>>3

[0173] Five weights are allowed in weighted average bidirectional prediction, w∈{-2,3,4,5,10}. For each bidirectional prediction CU, the weight w is determined in one of two ways: 1) For non-merged CUs, the weight index is signaled after the motion vector difference; 2) For merged CUs, the weight index is inferred from neighboring blocks based on the merge candidate index. BCW is only applied to CUs with 256 or more luma samples (i.e., CU width multiplied by CU height is greater than or equal to 256). For low-latency pictures, all 5 weights are used. For non-low-latency pictures, only 3 weights (w∈{3, 4, 5}) are used.

[0174] At the encoder, fast search algorithms are applied to find the weight indices without significantly increasing the encoder complexity. These algorithms are summarized as follows. - When combined with AMVR, unequal weights are only conditionally checked for 1-pixel and 4-pixel motion vector precision if the current picture is a low-latency picture. - When combined with affine, affine ME will be performed for unequal weights if and only if the affine mode is selected as the current best mode. - Unequal weights are only conditionally checked when the two reference pictures in bidirectional prediction are the same.

[0175] Unequal weights are not searched when certain conditions are met, which depend on the POC distance between the current picture and its reference pictures, the coding QP, and the temporal level.

[0176] The BCW weight index is encoded using one context coded binary bit followed by a bypass coded binary bit. The first context coded binary bit indicates whether equal weights are used; if unequal weights are used, an additional binary bit is signaled using bypass coding to indicate which unequal weights are used.

[0177] Weighted prediction (WP) is a coding tool supported by the H.264 / AVC and HEVC standards to efficiently encode video content using fading. Support for WP is also added to the VVC standard. WP allows weighting parameters (weights and offsets) to be signaled for each reference picture in each of the reference picture lists L0 and L1. Then, during motion compensation, the weights and offsets of the corresponding reference pictures are applied. WP and BCW are designed for different types of video content. In order to avoid the interaction between WP and BCW, which will complicate the VVC decoder design, if the CU uses WP, the BCW weight index is not signaled, and w is inferred to be 4 (ie, equal weights are applied). For the merged CU, the weight index is inferred from the neighboring blocks based on the merge candidate index. This can be applied to the normal merge mode and the inherited affine merge mode. For the constructed affine merge mode, affine motion information is constructed based on the motion information of up to 3 blocks. The BCW index for the CU using the constructed affine merge mode is simply set to be equal to the BCW index of the first control point MV.

[0178] In VVC, CIIP and BCW cannot be applied jointly to a CU. When a CU is encoded using CIIP mode, the BCW index of the current CU is set to 2, such as equal weight.

[0179] Combined Inter and Intra Prediction (CIIP)

[0180] In VVC, when a CU is encoded in merge mode, if the CU contains at least 64 luma samples (i.e., the CU width multiplied by the CU height is equal to or greater than 64), and if both the CU width and the CU height are less than 128 luma samples, an additional flag is signaled to indicate whether the combined inter- and intra-prediction (CIIP) mode applies to the current CU. As its name indicates, CIIP prediction combines the inter-prediction signal with the intra-prediction signal. The inter-prediction signal P in CIIP mode inter The intra prediction signal P is derived using the same inter prediction process applied to the normal merge mode; and intra It is derived after the regular intra prediction process with planar mode. The intra and inter prediction signals are then combined using weighted averaging, where the weight values ​​depend on the top and left neighboring blocks (in Fig.12 The coding mode is calculated as follows: - if the top neighbor is available and is intra-coded, set isIntraTop to 1, otherwise set isIntraTop to 0; - if the left neighbor is available and is intra-coded, set isIntraLeft to 1, otherwise set isIntraLeft to 0; - If (isIntraLeft + isIntraTop) is equal to 2, then set wt to 3; - Otherwise, if (isIntraLeft + isIntraTop) is equal to 1, set wt to 2; - Otherwise, set wt to 1.

[0181] The CIIP forecast is formed as follows: P CIIP =((4-wt)*P inter +wt*P intra +2)>>2

[0182] Local Illumination Compensation (LIC)

[0183] LIC is an inter-prediction technique to model the local illumination variation between the current block and its prediction block as a function of the local illumination variation between the current block template and the reference block template. The parameters of the function can be represented by a scale α and an offset β, which form a linear equation, i.e., α*p[x]+β to compensate for illumination variation, where p[x] is the reference sample pointed to by MV at position x on the reference picture. Since α and β can be derived based on the current block template and the reference block template, no signaling overhead is required for them, except for signaling the LIC flag for AMVP mode to indicate the use of LIC.

[0184] The local illumination compensation proposed in JVET-O0066 is used in ECM for unidirectionally predicted inter CUs with the following modifications. Neighboring samples within the frame can be used to derive LIC parameters; Disable LIC for blocks with less than 32 luma samples; For both non-sub-block and affine modes, LIC parameter derivation is performed based on the template block samples corresponding to the current CU instead of the partial template block samples corresponding to the first top-left 16x16 unit; Generate samples of the reference block template by using MC with the block MV without rounding it to integer pixel precision.

[0185] Inter-coding in modern video coding standards such as H.266 uses single-tree motion compensated prediction: regions from one or more reference frames are copied to the target frame using sub-pixel interpolation. Additional tools such as weighted prediction (WP), bidirectional prediction with CU-level weights (BCW), combined inter- and intra-frame prediction (CIIP), or local illumination compensation (LIC) are used to account for changes in illumination and / or color difference between the reference frame and the target frame. Single-tree coding signals a set of partition information per block, and the partition is shared between luma and chroma components. Due to the small amount of chroma-specific coding tools, inter-coding usually implicitly sets a higher priority for luma blocks compared to chroma. Even if chroma-specific tools such as CCCM or CCLM are used, they will not be very effective because chroma blocks in reference frames are usually of higher quality than cross-component blocks predicted by intra-frames. In order to benefit from cross-component tools in inter prediction, there should be a way to model chroma using mainly co-located motion compensated luma and chroma.

[0186] We now introduce an improved cross-component residual model for prediction.

[0187] According to an embodiment, a method for directly mapping luma residuals into chroma predictions is provided. Specifically, the mapping can be applied in single-tree inter-coded blocks, where the same partition is shared between luma blocks and chroma blocks. An intuitive way to perform the mapping after decoding the luma residuals of a single-tree inter-block is disclosed. The method can also be extended to dual-tree blocks or any other partitioning scheme. Alternative configurations such as chroma to chroma are also possible.

[0188] Single-tree coding shares the same partitions and motion between luma (Y) and chroma (Cb, Cr) components. Typically, the reconstruction process for single-tree inter-coded blocks is as follows: first, motion compensated prediction is performed using reference frames and sub-pixel motion vectors; second, additional adjustments are made using tools such as WP, BCW, CIIP, or LIC; third, residuals are applied. These steps can be applied to luma and chroma components consecutively. Cross-component prediction has been shown to locally model chroma using linear or nonlinear models that map luma components into chroma. Such a model can be formulated as: C=f(Y)+∈ c , where C is the chromaticity, f(·) is the function that maps luminance to chromaticity, and ∈ c is the remaining chroma residual. In existing cross-component tools (such as CCCM), the function f(·) is obtained using neighboring reference rows, and the function f(·) is applied using the co-located luma value as input. Typically, cross-component tools are used for dual-tree or single-tree intra coding. For single-tree inter coding, the function f(·) can be obtained using the co-located luma after inter prediction, including motion compensation and optionally any additional tools such as WP / BCW / CIIP / LIC, but in applying the luma residual ∈ y Before, the final chrominance prediction maps the luma residual to the chrominance prediction, C=f(Y+∈ y )+∈ c .

[0189] The parameters of the model f(·) may be obtained using CCCM, CCLM or any other cross-component method. The model may be one-dimensional (1D) or two-dimensional (2D) and may include nonlinear terms.

[0190] According to an embodiment, there are mainly four steps of cross-component residual model (CCRM). These steps are listed as follows: 1) Perform motion compensated prediction on the luma component Y and chroma components Cb, Cr (optionally including additional refinements such as WP / BCW / CIIP / LIC). 2) Use the co-located luminance prediction and chrominance prediction to obtain the function f for the chrominance components Cb and Cr respectively Cb 、f Cr . 3) Reconstruct the brightness residual ∈ y . 4) Obtain chromaticity prediction C Cb =f Cb (Y+∈ y ) and C Cr =f Cr (Y+∈ y ).

[0191] Signaling for CCRM can be done using, for example, TU / PU / CU level flags (joint flag for Cb, Cr) or flags (separate flags for Cb, Cr). Alternatively, tools that provide additional prediction refinement can be applied after CCRM. In this case, the changed steps would be, 1. Perform motion compensated prediction with additional refinements for the luma component Y and the chroma components Cb, Cr, where additional refinements such as WP / BCW / CIIP / LIC are applied only to the luma component Y. 2. (Unchanged) Use the co-located luminance prediction and chrominance prediction to obtain the function f for the chrominance components Cb and Cr respectively Cb 、f Cr . 3. (Unchanged) Reconstruct the brightness residual ∈ y . 4. (Unchanged) Get chromaticity prediction Chromaticity prediction C Cb =f Cb (Y+∈ y ) and C Cr =f Cr (Y+∈ y ). 5. Apply additional refinements such as WP / BCW / CIIP / LIC for Cb and Cr prediction.

[0192] After the last step, which is step 4 or 5, depending on the above two cases, the decoder can reconstruct and add the chroma residual ∈ Cb ,∈ Cr , to further refine the reconstruction. The above listing predicts chrominance from luma, but the CCRM can also be configured to perform chrominance-to-chrominance or chrominance-to-luminance prediction, as will be shown later.

[0193] Even after adding the luma residual, the derived model f(·) should maintain its stability. The following step 4a (which is an alternative to step 4 above) can be used to mitigate possible problems caused by large magnitude or high variance of the luma residual: 4a. Obtain chroma predictions and in represents a filtered version of the original luma residual. This filtering can be, for example, clipping with a large magnitude or spatial operation such as a low pass filter, or multiplying the residual by a predetermined or signaled value, for example, by 0.5 or 0.75. The filtering does not have to be the same for the chroma components Cb and Cr.

[0194] For some cases, it may be beneficial to perform step 4 as a weighted sum of the original chrominance prediction and the CCRM prediction, for example as follows: 4b. Get chromaticity prediction and in and is the initial inter-frame predicted chrominance block, and α, β are the mixing weights defined for the chrominance components Cb and Cr respectively.

[0195] As another example, step 4 may be performed by applying the mapping function f(·) only to the luma residual, e.g., as follows: 4c. Get chromaticity prediction and in and is the initial inter-frame predicted chrominance block, and α, β are the mixing weights defined for the chrominance components Cb and Cr respectively.

[0196] Any combination of steps 4, 4a, 4b and 4c above is possible, for example a weighted sum w0*4a+w1*4b can be used, where w0 and w1 are weights. The selection between 4, 4a, 4b and 4c can occur at the sub-block level or even at the sample level. A further extension of the above approach is to include chroma residuals in the model, leading to chroma-to-chroma prediction: or in and Map Cr to Cb and Cb to Cr, respectively. Typically, due to processing order, the Cb to Cr model is used. In some cases, it may be advantageous to derive a weighted sum of two different cross-component models as: or Among them, α and β are in C Cb With C Cr When the processing order of the components allows prediction of luma from chroma, such a predictor can be designed as: Y=f(C+∈ c ), where C can be either Cb or Cr. In chroma-to-luminance prediction, the input or output of f(·) can be upsampled to match the luma grid. Alternatively, we can also use the two chroma components to predict luma as: Y=α*f0(Cb+∈ Cb )+β*f1(Cr+∈ Cr ), where α and β are weights, and f0, f1 map Cb to Y and Cr to Y respectively.

[0197] Some example embodiments will be described below.

[0198] In an embodiment, the co-located luma component may be downsampled to match the subsampled chroma grid.

[0199] In an embodiment, the co-located luminance component (including the residual) may be filtered with any 1D or 2D filter.

[0200] In an embodiment, the mapping function f(·) may be obtained using any method for deriving a cross-component model such as CCCM or CCLM.

[0201] In an embodiment, the mapping function f(·) may include any number of filter taps and may include non-linear terms, such as quadratic terms.

[0202] In an embodiment, the mapping function f(·) may be 1D or 2D.

[0203] In an embodiment, the mapping function f(·) may map luma to chroma or chroma to chroma. Any weighted sum of the two versions may be used to obtain the final prediction.

[0204] In an embodiment, the use of CCRM may be signaled from the encoder to the decoder at TU, PU, ​​CU, CTU, slice, frame or sequence level.

[0205] In an embodiment, the use of CCRM may be signaled for different components such as Cb, Cr and Y individually or jointly.

[0206] In an embodiment, the luma residual may be clipped or spatially filtered before applying the cross-component model.

[0207] In an embodiment, the type and amount of filtering applied to the luma residual in step 4a may be determined separately for Cb and Cr.

[0208] In an embodiment, the type and amount of filtering applied to the luma residual in step 4a may be determined based on the output of f(·).

[0209] In an embodiment, the type and amount of filtering applied to the luma residual in step 4a may be applied differently at the sub-block or sample level.

[0210] In an embodiment, the mixing weights α and β of step 4b may be defined independently or jointly for Cb and Cr.

[0211] In an embodiment, the mixing weights α and β of step 4b may be defined at a sub-block level or even at a sample level.

[0212] In an embodiment, the mixing weights α and β of step 4b may be signaled from the encoder to the decoder at TU, CU, CTU, slice, frame or sequence level.

[0213] In an embodiment, the block prediction may be obtained as a weighted sum of different mapping functions such as the mapping functions described in steps 4, 4a and 4b.

[0214] In an embodiment, block prediction may use different mapping functions at the sub-block level.

[0215] In an embodiment, block prediction may use different mapping functions based on thresholds derived on luma, luma residual or chroma. For example, a threshold (based on, for example, the average luma over the block) may be set, and below or above the threshold, different mapping functions are derived and used.

[0216] In an embodiment, the mapping function f(·) may map chrominance to luma. In chrominance-to-luma prediction, the input of f(·) may be upsampled to match the luma grid. In chrominance-to-luma prediction, the output of f(·) may be upsampled to match the luma grid.

[0217] Fig. 9 A method according to an aspect is shown in the figure, wherein the method includes: receiving (900) an image block unit of a frame, the image block unit including samples in a first chroma channel, a second chroma channel and a luma channel; defining (902) a co-located reference area on a luma component of the luma channel, a chroma component of the first chroma channel and a chroma component of the second chroma channel; performing (904) motion-compensated prediction on the luma component, the first chroma component and the second chroma component using a reference frame and a motion vector; obtaining (906) a first mapping function and a second mapping function using the co-located luma prediction and chroma prediction, the first mapping function mapping the luma component to the first chroma component and the second mapping function mapping the luma component to the second chroma component; reconstructing (908) a luma residual; and obtaining (910) a first chroma prediction by using the first function and the luma residual, and obtaining a second chroma prediction by using the second function and the luma residual.

[0218] According to one aspect, an apparatus includes: a component for receiving an image block unit of a frame, the image block unit including samples in color channels including at least one chrominance channel and a luma channel; a component for determining an intra-prediction direction based on the samples of the luma channel of the image block unit; a component for selecting a low-frequency inseparable transform index for a low-frequency inseparable transform using the determined intra-prediction direction; and a component for performing the low-frequency inseparable transform using the selected index.

[0219] According to an embodiment, the device comprises means for downsampling the samples of the luma channel to correspond to the size of the samples of the chroma samples before determining the filter coefficients.

[0220] As another aspect, a device is provided, comprising: at least one processor and at least one memory, wherein the at least one memory stores codes, wherein when the codes are executed by the at least one processor, the device at least performs: a component of receiving an image block unit of a frame, wherein the image block unit includes samples in color channels including at least one chrominance channel and a luminance channel; reconstructing samples of the luminance channel of the image block unit; determining an intra-frame prediction direction based on the samples of the luminance channel of the image block unit; selecting a low-frequency inseparable transform index for a low-frequency inseparable transform using the determined intra-frame prediction direction; and performing the low-frequency inseparable transform using the selected index.

[0221] Such means may include, for example, Figure 1 , Figure 2 , Figure 4a and Figure 4b The functional units disclosed in any one of the for implementing the embodiments.

[0222] Such an apparatus also includes code stored in the at least one memory, which, when executed by the at least one processor, causes the apparatus to perform one or more embodiments disclosed herein.

[0223] Fig.1315 is a graphical representation of an example multimedia communication system in which various embodiments can be implemented. Data source 1510 provides source signals in analog, uncompressed digital or compressed digital formats or any combination of these formats. Encoder 1520 may include preprocessing or be connected to preprocessing, such as data format conversion and / or filtering of source signals. Encoder 1520 encodes source signals into coded media bitstreams. It should be noted that the bitstream to be decoded can be received directly or indirectly from a remote device located in virtually any type of network. In addition, bitstreams can be received from local hardware or software. Encoder 1520 is capable of encoding more than one media type (such as audio and video), or more than one encoder 1520 may be required to encode different media types of source signals. Encoder 1520 can also obtain synthetically generated inputs, such as graphics and text, or it can generate coded bitstreams of synthetic media. In the following, only the processing of a coded media bitstream of a media type is considered to simplify the description. However, it should be noted that usually real-time broadcast services include several streams (usually at least one audio, video and text subtitle stream). It should also be noted that the system may include many encoders, but in the figure, only one encoder 1520 is shown to simplify the description without lack of generality. It should also be understood that although the text and examples contained herein may specifically describe the encoding process, those skilled in the art will understand that the same concepts and principles also apply to the corresponding decoding process, and vice versa.

[0224] The coded media bitstream may be transmitted to a storage device 1530. The storage device 1530 may include any type of mass storage for storing coded media bitstreams. The format of the coded media bitstream in the storage device 1530 may be a basic self-contained bitstream format, or one or more coded media bitstreams may be encapsulated into a container file, or the coded media bitstream may be encapsulated into a fragment format suitable for DASH (or a similar streaming system) and stored as a fragment sequence. If one or more media bitstreams are encapsulated in a container file, a file generator (not shown in the figure) may be used to store one or more media bitstreams in a file and create file format metadata that may also be stored in the file. The encoder 1520 or the storage device 1530 may include a file generator, or the file generator may be operably attached to the encoder 1520 or the storage device 1530. Some systems operate "on-site", i.e., storage is omitted and the coded media bitstream is transmitted directly from the encoder 1520 to the sender 1540. The coded media bitstream may then be transmitted to the sender 1540, also referred to as a server, based on demand. The format used in the transmission can be an elementary self-contained bitstream format, a packetized stream format, a fragment format suitable for DASH (or similar streaming systems), or one or more coded media bitstreams can be encapsulated into a container file. The encoder 1520, storage device 1530, and server 1540 can reside in the same physical device, or they can be included in separate devices. The encoder 1520 and server 1540 can operate with live real-time content, in which case the coded media bitstreams are typically not stored permanently, but are cached for small time periods in the content encoder 1520 and / or server 1540 to smooth out processing delays, transmission delays, and variations in the coded media bitrate.

[0225] The server 1540 sends the coded media bitstream using a communication protocol stack. The stack may include, but is not limited to, one or more of a real-time transport protocol (RTP), a user datagram protocol (UDP), a hypertext transfer protocol (HTTP), a transmission control protocol (TCP), and an Internet protocol (IP). When the communication protocol stack is packet-oriented, the server 1540 encapsulates the coded media bitstream into packets. For example, when using RTP, the server 1540 encapsulates the coded media bitstream into RTP packets according to the RTP payload format. Typically, each media type has a dedicated RTP payload format. It should be noted again that the system may include more than one server 1540, but for simplicity, the following description only considers one server 1540.

[0226] If the media content is encapsulated in a container file for storage device 1530 or for inputting data to sender 1540, sender 1540 may include or be operably attached to a "send file parser" (not shown in the figure). In particular, if the container file is not transmitted as such, but at least one of the contained coded media bitstreams is encapsulated for transmission over a communication protocol, the send file parser locates the appropriate portion of the coded media bitstream to be transmitted over the communication protocol. The send file parser can also help create the correct format for the communication protocol, such as packet headers and payloads. The multimedia container file may contain encapsulation instructions, such as a hint track in ISOBMFF, for encapsulating at least one of the contained media bitstreams on the communication protocol.

[0227] Server 1540 may or may not be connected to gateway 1550 via a communication network (e.g., a CDN, the Internet, and / or a combination of one or more access networks). A gateway may also or alternatively be referred to as a middlebox. For DASH, a gateway may be an edge server (of a CDN) or a web proxy. Note that a system may typically include any number of gateways or the like, but for simplicity, the following description considers only one gateway 1550. Gateway 1550 may perform different types of functions, such as converting a packet stream from one communication protocol stack to another, merging and forking data streams, and manipulating data streams according to downlink and / or receiver capabilities, such as controlling the bit rate of a forwarded stream according to prevailing downlink network conditions. In various embodiments, gateway 1550 may be a server entity.

[0228] The system includes one or more receivers 1560, which are generally capable of receiving, demodulating and decapsulating the transmitted signal into a coded media bitstream. The coded media bitstream can be transmitted to a recording storage device 1570. The recording storage device 1570 may include any type of large-capacity storage for storing coded media bitstreams. The recording storage device 1570 may alternatively or additionally include a computing memory, such as a random access memory. The format of the coded media bitstream in the recording storage device 1570 may be a basic self-contained bitstream format, or one or more coded media bitstreams may be encapsulated into a container file. If there are multiple coded media bitstreams associated with each other (e.g., audio streams and video streams), a container file is generally used, and the receiver 1560 includes or is attached to a container file generator that generates a container file from an input stream. Some systems operate "on-site", i.e., the recording storage device 1570 is omitted and the coded media bitstream is transmitted directly from the receiver 1560 to the decoder 1580. In some systems, only the most recent portion of the recorded stream (eg, the most recent 10 minute excerpt of the recorded stream) is retained in the recording storage device 1570, while any earlier recorded data is discarded from the recording storage device 1570.

[0229] The coded media bitstream may be transferred from the recording storage device 1570 to the decoder 1580. If there are many coded media bitstreams (e.g., audio streams and video streams) associated with each other and encapsulated in a container file or a single media bitstream is encapsulated in a container file, for example, for easier access, a file parser (not shown in the figure) is used to decapsulate each coded media bitstream from the container file. The recording storage device 1570 or the decoder 1580 may include a file parser, or the file parser is attached to the recording storage device 1570 or the decoder 1580. It should also be noted that the system may include many decoders, but only one decoder 1570 is discussed here to simplify the description without lack of generality.

[0230] The coded media bitstream may be further processed by a decoder 1570, the output of which is one or more uncompressed media streams. Finally, a renderer 1590 may reproduce the uncompressed media streams using a speaker or display, for example. The receiver 1560, the recording storage device 1570, the decoder 1570, and the renderer 1590 may reside in the same physical device, or they may be included in separate devices.

[0231] The sender 1540 and / or the gateway 1550 may be configured to perform switching between different representations, such as switching between different viewports of 360-degree video content, view switching, bit rate adaptation and / or fast start, and / or the sender 1540 and / or the gateway 1550 may be configured to select the representation to be sent. The switching between different representations may occur for a variety of reasons, such as in response to a request from the receiver 1560 or the general conditions of the network transmitting the bitstream, such as throughput. In other words, the receiver 1560 may initiate switching between representations. The request from the receiver may be, for example, a request for a fragment or sub-fragment from a representation different from an earlier representation, a request for a change in the scalability layer and / or sub-layer sent, or a change in a rendering device with different capabilities compared to a previous one. The request for the fragment may be an HTTP GET request. The request for the fragment may be an HTTP GET request with a byte range. Additionally or alternatively, bitrate adjustment or bitrate adaptation may be used, for example, to provide so-called fast start in streaming services, where the bitrate of the stream sent is lower than the channel bitrate after starting or randomly accessing the streaming, so as to start playback immediately and achieve a buffer occupancy level that tolerates occasional packet delays and / or retransmissions. Bitrate adaptation may include multiple representation or layer up-switching and representation or layer down-switching operations occurring in various orders.

[0232] Decoder 1580 can be configured to switch between different representations, such as for switching between different viewports of 360-degree video content, view switching, bit rate adaptation and / or fast start, and / or decoder 1580 can be configured to select the representation sent. Switching between different representations can occur for a variety of reasons, such as to achieve faster decoding operations or to adapt the sent bitstream (e.g., based on the bitrate) to the prevailing conditions (e.g., throughput) of the network that transmits the bitstream. For example, if a device including decoder 1580 is multitasking and uses computing resources for purposes other than decoding video bitstreams, faster decoding operations may be required. In another example, when content is played back at a faster speed than normal playback speed, for example, twice or three times faster than the conventional real-time playback rate, faster decoding operations may be required.

[0233] In the above, some embodiments have been described with reference to and / or using the terminology of HEVC and / or VVC. It should be understood that the embodiments may be implemented similarly to any video encoder and / or video decoder.

[0234] Where example embodiments have been described above with reference to an encoder, it will be appreciated that the resulting bitstream and decoder may have corresponding elements therein. Likewise, where example embodiments have been described with reference to a decoder, it will be appreciated that the encoder may have a structure and / or computer program for generating a bitstream to be decoded by a decoder. For example, some embodiments have been described relating to generating prediction blocks as part of encoding. Embodiments may be similarly implemented by generating prediction blocks as part of decoding, where encoding parameters decoded from the bitstream are different from encoding parameters determined by the encoder, such as horizontal offsets and vertical offsets.

[0235] The embodiments of the present invention described above describe the codec according to separate encoder and decoder devices to help understand the processes involved. However, it should be understood that the devices, structures and operations can be implemented as a single encoder-decoder device / structure / operation. In addition, the encoder and decoder can share some or all common elements.

[0236] Although the above examples describe embodiments of the present invention operating within a codec within an electronic device, it should be understood that the present invention as defined in the claims may be implemented as part of any video codec. Thus, for example, embodiments of the present invention may be implemented in a video codec that may implement video encoding via a fixed or wired communication path.

[0237] Hence, the user equipment may comprise a video codec such as those described in the embodiments of the invention above.It should be understood that the term "user equipment" is intended to cover any suitable type of wireless user equipment, such as a mobile phone, a portable data processing device or a portable web browser.

[0238] Furthermore, elements of a public land mobile network (PLMN) may also include a video codec as described above.

[0239] In general, various embodiments of the present invention may be implemented in hardware or dedicated circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Although various aspects of the present invention may be shown and described as block diagrams, flow charts, or using some other graphical representation, it should be understood that the boxes, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general hardware or controllers or other computing devices, or some combination thereof in non-limiting examples.

[0240] Embodiments of the present invention may be implemented by computer software executed by a data processor of a mobile device, such as in a processor entity, or by hardware, or by a combination of software and hardware. Furthermore, in this regard, it should be noted that any box of the logic flow in the accompanying drawings may represent program steps, or interconnected logic circuits, boxes and functions, or a combination of program steps and logic circuits, boxes and functions. The software may be stored on such physical media as a memory chip, or a memory block implemented within a processor, a magnetic medium (such as a hard disk or floppy disk), and an optical medium (such as, for example, a DVD and its data variant CD).

[0241] The memory may be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. As non-limiting examples, the data processor may be of any type suitable for the local technical environment and may include one or more of a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture.

[0242] Embodiments of the present invention may be implemented in various components such as integrated circuit modules. The design of integrated circuits is a highly automated process. Complex and powerful software tools are available to convert a logic level design into a semiconductor circuit design that is ready to be etched and formed on a semiconductor substrate.

[0243] Programs such as those offered by Synopsys, Inc. of Mountain View, Calif., automatically route conductors and locate components on a semiconductor chip using well-established design rules and a library of pre-stored design modules. Once a design for a semiconductor circuit is complete, the resulting design in a standardized electronic format (e.g., Opus, GDSII, etc.) can be transmitted to a semiconductor manufacturing facility or "fab" for fabrication.

[0244] The above description provides a complete and informative description of exemplary embodiments of the present invention by way of exemplary and non-limiting examples. However, in view of the foregoing description, various modifications and adaptations may become apparent to those skilled in the relevant art when read in conjunction with the accompanying drawings and the appended claims. However, all such and similar modifications of the teachings of the present invention will still fall within the scope of the present invention.

Claims

1. A device comprising: means for receiving an image block unit of a frame, the image block unit comprising samples in a first chroma channel, a second chroma channel, and a luma channel; means for defining a co-located reference region on a luma component of the luma channel, a chroma component of the first chroma channel, and a chroma component of the second chroma channel; means for performing motion-compensated prediction on the luma component, the first chroma component and the second chroma component using a reference frame and a motion vector; means for obtaining a first mapping function and a second mapping function using co-located luma prediction and chroma prediction, the first mapping function mapping the luma component into the first chroma component and the second mapping function mapping the luma component into the second chroma component; A component for reconstructing the brightness residual; as well as Means for obtaining a first chroma prediction by using a first function and said luma residual and obtaining a second chroma prediction by using a second function and said luma residual.

2. The device according to claim 1, comprising: Components for downsampling the co-located luma components to match the subsampled chroma grid.

3. The device according to claim 1 or 2, comprising: means for filtering said co-located luminance component using a one-dimensional filter or a two-dimensional filter, wherein texture analysis comprises gradient calculation for said samples.

4. The device according to claim 1, 2 or 3, comprising: Means for mapping the luma component to the first chroma component or the second chroma or the first chroma component to the second chroma component using the first mapping function or the second mapping function.

5. The device according to any one of claims 1 to 4, comprising: Means for signaling usage of cross-component residual models from an encoder to a decoder at transform unit, prediction unit, coding unit, coding tree unit, slice, frame, or sequence level.

6. The device according to any one of claims 1 to 5, comprising: Means for cropping or spatially filtering said luma residual prior to applying said cross-component model.

7. The device according to any one of claims 1 to 6, comprising: Means for determining a type and amount of filtering to be applied to the luma residual for the first chroma component and the second chroma component, respectively.

8. The device according to any one of claims 1 to 7, comprising: Means for determining a type and amount of filtering to be applied to the luma residual based on an output of the first mapping function or an output of the second mapping function.

9. The device according to any one of claims 1 to 8, comprising: Means for applying the type and amount of filtering differently to the luma residual at a sub-block or sample level.

10. The device according to any one of claims 1 to 9, comprising: Means for using different mapping functions in block prediction based on one or more thresholds derived on luma, luma residual, or chroma.

11. The device according to any one of claims 1 to 10, comprising: A component used to map chroma components to luma components.

12. The device according to any one of claims 1 to 11, comprising at least one of the following: A component for upsampling the input of the mapping function to match the brightness grid; Means for upsampling the output of the mapping function to match the luminance grid.

13. The device according to any one of claims 1 to 12, comprising: Means for mapping the first chrominance component to the second chrominance component.

14. An apparatus for encoding, comprising at least one processor, a memory including computer program code, the memory and the computer program code being configured to, with the at least one processor, cause the apparatus to at least perform the following: receiving an image block unit of a frame, the image block unit comprising samples in a first chrominance channel, a second chrominance channel, and a luma channel; defining a co-located reference region on a luma component of the luma channel, a chroma component of the first chroma channel, and a chroma component of the second chroma channel; performing motion-compensated prediction on the luma component, the first chroma component, and the second chroma component using a reference frame and a motion vector; Obtaining a first mapping function and a second mapping function using co-located luma prediction and chroma prediction, wherein the first mapping function maps the luma component to the first chroma component, and the second mapping function maps the luma component to the second chroma component; Reconstruct brightness residual; as well as A first chroma prediction is obtained by using a first function and the luma residual, and a second chroma prediction is obtained by using a second function and the luma residual.

15. A method comprising: receiving an image block unit of a frame, the image block unit comprising samples in a first chrominance channel, a second chrominance channel, and a luma channel; defining a co-located reference region on a luma component of the luma channel, a chroma component of the first chroma channel, and a chroma component of the second chroma channel; performing motion-compensated prediction on the luma component, the first chroma component, and the second chroma component using a reference frame and a motion vector; Obtaining a first mapping function and a second mapping function using co-located luma prediction and chroma prediction, wherein the first mapping function maps the luma component to the first chroma component, and the second mapping function maps the luma component to the second chroma component; Reconstruct brightness residual; as well as A first chroma prediction is obtained by using a first function and the luma residual, and a second chroma prediction is obtained by using a second function and the luma residual.

Images