Apparatus, method, and computer program for video coding and decoding

Abstract

A method comprising receiving an image block unit of a frame (1500), wherein the image block unit includes a sample in a color channel, the color channel includes at least one chrominance channel and one luminance channel; reconstructing the sample in the aforementioned luminance channel of the image block unit (1502); determining a reference area for predicting a target sample in at least one color channel of the image block unit (1504), wherein the aforementioned reference area includes one or more reference samples in the current block or in a block at the same location, in the current color channel / frame or in a reference color channel / frame, encoded using an intrablock copy (IBC) method; determining a block vector that indicates the spatial distance from the area of ​​the target sample to the reference area (1506); and predicting the aforementioned target sample in at least one color channel of the image block unit using an inter-component prediction model based on the reference sample in the aforementioned reference area indicated by the aforementioned block vector (1508).

Description

[Technical Field]

[0001] The present invention relates to an apparatus, method, and computer program for coding and decoding video. [Background technology]

[0002] In video coding, video and image samples are encoded using color representations such as YUV or YCbCr, which typically consist of one luminance (luma) channel and two chrominance (chroma) channels. In such cases, the luminance channel, which primarily represents the lighting of the scene, is usually coded at a specific resolution, while the chrominance channel, which typically represents the differences between specific color components, is often coded at a second resolution lower than the resolution of the luminance signal. The intention of this type of differential representation is to eliminate the correlation between color components, allowing for more efficient data compression.

[0003] Cross-Component Linear Models (CCLM) and Convolutional Cross-Component Models (CCCM) are used to predict samples in chroma channels (e.g., Cb and Cr) using inter-channel correlations (e.g., using luminance samples). Model parameters are derived based on reconstructed samples in the vicinity of a chroma block, adjacent samples at the same location within a chroma block, and reconstructed samples within the same location within a chroma block.

[0004] The assumption is that adjacent reconstructed samples have a good correlation with the samples in the prediction block. However, if adjacent samples do not correlate with the samples in the current block, or if the correlation is very small, the calculated model may not be able to efficiently predict the samples in the block. [Overview of the project]

[0005] This specification introduces improved methods for improving the efficiency of intercomponent prediction tools, in order to at least mitigate the problems described above.

[0006] The scope of protection required with respect to various embodiments of the present invention is presented by the independent claims. Embodiments and features described herein that do not fall within the scope of the independent claims should be construed as examples that are helpful in understanding various embodiments of the present invention, if any.

[0007] A method according to a first embodiment includes receiving an image block unit of a frame, wherein the image block unit includes a sample in a color channel, the color channel including at least one chrominance channel and one luminance channel; reconstructing the sample in the aforementioned luminance channel of the image block unit; determining a reference area for predicting a target sample in at least one color channel of the image block unit, wherein the aforementioned reference area includes one or more reference samples in the current block or in a block at the same location, in the current color channel or in the current frame or in a reference color channel or in a reference frame, encoded using an intra-block copy (IBC) method; determining a block vector that indicates the spatial distance from the area of ​​the target sample to the reference area; and predicting the aforementioned target sample in at least one color channel of the image block unit using an inter-component prediction model based on the reference sample in the aforementioned reference area indicated by the aforementioned block vector.

[0008] An apparatus according to a second embodiment comprises means for receiving an image block unit of a frame, wherein the image block unit includes a sample in a color channel, the color channel including at least one chrominance channel and one luminance channel; means for reconstructing the sample of the luminance channel of the image block unit; means for determining a reference area for predicting a target sample of at least one color channel of the image block unit, wherein the reference area includes one or more reference samples in the current block or in a block at the same location, in the current color channel or in the current frame or in a reference color channel or in a reference frame, encoded using an intra-block copy (IBC) method; means for determining a block vector that indicates the spatial distance from the area of ​​the target sample to the reference area; and means for predicting the target sample of at least one color channel of the image block unit using an inter-component prediction model based on the reference sample in the reference area indicated by the block vector.

[0009] According to the embodiment, the inter-component prediction method is an inter-component linear model (CCLM) or a convolutional inter-component model (CCCM).

[0010] According to one embodiment, the apparatus includes means for deriving parameters of an intercomponent model using a reference block in the current channel and a sample from a reference block in a reference channel.

[0011] According to one embodiment, the apparatus includes means for estimating a block vector from a block at the same location in a reference channel, and means for scaling the block vector to match the sampling density of the current channel.

[0012] According to one embodiment, the block vectors estimated from the reference channel are not at least partially aligned with the coordinates of blocks at the same location.

[0013] According to an embodiment, the apparatus comprises means for identifying the closest match of a reference area within the current channel using a search based on template matching within the current channel.

[0014] According to an embodiment, the apparatus calculates inter-component model parameters and applies the calculated model to - one or more samples from a reference block within the current channel, - one or more samples from a reference block within a reference channel, - one or more samples from a block at the same position within the reference channel, - one or more samples from the vicinity of a reference block within the current channel, - one or more samples from the vicinity of a reference block within the reference channel, - one or more samples from the vicinity of the current block within the current channel, - means for applying to a prediction based on one or more of one or more samples from the vicinity of a block at the same position within the reference channel.

[0015] According to an embodiment, the apparatus calculates inter-component model parameters and applies the calculated model to - the horizontal and / or vertical coordinates of the aforementioned samples, - means for applying to a prediction based on one or more of the direction gradient values of the aforementioned samples.

[0016] According to an embodiment, the apparatus comprises means for using the block vector of a block at the same position estimated from a reference channel as the initial block vector of the current block.

[0017] According to one embodiment, the apparatus includes means for generating a list of candidate block vectors from blocks encoded using an intrablock copy (IBC) method within a block at the same location in a reference channel and / or in the vicinity of the current block in the current channel.

[0018] According to the embodiment, the apparatus comprises means for obtaining the final inter-component prediction parameters of the current block using a two-step derivation, the first step of which is performed using a reference block in a reference channel pointed to by a block vector, and the second step of which is performed in the vicinity of the current block.

[0019] In a third embodiment, a device is provided comprising at least one processor and at least one memory, wherein the at least one memory stores code, which, when executed by the at least one processor, causes the device to receive an image block unit of a frame, wherein the image block unit includes a sample in a color channel, the color channel including at least one chrominance channel and one luminance channel; reconstruct the sample of the luminance channel of the image block unit; determine a reference area for predicting a target sample of at least one color channel of the image block unit, wherein the reference area includes one or more reference samples in the current block or in a block at the same location, in the current color channel or in the current frame or in a reference color channel or in a reference frame, encoded using an intra-block copy (IBC) method; determine a block vector that indicates the spatial distance from the area of ​​the target sample to the reference area; and predict the target sample of at least one color channel of the image block unit using an inter-component prediction model based on the reference sample in the reference area indicated by the block vector.

[0020] Therefore, the computer-readable storage medium containing the device and code is arranged to perform one or more of the methods and related embodiments described above, as stated above.

[0021] To better understand the present invention, references are made here to the accompanying drawings as an example. [Brief explanation of the drawing]

[0022] [Figure 1] This figure schematically shows an electronic device employing an embodiment of the present invention. [Figure 2] This figure schematically shows a user device suitable for employing embodiments of the present invention. [Figure 3] This figure further schematically illustrates an electronic device employing an embodiment of the present invention, connected using wireless and wired network connections. [Figure 4a] This figure schematically shows an encoder and decoder suitable for carrying out embodiments of the present invention. [Figure 4b] This figure schematically shows an encoder and decoder suitable for carrying out embodiments of the present invention. [Figure 5] This figure shows the sample locations used to derive the parameters of the component-cluster linear model (CCLM). [Figure 6a] This figure illustrates examples of classifying Luma samples into two classes, one within the sample domain and the other within the spatial domain. [Figure 6b] This figure illustrates examples of classifying Luma samples into two classes, one within the sample domain and the other within the spatial domain. [Figure 7] This figure shows an example of a reference sample area at the same location consisting of reconstructed luma and chroma samples, defined for both luma and chroma in the convolutional component model (CCCM). [Figure 8] This figure shows various examples of filter kernel dimensions in CCCM. [Figure 9] This figure shows an example of four reference lines adjacent to a prediction block. [Figure 10] This figure shows the matrix-weighted intra-prediction process. [Figure 11] This figure shows an example of a low-frequency non-separable transform (LFNST) process. [Figure 12] This figure shows an example of intra-template matching prediction (intra-TMP). [Figure 13a] This diagram shows an example of defining a reference area in Intrablock Copy (IBC) mode. [Figure 13b] This diagram shows an example of defining a reference area in Intrablock Copy (IBC) mode. [Figure 14a] This diagram shows the methods for horizontal and vertical flipping used in Reconstruction-Reordered IBC. [Figure 14b] This diagram shows the methods for horizontal and vertical flipping used in Reconstruction-Reordered IBC. [Figure 15] This figure shows a flowchart of a method for predicting a sample in at least one color channel according to an embodiment of the present invention. [Figure 16] This diagram illustrates the basic principles of using block vectors in Intrablock Copy (IBC) mode with the current block and referenced blocks. [Figure 17a] This figure shows an example of using block vectors of IBC coding blocks to indicate a reference area for parameter derivation of intercomponent predictions, according to various embodiments of the present invention. [Figure 17b] This figure shows an example of using block vectors of IBC coding blocks to indicate a reference area for parameter derivation of intercomponent predictions, according to various embodiments of the present invention. [Figure 17c] This figure shows an example of using block vectors of IBC coding blocks to indicate a reference area for parameter derivation of intercomponent predictions, according to various embodiments of the present invention. [Figure 17d] This figure shows an example of using block vectors of IBC coding blocks to indicate a reference area for parameter derivation of intercomponent predictions, according to various embodiments of the present invention. [Figure 17e] This figure shows an example of using block vectors of IBC coding blocks to indicate a reference area for parameter derivation of intercomponent predictions, according to various embodiments of the present invention. [Figure 17f] This figure shows an example of using block vectors of IBC coding blocks to indicate a reference area for parameter derivation of intercomponent predictions, according to various embodiments of the present invention. [Figure 18] This figure shows a schematic diagram of an exemplary multimedia communication system in which various embodiments may be implemented. [Modes for carrying out the invention]

[0023] The following describes in more detail suitable apparatuses and possible mechanisms for improving the efficiency of the inter-component prediction tool. In this regard, references to Figures 1 and 2 are first made, with Figure 1 showing a block diagram of a video coding system according to an embodiment, as a schematic block diagram of an exemplary apparatus or electronic device 50 into which the codec according to an embodiment of the present invention can be incorporated. Figure 2 shows the layout of the apparatus according to an embodiment. The elements of Figures 1 and 2 are then described.

[0024] The electronic device 50 may be, for example, a portable terminal or user device of a wireless communication system. However, it will be understood that embodiments of the present invention may be carried out in any electronic device or apparatus that may require encoding and decoding or encoding or decoding of video images.

[0025] The device 50 may include a housing 30 for housing and protecting the device. The device 50 may further include a display 32 in the form of a liquid crystal display. In other embodiments of the present invention, the display may be any suitable display technology suitable for displaying images or videos. The device 50 may further include a keypad 34. In other embodiments of the present invention, any suitable data or user interface mechanism may be employed. For example, the user interface may be implemented as a virtual keyboard or data input system as part of a touch-sensitive display.

[0026] The device may be equipped with a microphone 36 or any suitable audio input which may be a digital or analog signal input. The device 50 may further be equipped with an audio output device which, in embodiments of the present invention, may be an earphone 38, a speaker, or any one of an analog audio output connection or a digital audio output connection. The device 50 may also be equipped with a battery (or, in other embodiments of the present invention, the device may be powered by any suitable mobile energy device such as a solar cell, a fuel cell, or a spring-driven generator). The device may further be equipped with a camera which may record or capture images and / or video. The device 50 may further be equipped with an infrared port for short-range line-of-sight communication to other devices. In other embodiments, the device 50 may further be equipped with any suitable short-range communication solution such as a Bluetooth wireless connection or a USB / Firewire wired connection.

[0027] The device 50 may include a controller 56, a processor, or a processor circuit for controlling the device 50. The controller 56 may be connected to a memory 58, which in embodiments of the present invention may store both image data and audio data, and / or instructions for execution by the controller 56. The controller 56 may be further connected to a codec circuit 54, which is suitable for performing coding and decoding of audio and video data, or assists in coding and decoding performed by the controller.

[0028] The device 50 may further include a card reader 48 and a smart card 46, such as a UICC and a UICC reader, which are suitable for providing user information and for providing authentication information for user authentication and authorization on a network.

[0029] The device 50 may include a radio interface circuit 52 connected to a controller and suitable for generating wireless communication signals for communication with, for example, a cellular communication network, a wireless communication system, or a wireless local area network. The device 50 may further include an antenna 44 connected to the radio interface circuit 52 for transmitting radio frequency signals generated by the radio interface circuit 52 to other devices and for receiving radio frequency signals from other devices.

[0030] The device 50 may include a camera capable of recording or detecting individual frames, which are then passed to a codec 54 or controller for processing. The device may receive video image data for processing from another device before transmission and / or storage. The device 50 may also receive images for coding / decoding wirelessly or via a wired connection. The structural elements of the device 50 described above represent examples of means for performing the corresponding functions.

[0031] With respect to Figure 3, an example of a system in which embodiments of the present invention may be used is shown. System 10 comprises multiple communication devices that can communicate over one or more networks. System 10 may comprise any combination of wired or wireless networks, including but not limited to wireless cellular telephone networks (such as GSM, UMTS, and CDMA networks), wireless local area networks (WLANs) as defined by any of the IEEE 802.x standards, Bluetooth personal area networks, Ethernet local area networks, Token Ring local area networks, wide area networks, and the Internet.

[0032] System 10 may include both wired and wireless communication devices and / or apparatus 50 suitable for carrying out embodiments of the present invention.

[0033] For example, the system shown in Figure 3 represents a cellular network 11 and the Internet 28. Connection to the Internet 28 may include, but is not limited to, long-range wireless connections, short-range wireless connections, and various wired connections including, but not limited to, telephone lines, cable lines, power lines, and similar communication paths.

[0034] The exemplary communication devices shown in System 10 may include, but are not limited to, electronic devices or apparatus 50, a combination of a personal digital assistant (PDA) and a mobile phone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, and a notebook computer 22. Apparatus 50 may be fixed in place or movable when carried by a moving individual. Apparatus 50 may also be located in a means of transport including, but not limited to, a car, truck, taxi, bus, train, boat, airplane, bicycle, motorcycle, or any similar suitable means of transport.

[0035] Embodiments may be implemented in set-top boxes, i.e., digital television receivers, which may or may not have display or wireless capabilities; in tablets or (laptop) personal computers (PCs), which may include the implementation of hardware or software or a combination of encoders / decoders; in various operating systems; and in chipsets, processors, DSPs, and / or embedded systems that provide hardware / software-based coding.

[0036] Some or more devices may send and receive calls and messages and communicate with service providers via a wireless connection 25 to a base station 24. The base station 24 may be connected to a network server 26 that enables communication between the cellular network 11 and the internet 28. This system may include additional communication devices and various types of communication devices.

[0037] Communication devices may communicate using a variety of transmission technologies, including but not limited to, code division multiple access (CDMA), global systems for mobile communications (GSM), universal mobile telecommunications system (UMTS), time divisional multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-internet protocol (TCP-IP), short messaging service (SMS), multimedia messaging service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.11, and any similar wireless communication technologies. Communication devices involved in carrying out various embodiments of the present invention may communicate using a variety of media, including but not limited to radio, infrared, laser, cable connections, and any suitable connections.

[0038] In telecommunications and data networks, a channel may refer to either a physical channel or a logical channel. A physical channel may refer to a physical transmission medium such as a wire, while a logical channel may refer to a logical connection on a multiplexed medium capable of transmitting multiple logical channels. A channel may be used to transmit information signals, such as a bitstream, from one or more senders (or transmitters) to one or more receivers.

[0039] An MPEG-2 transport stream (TS), as defined in ISO / IEC 13818-1 or equivalently in ITU-T Recommendation H.222.0, is a multi-stream format for carrying audio, video, and other media, as well as program metadata or other metadata. Packet identifiers (PIDs) are used to identify the underlying stream (also known as the packetized underlying stream) within the TS. Therefore, logical channels within an MPEG-2 TS may be considered to correspond to specific PID values.

[0040] The available media file format standards include ISO-based media file formats (ISO / IEC 14496-12, sometimes abbreviated as ISOBMFF) and the NAL unit structured video file format (ISO / IEC 14496-15), which is derived from ISOBMFF.

[0041] A video codec consists of an encoder, which converts the input video into a compressed representation suitable for storage / transmission, and a decoder, which can return the compressed video representation to a viewable form. The video encoder and / or video decoder may also be separate from each other, i.e., they may not need to form a codec. Typically, the encoder discards some information from the original video sequence in order to represent the video in a more compact form (i.e., at a lower bitrate).

[0042] Typical hybrid video encoders, such as many implementations of ITU-T H.263 and H.264, encode video information in two stages. First, pixel values ​​within a particular image area (or "block") are predicted, for example, by motion compensation (finding and indicating an area in one of the previously coded video frames that precisely corresponds to the block being coded) or by spatial means (using pixel values ​​around the block being coded in a specified way). Next, the prediction error, i.e., the difference between the predicted block of pixels and the original block of pixels, is coded. This is usually done by transforming the difference in pixel values ​​using a specified transformation (e.g., the Discrete Cosine Transform (DCT) or a variation thereof), quantizing the coefficients, and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (image quality) and the size of the resulting coded video representation (file size or transmission bitrate).

[0043] In time prediction, the source of the prediction is a previously decoded image (also known as a reference image). In intra-block copy (IBC), also known as intra-block copy prediction, the prediction is applied similarly to time prediction, except that the reference image is the current image, and only previously decoded samples may be referenced in the prediction process. Inter-layer or inter-view prediction may be applied similarly to time prediction, but the reference image is an image decoded from another scalable layer or from another view, respectively. In some cases, inter-prediction may refer only to time prediction, but in other cases, inter-prediction may collectively refer to time prediction, and any of intra-block copy, inter-layer, and inter-view prediction, provided that they are performed in the same or similar process as time prediction. Inter-prediction or time prediction is sometimes called motion-compensated or motion-compensated prediction.

[0044] Motion compensation can be performed with either full-sample accuracy or sub-sample accuracy. In full-sample accuracy motion compensation, motion can be represented as a motion vector containing integer values ​​of horizontal and vertical displacements, and the motion compensation process uses these displacements to effectively copy a sample from a reference image. In sub-sample accuracy motion compensation, the motion vector is represented by decimal or decimal values ​​of its horizontal and vertical components. If the motion vector refers to a non-integer position in the reference image, a sub-sample interpolation process is typically invoked to calculate a predicted sample value based on the reference sample and the selected sub-sample position. The sub-sample interpolation process typically consists of horizontal filtering, which compensates for the horizontal offset with respect to the full sample position, followed by vertical filtering, which compensates for the vertical offset with respect to the full sample position. However, in some environments, vertical processing may be performed before horizontal processing.

[0045] Interpretation, sometimes called time prediction, motion compensation, or motion-compensated prediction, reduces temporal redundancy. In interpretation, the source of prediction is a previously decoded image. Intrapretation takes advantage of the fact that adjacent pixels within the same image are likely to be correlated. Intrapretation can be performed in the spatial domain or the transformation domain, meaning that either sample values ​​or transformation coefficients can be predicted. Intrapretation is typically used in intracoding where interpretation is not applicable.

[0046] One result of the coding procedure is a set of coding parameters, such as motion vectors and quantized transform coefficients. Many parameters can be entropy-coded more efficiently when they are first predicted from spatially or temporally adjacent parameters. For example, motion vectors may be predicted from spatially adjacent motion vectors, and only the relative difference with respect to this motion vector predictor may be coded. The prediction of coding parameters and intra prediction are sometimes collectively referred to as picture prediction.

[0047] FIGS. 4a and 4b show an encoder and a decoder suitable for employing embodiments of the present invention. A video codec is composed of an encoder that converts an input video into a compressed representation suitable for storage / transmission, and a decoder that can return the compressed video representation to a displayable form. Usually, the encoder discards and / or loses some information in the original video sequence in order to represent the video in a more compact form (i.e., at a lower bitrate). An example of the encoding process is shown in FIG. 4a. FIG. 4a shows the image (I n ) to be encoded, the predicted representation (P ’n ) of the image block, the prediction error signal (D n ), the reconstructed prediction error signal (D ’n ), the preliminary reconstructed image (I ’n ), the final reconstructed image (R ’n ), the transform (T) and inverse transform (T -1 ), quantization (Q) and inverse quantization (Q -1 ), entropy encoding (E), reference frame memory (RFM: reference frame memory), inter prediction (P inter ), intra prediction (P intra ), mode selection (MS: mode selection) and filtering (F: filtering).

[0048] An example of the decoding process is shown in FIG. 4b. FIG. 4b shows the predicted representation (P ’n), reconstructed prediction error signal (D ’n ), a backup reconstructed image (I ’n ), the final reconstructed image (R ’n ), inverse transform (T -1 ), inverse quantization (Q -1 ), entropy decoding (E -1 This shows reference frame memory (RFM), prediction (either inter or intra) (P), and filtering (F).

[0049] Many hybrid video encoders encode video information in two stages. First, pixel values ​​within a particular image area (or "block") are predicted, for example, by motion compensation (finding and indicating an area in one of the previously coded video frames that precisely corresponds to the block being coded) or by spatial means (using pixel values ​​around the block being coded in a specified way). Next, the prediction error, i.e., the difference between the predicted block of pixels and the original block of pixels, is coded. This is usually done by transforming the difference in pixel values ​​using a specified transform (e.g., the discrete cosine transform (DCT) or a variation thereof), quantizing the coefficients, and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (image quality) and the size of the resulting coded video representation (file size or transmission bitrate). Video codecs may also provide transform skip modes that the encoder can choose to use. In transformation skip mode, prediction errors are coded within the sample region, for example, by deriving sample-by-sample difference values ​​relative to specific adjacent samples and coding these sample-by-sample difference values ​​using an entropy coder.

[0050] Entropy coding / decoding may be performed in many ways. For example, context-based coding / decoding may be applied in both the encoder and decoder, which modifies the context state of coding parameters based on previously coded / decoded coding parameters. Context-based coding may be, for example, context-adaptive binary arithmetic coding (CABAC) or context-based variable length coding (CAVLC) or any similar entropy coding. Entropy coding / decoding may also be performed using variable-length coding schemes such as Huffman coding / decoding or Exp-Golomb coding / decoding, either as an alternative or additional method. Decoding coding parameters from an entropy-coded bitstream or codeword is sometimes called parsing.

[0051] The phrase "along a bitstream" (for example, indicating that something is along a bitstream) may be defined to refer to an out-of-band transmission, signaling, or storage in a manner that the out-of-band data is associated with the bitstream. The phrase "along a bitstream decoding" or similar phrases may refer to decoding referenced out-of-band data associated with the bitstream (which may be obtained from an out-of-band transmission, signaling, or storage). For example, an instruction "along a bitstream" may refer to metadata within a container file that encapsulates the bitstream.

[0052] The H.264 / AVC standard was developed by the Joint Video Team (JVT) of the Video Coding Experts Group (VCEG) of the Telecommunications Standardization Sector of the International Telecommunication Union (ITU-T) and the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard is published by both parent standardization organizations and is known as ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10 (also known as MPEG-4 Part 10 Advanced Video Coding (AVC)). Multiple versions of the H.264 / AVC standard exist, each incorporating new extensions or features into the specification. These extensions include Scalable Video Coding (SVC) and Multiview Video Coding (MVC).

[0053] Version 1 of the H.265 / HEVC (High Efficiency Video Coding) standard, also known as HEVC, was developed by the Joint Collaborative Team - Video Coding (JCT-VC) of VCEG and MPEG. This standard is published by both parent standardization organizations and is known as ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2 (also known as MPEG-H Part 2 High Efficiency Video Coding (HEVC)). Subsequent versions of H.265 / HEVC include extensions for scalable, multiview, fidelity range, 3D, and screen content coding, which are sometimes abbreviated as SHVC, MV-HEVC, REXT, 3D-HEVC, and SCC, respectively.

[0054] Versatile Video Coding (VVC) (MPEG-I Part 3), also known as ITU-T H.266, is a video compression standard developed by the Joint Video Experts Team (JVET) of the Video Experts Group (MPEG) (officially ISO / IEC JTC1 SC29 WG11) and the Video Coding Experts Group (VCEG) of the International Telecommunication Union (ITU) to succeed HEVC / H.265.

[0055] Some key definitions, bitstream and coding structures, and concepts of H.264 / AVC and HEVC are described in this section as examples of video encoders, decoders, encoding methods, decoding methods, and bitstream structures on which embodiments may be implemented. Some key definitions, bitstream and coding structures, and concepts of H.264 / AVC are the same as those of HEVC and are therefore described together below. The embodiments of the present invention are not limited to H.264 / AVC or HEVC, and a description of one possible basis is given on which the present invention may be implemented in part or in whole.

[0056] As with many earlier video coding standards, the syntax and semantics of the bitstream, as well as the decoding process for an error-free bitstream, are specified in H.264 / AVC and HEVC. While the encoding process is not specified, the encoder must produce a compliant bitstream. The compatibility of the bitstream and decoder can be verified using a Hypothetical Reference Decoder (HRD). These standards include coding tools to help address transmission errors and losses; however, the use of these tools in encoding is optional, and the decoding process for erroneous bitstreams is not specified.

[0057] The basic unit for input to an H.264 / AVC or HEVC encoder and output to an H.264 / AVC or HEVC decoder is, in each case, an image. The image given as input to the encoder is sometimes called the source image, and the image decoded by the decoder is sometimes called the decoded image.

[0058] The source image and the decoded image each consist of one or more sample sequences, such as one of the following sets of sample sequences. - Luma (Y) only (monochrome). - Luma and two chromosomes (YCbCr or YCgCo). - Green, blue, and red (also known as GBR or RGB). - An array representing a sample of any other unspecified monochrome or tristimulus color (e.g., also known as YZX, XYZ).

[0059] In H.264 / AVC and HEVC, an image may be either a frame or a field. A frame contains a matrix of luma samples and, optionally, corresponding chroma samples. A field is a set of alternative sample rows for a frame and may be used as an encoder input when the source signal is interlaced. A chroma sample sequence may be absent (and therefore monosampling may be used), or the chroma sample sequence may be subsampled when compared to a luma sample sequence. The chroma format can be summarized as follows: - In monochromatic sampling, there is only one sample sequence, which can nominally be considered a luma sequence. - In 4:2:0 sampling, each of the two chroma sequences has half the height and half the width of the luma sequence. - In 4:2:2 sampling, each of the two chroma sequences has the same height and half the width as the luma sequence. - In 4:4:4 sampling, if separate color planes are not used, each of the two chroma arrays will have the same height and width as the luma array.

[0060] In H.264 / AVC and HEVC, the sample sequence is coded into a bitstream as separate color planes, and each coded color plane can be decoded separately from the bitstream. When separate color planes are used, each of those color planes is processed separately as an image (by the encoder and / or decoder) using monochrome sampling.

[0061] A split may also be defined as a division of a set into subsets such that each element of the set is contained in exactly one of the subsets.

[0062] The following terms may be used when describing the operation of HEVC encoding and / or decoding: A coding block may be defined as an N×N block of samples for a certain value N, such that the division of a coding tree block into a coding block is a division. A coding tree block (CTB) may be defined as an N×N block of samples for a certain value N, such that the division of components into a coding tree block is a division. A coding tree unit (CTU) may be defined as a coding tree block of a luma sample, two corresponding coding tree blocks of a chroma sample of an image containing three sample sequences, or a coding tree block of a sample of an image coded using a syntactic structure used to code a monochrome image or three separate color planes and samples. A coding unit (CU) may be defined as a coding block of a luma sample, two corresponding coding blocks of a chroma sample of an image containing three sample sequences, or a coding block of a sample of an image coded using a syntactic structure used to code a monochrome image or three separate color planes and samples. The CU with the largest allowable size may be named LCU (largest coding unit) or coding tree unit (CTU), and the video image is divided into non-overlapping LCUs.

[0063] A CU consists of one or more prediction units (PUs) that define the prediction process for samples within the CU, and one or more transform units (TUs) that define the prediction error coding process for samples within the aforementioned CU. Typically, a CU consists of square blocks of samples with sizes selectable from a predefined set of possible CU sizes. Each PU and TU can be further subdivided into smaller PUs and TUs, respectively, to increase the granularity of the prediction and prediction error coding processes. Each PU has prediction information associated with it (e.g., motion vector information for inter-predicted PUs and intra-prediction direction information for intra-predicted PUs) that defines the type of prediction applied to the pixels within that PU.

[0064] Each TU may be associated with information describing the prediction error decoding process for the samples within the aforementioned TU (e.g., including DCT coefficient information). Typically, whether prediction error coding is applied to each CU is signaled at the CU level. If there is no residual of the prediction error associated with a CU, the TU for the aforementioned CU may be considered nonexistent. The division of the image into CUs, as well as the division of CUs into PUs and TUs, is typically signaled in a bitstream, allowing the decoder to reconstruct the intended structure of these units.

[0065] In HEVC, an image can be divided into rectangular tiles containing an integer number of LCUs. In HEVC, the division into tiles forms a regular grid, and the height and width of the tiles differ from each other by a maximum of 1 LCU. In HEVC, a slice is defined as an integer number of coding tree units contained in one independent slice segment and all subsequent dependent slice segments (if any) that precede the next independent slice segment (if any) within the same access unit. In HEVC, a slice segment is defined as an integer number of coding tree units contained in a single NAL unit that is sequentially ordered in a tile scan. Each division of an image into slice segments is a division. In HEVC, an independent slice segment is defined as a slice segment in which the values ​​of the syntactic elements of the slice segment header are not inferred from the values ​​of the preceding slice segment, and a dependent slice segment is defined as a slice segment in which the values ​​of some syntactic elements of the slice segment header are inferred from the values ​​of the preceding independent slice segment in the decoding order. In HEVC, a slice header is defined as the slice segment header of an independent slice segment, which is either the current slice segment or an independent slice segment preceding the current dependent slice segment. A slice segment header is defined as part of a coded slice segment, containing data elements for the first or all coding tree units represented within the slice segment. CUs are scanned in the raster scan order of LCUs within a tile, or, if tiles are not used, within the image. Within an LCU, CUs have a specific scan order.

[0066] The decoder reconstructs the output video by applying predictive means similar to those of the encoder to form a predictive representation of pixel blocks (using motion or spatial information created by the encoder and stored in a compressed representation), and predictive error decoding (the inverse operation of predictive error coding, which recovers the quantized predictive error signal within the spatial pixel region). After applying the predictive and predictive error decoding means, the decoder aggregates the predictive and predictive error signals (pixel values) to form an output video frame. The decoder (and encoder) may also apply additional filtering means to improve the quality of the output video before passing the output video for display and / or storing it as a predictive reference for the next frame in the video sequence.

[0067] Filtering may include, for example, one or more of the following: deblocking, sample adaptive offset (SAO), and / or adaptive loop filtering (ALF). H.264 / AVC includes deblocking, while HEVC includes both deblocking and SAO.

[0068] In typical video codecs, motion information is represented using motion vectors associated with each motion-compensated image block, such as prediction units. Each of these motion vectors represents the displacement of the image block in the image being coded (on the encoder side) or decoded (on the decoder side), and the prediction source block in one of the previously coded or decoded images. To efficiently represent motion vectors, they are typically coded differentially with respect to the block-specific prediction motion vector. In typical video codecs, the predicted motion vector is created in a predefined manner, for example, by calculating the median of the encoded or decoded motion vectors of adjacent blocks. Another method for creating motion vector predictions is to generate a list of candidate predictions from adjacent blocks and / or blocks at the same position in the time-reference image, and signal the selected candidates as motion vector predictors. In addition to predicting motion vector values, it is possible to predict which reference images will be used for motion compensation prediction, and this prediction information may be represented, for example, by the reference index of a previously coded / decoded image. Reference indices are typically predicted from adjacent blocks and / or blocks at the same location within the time-referenced image. Furthermore, representative high-efficiency video codecs employ an additional motion information coding / decoding mechanism, often called merge / merge mode, where all motion field information, including motion vectors and corresponding reference image indices, is predicted for each available list of reference images and used without modification / correction. Similarly, predicting motion field information is performed using motion field information from adjacent blocks and / or blocks at the same location within the time-referenced image, and the motion field information used is signaled within a list of motion field candidate lists filled with motion field information from available adjacent / same-location blocks.

[0069] In typical video codecs, the predicted residual after motion compensation is first transformed using a transformation kernel (such as DCT), and then coded. This is because there is often still some correlation between the residual and the transformation, and reducing this correlation can often help lead to more efficient coding.

[0070] Video coding standards and specifications may allow encoders to divide coded images into coded slices or similar. In-picture prediction is typically disabled across slice boundaries. Therefore, slices can be considered a method for dividing coded images into independently decodeable elements. In H.264 / AVC and HEVC, in-picture prediction may be disabled across slice boundaries. Therefore, slices can be considered a method for dividing coded images into independently decodeable elements, and thus slices are often considered the basic unit of transmission. Often, encoders can indicate in the bitstream which types of in-picture prediction are turned off across slice boundaries, and decoder operation takes this information into consideration, for example, when concluding which prediction sources are available. For example, a sample from an adjacent CU may be considered unavailable for intra-prediction if the adjacent CUs are in different slices.

[0071] The basic unit for the output of an H.264 / AVC or HEVC encoder and the input of an H.264 / AVC or HEVC decoder is the Network Abstraction Layer (NAL) unit. For transport over a packet-oriented network or storage in a structured file, the NAL unit may be encapsulated in a packet or similar structure. H.264 / AVC and HEVC specify a byte-stream format for transmission or storage environments that do not provide a framing structure. The byte-stream format separates NAL units from each other by attaching a start code before each NAL unit. To prevent false detection of NAL unit boundaries, the encoder implements a byte-oriented start code emulation prevention algorithm, adding an emulation prevention byte to the NAL unit payload if the start code occurs otherwise. Start code emulation prevention may always be performed, regardless of whether the byte-stream format is used, to enable simple gateway operation between packet-oriented and stream-oriented systems. A NAL unit may be defined as a syntactic structure that includes the type of data to track and, if necessary, an indication of the bytes containing that data in the form of an RBSP interspersed with an emulation prevention byte. A raw byte sequence payload (RBSP) may be defined as a syntactic structure that includes an integer number of bytes encapsulated within a NAL unit. An RBSP may be empty or take the form of a data bit string containing syntactic elements, followed by an RBSP stop bit, and then zero or more subsequent bits equal to 0.

[0072] A NAL unit consists of a header and a payload. In H.264 / AVC and HEVC, the NAL unit header indicates the type of NAL unit.

[0073] In HEVC, a 2-byte NAL unit header is used for all specified NAL unit types. The NAL unit header contains one reserved bit, a 6-bit NAL unit type indicator, a 3-bit nuh_temporal_id_plus1 indicator for the time level (which may be required to be 1 or greater), and a 6-bit nuh_layer_id syntax element. The temporal_id_plus1 syntax element may be considered a time identifier for the NAL unit, and a zero-based TemporalId variable may be derived as TemporalId=temporal_id_plus1-1. The abbreviation TID may be used interchangeably with the TemporalId variable. A TemporalId equal to 0 corresponds to the lowest time level. The value of temporal_id_plus1 must be non-zero to prevent start code emulation that includes the 2 bytes of the NAL unit header. A bitstream created by excluding all VCL NAL units with a TemporalId greater than or equal to a selected value and including all other VCL NAL units remains conforming. Therefore, images with a TemporalId equal to tid_value will not use images with a TemporalId greater than tid_value as interpredictive references. A sublayer or time sublayer may be defined as a time-scalable layer (or time layer TL (temporal layer)) of a time-scalable bitstream, consisting of VCL NAL units and associated non-VCL NAL units having a specific value for the TemporalId variable. nuh_layer_id may be understood as a scalability layer identifier.

[0074] NAL units can be classified into Video Coding Layer (VCL) NAL units and non-VCL NAL units. VCL NAL units are typically coded slice NAL units. In HEVC, a VCL NAL unit contains syntactic elements representing one or more CUs.

[0075] Non-VCL NAL units may be, for example, sequence parameter sets, image parameter sets, supplemental enhancement information (SEI) NAL units, access unit delimiters, sequence end NAL units, bitstream end NAL units, or filler data NAL units. Parameter sets may be required for the reconstruction of decoded images, while many other non-VCL NAL units are not required for the reconstruction of decoded sample values.

[0076] Parameters that do not change throughout the coding video sequence may be included in the sequence parameter set. In addition to parameters that may be required by the decoding process, the sequence parameter set may optionally include video usability information (VUI), which may include parameters that are important for buffering, image output timing, rendering, and resource reservation. In HEVC, the sequence parameter set RBSP includes parameters that may be referenced by one or more image parameter sets RBSP or one or more SEI NAL units that contain buffering period SEI messages. The image parameter set includes parameters that are likely to remain unchanged across multiple coded images. The image parameter set RBSP may include parameters that may be referenced by coded slice NAL units of one or more coded images.

[0077] In HEVC, a video parameter set (VPS) may be defined as a syntactic structure containing syntactic elements that apply to zero or more coding video sequences, determined by the content of syntactic elements found in the SPS, which are referenced by syntactic elements found in the P

[0078] A video parameter set RBSP may include parameters that can be referenced by one or more sequence parameter sets RBSP.

[0079] The relationships and hierarchy between video parameter sets (VPS), sequence parameter sets (SPS), and picture parameter sets (PPS) may be described as follows: The VPS exists one level above the SPS within the parameter set hierarchy, and in the context of scalability and / or 3D video. The VPS may contain parameters common to all slices across all (scalability or view) layers in the entire coding video sequence. The SPS contains parameters common to all slices within a particular (scalability or view) layer in the entire coding video sequence, and which may be shared by multiple (scalability or view) layers. The PPS contains parameters common to all slices in a particular layer representation (a representation of one scalability or view layer within one access unit), and which are likely to be shared by all slices across multiple layer representations.

[0080] The VPS may provide information about layer dependencies in the bitstream, and much other information applicable to all slices across all (scalability or view) layers in the entire coding video sequence. The VPS may be considered to consist of two parts: a base VPS and a VPS extension, the VPS extension being optional.

[0081] Out-of-band transmission, signaling, or storage may be used additionally or alternatively for purposes other than tolerance to transmission errors, such as ease of access or session negotiation. For example, sample entries for tracks in a file conforming to an ISO-based media file format may include parameter sets, while coded data in a bitstream is stored elsewhere in the file or in a separate file. The phrases "along the bitstream" (e.g., indicating that it is along the bitstream) or "along the coded units of the bitstream" (e.g., indicating that it is along the coded tiles) may be used in the claims and embodiments described to refer to out-of-band transmission, signaling, or storage in a manner in which out-of-band data is associated with the bitstream or coded units, respectively. The phrases "decoding along the bitstream" or "decoding along the coded units of the bitstream," or similar phrases, may refer to decoding referenced out-of-band data (which may be obtained from out-of-band transmission, signaling, or storage) associated with the bitstream or coded units, respectively.

[0082] SEI NAL units may contain one or more SEI messages that are not required for decoding the output image but can be useful for related processes such as image output timing, rendering, error detection, error hiding, and resource reservation.

[0083] A coded image is a coded representation of an image.

[0084] In HEVC, a coded image may be defined as a coded representation of an image, including all coding tree units of the image. In HEVC, an access unit (AU) may be defined as a set of NAL units that are related to each other according to a defined classification rule, are consecutive in the decoding order, and contain at most one image having any particular value of nuh_layer_id. In addition to containing VCL NAL units of coded images, an access unit may also contain non-VCL NAL units. The aforementioned defined classification rule may, for example, associate images with the same output time or picture output count value with the same access unit.

[0085] A bitstream may be defined as a sequence of bits in the form of a NAL unit stream or byte stream that forms a representation of the coded image and associated data that make up one or more coded video sequences. A second bitstream may follow a first bitstream within the same logical channel, such as within the same file or on the same connection of a communication protocol. The base stream may be defined (in the context of video coding) as a sequence of one or more bitstreams. The end of the first bitstream may be indicated by a specific NAL unit, sometimes called the end of bitstream (EOB) NAL unit, which is the last NAL unit of the bitstream. HEVC and its current draft extensions require that the EOB NAL unit have a nuh_layer_id equal to 0.

[0086] In H.264 / AVC, a coded video sequence is defined as a sequence of consecutive access units in the decoding order, from one IDR access unit (inclusive) to the next IDR access unit (exclusive) or the end of the bitstream, whichever comes first.

[0087] In HEVC, a coded video sequence (CVS) may be defined as a sequence of access units consisting of zero or more access units that are not IRAP access units with a NoRaslOutputFlag equal to 1, including, for example, an IRAP access unit with a NoRaslOutputFlag equal to 1 in the decoding order, and all subsequent access units that follow it, but do not include any subsequent access units that are IRAP access units with a NoRaslOutputFlag equal to 1. An IRAP access unit may be defined as an access unit whose base layer image is an IRAP image. The value of NoRaslOutputFlag is equal to 1 for each IDR image, each BLA image, and each IRAP image, where the IRAP image is the first image in that particular layer in the bitstream in the decoding order and the first IRAP image following a sequence-ending NAL unit with the same value of nuh_layer_id in the decoding order. Means may exist for providing the decoder with a HandleCraAsBlaFlag value from an external entity such as a player or receiver that can control the decoder. HandleCraAsBlaFlag may be set to 1 by a player, for example, by seeking to a new position in the bitstream or tuning into a broadcast to begin decoding, and then starting decoding from the CRA image. If HandleCraAsBlaFlag for a CRA image is equal to 1, the CRA image is processed and decoded as if it were a BLA image.

[0088] In HEVC, a coded video sequence may be specified (additionally or alternatively to the above specification) to terminate when a certain NAL unit, sometimes called an end of sequence (EOS) NAL unit, appears in the bitstream and has a nuh_layer_id equal to 0.

[0089] A group of pictures (GOP) and its characteristics may be defined as follows: A GOP can be decoded regardless of whether previous images have been decoded. An open GOP is a group of images such that, when decoding starts with the first intra-image of the open GOP, images preceding the first intra-image in the output order may not be correctly decoded. In other words, images in an open GOP may refer to images belonging to a previous GOP (in interpretation). The HEVC decoder can recognize the intra-image that starts an open GOP because a specific NAL unit type, the CRA NAL unit type, may be used for its coded slice. A closed GOP is a group of images such that, when decoding starts with the first intra-image of the closed GOP, all images can be correctly decoded. In other words, images in a closed GOP do not refer to images in a previous GOP. In H.264 / AVC and HEVC, a closed GOP may start with an IDR image. In HEVC, a closed GOP may also start with a BLA_W_RADL or BLA_N_LP image. Open GOP coding structures may be more efficient in terms of compression compared to closed GOP coding structures due to their greater flexibility in the selection of reference images.

[0090] A Decoded Picture Buffer (DPB) may be used in the encoder and / or decoder. There are two reasons for buffering the decoded image: for reference in interpretation and for sorting the decoded image in output order. Since H.264 / AVC and HEVC offer great flexibility in both marking the reference image and sorting the output, separate buffers for buffering the reference image and the output image can waste memory resources. Therefore, the DPB may include a unified decoded image buffering process for both the reference image and output sorting. The decoded image may be removed from the DPB when it is no longer used as a reference and is no longer required for output.

[0091] In many coding modes of H.264 / AVC and HEVC, the reference images for interpretation are indicated using an index to a list of reference images. The index may be coded using variable-length coding, which usually results in smaller indices having shorter values ​​for the corresponding syntactic elements. In H.264 / AVC and HEVC, two list of reference images (reference image list 0 and reference image list 1) are generated for each bipredicted (B) slice, and one list of reference images (reference image list 0) is formed for each intercoded (P) slice.

[0092] Many coding standards, including H.264 / AVC and HEVC, may have a decoding process for deriving a reference image index into a reference image list, which may be used to indicate which of several reference images is used for interpretation of a particular block. The reference image index may be coded into the bitstream by an encoder in some intercoding mode, or it may be derived (by an encoder and decoder) using adjacent blocks in some other intercoding mode, for example.

[0093] The motion parameter types or motion information may include, but are not limited to, one or more of the following types: - Indication of the type of prediction (e.g., intra prediction, single prediction, dual prediction) and / or the number of reference images, - Instructions for prediction directions, such as interpretation (also known as time prediction), interlayer prediction, interview prediction, view synthesis prediction (VSP), and intercomponent prediction (this may be indicated per reference image and / or per prediction type, and in some embodiments, interview prediction and view synthesis prediction may be considered together as one prediction direction), and / or - Indication of the type of reference image, such as short-term reference images and / or long-term reference images and / or interlayer reference images (this may, for example, be indicated for each reference image). - A reference index to the reference image list and / or any other identifier for the reference image (this may, for example, be shown for each reference image, its type may depend on the prediction direction and / or the type of reference image, and may be accompanied by other relevant information such as the reference image list to which the reference index applies), - Horizontal motion vector components (which may be shown, for example, for each prediction block, or for each reference index, or similarly), - Vertical motion vector component (this may be shown, for example, for each prediction block, or for each reference index, or similarly), - One or more parameters, including motion parameters, or associated motion parameters, which may be used to scale the horizontal motion vector components and / or vertical motion vector components in one or more motion vector prediction processes, such as the difference in image order counts between an image and its reference image and / or relative camera separation (the aforementioned one or more parameters may be shown, for example, for each reference image or each reference index, or similarly), - The coordinates of the block to which the motion parameters and / or motion information are applied, for example, the coordinates of the top-left sample of the block in luma samples. - The range of the block to which motion parameters and / or motion information apply (e.g., width and height).

[0094] Compared to previous video coding standards, the versatile video codec (H.266 / VVC) introduces several new coding tools, such as: Intra prediction - 67 Intra Mode with Wide-Angle Mode Extension - Block size and mode-dependent 4-tap interpolation filter - Position-dependent intra-prediction combination (PDPC) - Intercomponent Linear Model Intraprediction (CCLM) - Multiple reference line intra prediction - Intra sub-partitions - Weighted intra prediction with matrix multiplication Inter-image prediction - Copying block movement using spatial, temporal, historical, and pairwise average merge candidates - Affine motion interface prediction - Subblock-based time-motion vector prediction - Adaptive motion vector resolution - 8x8 block-based motion compression for time-motion prediction - High-precision (1 / 16 pixel) motion vector storage and motion compensation with an 8-tap interpolation filter for the luma component and a 4-tap interpolation filter for the chroma component. - Triangulation - Prediction of combined intra and inter connections - Merge with MVD (MMVD) - Symmetric MVD coding - Bidirectional optical flow - Improvement of the decoder's motion vector - CU-level weighted biprediction • Coding of transformations, quantization, and coefficients - Multiple primary transformation selections using DCT2, DST7, and DCT8 - Secondary conversion of the low-frequency zone - Subblock transformation of interpreted residuals - Dependency quantization with maximum QP increased from 51 to 63 - Transformation coefficient coding with code data hiding - Conversion skip residual coding Entropy coding - Arithmetic coding engine with adaptive double windows probability update • In-loop filter - Inloop Reshape - Deblocking filters that use stronger, longer filters - Sample-adaptive offset - Adaptive loop filter • Screen content coding - Current image reference with reference area limitations 360-degree video coding - Horizontal wrap-around motion compensation • High-level syntax and parallel processing - Reference image management with direct reference image list signal transmission - Tile group containing rectangular tile groups

[0095] In VVC, partitioning is performed similarly to HEVC, i.e., each image is partitioned into coding tree units (CTUs). Images may also be partitioned into slices, tiles, bricks, and sub-images. CTUs may be partitioned into smaller CUs using a quadtree structure. Each CU may be partitioned using a quadtree, as well as nested multi-type trees including 3-partitions and 2-partitions. However, there are specific rules for inferring partitions at image boundaries, and in nested multi-type partitions, redundant partition patterns are not permitted.

[0096] In the new coding tool described above, the Intercomponent Linear Model (CCLM) prediction mode is used in VVC to reduce redundancy between components. There, the chroma sample is predicted based on the reconstructed chroma sample of the same CU by using a linear model such as the following: Nod C (i,j)=α·rec L '(i,j)+β (Equation 1a)

[0097] Here, pred C (i,j) represents the predicted chroma sample within the CU, and rec L '(i,j) represents a downsampled and reconstructed luma sample of the same CU.

[0098] Alternatively, the following equation may be used for CCLM.

number

[0099] The CCLM parameters (α and β) are derived using up to four adjacent chroma samples and their corresponding downsampled chroma samples. Assuming the dimensions of the current chroma block are W×H, W' and H' are set as follows: - When LM mode is applied, W'=W and H'=H. - When LM-A mode is applied, W'=W+H. - When LM-L mode is applied, H' = H + W.

[0100] In this specification, LM-A mode refers to the linear model - above, and only the upper template (i.e., sample values ​​from adjacent positions above the CU) is used to calculate the linear model coefficients. To obtain more samples, the upper template is extended to (W+H). Next, LM-L mode refers to the linear model - left, and only the left template (i.e., sample values ​​from adjacent positions to the left of the CU) is used to calculate the linear model coefficients. To obtain more samples, the left template is extended to (H+W). For non-square blocks, the upper template is extended to W+W and the left template is extended to H+H.

[0101] The adjacent positions above are denoted as S[0,-1]...S[W'-1,-1], and the adjacent positions to the left are denoted as S[-1,0]...S[-1,H'-1]. Next, four samples are selected as follows: - If LM mode is applied and both adjacent samples above and to the left are available, then S[W' / 4,-1], S[3*W' / 4,-1], S[-1,H' / 4], S[-1,3*H' / 4]. - If LM-A mode is applied, or if only the above adjacent samples are available, then S[W' / 8,-1], S[3*W' / 8,-1], S[5*W' / 8,-1], S[7*W' / 8,-1]. -LM-L mode is applied, or if only the left adjacent sample is available, S[-1,H' / 8], S[-1,3*H' / 8], S[-1,5*H' / 8], S[-1,7*H' / 8].

[0102] Four adjacent chroma samples at the selected location are downsampled and compared four times to find two smaller values, x0A and x1A, and two larger values, x0B and x1B. The corresponding chroma sample values ​​are denoted as y0A, y1A, y0B, and y1B. Then xA, xB, yA, and yB are derived as follows: X a =(x 0 A +x 1 A +1)>>1, X b =(x 0 B +x 1 B +1)>>1, Y a =( y 0 A +y 1 A +1)>>1, Y b =( y 0 B +y 1 B +1)>>1 (equation 2)

[0103] Finally, the linear model parameters α and β are obtained according to the following equations.

number

[0104] Figure 5 shows examples of the sample locations in the left and top samples involved in the CCLM mode, as well as the sample location in the current block.

[0105] The division operation to calculate the parameter α is performed using a lookup table. To reduce the memory required to store the table, the diff value (the difference between the maximum and minimum values) and the parameter α are represented in exponential notation. For example, diff is approximated using a 4-bit valid part and exponent. As a result, the 1 / diff table is reduced to 16 elements for 16 values ​​in the valid part, as follows: DivTable[]={0,7,6,5,5,4,4,3,3,2,2,1,1,1,1,0} (Equation 5)

[0106] This not only reduces the complexity of the calculations but also has the advantage of reducing the amount of memory required to store the necessary tables.

[0107] To match the chroma sample positions in a 4:2:0 video sequence, two downsampling filters are applied to the chroma samples to achieve a 2:1 downsampling ratio in both the horizontal and vertical directions. The selection of the downsampling filters is specified by the SPS level flag. The two downsampling filters are as follows, corresponding to "Type 0" and "Type 2" content, respectively.

[0108]

number

number

[0109] Note that when the upper reference line is at the CTU boundary, only one luma line (a common line buffer in intra prediction) is used to create a downsampled luma sample.

[0110] This parameter calculation is performed not only as part of the encoder's lookup operation, but also as part of the decoding process. As a result, no syntax is used to communicate the values ​​of α and β to the decoder.

[0111] For chroma intra-mode coding, a total of eight intra-modes are permitted for chroma intra-mode coding. These modes include five conventional intra-modes and three inter-component linear model modes (CCLM, LM_A, and LM_L). The signal transfer and derivation processes for chroma modes are shown in Table 1. Chroma mode coding directly depends on the intra-prediction mode of the corresponding luma block. In I-slice, separate block partitioning structures for luma and chroma components are enabled, so one chroma block may correspond to multiple luma blocks. Therefore, for chroma DM modes, the intra-prediction mode of the corresponding luma block covering the center position of the current chroma block is directly inherited.

[0112] [Table 1]

[0113] As shown in Table 2, a single binarization table is used regardless of the value of sps_cclm_enabled_flag. [Table 2]

[0114] Table 2 shows that the first binary number indicates either standard mode (0) or LM mode (1). If it is LM mode, the next binary number indicates whether it is LM_CHROMA (0). If it is not LM_CHROMA, the next binary number indicates whether it is LM_L (0) or LM_A (1). In this case, when sps_cclm_enabled_flag is 0, the first binary number in the corresponding intra_chroma_pred_mode binarization table may be discarded before entropy coding. Or, in other words, the first binary number is assumed to be 0 and therefore not coded. This single binarization table is used for both cases where sps_cclm_enabled_flag is equal to 0 and 1. The first two binary numbers in Table 2 are context-coded using their own context model, and the remaining binary numbers are bypass-coded.

[0115] In addition, to reduce luma-chroma latency in dual trees, if a 64x64 luma coding tree node is not partitioned (and intra sub-partitions (ISPs) are not used for the 64x64 CU) or is partitioned using QT, then chroma CUs in 32x32 / 32x16 chroma coding tree nodes are permitted to use CCLM in the following manner: - If a 32x32 chroma node is not partitioned, or is partitioned by QT partitioning, all chroma CUs within the 32x32 node can use CCLM. - If a 32x32 chroma node is divided using horizontal BT and the 32x16 child nodes are not divided or use vertical BT division, then all chroma CUs within the 32x16 chroma node can use CCLM.

[0116] In all other luma and chroma coding partitioning conditions, CCLM is not permitted for chroma CU.

[0117] Multi-model LM (MMLM) The CCLM included in VVC is extended by adding three multi-model LM (MMLM) modes. In each MMLM mode, reconstructed adjacent samples are classified into two classes using a threshold where Luma is the average of reconstructed adjacent samples. A linear model for each class is derived using the Least-Mean-Square (LMS) method. In the case of CCLM mode, the LMS method is also used to derive the linear model. Figures 6a and 6b show the two Luma-Chroma intermodels obtained for 17 Luma (Y) thresholds in the sample and spatial domains, respectively. Each Luma-Chroma intermodel has its own linear model parameters α and β. As shown in Figure 6b, each Luma-Chroma intermodel corresponds to the spatial segmentation of the content (i.e., to different objects or textures in the scene).

[0118] Convolutional cross-component model (CCCM) An improved version of intercomponent prediction, known as CCCM, uses a 2D filter kernel to derive a luma-chroma intercomponent model. Using a reconstructed set of input data and chroma samples, the filter coefficients are derived on the decoder side. A reference sample area (consisting of reconstructed luma and chroma samples) at the same location is defined with respect to both luma and chroma, as shown in Figure 7, where the commonly used 4:2:0 chroma downsampling is applied for the derivation of the filter coefficients. The reference sample area for a particular block can be, for example, the six lines above and to the left, as shown in Figure 7, but any number of reference lines (which can be realized by both the encoder and decoder) can be used. Generally, the reference samples can include any chroma and luma samples reconstructed by both the encoder and decoder. After the reference samples are determined, the filter coefficients can be derived using various types of linear regression tools, such as standard least squares estimation, orthogonal matching track, optimized orthogonal matching track, ridge regression, or minimum absolute contraction and selection operators.

[0119] The dimensions of the filter kernel can be, for example, 1x3 (1D vertical), 3x1 (1D horizontal), 3x3, 7x7, or any other dimensions, and can be shaped as a cross or a rhombus (as shown in Figure 8) or any specific shape (by selecting only a subset of all possible kernel positions). When referring to samples within the filter kernel, the letters N, E, S, W, and C are used to represent North (top), East (right), South (bottom), West (left), and Center, respectively, as shown in Figure 8.

[0120] The entire method for reconstructing chroma samples using the convolution between the filter kernel acquired on the decoder side and the input data set is referred to herein as the Interconvolutional Component Model (CCCM). The CCCM can be implemented by applying the following steps. 1) Define reference areas at the same location on the luma and chroma components. 2) Downsample the luma sample to match the chroma grid (optional). 3) Scan the luma and chroma samples of the reference area and collect available statistics (such as autocorrelation matrices and cross-correlation vectors) based on the filter shape. 4) Solve the filter coefficients by minimizing the squared error (or any other metric) based on available statistics (such as the autocorrelation matrix and cross-correlation vector). 5) The predicted chroma block is computed by convolving the downsampled chroma samples with the filter kernel.

[0121] The chroma samples (which may be downsampled in some cases) are defined as a 2D array Y(x,y) indexed using the horizontal x-coordinate and vertical y-coordinate. The chroma samples at the same positions are also defined as a 2D array C(x,y), and the filter kernel (i.e., coefficients) is defined as a 3x3 array F(i,j). At the sample level, the convolution between Y and F is defined as follows:

[0122]

number

number

number

number

[0123] Multiple reference line (MRL) intra-prediction Multiple Reference Lines (MRL) intra-prediction uses more reference lines for intra-prediction. Figure 9 shows an example with four reference lines, where samples from segments A and F are embedded with the nearest samples from segments B and E, respectively, rather than being taken from reconstructed adjacent samples. HEVC intra-image prediction uses the nearest reference line (i.e., reference line 0). MRL uses two additional lines (reference line 1 and reference line 3).

[0124] The index of the selected reference line (mrl_idx) is transmitted and used to generate the intra-predictor. If the reference line idx is greater than 0, only the MPM index is transmitted, including only the additional reference line modes in the MPM list and excluding the remaining modes. If the reference line index is transmitted before the intra-prediction mode and a non-zero reference line index is transmitted, the planar mode is excluded from the intra-prediction mode.

[0125] In the first line of a block within a CTU, MRL is disabled to prevent the use of extended reference samples outside the current CTU line. Additionally, PDPC is disabled if additional lines are used. In MRL mode, the derivation of DC values ​​for non-zero reference line indices in DC intra-prediction mode is aligned with the derivation for reference line index 0. MRL requires the storage of three adjacent Luma reference lines along with the CTU to generate predictions. The CCLM tool also requires three adjacent Luma reference lines for its downsampling filter. The MRL definition for using the same three lines is aligned with CCLM to reduce the decoder's storage requirements.

[0126] Intranet Subdivision (ISP) Intra subdivision (ISP) divides a luma intra predictive block into two or four subdivisions, vertically or horizontally, depending on the block size. For example, the minimum block size for an ISP is 4x8 (or 8x4). If the block size is larger than 4x8 (or 8x4), the corresponding block is divided into four subdivisions. Note that ISP blocks of Mx128 (M≦64) and 128xN (N≦64) can cause potential problems with 64x64 VDPUs. For example, an Mx128 CU in the case of a single tree contains an Mx128 luma TB and two corresponding M / 2x64 chroma TBs. If the CU uses an ISP, the luma TB is divided into four Mx32 TBs (only horizontal division is possible), each of which is smaller than a 64x64 block. However, in the current ISP design, chroma blocks are not divided. Therefore, both chroma components have a size larger than a 32x32 block. Similarly, a similar situation can be created using ISP with a 128xN CU. Thus, these two cases are problematic for a 64x64 decoder pipeline. For this reason, the size of the CU in which ISP can be used is limited to the maximum of 64x64. All subdivisions satisfy the condition that they contain at least 16 samples.

[0127] Matrix-weighted intra-prediction (MIP) Matrix-weighted intra-prediction (MIP) is a new intra-prediction technique added to VVC. To predict samples of a rectangular block of width W and height H, Matrix-weighted intra-prediction (MIP) takes as input a reconstructed adjacent boundary sample of one line H on the left side of the block and a reconstructed adjacent boundary sample of one line W above the block. If reconstructed samples are not available, reconstructed samples are generated in the same way as in conventional intra-prediction. The generation of the prediction signal is based on three steps: averaging, matrix-vector multiplication, and linear interpolation, as shown in Figure 10.

[0128] Decoder-side intra-mode derivation (DIMD) When DIMD is applied, two intra-modes are derived from the reconstructed neighboring samples, and these two predictors are combined with the plane mode predictors, which have weights derived from the gradient, as described in JVET-O0449. The division operation in weight derivation is performed using the same lookup table (LUT)-based integerization scheme used by CCLM. For example, the division operation in orientation calculation. Orient=G y / G x This is calculated using the following LUT-based method. x = Floor(Log2(Gx)) normDiff=((Gx<<4)>>x)&15 x + = (3 + (normDiff != 0) ? 1 : 0) Orient=(Gy*(DivSigTable[normDiff]|8)+(1<<(x-1)))>>x Here, DivSigTable

[16] ={0,7,6,5,5,4,4,3,3,2,2,1,1,1,1,0}

[0129] The derived intra-mode is included in the primary list of most probable intra-modes (MPMs), and therefore the DIMD process runs before the MPM list is constructed. The derived primary intra-mode of a DIMD block is stored with the block and used to construct the MPM list for adjacent blocks.

[0130] Fusion for template-based intra-mode derivation (TIMD) For each intra-prediction mode in MPM, the SATD between the template's predicted sample and the reconstructed sample is calculated. The first two intra-prediction modes with the minimum SATD are selected as TIMD modes. These two TIMD modes are fused with weights after applying the PDPC process, and such weighted intra-predictions are used for coding the current CU. Position-dependent intra-prediction combinations (PDPC) are included in the derivation of TIMD modes.

[0131] The costs of the two selected modes are compared to a threshold, and in the test, a cost coefficient of 2 is applied as follows: costMode2 < 2 * costMode1.

[0132] If this condition is true, fusion is applied; otherwise, only mode 1 is used.

[0133] The mode weights are calculated from the mode's SATD cost as follows: weight1=costMode2 / (costMode1+costMode2) weight2 = 1 - weight1

[0134] Division operations are performed using the same lookup table (LUT)-based integerization scheme used by CCLM.

[0135] Low-frequency non-separated conversion (LFNST) In VVC, as shown in Figure 11, LFNST is applied between the forward linear transformation and quantization (on the encoder side) and between the inverse quantization and inverse linear transformation (on the decoder side). LFNST applies either a 4x4 inseparable transformation or an 8x8 inseparable transformation depending on the block size. For example, a 4x4 LFNST is applied to smaller blocks (i.e., min(width, height) < 8), and an 8x8 LFNST is applied to larger blocks (i.e., min(width, height) > 4).

[0136] The application of the non-separated transform used in LFNST is explained as follows, using the input as an example: To apply LFNST to a 4x4 input block X

number

number

number

[0137] Inseparable transformations are,

number

number

number

[0138] Shrink-and-Unseparate Transformation LFNST (Low Frequency Non-Separable Transform) applies non-separable transform based on the direct matrix multiplication method so that it can be implemented in one pass instead of multiple iterations. However, in order to minimize the computational complexity and the memory space for storing the transform coefficients, it is necessary to reduce the dimension of the non-separable transform matrix. Therefore, the reduced non-separable transform (or RST: reduced non-separable transform) method is used in LFNST. The main idea of the reduced non-separable transform is to map an N-dimensional vector (where N is generally equal to 64 in the case of an 8×8 NSST) to an R-dimensional vector in another space, and N / R (R < N) is the reduction coefficient. Therefore, instead of an N×N matrix, the RST matrix becomes an R×N matrix as follows.

Number

[0139] The inverse transform matrix of RT is the transpose of its forward transform. In the case of 8x8 LFNST, a reduction factor of 4 is applied, reducing the direct matrix size of 64x64, which is the size of the conventional 8x8 non-separable transform matrix, to a direct matrix of 16x48. Thus, a 48x16 inverse RST matrix is ​​used on the decoder side to generate the core (linear) transform coefficients in the upper left region of the 8x8. When a 16x48 matrix is ​​applied instead of a 16x64 matrix with the same transform set configuration, each of those matrices receives 48 input data from three 4x4 blocks within the upper left 8x8 block, excluding the lower right 4x4 block. Thanks to the reduced dimension, the memory usage for storing all LFNST matrices is reduced from 10KB to 8KB, with a reasonable performance degradation. To reduce complexity, LFNST is restricted to being applicable only if all coefficients except the first coefficient subgroup are not significant. Thus, when LFNST is applied, all linear-only transform coefficients must be zero. This allows for adjustment of LFNST index signal transmission at the last significant position, thus avoiding the extra coefficient scans required in current LFNST designs to check significant coefficients only at specific positions. The worst-case handling of LFNST (in terms of pixel-by-pixel multiplication) restricts the non-separable transforms of 4x4 and 8x8 blocks to 8x16 and 8x48 transforms, respectively. In such cases, the last significant scan position must be less than 8 when LFNST is applied, and less than 16 for the other sizes. For blocks with 4xN and Nx4 shapes where N>8, the proposed restriction means that LFNST is applied only once, and only to the top-left 4x4 region. In such cases, the number of operations required for the linear transform is reduced, as all linear-only coefficients are zero when LFNST is applied. From the encoder's perspective, coefficient quantization is significantly simplified when the LFNST transform is tested. Rate-distortion-optimized quantization needs to be performed on at most the first 16 coefficients (in scan order), and the remaining coefficients are forced to zero.

[0140] LFNST conversion selection There are a total of four transformation sets, and two inseparable transformation matrices (kernels) are used in LFNST for each transformation set. The mapping from intra-prediction modes to transformation sets is predefined, as shown in the table below. If one of the three CCLM modes (INTRA_LT_CCLM, INTRA_T_CCLM, or INTRA_L_CCLM) is used for the current block (81 <= predModeIntra <= 83), transformation set 0 is selected for the current chroma block. For each transformation set, the selected inseparable quadratic transformation candidate is further specified by an LFNST index that is explicitly signaled. This index is signaled in the bitstream once per intra-CU after the transformation coefficients.

[0141] [Table 3]

[0142] LFNST index signaling and interaction with other tools Because LFNST is restricted to being applicable only when all coefficients except the first coefficient subgroup are unsignificant, LFNST index coding depends on the position of the last significant coefficient. In addition, LFNST indices are context-coded, but not dependent on the intra-prediction mode, and only the first binary digit is context-coded. Furthermore, LFNST applies to intra-CU in both intra-slice and inter-slice, as well as to both luma and chroma. When dual-tree is enabled, LFNST indices for luma and chroma are signaled separately. In the case of inter-slice (with dual-tree disabled), a single LFNST index is signaled and used for both luma and chroma.

[0143] Given the existing maximum conversion size limit (64x64), and considering that larger CUs exceeding 64x64 are implicitly partitioned (TU tiling), LFNST index lookup could quadruple the data buffering for a certain number of decoding pipeline stages. Therefore, the maximum size allowed for LFNST is limited to 64x64. Note that LFNST is enabled only in conjunction with DCT2. LFNST index signaling is performed before MTS index signaling.

[0144] The use of scaling matrices for perceptual quantization is not permitted because it is not clear that a scaling matrix specified for a linear matrix can be useful for LFNST coefficients. Therefore, the use of scaling matrices for LFNST coefficients is not permitted. In single-tree partition mode, chroma LFNST is not applicable.

[0145] Improved Multiple Transform Selection (MTS) for intracoding In current VVC designs, the MTS utilizes only the DST7 and DCT8 translation kernels used for intracoding and intercoding.

[0146] Additional linear transformations are employed, including DCT5, DST4, DST1, and the identity transform (IDT). Additionally, MTS sets are created based on the TU size and intra-mode information. Sixteen different TU sizes are considered, and for each TU size, five different classes are considered based on the intra-mode information. For each class, one, four, or six different transformation pairs are considered. The number of intra-MTS candidates is adaptively selected (from one, four, and six MTS candidates) based on the sum of the absolute values ​​of the transformation coefficients. This sum is compared to two fixed thresholds, as follows, to determine the total number of acceptable MTS candidates. One candidate: Total <= th0 4 candidates: th0 < total <= th1 6 candidates: Total > th1

[0147] A total of 80 different classes are considered, but note that some of these different classes often share the exact same set of transformations. Therefore, the resulting LUT will have 58 (less than 80) unique entries.

[0148] For angular modes, the bond symmetry across TU shape and intra-prediction is considered. Therefore, mode i (i>34) with TU shape A×B is mapped to the same class as mode j=(68-i) with TU shape B×A. However, for each transformation pair, the order of the horizontal and vertical transformation kernels is swapped. For example, a 16×4 block with mode 18 (horizontal prediction) and a 4×16 block with mode 50 (vertical prediction) are mapped to the same class, but the vertical and horizontal transformation kernels are swapped. For wide-angle modes, the nearest conventional angular mode is used to determine the transformation set. For example, mode 2 is used for all modes from -2 to -14. Similarly, mode 66 is used for modes 67 to 80.

[0149] Optimization of Intermultiplexing Selection (MTS) For the MTS of the intercoding CU, four candidates are used per CU: {(DST7,DST7),(DST7,DCT8),(DCT8,DST7),(DCT8,DCT8)}. For sequences with higher resolution (width > 1080), the maximum CU size for interMTS use is set to 32 (i.e., interMTS is used for CUs with width <= 32 and height <= 32), and for the remaining sequences (smaller resolutions), the maximum CU size is set to 16. For 4pt, 8pt, and 16pt conversions, the current AMT conversion cores, namely DST-7 and DCT-8, are replaced with separable KLTs, as proposed in JVET-J0021.

[0150] Intra template matching Intra-template matching prediction (Intra-TMP) is a special intra-prediction mode that copies the best prediction block with an L-shaped template matching the current template from the reconstructed portion of the current frame. For a predefined search range, the encoder searches the reconstructed portion of the current frame for the template most similar to the current template and uses the corresponding block as the prediction block. The encoder then signals the use of this mode, and the decoder performs the same prediction operation.

[0151] A prediction signal is generated by matching the neighborhood representing the L-shaped causal relationship of the current block with another block within a predefined search area shown in Figure 12, which is composed of the following components. - R1: Current CTU - R2: CTU in the upper left - R3: Upper CTU - R4: Left CTU

[0152] The sum of absolute differences (SAD) is used as the cost function.

[0153] Within each region, the decoder searches for the template with the smallest SAD for the current template and uses the corresponding block as the prediction block.

[0154] To ensure a fixed number of SAD comparisons per pixel, the dimensions of all regions (SearchRange_w, SearchRange_h) are set proportionally to the block dimensions (BlkW, BlkH). That is, - SearchRange_w=a*BlkW - SearchRange_h=a*BlkH Here, "a" is a constant that controls the trade-off between gain and complexity. In practice, "a" is equal to 5.

[0155] The intra-template matching tool is enabled for CUs with a width and height of 64 or less. This maximum CU size for intra-template matching is configurable.

[0156] If DIMD is not used for the current CU, the intra-template matching prediction mode is signaled at the CU level via a dedicated flag.

[0157] Intrablock copying using template matching (IBC) In IBC, template matching is used in both IBC merge mode and IBC AMVP mode.

[0158] The IBC-TM merge list is compared to the merge list used by the standard IBC merge mode and modified so that candidates are selected according to a decimation method that uses the distance moved between candidates, similar to the standard TM merge mode. The realization of movement of trailing zeros is replaced with movement vectors to the left (-W,0), up (0,-H), and top-left (-W,-H), where W is the width of the current CU and H is the height.

[0159] In IBC-TM merge mode, selected candidates are refined using a template matching method before the RDO or decoding process. IBC-TM merge mode competes with standard IBC merge mode, and the TM merge flag is signaled.

[0160] In IBC-TM AMVP mode, up to three candidates are selected from the IBC-TM merge list. Each of these three selected candidates is improved using a template matching method and sorted according to the resulting template matching cost. Then, only the first two candidates are considered in the motion estimation process, as usual.

[0161] As shown in Figure 13a, since the motion vectors of IBC are constrained to be (i) integers and (ii) within the reference region, template matching improvements in both IBC-TM merge mode and AMVP mode are quite straightforward. Therefore, in IBC-TM merge mode, all improvements are performed with integer precision, and in IBC-TM AMVP mode, improvements are performed with either integer or 4-pixel precision depending on the AMVR value. Such improvements access only samples without interpolation. In both cases, the improved motion vectors and templates used in each improvement step must adhere to the reference region constraints.

[0162] IBC reference area The IBC reference area is extended to the two CTU rows shown above. Figure 13b shows the reference area for coding CTU(m,n). In detail, when CTU(m,n) is coded, the reference area contains CTUs with indices (m-2,n-2)...(W,n-2),(0,n-1)...(W,n-1),(0,n)...(m,n), where W indicates the maximum horizontal index in the current tile, slice, or image. If the CTU size is 256, the reference area is limited to the single CTU row shown above. This setting ensures that the IBC does not require additional memory on the current ETM platform when the CTU size is 128 or 256. The sample-by-sample block vector search (or local search) range is limited to [-(C<<1),C>>2] horizontally and [-C,C>>2] vertically to adapt to the expansion of the reference area, where C indicates the CTU size.

[0163] Reconstruction-Reordered IBC (RR-IBC) The IBC coding block allows for Re-ordered IBC (RR-IBC) mode. When RR-IBC is applied, the samples within the reconstructed block are reversed according to the type of reversal of the current block. On the encoder side, the original block is reversed before motion search and residual calculation, while the predicted block is derived without reversal. On the decoder side, the reconstructed block is reversed again to restore the original block.

[0164] RR-IBC coding blocks support two inversion methods: horizontal inversion and vertical inversion. For IBC AMVP coding blocks, a syntax flag is first propagated to indicate whether the reconstruction is inverted, and if so, another flag specifying the type of inversion is further propagated. For IBC merges, the inversion type is inherited from adjacent blocks without syntax propagation. Considering horizontal or vertical symmetry, the current block and referenced blocks are typically aligned horizontally or vertically. Therefore, when horizontal inversion is applied, the vertical component of the BV is not propagated and is assumed to be equal to 0. Similarly, when vertical inversion is applied, the horizontal component of the BV is not propagated and is assumed to be equal to 0.

[0165] To better utilize symmetry and improve block vector candidates, a BV adjustment technique that recognizes inversion is applied. For example, as shown in Figure 14, (xnbr, ynbr) and (xcur, ycur) represent the coordinates of the center samples of the adjacent block and the current block, respectively, and BVnbr and BVcur represent the BVs of the adjacent block and the current block, respectively. The horizontal component of BVcur is calculated by adding a motion shift to the horizontal component of BVnbr (shown as BVnbrh) if the adjacent block is coded by horizontal inversion, instead of directly inheriting the BV from the adjacent block, i.e., BVcurh = 2(xnbr - xcur) + BVnbrh. Similarly, the vertical component of BVcur is calculated by adding a motion shift to the vertical component of BVnbr (shown as BVnbrv) if the adjacent block is coded by vertical inversion, i.e., BVcurv = 2(ynbr - ycur) + BVnbrv.

[0166] IBC merge mode with block vector differences (IBC-MBVD: IBC merge mode with block vector differences) As extensions to the standard MMVD mode, affine MMVD and GPM-MMVD were adopted for ECMs. Extending the MMVD mode to IBC merge mode is a natural progression.

[0167] In IBC-MBVD, the distance set is {1 pixel, 2 pixels, 4 pixels, 8 pixels, 12 pixels, 16 pixels, 24 pixels, 32 pixels, 40 pixels, 48 ​​pixels, 56 pixels, 64 pixels, 72 pixels, 80 pixels, 88 pixels, 96 pixels, 104 pixels, 112 pixels, 120 pixels, 128 pixels}, and the BVD direction consists of two horizontal and two vertical directions.

[0168] The base candidate is selected from the first five candidates in the reordered IBC merge list. Additionally, all possible MBVD improvement positions (20x4) for each base candidate are reordered based on the SAD cost between the template (one row above and one column to the left of the current block) and its reference for each improvement position. Finally, the top eight improvement positions with the lowest template SAD cost are retained as positions available for MBVD index coding. The MBVD index is binarized by a Rice Code containing parameters equal to 1.

[0169] IBC-MBVD coding blocks do not inherit the inversion type from neighboring RR-IBC coding blocks.

[0170] The aforementioned intercomponent intra-prediction tools, such as CCLM and CCCM, use samples from adjacent reconstructed samples to calculate the parameters of the prediction model. This assumes that the adjacent reconstructed samples have a good correlation with the samples in the prediction block. However, if the adjacent samples do not correlate with the samples in the current block, or if the correlation is very small, the calculated model may not be able to efficiently predict the samples in the block.

[0171] Here, an improved method is introduced to enhance the efficiency of the intercomponent prediction tool.

[0172] One embodiment of the method is shown in Figure 15, which includes receiving an image block unit of a frame (1500), wherein the image block unit includes a sample in a color channel, the color channel includes at least one chrominance channel and one luminance channel; reconstructing the sample in the aforementioned luminance channel of the image block unit (1502); determining a reference area for predicting a target sample in at least one color channel of the image block unit (1504), wherein the aforementioned reference area includes one or more reference samples in the current block or in a block at the same location, in the current color channel / frame or in a reference color channel / frame, encoded using an intrablock copy (IBC) method; determining a block vector that indicates the spatial distance from the area of ​​the target sample to the reference area (1506); and predicting the aforementioned target sample in at least one color channel of the image block unit using an inter-component prediction model based on the reference sample in the aforementioned reference area indicated by the aforementioned block vector (1508).

[0173] Therefore, in this method, when coding a block or area in intercomponent prediction mode, if a block or area at the same location in the reference channel is coded using IBC mode, the corresponding block vector in IBC mode is used to indicate the reference region for intercomponent prediction. Then, samples of the reference region in the current channel and the reference channel may be used to compute the prediction model parameters.

[0174] Figure 16 illustrates the basic principle of using block vectors in Intra-Block Copy (IBC) mode for the current and reference blocks. In the IBC method, block prediction is achieved by copying a sample reconstructed from a different region of the image containing the content that best matches the current block. This can be done by various means, such as lookup on the encoder side and signaling the displacement vector or block vector or the index of the best block vector from a list of block vector candidates for matching blocks in the bitstream, with or without explicit signaling of vector differences; or on the decoder side, using a method based on template matching without signaling the block vector; or a combination of both.

[0175] In the intra-block copy method, the prediction for the current block is obtained by copying a reconstructed sample from another block or area. The spatial distance from the current block to the reference block is indicated by the block vector (BV).

[0176] According to the embodiment, the inter-component prediction method is an inter-component linear model (CCLM) or a convolutional inter-component model (CCCM).

[0177] The following primarily describes various embodiments in the context of improving the performance of the convolutional component-inter-component model (CCCM). It should be understood that the CCCM method is given as an example in this invention, and the proposed method and embodiments may be used in any other method having a similar concept, such as local illumination compensation (LIC) and other intra-prediction or inter-prediction methods. It should be further noted that inter-channel prediction may be performed from one chroma channel to another (e.g., from Cb to Cr, or vice versa), or from a chroma channel to a luma channel (Cb / Cr to Y, or vice versa).

[0178] Figure 17a shows an example of using the BV of the IBC coding block located in the same position to indicate the reference area for parameter derivation of intercomponent predictions.

[0179] According to this embodiment, the parameter derivation of the intercomponent model is performed using a reference block in the current channel and a sample from a reference block in the reference channel. After the parameters for the intercomponent prediction have been derived (also called trained), a sample from a block at the same location in the reference channel is used to predict the current block using the derived model. Figure 17b shows an example illustrating this embodiment.

[0180] According to one embodiment, this method includes estimating a block vector from a block at the same location in a reference channel and scaling the block vector to match the sampling density of the current channel.

[0181] Therefore, the block vector estimated from the same position in the reference channel may be scaled to match the sampling density of the current channel. For example, in 4:2:0 or 4:2:2 YUV / YCbCr chroma formats, the BV estimated from the same position in the chroma channel may be scaled down to obtain a matching reference block in the current channel. Figure 17c shows examples of scaled BVs when the sampling formats of the current channel and the reference channel are different.

[0182] According to one embodiment, this method includes identifying the closest match of a reference area within the current channel using a search based on template matching within the current channel.

[0183] Therefore, for example, a template matching (TM)-based search, similar to a TM-based search in IBC, may be used within the current channel to identify the nearest match of a reference region or reference block within the current channel. Next, the BV of the TM-based search is used to identify the corresponding region within the reference channel. Some or all of the reference blocks within the current channel and the samples within the reference blocks within the reference channel are used to compute the parameters of the intercomponent prediction model.

[0184] According to one embodiment, the block vectors estimated from the reference channel are not at least partially aligned with the coordinates of blocks at the same location.

[0185] Therefore, the BV derived from the reference channel does not need to be precisely aligned with the coordinates of blocks at the same location. The BV may be derived from one or more blocks within a region at the same location within the reference channel. A region at the same location may be defined as the area containing the blocks at the same location, and an area extended in different directions from the blocks at the same location. The size of the extended area may be predefined or determined using various criteria such as block size, channel type, etc.

[0186] Figure 17d shows an example where a TM-based search is used within the current channel to identify the best-matching reference block within the current channel. As shown in Figure 17d, the BV of the TM-based search process is then used to identify the reference block within the reference channel.

[0187] According to one embodiment, this method calculates the inter-component model parameters and the calculated model, - One or more samples from the reference block in the current channel, - One or more samples from the reference block within the reference channel, - One or more samples from the block at the same location in the reference channel, - One or more samples from the vicinity of the reference block in the current channel, - One or more samples from the vicinity of the reference block in the reference channel, - One or more samples from the neighborhood of the current block in the current channel, - This includes applying it to predictions based on one or more samples from the neighborhood of a block at the same location within a reference channel.

[0188] According to one embodiment, this method calculates the inter-component model parameters and the calculated model, - The horizontal and / or vertical coordinates of the aforementioned sample, including applying to a prediction based on one or more of the foregoing sample directional gradient values.

[0189] Thus, with respect to the samples referred to in the previous embodiments, for example, those samples may be applied by considering their horizontal and / or vertical coordinates and / or directional gradients. The gradient values may be calculated in different directions. For example, a horizontal gradient may be calculated by subtracting a sample on the left from a sample on the right, or vice versa, and a vertical gradient may be calculated by subtracting a sample on the upper side from a sample on the lower side, or vice versa.

[0190] According to an embodiment, the method includes using, as an initial block vector of a current block, a block vector of a block at the same position estimated from a reference channel.

[0191] Herein, the BV of an IBC coding block at the same position from a reference channel may be used as the initial BV of the current block. Next, an improvement based on TM is performed on the initial BV in the current channel to find the best match within the current channel. When implementing the improvement process, a predefined search range may be defined.

[0192] According to an embodiment, the method includes generating a list of block vector candidates from blocks encoded using an in-block copy (IBC) method within a region at the same position within a reference channel and / or in the vicinity of the current block within the current channel.

[0193] Therefore, the list of BV candidates may be generated from IBC coding blocks in the same region within the reference channel and / or from IBC coding blocks in the vicinity of the current block within the current channel. The encoder may test all BV candidates in the list by generating different inter-component predictions, as described above, and the index of the best-performing candidate is signaled to the bitstream. Alternatively, the best-performing candidate may be determined by a TM-based process on both the encoder and decoder sides.

[0194] According to one embodiment, this method includes combining two or more predictions to obtain the final prediction for the current block.

[0195] Therefore, the final prediction for a block may be obtained by combining two or more different predictions. For example, two different BVs may be used to derive two different intercomponent models. Then, two predictions are generated using the models, and the final prediction is obtained by combining the predictions. The weights of the combination may be predefined and signaled to the bitstream, or they may be determined on the decoder side.

[0196] According to the embodiment, multiple predictions may be obtained with respect to a combination of predictions based on one or more of the following: - One or more inter-component predictions may be obtained by block vectors determined from IBC coding blocks in the same region within the reference channel. - One or more inter-component predictions may be obtained by block vectors determined from IBC coding blocks in the vicinity of the current block within the current channel. - One or more predictions obtained by copying a sample from a reference block in the current channel, using a block vector determined from an IBC coding block in the same region within the reference channel. - One or more predictions obtained by copying a sample from a reference block in the current channel, using a block vector determined from an IBC coding block in the vicinity of the current block in the current channel.

[0197] According to one embodiment, some or all of the samples from the reference channel may be resampled before using those samples in the parameter derivation process for intercomponent prediction.

[0198] According to one embodiment, some or all of the samples from the reference channel and / or the current channel may be filtered before being used in the parameter derivation process for inter-component prediction.

[0199] According to one embodiment, the type of intercomponent model is estimated from a reference block or reference area.

[0200] For example, if the BV used points to a reference block coded using the CCLM or CCCM method, the same type of inter-component prediction may be used for the current block.

[0201] According to one embodiment, if the reference channel of a block at the same location is coded by an IBC, the corresponding IBC BV of the current channel (scaled according to the relationship between channels) may be used to check the reference area in the current channel and determine whether there are one or more blocks coded by inter-component predictions. If positive, the inter-component prediction parameters of these blocks may be directly inherited by the current block using a location-based scan order or some other priority criterion. Alternatively, these parameters may be added to a list of candidate inter-component predictors for the current mode, from which the encoder can select an index and transmit the signal.

[0202] In some embodiments, one or more parameters inherited from the corresponding reference area / reference block, determined according to a previous embodiment, may be modified before being used in the current block. For example, the bias term of the predictive model may be recalculated based on the adjacent reference samples in the current block.

[0203] According to one embodiment, this method involves obtaining the final inter-component prediction parameters of the current block using a two-step derivation, the first step of which is performed using a reference block in a reference channel pointed to by a block vector, and the second step of which is performed in the neighborhood of the current block.

[0204] Therefore, the final inter-component prediction parameters for the current block can be obtained using a two-step derivation. The first step of derivation is performed using the reference block pointed to by the IBC BV in the reference channel, and the second step of derivation operates on the neighborhood of the current block. Figure 17e shows an example of the process according to an embodiment. The parameters derived in the first step are used to predict the neighborhood of the current block in the current channel, and then the parameters of the second step are derived to predict the difference between the neighborhood of the current block and its prediction. The final prediction is a combination of predictions obtained by applying the parameters of the first and second steps to the block at the same location in the reference channel.

[0205] According to the embodiment, the reference block and the current block can be divided into subblocks, each subblock having its own CCLM or CCCM model. Overlapping subblocks may be used to smooth the obtained predictions.

[0206] According to one embodiment, a CCLM or CCCM model derived for the template area of ​​the current block may be mixed with a model derived for a reference block (such as one indicated by BV). For example, the prediction may mix the outputs of the two models. The weights used in the mixing may vary as a function of position within the block, for example, giving more weight to the template-based model in the upper left region and less weight towards the lower right. The weights of the combination may be predefined and signaled to the bitstream, or determined on the decoder side.

[0207] According to one embodiment, one or more intra-prediction modes of the reference block pointed to by the BV may be used to predict the current block. Alternatively, one or more intra-prediction modes from the reference block pointed to by the BV may be combined with inter-component predictions obtained from samples of the reference block and / or inter-component predictions obtained from adjacent reference samples of the current block.

[0208] According to one embodiment, this method includes using the residuals of intra-block copy coding blocks located at the same position as segmentation for model derivation.

[0209] Therefore, residuals of IBC coding blocks in the same location can be used as segmentation to guide the derivation of a CCLM or CCCM model. For example, only samples corresponding to significant residual activity are used in the model derivation, and darker, more fragmented areas within the block indicate significant residuals, as shown in Figure 17f. In this context, significant can mean residuals that are greater than zero or greater than some estimated, fixed, predetermined, or signaled threshold. In another embodiment, the model derivation may give more weight to samples defined by areas of significant residual activity.

[0210] According to an embodiment, samples that do not belong to the area defined by the significant residual activity according to the previous embodiment can be predicted using samples within the current channel determined by block copy and (possibly scaled) BV, or vice versa.

[0211] According to an embodiment, the reference block and the block at the same position can be divided into a plurality of regions based on the connected component labeling of the residuals. For example, two blobs can be used to derive two models. The connected component labeling can be based on, for example, the magnitude of the residual, and the blobs are considered separated only if the corresponding residual values are significantly different (which may indicate two different objects).

[0212] An apparatus according to one aspect includes means for receiving an image block unit of a frame, the image block unit including samples in a color channel, the color channel including at least one chrominance channel and one luminance channel; means for reconstructing samples of the luminance channel of the image block unit; means for determining a reference area for predicting target samples of at least one color channel of the image block unit, the reference area including one or more of the reference samples within the current block or the block at the same position within the current color channel / frame or the reference color channel / frame encoded using an in-block copy (IBC) method; means for determining a block vector indicating the spatial distance from the area of the target sample to the reference area; and means for predicting the target samples of at least one color channel of the image block unit using an inter-component prediction model based on the reference samples within the reference area indicated by the block vector.

[0213] According to the embodiment, the inter-component prediction method is an inter-component linear model (CCLM) or a convolutional inter-component model (CCCM).

[0214] According to one embodiment, the apparatus includes means for deriving parameters of an intercomponent model using a reference block in the current channel and a sample from a reference block in a reference channel.

[0215] According to one embodiment, the apparatus includes means for estimating a block vector from a block at the same location in a reference channel, and means for scaling the block vector to match the sampling density of the current channel.

[0216] According to one embodiment, the block vectors estimated from the reference channel are not at least partially aligned with the coordinates of blocks at the same location.

[0217] According to one embodiment, the device includes means for identifying the nearest match of a reference area in the current channel using a search based on template matching within the current channel.

[0218] According to the embodiment, the apparatus calculates the inter-component model parameters and the calculated model, - One or more samples from the reference block in the current channel, - One or more samples from the reference block within the reference channel, - One or more samples from the block at the same location in the reference channel, - One or more samples from the vicinity of the reference block in the current channel, - One or more samples from the vicinity of the reference block in the reference channel, - One or more samples from the neighborhood of the current block in the current channel, - It includes means for applying to predictions based on one or more samples from the neighborhood of a block at the same location within a reference channel.

[0219] According to the embodiment, the apparatus calculates the inter-component model parameters and the calculated model, - The horizontal and / or vertical coordinates of the aforementioned sample, - It includes means for applying to predictions based on one or more of the aforementioned sample directional gradient values.

[0220] According to one embodiment, the device includes means for using the block vector of a block at the same location, estimated from a reference channel, as the initial block vector of the current block.

[0221] According to one embodiment, the apparatus includes means for generating a list of candidate block vectors from a block encoded using an intrablock copy (IBC) method within a region at the same location in a reference channel and / or in the vicinity of the current block in the current channel.

[0222] According to the embodiment, the apparatus comprises means for obtaining the final inter-component prediction parameters of the current block using a two-step derivation, the first step of which is performed using a reference block in a reference channel pointed to by a block vector, and the second step of which is performed in the vicinity of the current block.

[0223] In a further embodiment, a device is provided comprising at least one processor and at least one memory, wherein the at least one memory stores code, which, when executed by the at least one processor, causes the device to receive an image block unit of a frame, wherein the image block unit includes a sample in a color channel, the color channel includes at least one chrominance channel and one luminance channel; reconstruct the sample of the luminance channel of the image block unit; determine a reference area for predicting a target sample of at least one color channel of the image block unit, wherein the reference area includes one or more reference samples in the current block or in a block at the same location, in the current color channel / frame or in a reference color channel / frame, encoded using an intra-block copy (IBC) method; determine a block vector that indicates the spatial distance from the area of ​​the target sample to the reference area; and predict the aforementioned target sample of at least one color channel of the image block unit using an inter-component prediction model based on the reference sample in the aforementioned reference area indicated by the block vector.

[0224] According to the embodiment, the inter-component prediction method is an inter-component linear model (CCLM) or a convolutional inter-component model (CCCM).

[0225] According to one embodiment, the device includes code configured to cause the device to derive parameters of an intercomponent model using a reference block in the current channel and a sample from a reference block in a reference channel.

[0226] According to one embodiment, the device includes code configured to cause the device to estimate a block vector from a block at the same location in a reference channel and to scale the block vector to match the sampling density of the current channel.

[0227] According to one embodiment, the block vectors estimated from the reference channel are not at least partially aligned with the coordinates of blocks at the same location.

[0228] According to one embodiment, the device includes code configured to cause the device to identify the nearest match of a reference area in the current channel using a search based on template matching within the current channel.

[0229] According to the embodiment, the device calculates the inter-component model parameters and the calculated model, - One or more samples from the reference block in the current channel, - One or more samples from the reference block within the reference channel, - One or more samples from the block at the same location in the reference channel, - One or more samples from the vicinity of the reference block in the current channel, - One or more samples from the vicinity of the reference block in the reference channel, - One or more samples from the neighborhood of the current block in the current channel, - Includes code configured to be applied to predictions based on one or more samples from the neighborhood of a block at the same location within a reference channel.

[0230] According to the embodiment, the device calculates the inter-component model parameters and the calculated model, - The horizontal and / or vertical coordinates of the aforementioned sample, - Includes code configured to be applied to predictions based on one or more of the directional gradient values ​​of the aforementioned samples.

[0231] According to one embodiment, the device includes code configured to cause the device to use the block vector of a block at the same location, estimated from a reference channel, as the initial block vector of the current block.

[0232] According to one embodiment, the device includes code configured to cause the device to generate a list of candidate block vectors from a block encoded using an intrablock copy (IBC) method within a region at the same location in a reference channel and / or in the vicinity of the current block in the current channel.

[0233] According to one embodiment, the device includes code configured to cause the device to obtain the final inter-component prediction parameters of the current block using a two-step derivation, the first step of which is performed using a reference block in a reference channel pointed to by a block vector, and the second step of which is performed in the vicinity of the current block.

[0234] Such a device may include, for example, a functional unit disclosed in any of Figures 1, 2, 4a, and 4b for carrying out an embodiment.

[0235] Such a device further comprises code stored in at least one memory, which, when executed by the at least one processor, causes the device to execute one or more of the embodiments disclosed herein.

[0236] Figure 18 is a graphic representation of an exemplary multimedia communication system in which various embodiments may be implemented. Data source 1510 provides a source signal in analog, uncompressed digital, or compressed digital format, or any combination thereof. Encoder 1520 may include, or be connected to, preprocessing such as data format conversion and / or filtering of the source signal. Encoder 1520 encodes the source signal into a coded media bitstream. It should be noted that the bitstream to be decoded may be received directly or indirectly from a remote device in virtually any type of network. Furthermore, the bitstream may be received from local hardware or software. Encoder 1520 may encode two or more media types, such as audio and video, or two or more encoders 1520 may be required to encode different media types of the source signal. Encoder 1520 may also take synthetically generated inputs, such as graphics and text, or produce coded bitstreams of synthetic media. In the following, for the sake of simplicity, only the processing of one coded media bitstream of one media type will be considered. However, it should be noted that real-time broadcast services typically include multiple streams (usually at least one audio, video, and text subtitle stream). While a system may include many encoders, it should also be noted that, for the sake of simplicity and generality, only one encoder 1520 is represented in this diagram. Although the text and examples included herein may specifically describe the encoding process, it should be understood that those skilled in the art will recognize that the same concepts and principles apply to the corresponding decoding process, and vice versa.

[0237] The coded media bitstream may be transferred to storage 1530. Storage 1530 may have any kind of mass memory for storing the coded media bitstream. The format of the coded media bitstream in storage 1530 may be a basic self-contained bitstream format, or one or more coded media bitstreams may be encapsulated in a container file, or the coded media bitstream may be encapsulated in a segment format suitable for DASH (or a similar streaming system) and stored as a sequence of segments. If one or more media bitstreams are encapsulated in a container file, a file generator (not shown in the figure) may be used to store one or more media bitstreams in a file and create metadata for a file format that can also be stored in a file. The encoder 1520 or storage 1530 may have a file generator, or the file generator may be operably mounted on either the encoder 1520 or storage 1530. Some systems operate "live," i.e., omitting storage and transferring the coded media bitstream directly from encoder 1520 to transmitter 1540. The coded media bitstream may then be forwarded to the transmitter 1540, also called the server, as needed. The format used in transmission may be a basic self-contained bitstream format, a packet stream format, a segment format suitable for DASH (or a similar streaming system), or one or more coded media bitstreams may be encapsulated in a container file. The encoder 1520, storage 1530, and server 1540 may reside on the same physical device or be contained in separate devices.The encoder 1520 and server 1540 may also operate with live real-time content, in which case the coded media bitstream is typically buffered for a short time within the content encoder 1520 and / or server 1540 to smooth out processing delays, transfer delays, and fluctuations in the coded media bitrate, rather than being stored permanently.

[0238] Server 1540 transmits a coded media bitstream using a communication protocol stack. The stack may include, but is not limited to, one or more of the following: Real-Time Transport Protocol (RTP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Transmission Control Protocol (TCP), and Internet Protocol (IP). If the communication protocol stack is packet-oriented, Server 1540 encapsulates the coded media bitstream in packets. For example, if RTP is used, Server 1540 encapsulates the coded media bitstream in RTP packets according to the RTP payload format. Typically, each media type has its own dedicated RTP payload format. The system may include two or more Server 1540s, but for simplicity, it should be noted that the following description will consider only one Server 1540.

[0239] If media content is encapsulated in a container file for storage 1530 or for inputting data to transmitter 1540, transmitter 1540 may have a “transmit file parser” (not shown in the figure) or may be operably attached to a “transmit file parser”. In particular, if the container file is not transmitted in that manner, but at least one of the coded media bitstreams it contains is encapsulated for transport over the communication protocol, the transmit file parser finds the appropriate portion of the coded media bitstream to be transmitted over the communication protocol. The transmit file parser may also help to create the correct format of the communication protocol, such as packet headers and payloads. The multimedia container file may include encapsulation instructions, such as hint tracks in ISOBMFF, for the encapsulation of at least one of the contained media bitstreams over the communication protocol.

[0240] Server 1540 may or may not be connected to Gateway 1550 via a communication network which may be, for example, a CDN, the Internet, and / or a combination of one or more access networks. The gateway may, additionally or alternatively, be called a middlebox. In the case of DASH, the gateway may be an edge server (of the CDN) or a web proxy. Note that the system may generally have any number of gateways or similar, but for simplicity, the following description will consider only one gateway 1550. Gateway 1550 may perform various kinds of functions, such as translating packet streams following one communication protocol stack to another communication protocol stack, merging and forking data streams, and manipulating data streams according to downlink capabilities and / or receiver capabilities, such as controlling the bitrate of streams forwarded according to general downlink network conditions. In various embodiments, Gateway 1550 may be a server entity.

[0241] The system typically includes one or more receivers 1560 that can receive, demodulate, and decapsulate transmitted signals into coded media bitstreams. The coded media bitstreams may be transferred to recording storage 1570. Recording storage 1570 may have any kind of mass memory for storing the coded media bitstreams. Recording storage 1570 may optionally or additionally have computational memory, such as random access memory. The format of the coded media bitstreams in recording storage 1570 may be a basic self-contained bitstream format, or one or more coded media bitstreams may be encapsulated in a container file. If there are multiple coded media bitstreams, such as audio and video streams, that are associated with each other, a container file is typically used, and receiver 1560 has or is attached to a container file generator that produces a container file from the input stream. Some systems operate "live," i.e., omitting recording storage 1570 and transferring the coded media bitstreams directly from receiver 1560 to decoder 1580. In some systems, only the most recent portion of a recorded stream, for example, an excerpt of the most recent 10 minutes of the recorded stream, is retained in the recording storage 1570, while previously recorded data is discarded from the recording storage 1570.

[0242] The coded media bitstream may be transferred from the recording storage 1570 to the decoder 1580. If there are many coded media bitstreams, such as audio and video streams, associated with each other and encapsulated in a container file, or if a single media bitstream is encapsulated in a container file for easier access, for example, a file parser (not shown in the diagram) is used to decapsulate each coded media bitstream from the container file. The recording storage 1570 or decoder 1580 may include a file parser, or the file parser may be mounted on either the recording storage 1570 or decoder 1580. It should also be noted that while the system may include many decoders, only one decoder 1570 is described herein for the sake of simplicity without loss of generality.

[0243] The coded media bitstream is further processed by the decoder 1570, whose output is one or more uncompressed media streams. Finally, the renderer 1590 may play the uncompressed media streams, for example, using a loudspeaker or display. The receiver 1560, recording storage 1570, decoder 1570, and renderer 1590 may reside in the same physical device or be contained in separate devices.

[0244] Transmitter 1540 and / or gateway 1550 may be configured to perform switching between different representations, view switching, bitrate adaptation, and / or fast startup, for example, to switch between different viewports of 360-degree video content, and / or transmitter 1540 and / or gateway 1550 may be configured to select the transmitted representation. Switching between different representations may be done for several reasons, such as in response to a request from receiver 1560 or for general conditions such as the throughput of the network through which the bitstream is transmitted. In other words, receiver 1560 may initiate switching between representations. Requests from receivers may be, for example, requests for a segment or subsegment from a different representation than previously, requests for changes to the transmitted scalability layer and / or sublayer, or requests for changes to rendering devices having different capabilities compared to previous capabilities. A segment request may be an HTTP GET request. A subsegment request may be an HTTP GET request with a byte range. Additionally or alternatively, bitrate adjustment or bitrate adaptation may be used, for example, in a streaming service to provide so-called fast startup, where the bitrate of the transmitted stream is lower than the channel bitrate at the start of streaming or after random access in order to begin playback immediately and to achieve a buffer occupancy level that allows for occasional packet delays and / or retransmissions. Bitrate adaptation may include up-switching operations of multiple representations or layers, and down-switching operations of representations or layers, performed in various orders.

[0245] Decoder 1580 may be configured to perform switching between different representations, view switching, bitrate adaptation, and / or fast startup, for example, to switch between different viewports of 360-degree video content, and / or decoder 1580 may be configured to select the transmitted representation. Switching between different representations may be done for several reasons, such as to achieve faster decoding operation or to adapt the transmitted bitstream to common conditions, such as the throughput of the network through which the bitstream is transmitted, with respect to the bitrate. Faster decoding operation may be required, for example, when the device containing decoder 1580 is multitasking and uses computing resources for purposes other than decoding the video bitstream. In another example, faster decoding operation may be required when the content is played back at a faster pace than normal playback speed (e.g., twice or three times faster than conventional real-time playback speed).

[0246] In the above, some embodiments have been described with reference to and / or use of the terms HEVC and / or VVC. It should be understood that embodiments can be similarly implemented using any video encoder and / or video decoder.

[0247] Where an embodiment is described above with reference to an encoder, it should be understood that the resulting bitstream and decoder may include corresponding elements. Similarly, where an embodiment is described with reference to a decoder, it should be understood that the encoder may include a structure and / or computer program for generating a bitstream to be decoded by the decoder. For example, some embodiments have been described in relation to generating prediction blocks as part of encoding. Embodiments can similarly be realized by generating prediction blocks as part of decoding, but with the difference that coding parameters such as horizontal and vertical offsets are decoded from the bitstream rather than being determined by the encoder.

[0248] The embodiments of the present invention described above will explain the codec in terms of separate encoder and decoder devices to aid in understanding the processes involved. However, it will be understood that the device, structure, and operation may be carried out as a single encoder / decoder device / structure / operation. Furthermore, it is possible that the coder and decoder may share some or all common elements.

[0249] While the above examples illustrate embodiments of the present invention operating within a codec in an electronic device, it will be understood that the present invention, as defined in the claims, can be implemented as part of any video codec. Therefore, for example, embodiments of the present invention may be implemented in a video codec capable of performing video coding over a fixed or wired communication path.

[0250] Therefore, the user device may be equipped with a video codec such as the video codec described in the embodiments of the present invention described above. It should be understood that the term "user device" is intended to refer to any suitable type of wireless user device, such as a mobile phone, a portable data processing device, or a portable web browser.

[0251] Furthermore, elements of the public land mobile network (PLMN) may also be equipped with video codecs as described above.

[0252] In general, various embodiments of the present invention may be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. For example, some embodiments may be implemented in hardware, while others may be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device, but the present invention is not limited thereto. Various embodiments of the present invention may be illustrated and explained using block diagrams, flowcharts, or any other graphical representation, but it should be understood that these blocks, devices, systems, techniques, or methods described herein may, in non-limiting examples, be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or any combination thereof.

[0253] Embodiments of the present invention may be implemented by computer software executable by a data processor of a mobile device, such as a processor entity, by hardware, or by a combination of software and hardware. Furthermore, it should be noted that any block of the logic flow shown in the figure may represent a program step, or an interconnected set of logic circuits, blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on a physical medium such as a memory chip, or a memory block implemented within the processor, on a magnetic medium such as a hard disk or floppy disk, or on an optical medium such as a DVD and its data variant, or a CD.

[0254] Memory may be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The data processor may be of any type suitable for the local technical environment and may include, in non-limiting examples, one or more of general-purpose computers, dedicated computers, microprocessors, digital signal processors (DSPs), and processors based on multi-core processor architectures.

[0255] Embodiments of the present invention may be practiced in various components, such as integrated circuit modules. Integrated circuit design is generally a highly automated process. Complex and powerful software tools are available to translate logic-level designs into ready-to-form semiconductor circuit designs that are etched onto semiconductor substrates.

[0256] Programs such as those offered by Synopsys, Inc. in Mountain View, California, and Cadence Design Systems, Inc. in San Jose, California, use established design rules and a library of pre-stored design modules to automatically route conductors and place components on a semiconductor chip. After the semiconductor circuit design is complete, the resulting design in a standardized electronic format (e.g., Opus, GDSII, etc.) may be sent to a semiconductor manufacturing facility or "fab" for production.

[0257] The foregoing description has provided a complete and useful explanation of embodiments of the present invention, as exemplary and non-limiting examples. However, various modifications and adaptations may become apparent to those skilled in the art when considered in conjunction with the accompanying drawings and claims. However, all such modifications and similar modifications of the teachings of the present invention remain within the scope of the invention.

Claims

1. Means for receiving an image block unit of a frame, wherein the image block unit includes a sample in a color channel, and the color channel includes at least one chrominance channel and one luminance channel; Means for reconstructing the luminance channel samples in the image block units, Means for determining a reference area for predicting a target sample of at least one color channel in the image block unit, wherein the reference area includes one or more reference samples in the current block or in a block at the same location, within the current color channel or in the current frame or within the reference color channel or in the reference frame, encoded using an intrablock copy (IBC) method. Means for determining a block vector that indicates the spatial distance from the area of ​​the target sample to the reference area, An apparatus comprising means for predicting the target sample of at least one color channel of the image block unit using an inter-component prediction model based on the reference sample in the reference area indicated by the block vector.

2. The apparatus according to claim 1, wherein the inter-component prediction method is an inter-component linear model (CCLM) or a convolutional inter-component model (CCCM).

3. The apparatus according to claim 1 or 2, comprising means for deriving the parameters of the intercomponent model using a reference block in the current channel and a sample from the reference block in the reference channel.

4. Means for estimating the block vector from the block at the same position within the reference channel, The apparatus according to any one of claims 1 to 3, further comprising means for scaling the block vector to match the current sampling density of the channel.

5. The apparatus according to claim 4, wherein the block vector estimated from the reference channel is not at least partially aligned with the coordinates of a block at the same location.

6. The apparatus according to any one of claims 1 to 5, comprising means for identifying the nearest match of the reference area in the current channel using a search based on template matching in the current channel.

7. The inter-component model parameters are calculated, and the calculated model is used. - One or more samples from the reference block in the current channel, - One or more samples from the reference block in the reference channel, - One or more samples from the block at the same location in the reference channel, - One or more samples from the vicinity of the reference block in the current channel, - One or more samples from the vicinity of the reference block in the reference channel, - One or more samples from the vicinity of the current block within the current channel, - The apparatus according to any one of claims 1 to 6, comprising means for applying to a prediction based on one or more samples from the vicinity of the block at the same location in the reference channel.

8. The inter-component model parameters are calculated, and the calculated model is used. - The horizontal and / or vertical coordinates of the aforementioned sample, - The apparatus according to claim 7, further comprising means for applying to a prediction based on one or more of the directional gradient values ​​of the sample.

9. The apparatus according to any one of claims 1 to 8, comprising means for using the block vector of the block at the same location estimated from the reference channel as the initial block vector of the current block.

10. The apparatus according to any one of claims 1 to 9, comprising means for generating a list of candidate block vectors from blocks encoded using the intrablock copy (IBC) method within the block at the same location in the reference channel and / or in the vicinity of the current block in the current channel.

11. The apparatus according to any one of claims 1 to 10, comprising means for obtaining the final inter-component prediction parameters of the current block using a two-step derivation, wherein the first step of derivation is performed using the reference block in the reference channel pointed to by the block vector, and the second step of derivation is performed in the vicinity of the current block.

12. Receiving an image block unit of a frame, wherein the image block unit includes a sample in a color channel, and the color channel includes at least one chrominance channel and one luminance channel. Reconstructing the sample of the luminance channel in the image block unit, Determining a reference area for predicting a target sample of at least one color channel in the image block unit, wherein the reference area includes one or more reference samples in the current block or in the same block, encoded using an intrablock copy (IBC) method, in the current color channel or in the current frame, or in the reference color channel or in the reference frame. Determine a block vector that indicates the spatial distance from the area of ​​the target sample to the reference area, A method comprising predicting the target sample for at least one color channel of the image block unit using an inter-component prediction model based on the reference sample in the reference area indicated by the block vector.

13. The method according to claim 12, wherein the intercomponent prediction method is an intercomponent linear model (CCLM) or a convolutional intercomponent model (CCCM).

14. The method according to claim 12 or 13, comprising deriving the parameters of the intercomponent model using a reference block in the current channel and a sample from a reference block in the reference channel.

15. Estimating the block vector from the block at the same location within the reference channel, The method according to any one of claims 12 to 14, comprising scaling the block vector to match the sampling density of the current channel.

Images