Apparatus, method, and computer program for determining cross-component parameters

Abstract

11. A method comprising: determining a reference value for a parameter of a mapping function from a first color component to a second color component; decoding a magnitude of an initial update term; decoding a syntax element to be used in determining a sign of the update term; determining the sign of the update term by interpreting the decoded syntax element based on the reference value; and determining a value of the parameter of the mapping function using the reference value, the decoded magnitude of the update term, and the determined sign of the update term.

Description

[Technical Field]

[0001] The present invention relates to an apparatus, method, and computer program for determining cross-component parameters in video encoding and decoding. [Background technology]

[0002] In video coding, video and image samples are typically encoded using color representations such as YUV or YCbCr, which consist of one luminance (luma) channel and two chrominance (chroma) channels. In these cases, the luminance channel, which primarily represents the illumination of the scene, is generally coded at a specific resolution, while the chrominance channel, which generally represents the difference between specific color components, is often coded at a second resolution lower than the resolution of the luminance signal. The intention of this type of difference representation is to decorrelated the color components, allowing for more efficient data compression.

[0003] The Versatile Video Coding (VVC / H.266) standard uses a cross-component linear model (CCLM) as the linear model for predicting samples in chroma channels (e.g., Cb and Cr). This process generates a linear model that can be used to map chroma sample values ​​to chroma sample values. The parameters of the linear model are constructed using the available reconstructed chroma and chroma reference samples outside the boundaries of the prediction block. Once the parameters are constructed, the linear model specified by those parameters is used to predict the chroma sample values ​​inside the prediction block.

[0004] Applying cross-component linear model prediction to predict chroma samples can be more efficient than intra-component space prediction, because if there is a strong linear correlation between luma and chroma channels, the reconstructed luma block texture can sometimes generate a very good predictor for the chroma texture. However, the reconstructed reference samples used to generate the linear model parameters may be noisy or may not adequately represent the content inside the actual prediction block. In these cases, the prediction fails, and the process results in coding efficiency that is not optimal for the content. [Overview of the project] [Means for solving the problem]

[0005] Next, to at least mitigate the aforementioned problems, an enhanced method for achieving a better correlation between the luminous and chroma channels is introduced herein.

[0006] In some embodiments, a method is provided for generating reference values ​​for parameters used in a lumane-to-chromane mapping model and determining update terms applied to refine the reference values. Determining the value of the update term may include reversing the sign of the displayed value of the update term depending on the sign or magnitude of the generated reference value.

[0007] Some embodiments may be applied to mapping models from a first color component to a second color component, for example, the first color component may be the luma component and the second color component may be the chroma component, or the first color component may be the chroma component and the second color component may be the luma component. Some embodiments may also be applied to mapping models that operate between different types of data. For example, some embodiments may be applied to mappings that convert image sample values ​​into depth map values ​​that indicate the distance of the sample from a specific point or plane.

[0008] The apparatus according to the first aspect: determines a reference value of a parameter of a mapping function from a first color component to a second color component; decodes the magnitude of an initial update term; decodes a syntax element to be used when determining the sign of an update term; determines the sign of an update term by interpreting the decoded syntax element based on the reference value; uses the reference value, the decoded magnitude of the update term, and the determined sign of the update term to determine the value of a parameter of the mapping function and comprises means for doing so.

[0009] According to one embodiment, the apparatus further comprises means for: comparing the reference value with a threshold value; determining that the sign of the update term is positive if the reference value is greater than the threshold value and the syntax element has a first display value, or if the reference value is less than or equal to the threshold value and the syntax element has a second display value different from the first display value, or determining that the sign of the update term is negative otherwise and comprises means for doing so.

[0010] According to one embodiment, the first display value is 0 and the second display value is 1, or the first display value is 1 and the second display value is 0.

[0011] According to one embodiment, the threshold value is 0.

[0012] According to one embodiment of the apparatus, the means for determining the sign comprises means for determining the sign such that the reference value is refined towards zero if the decoded binary syntax element representing the sign of the update term is 0, and the reference value is refined away from zero if the decoded binary syntax element representing the sign of the update term is 1.

[0013] According to one embodiment of the apparatus, the means for determining the sign comprises means for determining the sign such that the updated term refines the reference value toward zero when the decoded binary syntax element representing the sign of the updated term is 1, and refines the reference value toward zero when the decoded binary syntax element representing the sign of the updated term is 0.

[0014] According to one embodiment, the device includes means for determining the value of an update term by reversing the sign of the initial update term if the value of the reference parameter is greater than or equal to zero.

[0015] According to one embodiment, the apparatus includes means for determining the value of an update term by leaving the initial update term unchanged if the value of a reference parameter is less than or equal to zero.

[0016] According to one embodiment of the apparatus, the first color component is a luma component, and the second color component is a chroma component.

[0017] The method according to the second embodiment is: To determine the reference values ​​of the parameters of the mapping function from the first color component to the second color component; Decode the size of the initial update term; Decoding the syntax elements to be used when determining the sign of the update term; The sign of the update term is determined by interpreting the decoded syntax element based on the reference value; The values ​​of the mapping function parameters are determined using the reference value, the decoded size of the update term, and the determined sign of the update term. Includes.

[0018] An apparatus according to a third embodiment comprises at least one processor and at least one memory, wherein code is stored in the at least one memory, and when the code is executed by the at least one processor, the apparatus provides at least: To determine the reference values ​​of the parameters of the mapping function from the first color component to the second color component; Decode the size of the initial update term; Decoding the syntax elements to be used when determining the sign of the update term; The sign of the update term is determined by interpreting the decoded syntax element based on the reference value; The values ​​of the mapping function parameters are determined using the reference value, the decoded size of the update term, and the determined sign of the update term. Have them do it.

[0019] The apparatus according to the fourth aspect is: Determine the reference values ​​for the parameters of the mapping function from the first color component to the second color component; Determining the value of the initial update term for the reference value, If the reference value is greater than the threshold, the displayed value of the update term is determined such that the sign of the displayed value is opposite to the sign of the determined value of the update term, The system includes means for encoding the sign and magnitude of the displayed value of the update item into the video bitstream.

[0020] According to one embodiment, the device includes means for reversing the sign of an update term if the value of a reference parameter is greater than or equal to zero.

[0021] According to one embodiment, the device includes means for determining the displayed value of an update term by leaving the update term unchanged if the value of the reference parameter is less than or equal to zero.

[0022] According to one embodiment, this device is Rate distortion is determined to determine the value of the update term. The system includes means for determining the update items to be displayed based on the values ​​of the update items.

[0023] According to one embodiment, this device is Rate distortion is determined to determine the values ​​of the displayed update terms. The system includes means for determining the update item based on the displayed update item value.

[0024] The method according to the fifth aspect is: To determine the reference values ​​of the parameters of the mapping function from the first color component to the second color component; To determine the value of the initial update term for the reference value; If the reference value is greater than the threshold, the displayed value of the update term is determined such that the sign of the displayed value is opposite to the sign of the determined value of the update term; Encoding the sign and magnitude of the displayed values ​​in the update item into the video bitstream. Includes.

[0025] An apparatus according to the sixth aspect comprises at least one processor and at least one memory, wherein code is stored in the at least one memory, and when the code is executed by the at least one processor, the apparatus provides at least: To determine the reference values ​​of the parameters of the mapping function from the first color component to the second color component; To determine the value of the initial update term for the reference value; If the reference value is greater than the threshold, the displayed value of the update term is determined such that the sign of the displayed value is opposite to the sign of the determined value of the update term; Encoding the sign and magnitude of the displayed values ​​in the update item into the video bitstream. Have them do it.

[0026] Therefore, the device and the computer-readable storage medium in which the code is stored are configured to perform one or more of the methods and related embodiments described above.

[0027] For a better understanding of the present invention, the accompanying drawings are now referred to as examples. [Brief explanation of the drawing]

[0028] [Figure 1] This figure schematically illustrates an electronic device utilizing an embodiment of the present invention. [Figure 2] This figure schematically shows a user device suitable for using embodiments of the present invention. [Figure 3] This figure schematically illustrates electronic devices utilizing embodiments of the present invention, connected using wireless and wired network connections. [Figure 4a] This figure schematically shows an encoder suitable for carrying out an embodiment of the present invention. [Figure 4b] This figure schematically shows a decoder suitable for carrying out embodiments of the present invention. [Figure 5a] This figure shows a linear model mapping from luma values ​​to chroma values ​​according to one embodiment of the present disclosure. [Figure 5b] This is a diagram showing the updated mapping. [Figure 5c] This figure shows how to select control points pr outside the determined initial mapping line and perform gradient updates with respect to such points. [Figure 5d] This figure shows how to update only the offset parameter b without changing the gradient, using the control point pr outside the determined initial mapping line. [Figure 6a] This figure shows the selection of a reference sample R on block B according to one embodiment of the present disclosure. [Figure 6b] This figure shows the selection of a reference sample R on block B according to one embodiment of the present disclosure. [Figure 6c] This figure shows the selection of a reference sample R on block B according to one embodiment of the present disclosure. [Figure 7a] This is a flowchart of a method according to one embodiment. [Figure 7b] This is a flowchart of a method according to another embodiment. [Figure 8]This is a schematic diagram of an exemplary multimedia communication system in which various embodiments can be implemented. [Modes for carrying out the invention]

[0029] The following describes in more detail suitable devices and possible mechanisms for initiating viewpoint switching. In this regard, Figures 1 and 2 are first referenced, with Figure 1 showing a block diagram of a video coding system according to an exemplary embodiment as a schematic block diagram of an exemplary device or electronic device 50 that can incorporate a codec according to one embodiment of the present invention. Figure 2 shows the layout of the device according to an exemplary embodiment. The elements of Figures 1 and 2 are described below.

[0030] The electronic device 50 may be, for example, a portable terminal or user device of a wireless communication system. However, it will be understood that embodiments of the present invention may be implemented in electronic devices or apparatus that require video image encoding and decoding or encoding or decoding.

[0031] The device 50 may include a housing 30 for the incorporation and protection of the device. The device 50 may further include a display 32 in the form of a liquid crystal display. In other embodiments of the present invention, the display may be any suitable display technology suitable for displaying images or videos. The device 50 may further include a keypad 34. In other embodiments of the present invention, any suitable data or user interface mechanism may be utilized. For example, the user interface may be implemented as a virtual keyboard or data entry system as part of a touch-sensitive display.

[0032] The device may be equipped with a microphone 36 or any suitable audio input which may be a digital or analog signal input. In embodiments of the present invention, the device 50 may further be equipped with an audio output device which may be an earphone 38, a speaker, or any one of analog audio or digital audio output connections. The device 50 may further be equipped with a battery (or in other embodiments of the present invention, the device may be powered by any suitable portable energy device such as a solar cell, fuel cell, or clockwork generator). The device may further be equipped with a camera which may record or capture images and / or video. The device 50 may further be equipped with an infrared port for short-range line-of-sight communication with other devices. In other embodiments, the device 50 may further be equipped with any suitable short-range communication solution, such as a Bluetooth wireless connection or a USB / FireWire wired connection.

[0033] The device 50 may include a controller 56, a processor, or a processor circuit for controlling the device 50. In embodiments of the present invention, the controller 56 may be connected to a memory 58 that can store data in both the form of image and audio data, and / or further store instructions for implementation by the controller 56. The controller 56 may further be connected to a codec circuit 54 that is suitable for performing coding and decoding of audio and / or video data, or for assisting coding and decoding performed by the controller.

[0034] The device 50 may further include a card reader 48 and a smart card 46, for example, a UICC and a UICC reader suitable for providing user information and authentication information for user authentication and authorization on a network.

[0035] The device 50 may include a radio interface circuit 52 connected to a controller and suitable for generating radio communication signals for communication with, for example, a cellular communication network, a wireless communication system, or a wireless local area network. The device 50 may further include an antenna 44 connected to the radio interface circuit 52 for transmitting radio frequency signals generated by the radio interface circuit 52 to other devices and for receiving radio frequency signals from other devices.

[0036] The device 50 may include a camera capable of recording or detecting individual frames, which are then passed to a codec 54 or controller for processing. The device may receive video image data for processing from another device before transmission and / or storage. The device 50 may further receive images for coding / decoding wirelessly or via a wired connection. The structural elements of the device 50 described above illustrate examples of means for performing the corresponding functions.

[0037] With respect to Figure 3, an example of a system in which embodiments of the present invention may be used is shown. System 10 includes a number of communication devices that can communicate through one or more networks. System 10 may include, but is not limited to, any combination of wired or wireless networks, including wireless cellular telephone networks (such as GSM, UMTS, and CDMA networks), wireless local area networks (WLANs) such as those defined by any of the IEEE 802.x standards, Bluetooth personal area networks, Ethernet local area networks, Token Ring local area networks, wide area networks, and the Internet.

[0038] System 10 may include both wired and wireless communication devices and / or apparatus 50 suitable for carrying out embodiments of the present invention.

[0039] For example, the system shown in Figure 3 represents a mobile phone network 11 and the Internet 28. Connectivity to the Internet 28 may include, but are not limited to, long-range wireless connections, short-range wireless connections, and various wired connections, but are not limited to, telephone lines, cable lines, power lines, and similar communication paths.

[0040] The communication devices illustrated in System 10 may include, but are not limited to, electronic devices or apparatus 50, a combination of a personal digital assistant (PDA) and a mobile phone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, and a notebook computer 22. Apparatus 50 may be fixed in place or portable if carried by a moving individual. Apparatus 50 may also be located in, but are not limited to, a car, truck, taxi, bus, train, ship, airplane, bicycle, motorcycle, or other means of transport with similar appropriate means of transport.

[0041] Embodiments may also be implemented in set-top boxes; digital TV receivers which may or may not have a display or wireless capabilities; tablets or (laptop) personal computers (PCs) which have hardware, software, or a combination of encoder / decoder implementations; various operating systems; and chipsets, processors, DSPs, and / or embedded systems which provide hardware / software-based coding.

[0042] Some or more devices can send and receive calls and messages and communicate with service providers through a wireless connection 25 to a base station 24. The base station 24 may be connected to a network server 26 that enables communication between the cellular network 11 and the internet 28. The system may include additional communication devices and various types of communication devices.

[0043] Communication devices can communicate using a variety of transmission technologies, including, but not limited to, Code Division Multiple Access (CDMA), Global Systems for Mobile Communications (GSM), Universal Mobile Telecommunications System (UMTS), Time Division Multiple Access (TDMA), Frequency Division Multiple Access (FDMA), Transmission Control Protocol-Internet Protocol (TCP-IP), Short Message Service (SMS), Multimedia Message Service (MMS), Email, Instant Message Service (IMS), Bluetooth, IEEE 802.11, and any similar wireless communication technologies. Communication devices relating to various embodiments of the present invention can communicate using a variety of media, including, but not limited to, wireless, infrared, laser, cable connections, and any suitable connections.

[0044] In telecommunications and data networks, a channel can refer to either a physical channel or a logical channel. A physical channel can refer to a physical transmission medium such as a wire, while a logical channel can refer to a logical connection on a multiplexing medium that can carry several logical channels. A channel can be used to carry information signals, such as a bitstream, from one or more senders (or transmitters) to one or more receivers.

[0045] The MPEG-2 transport stream (TS), as defined in ISO / IEC 13818-1 or equivalently in ITU-T Recommendation H.222.0, is a format for transporting audio, video, and other media, as well as program metadata or other metadata, in a multiplexed stream. Packet identifiers (PIDs) are used to identify elementary streams (also known as packetized elementary streams) within the TS. Therefore, logical channels within an MPEG-2 TS can be considered to correspond to specific PID values.

[0046] Available media file format standards include the ISO-based media file format (ISO / IEC 14496-12, sometimes abbreviated as ISOBMFF) and the file format for NAL unit structured video (ISO / IEC 14496-15), which is derived from ISOBMFF.

[0047] A video codec consists of an encoder, which converts the input video into a compressed representation suitable for storage / transmission, and a decoder, which restores the compressed video representation back into a viewable form. The video encoder and / or video decoder may also be separate from each other, i.e., they do not need to form a codec. Generally, the encoder discards some information from the original video sequence in order to represent the video in a more compact form (i.e., at a lower bitrate).

[0048] A typical hybrid video encoder, such as many encoder implementations of ITU-T H.263 and H.264, encodes video information in two phases. First, the pixel values ​​of a particular picture area (or "block") are predicted, for example, by motion compensation (finding and indicating an area in one of the previously coded video frames that closely corresponds to the block to be coded) or by spatial means (using pixel values ​​around the block to be coded in a specified way). Second, the prediction error, i.e., the difference between the predicted block of pixels and the original block of pixels, is coded. This is generally done by transforming the difference in pixel values ​​using a specified transformation (e.g., the discrete cosine transform (DCT) or a variation thereof), quantizing the coefficients, and entropy coding the quantized coefficients. By changing the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (picture quality) and the size of the resulting coded video representation (file size or transmission bitrate).

[0049] In time prediction, the source of the prediction is a previously decoded picture (also known as a reference picture). In intra-block copy (IBC; also known as intra-block copy prediction), the prediction is applied similarly to time prediction, but the reference picture is the current picture, and only previously decoded samples can be referenced in the prediction process. Interlayer or interview prediction can be applied similarly to time prediction, but the reference picture is a decoded picture from another scalable layer or another view, respectively. In some cases, inter-prediction can only reference time prediction, but in other cases, inter-prediction can refer to time prediction and any of intra-block copy, interlayer prediction, and interview prediction together, provided that they are performed in the same or similar process as time prediction. Inter-prediction or time prediction is sometimes called motion-compensated or motion-compensated prediction.

[0050] Motion compensation can be performed with either full-sample accuracy or sub-sample accuracy. In full-sample accuracy motion compensation, motion can be represented as a motion vector with integer values ​​for horizontal and vertical displacements, and the motion compensation process uses these displacements to effectively copy the sample from the reference picture. In sub-sample accuracy motion compensation, the motion vector is represented by decimal or decimal values ​​for the horizontal and vertical components of the motion vector. If the motion vector points to a non-integer position in the reference picture, a sub-sample interpolation process is generally invoked to calculate a predicted sample value based on the reference sample and the selected sub-sample position. The sub-sample interpolation process generally consists of vertical filtering to compensate for vertical offsets relative to the full sample position, followed by horizontal filtering to compensate for horizontal offsets relative to the full sample position. However, vertical processing may also be performed before horizontal processing, depending on the environment.

[0051] Interpretation, sometimes called time prediction, motion compensation, or motion-compensated prediction, reduces temporal redundancy. In interpretation, the source of prediction is a previously decoded picture. Intrapretation takes advantage of the fact that adjacent pixels within the same picture are likely to be correlated. Intrapretation can be performed in the spatial domain or the transformation domain, i.e., either sample values ​​or transformation coefficients can be predicted. Intrapretation is generally used in intracoding, where interpretation is not applied.

[0052] One outcome of the coding procedure is a set of coding parameters, such as motion vectors and quantized transformation coefficients. Many parameters can be entropy coded more efficiently if they are first predicted from spatially or temporally adjacent parameters. For example, motion vectors can be predicted from spatially adjacent motion vectors, and only the difference to the motion vector predictor can be coded. The prediction and intra-prediction of coding parameters are sometimes collectively referred to as in-picture prediction.

[0053] Figures 4a and 4b show encoders and decoders suitable for utilizing embodiments of the present invention. A video codec consists of an encoder that converts input video into a compressed representation suitable for storage / transmission, and a decoder that can restore the compressed video representation back into a viewable form. Generally, encoders discard and / or lose some information of the original video sequence in order to represent the video in a more compact form (i.e., at a lower bitrate). An example of the encoding process is shown in Figure 4a. Figure 4a shows the image to be encoded (In); a predicted representation of the image block (P'n), a predicted error signal (Dn); a reconstructed predicted error signal (D'n); a preliminary reconstructed image (I'n), a final reconstructed image (R'n); a transform (T) and an inverse transform (T-1); quantization (Q) and inverse quantization (Q-1); entropy encoding (E); reference frame memory (RFM); inter-prediction (Pinter); intra-prediction (Pintra); mode selection (MS) and filtering (F).

[0054] An example of the decoding process is shown in Figure 4b. Figure 4b shows the predicted representation of an image block (P'n); the reconstructed predicted error signal (D'n); the preliminary reconstructed image (I'n); the final reconstructed image (R'n); the inverse transform (T-1); the inverse quantization (Q-1); the entropy decoding (E-1); the reference frame memory (RFM); the prediction (either inter or intra) (P); and the filtering (F).

[0055] Many hybrid video encoders encode video information in two phases. First, the pixel values ​​of a particular picture area (or "block") are predicted, for example, by motion compensation (finding and indicating an area in one of the previously coded video frames that closely corresponds to the block to be coded) or by spatial means (using pixel values ​​around the block to be coded in a specified way). Second, the prediction error, i.e., the difference between the predicted block of pixels and the original block of pixels, is coded. This is generally done by transforming the difference in pixel values ​​using a specified transformation (e.g., the discrete cosine transform (DCT) or a variation thereof), quantizing the coefficients, and entropy coding the quantized coefficients. By changing the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (picture quality) and the size of the resulting coded video representation (file size or transmission bitrate). Video codecs can also provide transformation skip modes that the encoder can choose to use. In conversion skip mode, the prediction error is coded in the sample domain, for example, by deriving the sample-by-sample difference value for a specific adjacent sample and coding the sample-by-sample difference value using an entropy coder.

[0056] Entropy coding / decoding can be performed in many ways. For example, context-based coding / decoding may be applied, where both the encoder and decoder modify the contextual state of the coding parameters based on previously coded / decoded coding parameters. Context-based coding can be, for example, context-adaptive binary arithmetic coding (CABAC) or context-based variable-length coding (CAVLC) or any similar entropy coding. Entropy coding / decoding may be performed using variable-length coding schemes such as Huffman coding / decoding or exponential Golomb coding / decoding, either as an alternative or additional method. Decoding coding parameters from an entropy-coded bitstream or codeword is sometimes called parsing.

[0057] The phrase "along a bitstream" (for example, indicating that something is aligned with a bitstream) can be defined to refer to out-of-band transmission, signaling, or storage in a way that the out-of-band data is associated with the bitstream. Phrases such as "along a bitstream decoding" can refer to the decoding of referenced out-of-band data (which may be derived from out-of-band transmission, signaling, or storage) associated with the bitstream. For example, "along a bitstream indication" can refer to metadata within a container file that encapsulates the bitstream.

[0058] In the following, several embodiments will be described in the context of a single video coding configuration. However, it should be noted that these embodiments are not necessarily limited to this particular configuration. The embodiments relate to a fine-grained gradual decoding refresh with a CTU-aligned boundary between the clean and dirty areas of a picture.

[0059] The Advanced Video Coding standard (AVC or sometimes abbreviated as H.264 / AVC) was developed by the Video Coding Expert Group (VCEG) of the Telecommunication Standardization Sector of the International Telecommunication Union (ITU-T) and the Joint Video Team (JVT) of the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard is published by both standards bodies and is known as ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10, and is also known as MPEG-4 Part 10 Advanced Video Coding (AVC). Numerous versions of the H.264 / AVC standard exist, each incorporating new extensions or features into the specification. These extensions include Scalable Video Coding (SVC) and Multiview Video Coding (MVC).

[0060] The High Efficiency Video Coding standard (sometimes abbreviated as HEVC or H.265 / HEVC) was developed by the Joint Cooperation Team for Video Coding (JCT-VC) of VCEG and MPEG. This standard is published by both standards bodies and is known as ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2, also known as MPEG-H Part 2 High Efficiency Video Coding (HEVC). Extensions to H.265 / HEVC include scalable, multi-view, 3D, and fidelity range extensions, which are sometimes referred to as SHVC, MV-HEVC, 3D-HEVC, and REXT, respectively. References to H.265 / HEVC, SHVC, MV-HEVC, 3D-HEVC, and REXT in this specification are made for the purpose of understanding the definitions, structure, or concepts of these standards and should be understood as references to the latest versions of these standards available prior to the date of this filing, unless otherwise indicated.

[0061] Multipurpose Video Coding (VVC, H.266, or sometimes abbreviated as H.266 / VVC) is a video compression standard developed as a successor to HEVC. VVC is defined in ITU-T Recommendation H.266 and is equivalently defined in ISO / IEC 23090-3, also known as MPEG-I Part 3.

[0062] The AV1 bitstream format and decoding process specification was developed by the Alliance of Open Media (AOM). The AV1 specification was published in 2018. The AOM is reportedly working on the AV2 specification.

[0063] Some important definitions, bitstreams, and coding structures, and concepts of H.264 / AVC, HEVC, VVC, and / or AV1, and some of their extensions, are described in this section as examples of video encoders, decoders, encoding methods, decoding methods, and bitstream structures in which embodiments may be implemented. The various embodiments are not limited to H.264 / AVC, HEVC, VVC, and / or AV1, or their extensions; rather, the description is given for one possible basis in which these embodiments may be realized in part or in whole.

[0064] A video codec may include an encoder, which converts input video into a compressed representation suitable for storage / transmission, and a decoder, which restores the compressed video representation back into a viewable form. The compressed representation is sometimes called a bitstream or video bitstream. The video encoder and / or video decoder may also be separate from each other, i.e., they do not need to form a codec. The encoder may discard some information from the original video sequence in order to represent the video in a more compact form (i.e., at a lower bitrate). The notation "(de)coder" refers to both the encoder and / or decoder.

[0065] A hybrid video codec, such as ITU-T H.264, can encode video information in two phases. First, the pixel values ​​of a particular picture area (or "block") are predicted, for example, by motion compensation (finding and indicating an area in one of the previously coded video frames that closely corresponds to the block to be coded) or by spatial means (using the pixel values ​​around the block to be coded in a specified way). Next, the prediction error, i.e., the difference between the predicted block of pixels and the original block of pixels, is coded. This can be done by transforming the difference in pixel values ​​using a specified transformation (e.g., the discrete cosine transform (DCT) or a variation thereof), quantizing the coefficients, and entropy coding the quantized coefficients. By changing the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (picture quality) and the size of the resulting coded video representation (file size or transmission bitrate).

[0066] In time prediction, the source of the prediction is a previously decoded picture (also known as a reference picture). In intrablock copy (IBC; also known as intrablock copy prediction or current picture reference), the prediction is applied similarly to time prediction, but the reference picture is the current picture, and only previously decoded samples can be referenced in the prediction process. Interlayer or interview prediction can be applied similarly to time prediction, but the reference picture is a decoded picture from another scalable layer or another view, respectively. In some cases, interpretation can only reference time prediction, but in other cases, interpretation can reference time prediction and any of intrablock copy, interlayer prediction, and interview prediction together, provided that they are performed in the same or similar process as time prediction. Interpretation or time prediction is sometimes called motion-compensated or motion-compensated prediction.

[0067] Intra prediction leverages the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in the spatial domain or the transformation domain; that is, either sample values ​​or transformation coefficients can be predicted. Intra prediction can be used in intra coding, where intra prediction is not applied.

[0068] One outcome of the coding procedure is a set of coding parameters, such as motion vectors and quantized transformation coefficients. Many parameters can be entropy coded more efficiently if they are first predicted from spatially or temporally adjacent parameters. For example, motion vectors can be predicted from spatially adjacent motion vectors, and only the difference to the motion vector predictor can be coded. The prediction and intra-prediction of coding parameters are sometimes collectively referred to as in-picture prediction.

[0069] Entropy coding / decoding can be performed in many ways. For example, context-based coding / decoding can be applied, where both the encoder and decoder modify the contextual state of the coding parameters based on previously coded / decoded coding parameters. Context-based coding can be, for example, context adaptive binary arithmetic coding (CABAC) or context-based variable length coding (CAVLC) or any similar entropy coding. Entropy coding / decoding can also be performed using variable-length coding schemes such as Huffman coding / decoding or exponential Golomb coding / decoding, either as an alternative or additional method. Decoding coding parameters from an entropy-coded bitstream or codeword is sometimes called parsing.

[0070] Video coding standards can specify bitstream syntax and semantics, as well as a decoding process for error-free bitstreams, while encoding processes may not be specified, although an encoder may simply be required to produce a compliant bitstream. Bitstream and decoder conformance can be verified with a Hypothetical Reference Decoder (HRD). Standards may include coding tools to help address transmission errors and losses, but using tools for encoding may be optional, and a decoding process for erroneous bitstreams may not be specified.

[0071] The basic units for the input to the encoder and the output to the decoder are, in most cases, pictures. The picture given as input to the encoder is sometimes called the source picture, and the picture decoded by the decoder is sometimes called the decoded picture or reconstructed picture.

[0072] The source picture and the decoded picture each consist of one or more sample arrays, for example, one of the following sets of sample arrays: - Luma (Y) only (single color) - Luma and two chromosomes (YCbCr or YCgCo). - Green, blue, and red (GBR, also known as RGB). - An array representing other unspecified monochromatic or tristimulus color sampling (e.g., also known as YZX, XYZ).

[0073] In the following, these arrays may be referred to as lumens (or L or Y) and chromens, regardless of the actual color representation method used, and the two chroma arrays may be referred to as Cb and Cr. The actual color representation method used may be indicated, for example, in an encoded bitstream using video usability information (VUI) syntax such as HEVC. Components may be defined as an array or a single sample from one of the three sample arrays (lumens and two chromens), or as an array or a single sample of an array that constitutes a picture in a monochromatic format.

[0074] A picture can be defined as either a frame or a field. A frame contains a matrix of luma samples and, optionally, corresponding chroma samples. A field is a set of alternating sample rows of a frame when the source signal is interlaced, and can be used as an encoder input. The chroma sample array may be optional (and therefore monochromatic sampling may be used), or the chroma sample array may be subsampled when compared to the luma sample array.

[0075] The chroma format can be summarized as follows: - In monochromatic sampling, only one sample array exists, which can be nominally considered a luma array. - In 4:2:0 sampling, each of the two chroma arrays has half the height and half the width of the chroma array. - In 4:2:2 sampling, each of the two chroma arrays has the same height as the luma array and half the width of the luma array. - In 4:4:4 sampling, when no separate color planes are used, each of the two chroma arrays has the same height and width as the chroma array.

[0076] A coding format or standard can encode a sample array into a bitstream as separate color planes, and allow each of these separately coded color planes to be decoded from the bitstream. When separate color planes are used, each of them is processed separately as a monochromatic sampled picture (by an encoder and / or decoder).

[0077] When chroma subsampling (e.g., 4:2:0 or 4:2:2 chroma sampling) is used, the location of the chroma sample relative to the luma sample may be determined on the encoder side (e.g., as a preprocessing step or as part of the encoding). The chroma sample location relative to the luma sample location may be predefined in a coding standard such as H.264 / AVC or HEVC, or may be indicated in the bitstream as part of the VUI of H.264 / AVC or HEVC.

[0078] Generally, a source video sequence provided as input for encoding can represent either interlaced or progressive source content. In interlaced source content, fields with opposite parity are captured at different times. Progressive source content consists of captured frames. Encoders can encode fields in interlaced source content in two ways: pairs of interlaced fields may be encoded within a coded frame, or fields may be encoded as coded fields. Similarly, encoders can encode frames in progressive source content in two ways: frames in progressive source content may be encoded into coded frames, or into pairs of coded fields. A field pair or complementary field pair can be defined as two fields adjacent to each other in decoding and / or output order, having opposite parity (i.e., one being the top field and the other the bottom field), and neither belonging to any other complementary field pair. Some video coding standards or schemes allow for the mixing of coded frames and coded fields within the same coded video sequence. Furthermore, predicting coded fields from fields within coded frames, and / or predicting coded frames for complementary field pairs (coded as fields), may be possible in encoding and / or decoding.

[0079] Partitioning can be defined as dividing a set into subsets such that each element of the set is exactly in one of the subsets.

[0080] In H.264 / AVC, a macroblock is a 16x16 block of chroma samples and a corresponding block of chroma samples. For example, in a 4:2:0 sampling pattern, a macroblock contains one 8x8 block of chroma samples for each chroma component. In H.264 / AVC, a picture is partitioned into one or more slice groups, and a slice group contains one or more slices. In H.264 / AVC, a slice consists of an integer number of macroblocks that are sequentially ordered in the raster scan within a given slice group.

[0081] When describing the operation of HEVC encoding and / or decoding, the following terms may be used: A coding block can be defined as an N×N block of samples for a given value N, such that the division of a coding tree block into coding blocks constitutes partitioning. A coding tree block (CTB) can be defined as an N×N block of samples for a given value N, such that the division of components into coding tree blocks constitutes partitioning. A coding tree unit (CTU) can be defined as a coding tree block of a lumina sample, two corresponding coding tree blocks of a chroma sample in a picture with three sample arrays, or a coding tree block of a sample in a monochromatic picture or a picture coded using three distinct color planes and syntax structures used to encode the sample. A coding unit (CU) can be defined as a coding block of a lumina sample, two corresponding coding blocks of a chroma sample in a picture with three sample arrays, or a coding block of a sample in a monochromatic picture or a picture coded using three distinct color planes and syntax structures used to encode the sample. The largest allowable CU may be named LCU (Largest Coding Unit) or Coding Tree Unit (CTU), and video pictures are divided into non-overlapping LCUs.

[0082] In some video codecs, such as the High Efficiency Video Coding (HEVC) codec, a video picture can be divided into coding units (CUs) that cover areas of the picture. A CU consists of one or more prediction units (PUs) that define the prediction process for samples within the CU, and one or more transformation units (TUs) that define the prediction error coding process for samples within the CU. A CU can consist of square blocks of samples with sizes selectable from a predefined set of possible CU sizes. The largest allowable CU size may be named LCU (Largest Coding Unit) or Coding Tree Unit (CTU), and the video picture is divided into non-overlapping LCUs. An LCU can be further split into smaller combinations of CUs, for example, by recursively splitting the LCU and the resulting CUs. Each resulting CU can have at least one PU and at least one associated TU. Each PU and TU can be further split into smaller PUs and TUs, respectively, to increase the granularity of the prediction and prediction error coding processes. Each PU has associated prediction information that defines what kind of prediction should be applied to the pixels within that PU (for example, motion vector information for interprediction PUs and intraprediction directionality information for intraprediction PUs).

[0083] Each TU may be associated with information describing the prediction error decoding process for the samples within the TU (e.g., including DCT coefficient information). Whether or not prediction error coding is applied to each CU is generally signaled at the CU level. If there is no prediction error residual associated with a CU, it can be assumed that the CU does not have a TU. The division of an image into CUs and the division of CUs into PUs and TUs are generally signaled in the bitstream so that the decoder can reconstruct the intended structure of these units.

[0084] In HEVC, a picture can be partitioned into tiles, which are rectangular and contain an integer number of LCUs. In HEVC, partitioning into tiles forms a regular grid, and the height and width of the tiles differ from each other by a maximum of 1 LCU. In HEVC, a slice is defined as an integer number of coding tree units contained in one independent slice segment and all subsequent dependent slice segments (if any) that precede the next independent slice segment (if any) within the same access unit. In HEVC, a slice segment is defined as an integer number of coding tree units that are sequentially ordered in a tile scan and contained in a single NAL unit. The division of each picture into slice segments is partitioning. In HEVC, an independent slice segment is defined as a slice segment in which the values ​​of the syntax elements in the slice segment header are not inferred from the values ​​of the preceding slice segments, and a dependent slice segment is defined as a slice segment in which the values ​​of some syntax elements in the slice segment header are inferred from the values ​​of the preceding independent slice segments in the decoding order. In HEVC, a slice header is defined as the slice segment header of an independent slice segment that is the current slice segment, or the slice segment header of an independent slice segment preceding the current dependent slice segment. A slice segment header is defined as part of a coded slice segment that contains data elements related to the first or all coding tree units represented within the slice segment. CUs are scanned in the raster scan order of LCUs within a tile, or within a picture if no tiles are used. Within an LCU, CUs have a specific scan order.

[0085] The decoder reconstructs the output video by applying prediction means similar to those of the encoder to form a predictive representation of pixel blocks (using motion or spatial information created by the encoder and stored in a compressed representation), and prediction error decoding (the inverse operation of prediction error decoding, which recovers the quantized prediction error signal in the spatial pixel domain). After applying the prediction and prediction error decoding means, the decoder sums the prediction and prediction error signals (pixel values) to form an output video frame. The decoder (and encoder) can further apply additional filtering means to improve the quality of the output video, then send it for display, and / or store it as a prediction reference for upcoming frames in the video sequence.

[0086] Filtering can include, for example, one or more of the following: deblocking, sample adaptive offset (SAO), and / or adaptive loop filtering (ALF). H.264 / AVC includes deblocking, while HEVC includes both deblocking and SAO.

[0087] Motion information can be represented by motion vectors associated with each motion-compensated image block, such as a prediction unit. Each of these motion vectors represents the displacement between the image block of the picture to be coded (on the encoder side) or decoded (on the decoder side) and one of the previously coded or decoded prediction source blocks. To efficiently represent motion vectors, they are generally coded differentially with respect to block-specific prediction motion vectors. In common video codecs, prediction motion vectors are created in a predefined way, for example, by calculating the median of the encoded or decoded motion vectors of adjacent blocks. Another way to create motion vector predictions is to generate a list of candidate predictions from adjacent blocks and / or blocks located in the same place within a time reference picture, and signal the selected candidates as motion vector predictors. In addition to predicting motion vector values, it may be predicted which reference picture will be used for motion-compensated prediction, and this prediction information may be represented, for example, by a reference index of previously coded / decoded pictures. Reference indices are generally predicted from adjacent blocks and / or blocks located in the same location within the time-referenced picture. Furthermore, common high-efficiency video codecs often utilize an additional motion information coding / decoding mechanism called merging mode, where all motion field information, including motion vectors and corresponding reference picture indices, is predicted for each available list of reference pictures and used without any modification / correction. Similarly, the prediction of motion field information is performed using motion field information from adjacent blocks and / or blocks located in the same location within the time-referenced picture, and the motion field information used is signaled in a list of motion field candidate lists filled with motion field information from available adjacent / same-location blocks.

[0088] In typical video codecs, the predicted residuals after motion compensation are first transformed by a transformation kernel (such as DCT) and then coded. This is because there is often still some correlation between the residuals, and the transformation can often help reduce this correlation and provide more efficient coding.

[0089] Video coding standards and specifications can allow encoders to divide coded pictures into coded slices, etc. In-picture prediction may be disabled when crossing slice boundaries. Therefore, a slice can be considered a way of splitting a coded picture into independently decodeable pieces. In H.264 / AVC and HEVC, in-picture prediction may be disabled when crossing slice boundaries. Therefore, a slice can be considered a way of splitting a coded picture into independently decodeable pieces, and thus, a slice can often be considered a basic unit of transmission. Often, the encoder can indicate in the bitstream which types of in-picture prediction have been stopped across slice boundaries, and the decoder operation takes this information into account when concluding, for example, which prediction sources are available. For example, if adjacent CUs are in different slices, samples from adjacent CUs may be considered unavailable for intra-prediction.

[0090] The basic units of output for H.264 / AVC or HEVC encoders and input for H.264 / AVC or HEVC decoders are Network Abstraction Layer (NAL) units. In packet-oriented network transport or storage to structured files, NAL units can be encapsulated in packets or similar structures. The byte-stream format is specified in H.264 / AVC and HEVC for transmission or storage environments that do not provide framing structures. The byte-stream format separates NAL units from each other by prefixing each NAL unit with a start code. To avoid false detection of NAL unit boundaries, encoders implement a byte-oriented start code emulation prevention algorithm, which adds an emulation prevention byte to the NAL unit payload if the start code occurs differently. Start code emulation prevention may always be performed, regardless of whether the byte-stream format is used, to enable direct gateway operation between packet-oriented and stream-oriented systems. A NAL unit can be defined as a syntax structure containing a byte that includes an indication of the type of data that follows, and a byte containing that data in the form of an RBSP, interspersed with an emulation prevention byte if necessary. A raw byte sequence payload (RBSP) can be defined as a syntax structure containing an integer number of bytes encapsulated in a NAL unit. An RBSP has the form of a string of data bits that are either empty or contain syntax elements, followed by an RBSP stop bit, and then zero or more subsequent bits equal to 0.

[0091] A NAL unit consists of a header and a payload. In H.264 / AVC and HEVC, the NAL unit header indicates the type of NAL unit.

[0092] In HEVC, a 2-byte NAL unit header is used for all specified NAL unit types. The NAL unit header contains one reserved bit, a 6-bit NAL unit type indicator, a 3-bit nuh_temporal_id_plus1 indicator at the time level (which may need to be 1 or greater), and a 6-bit nuh_layer_id syntax element. The temporal_id_plus1 syntax element can be considered the time identifier of the NAL unit, and the zero-based TemporalId variable can be derived as follows: TemporalId = temporal_id_plus1 - 1. The abbreviation TID can be used interchangeably with the TemporalId variable. A TemporalId equal to 0 corresponds to the lowest time level. The value of temporal_id_plus1 is required to be non-zero to avoid start code emulation that includes two NAL unit header bytes. A bitstream created by excluding all VCL NAL units with a TemporalId greater than or equal to a selected value and including all other VCL NAL units remains conforming. As a result, a picture with a TemporalId equal to tid_value will not use a picture with a TemporalId greater than tid_value as an interprediction reference. A sublayer or time sublayer can be defined as a time-scalable layer (or time layer, TL) of a time-scalable bitstream consisting of VCL NAL units with a specific value of the TemporalId variable and associated non-VCL NAL units. nuh_layer_id can be understood as a scalability layer identifier.

[0093] NAL units can be classified into Video Coding Layer (VCL) NAL units and non-VCL NAL units. VCL NAL units are generally coded slice NAL units. In HEVC, a VCL NAL unit contains syntax elements that represent one or more CUs.

[0094] Non-VCL NAL units can be, for example, one of the following types: sequence parameter sets, picture parameter sets, supplemental enhancement information (SEI) NAL units, access unit delimiters, end-of-sequence NAL units, end-of-bitstream NAL units, or filler data NAL units. Parameter sets may be required for the reconstruction of decoded pictures, while many other non-VCL NAL units are not required for the reconstruction of decoded sample values.

[0095] Parameters that remain immutable throughout the coded video sequence may be included in the sequence parameter set. In addition to parameters that may be required by the decoding process, the sequence parameter set may optionally include video usability information (VUI), which includes parameters that may be important for buffering, picture output timing, rendering, and resource reservation. In HEVC, the sequence parameter set RBSP includes parameters that may be referenced by one or more picture parameter sets RBSP or one or more SEI NAL units that include buffering period SEI messages. The picture parameter set includes parameters that are likely to be immutable across several coded pictures. The picture parameter set RBSP may include parameters that may be referenced by coded slice NAL units of one or more coded pictures.

[0096] In HEVC, a Video Parameter Set (VPS) can be defined as a syntax structure containing syntax elements that apply to zero or more coded video sequences, determined by the content of syntax elements found in the SPS, and the SPS is referenced by syntax elements found in the PPS, which in turn are referenced by syntax elements found in the slice segment header.

[0097] A video parameter set (RBSP) may contain parameters that can be referenced by one or more sequence parameter sets (RBSP).

[0098] The relationships and hierarchy between video parameter sets (VPS), sequence parameter sets (SPS), and picture parameter sets (PPS) can be described as follows: VPS resides one level above SPS in the parameter set hierarchy and in the context of scalability and / or 3D video. VPS can contain parameters common to all slices across all (scalability or view) layers in the entire coded video sequence. SPS contains parameters common to all slices of a particular (scalability or view) layer in the entire coded video sequence and may be shared by multiple (scalability or view) layers. PPS contains parameters common to all slices of a particular layer representation (a representation of a scalability or view layer in one access unit) and is likely to be shared by all slices of multiple layer representations.

[0099] A VPS can provide information about layer dependencies in a bitstream, as well as much other information applicable to all slices across all (scalability or view) layers in the entire coded video sequence. A VPS can be thought of as consisting of two parts: a base VPS and a VPS extension, with the VPS extension being optional.

[0100] Out-of-band transmission, signaling, or storage may be used, as an addition or alternative, for purposes other than tolerance to transmission errors, such as facilitating access or session negotiation. For example, sample entries for tracks in a file conforming to an ISO-based media file format may include parameter sets, while coded data for a bitstream is stored elsewhere in the file or in a separate file. The phrase "along the bitstream" (e.g., indicating "along the bitstream") or "along the coded unit of a bitstream" (e.g., indicating "along the coded tile") may be used in the claims and embodiments to refer to out-of-band transmission, signaling, or storage such that out-of-band data is associated with a bitstream or coded unit, respectively. Phrases such as "decoding along the bitstream" or "decoding along the coded unit of a bitstream" may refer to decoding of referenced out-of-band data (which may be obtained from out-of-band transmission, signaling, or storage) associated with a bitstream or coded unit, respectively.

[0101] The SEI NAL unit can contain one or more SEI messages, which are not required for decoding the output picture but can assist in related processes such as picture output timing, rendering, error detection, error hiding, and resource reservation.

[0102] A coded picture is a coded representation of a picture.

[0103] In HEVC, a coded picture can be defined as a coded representation of a picture that includes all of the picture's coding tree units. In HEVC, an access unit (AU) can be defined as a set of NAL units that are related to each other according to a specified classification rule, are consecutive in decoding order, and contain at most one picture with a particular value of nuh_layer_id. In addition to containing the VCL NAL units of a coded picture, an access unit can also contain non-VCL NAL units. The specified classification rule may, for example, associate pictures with the same output time or picture output count value with the same access unit.

[0104] A bitstream can be defined as a sequence of bits in the form of a NAL unit stream or byte stream that forms a representation of coded pictures and associated data that make up one or more coded video sequences. A first bitstream may be followed by a second bitstream within the same logical channel, such as within the same file or within the same connection of a communication protocol. An elementary stream (in the context of video coding) can be defined as a sequence of one or more bitstreams. The end of a first bitstream may be indicated by a specific NAL unit, which may be called the Bitstream End (EOB) NAL unit and is the last NAL unit of the bitstream. HEVC and its current draft extensions require that the EOB NAL unit have a nuh_layer_id equal to 0.

[0105] In H.264 / AVC, an encoded video sequence is defined as a sequence of consecutive access units in decoding order, from one IDR access unit (including itself) to the next IDR access unit (not including itself), or to the end of the bitstream, whichever comes first.

[0106] A coded video sequence (CVS) can be defined, for example, as a sequence of coded pictures in a decoding order, which are independently decodeable and followed by the end of another coded video sequence or bitstream.

[0107] In HEVC, an encoded video sequence may be specified, as an addition or alternative (to the specification above), to terminate when a certain NAL unit, sometimes called an End of Sequence (EOS) NAL unit, appears in the bitstream and has a nuh_layer_id equal to 0.

[0108] Picture groups (GOPs) and their characteristics can be defined as follows: A GOP can be decoded regardless of whether previous pictures have been decoded. An open GOP is a group of pictures in which, if decoding starts with the first intra-picture of the open GOP, pictures prior to the first intra-picture in output order may not be decoded correctly. In other words, pictures in an open GOP may refer to pictures belonging to a previous GOP (in interpretation). The HEVC decoder can recognize the intra-picture that starts an open GOP because certain NAL unit types, CRA NAL unit types, may be used for the coded slice. A closed GOP is a group of pictures in which, if decoding starts with the first intra-picture of the closed GOP, all pictures can be decoded correctly. In other words, pictures in a closed GOP do not refer to pictures in a previous GOP. In H.264 / AVC and HEVC, a closed GOP can start with an IDR picture. In HEVC, closed GOPs can also start with BLA_W_RADL or BLA_N_LP pictures. Open GOP coding structures are potentially more efficient in compression compared to closed GOP coding structures due to greater flexibility in the selection of reference pictures.

[0109] A Decoded Picture Buffer (DPB) may be used in encoders and / or decoders. There are two reasons for buffering the decoded picture: for reference in interpretation and for sorting the decoded picture in output order. Since H.264 / AVC and HEVC offer considerable flexibility in both reference picture marking and output sorting, separate buffers for reference picture buffering and output picture buffering can waste memory resources. Therefore, the DPB can include a unified decoded picture buffering process for both reference picture and output sorting. A decoded picture may be removed from the DPB when it is no longer used as a reference and is not required for output.

[0110] In many coding modes of H.264 / AVC and HEVC, reference pictures for interprediction are indicated by an index to a reference picture list. The index can be coded with variable-length coding, so that typically, a smaller index corresponds to a shorter value for the corresponding syntax element. In H.264 / AVC and HEVC, two reference picture lists (reference picture list 0 and reference picture list 1) are generated for each bi-predictive (B) slice, and one reference picture list (reference picture list 0) is formed for each intercoding (P) slice.

[0111] Many coding standards, including H.264 / AVC and HEVC, may have a decoding process for deriving a reference picture index to a reference picture list, which can be used to indicate which of a number of reference pictures is used for interpretation of a particular block. The reference picture index may be encoded into a bitstream by the encoder in some intercoding modes, or it may be derived using adjacent blocks (by the encoder and decoder) in some other intercoding modes, for example.

[0112] Motion parameter types or motion information may include, but are not limited to, one or more of the following types: - Prediction type (e.g., intra-prediction, uni-prediction, bi-prediction) and / or indication of several reference pictures, - Indications of prediction directions such as inter (also known as time) prediction, interlayer prediction, interview prediction, view composite prediction (VSP), and intercomponent prediction (these may be shown per reference picture and / or per prediction type, and depending on the embodiment, interview prediction and view composite prediction may be considered together as one prediction direction), and / or - Indications for reference picture types such as short-term reference pictures and / or long-term reference pictures and / or interlayer reference pictures (for example, which may be displayed for each reference picture), - A reference index to the reference picture list and / or other identifiers of the reference picture (for example, one for each reference picture, whose type may depend on the predicted direction and / or the reference picture type, and which may be accompanied by other relevant information such as the reference picture list to which the reference index applies); - Horizontal motion vector components (which may be shown, for example, per prediction block or per reference index); - Vertical motion vector components (which may be shown, for example, per prediction block or per reference index); - One or more parameters, such as the picture order count difference and / or relative camera separation between a picture containing or related to motion parameters and its reference picture, which may be used for scaling the horizontal motion vector components and / or vertical motion vector components in one or more motion vector prediction processes (where the one or more parameters may be indicated, for example, for each reference picture or for each reference index); - The coordinates of the block to which the motion parameters and / or motion information are applied, for example, the coordinates of the top-left sample of the block in the Luma sample unit; - The extent of the block to which motion parameters and / or motion information apply (e.g., width and height).

[0113] Compared to previous video coding standards, the Versatile Video Codec (H.266 / VVC) introduces several new coding tools, including the following: Intra prediction - 67 intra modes with wide-angle mode expansion - Block size and mode-dependent 4-tap interpolation filter - Location-dependent intra-predictive combination (PDPC) - Cross-component linear model intraprediction (CCLM) - Multi-reference line intra-prediction - Intra-subpartition - Weighted intra prediction using matrix multiplication Interpict prediction - Block motion copy with spatial, temporal, history-based, and pairwise average merge candidates - Affine motion interface prediction - Subblock-based temporal motion vector prediction - Adaptive motion vector resolution - 8x8 block-based motion compression for time-based motion prediction - High-precision (1 / 16 Pel) motion vector storage and motion compensation using an 8-tap interpolation filter for the luma component and a 4-tap interpolation filter for the chroma component. - Triangular partition - Combination of intra-prediction and inter-prediction - Merge with MVD (MMVD) - Symmetric MVD coding - Bidirectional optical flow - Decoder-side motion vector refinement - Biprediction with CU-level weights • Transformation, quantization, and coefficient coding - Numerous primary transformation options with DCT2, DST7, and DCT8 - Secondary conversion in the low-frequency zone - Subblock transformation of interpredicted residuals - Dependent quantization increased the maximum QP from 51 to 63. - Coding of conversion coefficients by obscuring coded data - Conversion skip residual coding Entropy coding - Arithmetic coding engine with adaptive double-window probability updates • In-loop filter - Inloop reshaping - Deblocking filter with a stronger, longer filter - Sample-adaptive offset - Adaptive loop filter • Screen content coding: - Reference to the current picture due to reference area limitations 360-degree video coding - Horizontal wrap-around motion compensation • High-level syntax and parallel processing - Reference picture management using direct reference picture list signaling - Tile group with rectangular tile group

[0114] Scalable video coding can refer to a coding structure in which a single bitstream can contain multiple representations of content, for example, at different bitrates, resolutions, or frame rates. In these cases, the receiver can extract the desired representation according to its characteristics (e.g., the resolution best suited to the display device). Alternatively, a server or network element can extract a portion of the bitstream to be transmitted to the receiver, for example, according to network characteristics or the receiver's processing power. A meaningful decoded representation can be created by decoding only a specific portion of the scalable bitstream. A scalable bitstream generally consists of a "base layer" that provides the lowest quality video available, and one or more enhancement layers that enhance the video quality when received and decoded together with the lower layers. To improve the coding efficiency of the enhancement layers, the coded representation of those layers generally depends on the lower layers. For example, motion and mode information of an enhancement layer can be predicted from the lower layers. Similarly, pixel data from the lower layers can be used to create predictions for the enhancement layer.

[0115] In some scalable video coding schemes, a video signal can be encoded into a base layer and one or more enhancement layers. The enhancement layers can enhance, for example, the quality of the video content represented by another layer or part thereof, such as temporal resolution (i.e., frame rate), spatial resolution, or simply another layer or part thereof. Each layer, together with all its dependent layers, is a single representation of the video signal at a particular spatial resolution, temporal resolution, and quality level. In this document, a scalable layer, together with all its dependent layers, is referred to as a “scalable layer representation.” The portion of the scalable bitstream corresponding to the scalable layer representation can be extracted and decoded to produce a representation of the original signal at a particular fidelity.

[0116] A scalability mode or scalability dimension can include, but is not limited to, the following: - Quality scalability: The base layer picture is coded with lower quality than the enhancement layer picture, which can be achieved, for example, by using larger quantization parameter values ​​in the base layer than in the enhancement layer (i.e., a larger quantization step size with respect to conversion coefficient quantization). Quality scalability can be further classified into fine-grained or fine-grained scalability (FGS), medium-grained or medium-grained scalability (MGS), and / or coarse-grained or coarse-grained scalability (CGS), as described below. - Spatial scalability: The base layer picture is encoded at a lower resolution (i.e., has fewer samples) than the enhancement layer picture. Spatial scalability and quality scalability, particularly its coarse-grained scalability type, can sometimes be considered the same type of scalability. - Bit depth scalability: The base layer picture is coded with a lower bit depth (e.g., 8 bits) than the enhancement layer picture (e.g., 10 or 12 bits). - Dynamic Range Scalability: Scalable layers represent images obtained using different dynamic ranges and / or different tone mapping functions and / or different optical transfer functions. - Chroma format scalability: The base layer picture provides lower spatial resolution in the chroma sample array (e.g., encoded in a 4:2:0 chroma format) than the enhancement layer picture (e.g., in a 4:4:4 format). - Color Gamut Scalability: The enhancement layer picture has a richer / wider color range than the base layer picture. For example, the enhancement layer may have the UHDTV (ITU-R BT.2020) color gamut, while the base layer may have the ITU-R BT.709 color gamut. - View scalability, sometimes called multi-view coding. The base layer represents the first view, while the enhancement layer represents the second view. A view can be defined as a sequence of pictures representing one camera or viewpoint. In stereoscopic or two-view video, one video sequence or view can be thought of as being presented to the left eye, while the parallel view is presented to the right eye. - Depth scalability, sometimes called depth-enhanced coding. One or more layers of the bitstream can represent a texture view, while one or more other layers can represent a depth view. - Scalability of areas of interest (as described below). - Interlace-to-progressive scalability (also known as field-to-frame scalability): Coded interlaced source content material in the base layer is enhanced by enhancement layers to represent progressive source content. Coded interlaced source content in the base layer can include coded fields, coded frames representing field pairs, or mixtures thereof. In interlace-to-progressive scalability, the base layer picture can be resampled so that it becomes a suitable reference picture for one or more enhancement layer pictures. - Hybrid codex scalability (also known as coding standard scalability): In hybrid codex scalability, the bitstream syntax, semantics, and decoding processes of the base layer and enhancement layer are defined by different video coding standards. Therefore, the base layer picture is coded according to a different coding standard or format than the enhancement layer picture. For example, the base layer may be coded using H.264 / AVC, and the enhancement layer may be coded using the HEVC multilayer extension.

[0117] For example, in the case of the scalability of the spatial format, bit depth format, and chroma format mentioned above, base layer information can be used to encode the enhancement layer to minimize additional bitrate overhead.

[0118] Scalability can be achieved in at least two basic ways: by introducing new coding modes to predict pixel values ​​or syntax from lower layers of a scalable representation, or by placing lower-layer pictures in a reference picture buffer (decoded picture buffer, DPB) of the upper layers. The first method can be more flexible and, in most cases, can provide better coding efficiency. However, the second reference frame-based scalability method can be implemented very efficiently with minimal changes to a single-layer codec while still achieving most of the available coding efficiency gains. According to one method, a reference frame-based scalability codec is implemented by utilizing the same hardware or software embodiment at all layers, with only DPB management being handled by external means.

[0119] To enable parallel processing, an image can be split into independently codeable and decodeable image segments (slices or tiles). A slice generally refers to an image segment consisting of a certain number of basic coding units that are processed in a default coding or decoding order, while a tile generally refers to an image segment defined as a rectangular image region that is processed at least to some extent as individual frames.

[0120] Video can be encoded in YUV or YCbCr color space because it has been found to reflect some characteristics of the human visual system, and because human perception is not as sensitive to the chrominance fidelity represented by the Cb and Cr channels, it allows for the use of lower-quality representations for the Cb and Cr channels.

[0121] Cross-component linear model prediction is used, for example, in the VVC / H.266 video codec. In its variations, there are three chroma prediction modes that use cross-component linear model prediction. One of these can be selected by the encoder as the prediction mode for a chroma prediction block and signaled to the decoder in the bitstream. The difference between the three modes lies in the set of reference samples used to generate the parameters of the linear model. One mode uses only the samples above the prediction block; one mode uses only the samples to the left of the prediction block; and one mode uses both the samples above and to the left of the prediction block. To keep the complexity of parameter generation low, the parameters are computed using only a subset of the reference samples available at the block boundaries.

[0122] The cross-component linear model prediction for VVC / H.266 applies the following formula to predict (or map) the luma sample value lumaVal to the predicted chroma sample value chromaVal: chromaVal=((lumaVal*a)>>k)+b Here, parameters a and k determine the gradient of the linear model, and b determines the offset value of the linear model. The notation ">>" is used to represent a right bit shift operation corresponding to division by a power of 2. Parameters a, k, and b are determined using a determined set of available reference samples.

[0123] An example of using linear regression to compute linear model parameters is given in JVET-D0110, a contribution to JVET (Joint Video Expert Team). In that embodiment, there may be two linear models operating in different ranges of the luma spectrum.

[0124] Some exemplary embodiments of this disclosure are described in more detail below.

[0125] First, some of the encoder's operations are described. The encoder receives the luminance and chrominance components of the pixels of the image to be encoded. The image is divided into smaller blocks, and the luminance and chrominance components of a single image can be processed on a block-by-block basis. Information from previously encoded and then decoded blocks or parts thereof of the image is stored by the encoder in reference frame memory (RFM) and can be used, for example, in predicting subsequent images.

[0126] A linear model for mapping luma values ​​to chroma values ​​can be used to generate predicted chroma values ​​based on the decoded luma values. This type of model can be given using a gradient parameter "a" and an offset parameter "b" as follows: chromaVal = lumaVal * a + b

[0127] The encoder can encode a gradient parameter "a" and an offset parameter "b" in the bitstream, and the decoder can obtain these parameters from the bitstream, or these parameters may be known to the decoder beforehand, and only changes to these parameters may be signaled to the decoder.

[0128] Figure 5a shows a mapping according to one embodiment of the present disclosure. Each effective luma sample value can be mapped to a chroma sample value using the model. In this example, two points or two luma-chroma pairs p0 and p1, having luma values ​​y0 and y1 and chroma values ​​c0 and c1, define the gradient parameter a and offset parameter b of the mapping function. In actual embodiments using integer arithmetic, the expression may further include a scaling parameter "k" that defines the basis or precision of the gradient parameter a: chromaVal=((lumaVal*a)>>k)+b

[0129] The scaling parameter k may be selected, for example, by an encoder during the process of generating the model parameters, or the scaling parameter k may also be presented to the decoder in a different way, or a fixed k may be used for all mappings performed by the encoder. Parameter k determines how many bits to shift down the result of the multiplication between lumaVal and the gradient parameter a using a bitwise right shift operation >> in order to reach the luma-chroma value space. In other words, the result is 2 k It is divided by .

[0130] In one embodiment, the update term "u" of the gradient parameter a is defined by the encoder.

[0131] The update term u can have a basis or precision "s" that is different from the basis or precision k determined for the gradient parameter a. For example, the update term u shifts down the received update term u by 3 bits (i.e., it is 2 3 It can have a basis of 3 corresponding to dividing by 8. As another example, the update term u can shift down the received update term u by 4 bits (i.e., it becomes 2 4It can have a basis of 4 corresponding to (dividing by 16). This basis may be fixed, signaled in the bitstream, or adaptively determined by the decoder depending on, for example, the size of the block to be predicted or processed. For example, the basis s may be larger in response to finer-grained update terms when the predicted block size is above a threshold, and smaller when the predicted block size is below a threshold.

[0132] For the update term u to be added to the gradient parameter a, the bases of these parameters must be equal. This can be achieved by shifting u bitwise up until its base matches the base of k, if k is greater than s, or by shifting a until its base matches the base of k, if k is less than s.

[0133] In this example, the updated linear model with parameters a', k', and b' is: chromaVal=((lumaVal*a')>>k')+b' It can be written as follows.

[0134] The updated linear model parameters can be calculated, for example, using the following pseudocode: a,b,k,refLuma=estimateModel(set of reference samples) u = decodeUpdateTerm(bitstream) k'=k if(k <s) { / / The final shift should be at least the size of the update precision. a = a << (sk) k' = s } else if (k > s) { / / Final shift is greater than the update precision: Scale the update up to the final precision. u = u << (ks) } a’ = a + u b’ = b - ((u * yr) >> k’)

[0135] The original linear model parameters a, b, and k can be calculated, for example, by the estimateModel function according to the process determined in the H.266 / VVC specification, or by an alternative method such as the use of linear regression. In addition to the original parameters a, b, and k, an additional reference luma parameter y r is determined and used as a reference value when calculating the updated offset parameter b’. y r The parameter can be determined in various ways. For example, if two luma-chroma pairs are used to calculate the linear model, y r can be set to the average of the luma values of those pairs. If four luma-chroma pairs are used to calculate the linear model, y r can be set, for example, to the average of the largest and smallest luma values of those pairs; or y r can be set to the average of the second-largest and third-largest luma values of those pairs; or y r can be set to the average of the luma values of those four pairs. As a further example, y r can be calculated as the average of the luma values within a determined set of reference luma-chroma pairs or luma reference values, the median of those values, or the average of the maximum and minimum of those values, or in other ways.

[0136] y r The way y is calculated may also be determined based on bitstream signaling. For example, whether the average of the determined luma reference samples is used as y r or whether y r is calculated as a weighted average of a set of reference luma values, and what those weights are, may be signaled. In addition to or instead of using conventional reference values obtained outside the block boundary, y rParameter determination may include reconfigured luma values ​​within a prediction block or prediction unit.

[0137] Figure 5b shows the updated mapping. The updated term u is the luma value y as the control point on which the mapping is rotated. r Reference point p r This is applied to the gradient of the mapping. The updated gradient parameter a' and updated offset parameter b' define the new mapping here. The scale parameters k and s are omitted for simplicity in the diagram.

[0138] The update term u can be signaled as two distinct components: magnitude and sign. The magnitude indicates the absolute value u of the update term u, i.e., |u|, and the sign indicates whether the actual value is positive, i.e., equal to the absolute value |u|, or negative, i.e., equal to -1*|u|. The magnitude can be binary-encoded and coded using arithmetic coding, or signaled as a variable-length or fixed-length codeword. The sign is coded as an arithmetic-encoded binary codeword, where either symbol 0 or symbol 1 is chosen to indicate that the update term is refining the original or reference value "a" toward zero; the other symbol is chosen to indicate that the reference value is being refined toward zero. Two alternative choices are summarized in Tables 1 and 2 below.

[0139] The expression "towards zero" means that the actual change in the reference value is such that the magnitude (absolute value) of the reference value decreases when the update term is added to it. Correspondingly, the expression "away from zero" means that the update term can be used in such a way that the actual change in the reference value is such that the magnitude (absolute value) of the reference value increases when the update term is added to it.

[0140] In Table 1, the sign of the update term u is indicated according to the sign and the syntax element representing the reference value a, where syntax element 0 represents the sign that refines the reference value toward zero, and syntax element 1 represents the sign that refines the reference value toward zero.

[0141] [Table 1]

[0142] For example, if the encoder determines that the reference value a is +4 and the update term is -2, the encoder can set the syntax element representing the sign to 0 and encode the size of the update term (2) and the syntax element 0 to indicate to the decoder that the update term is negative. As another example, if the encoder determines that the reference value a is -3 and the update term is -2, the encoder can set the syntax element representing the sign to 1 and encode the size of the update term (2) and the syntax element 1 to indicate to the decoder that the update term is negative.

[0143] In Table 2, the sign of the update term u is indicated according to the syntax element representing the sign and the reference value a, where syntax element 1 represents the sign that refines the reference value toward zero, and syntax element 0 represents the sign that refines the reference value toward zero.

[0144] [Table 2]

[0145] For example, if the encoder determines that the reference value a is +4 and the update term is -2, the encoder can set the syntax element representing the sign to 1. As another example, if the encoder determines that the reference value a is -3 and the update term is -2, the encoder can set the syntax element representing the sign to 0.

[0146] While zero is used as an example of a threshold, it should be noted that the same principle can be applied to non-zero thresholds as well. For example, the threshold could be 2, and the syntax element would indicate whether the reference value should be changed toward 2 or toward 2. Thus, in the leftmost column, zero would be replaced with a non-zero threshold, for example, 2 in this example. Still, the syntax element representing the sign along with the reference value (before adding the update term) would clarify whether the sign of the update term is positive or negative.

[0147] A syntax element value of 0 is sometimes called the first display value, and a syntax element value of 1 is sometimes called the second display value. Note that the first display value does not have to be 0, and the first display value does not have to be 1; other values ​​may be used instead.

[0148] Furthermore, the magnitude of the update term u, either independently or in combination with a reference value, can determine different zones, and different methods for coding the sign of the update term may be selected for such zones. For example, if the magnitude of the update term is below a threshold, the sign may be represented by fixed-length coding; otherwise, it may be represented by a sign-inverting technique.

[0149] At the time the syntax element is decoded, the decoder may not know the reference value a. In this case, the initial value of the update term u can be decoded for the update term u by setting the initial value to negative if the decoded syntax element representing the initial sign of the update term is 1, and setting the initial value to positive if the decoded syntax element is 0, or conversely, by setting the initial value to negative if the decoded syntax element representing the initial sign of the update term is 0, and setting the initial value to positive if the decoded syntax element is 1. Then, the final value of the update term u can be assigned once the reference value a is determined. In this case, the final value of the update term u can be determined using the following pseudocode, which inverts the sign of the decoded update term for positive reference value a, or maintains the sign of the decoded update term for negative and zero reference value a: uDec=decodeUpdateTerm(bitstream) a,b,k,refLuma=estimateModel(set of reference samples) u=a>0?-uDec:uDec

[0150] This can also be represented by the following pseudocode: uDec=decodeUpdateTerm(bitstream) a,b,k,refLuma=estimateModel(set of reference samples) if a>0: u = -uDec else: u=uDec

[0151] In other words, the encoded update term received in the bitstream is decoded to obtain uDec(size of u). The values ​​a, b, k, and refLuma are obtained based on the set of reference samples. Then, if the reference value a is greater than zero, the update term is set to the inverted value of the decoded update term. If the reference value a is not greater than zero (i.e., less than or equal to zero), the update term is set to the value of the decoded update term without inverting it.

[0152] In one embodiment, an update term "u" for the gradient parameter a is defined by an encoder, encoded into a bitstream by the encoder, received from the bitstream by a decoder, a reference point consisting of a luma-chroma value pair is determined, and parameters a, k, and b are updated by the decoder based on the update term and the reference point.

[0153] In one embodiment, the update term u of the gradient parameter a is encoded into a bitstream by an encoder, received from the bitstream by a decoder, a reference luma value is determined, and parameters a, k, and b are updated by the decoder based on the update term and the reference luma value.

[0154] In one embodiment, the video or image decoder performs the following steps:

[0155] The sign and magnitude of the initial update term are decoded. The value of the initial update term is determined based on the decoded sign and magnitude of the initial update term. The reference value of the parameter of the luma-to-chroma mapping function is determined. The value of the update term is determined by reversing the sign of the initial update term if the value of the reference parameter is greater than zero, or otherwise the sign is not reversed. The value of the parameter of the luma-to-chroma mapping function is determined using the reference value and the value of the update term.

[0156] In one embodiment, the value of the update term is determined by reversing the sign of the initial update term if the value of the reference parameter is greater than or equal to zero.

[0157] In one embodiment, the value of the update term is determined by leaving the initial update term unchanged if the value of the reference parameter is less than or equal to zero.

[0158] In one embodiment, the value of the update term is determined by leaving the initial update term unchanged if the value of the reference parameter is less than zero.

[0159] In an alternative embodiment, the video or image decoder performs the following steps:

[0160] The reference values ​​for the parameters of the luma-to-chroma mapping function are calculated. The magnitude of the update term for the parameters of the luma-to-chroma mapping function is determined. The binary syntax element to be used to determine the sign of the update term is also determined. The sign of the update term is determined by interpreting the binary syntax element based on the reference value. The values ​​of the parameters of the luma-to-chroma mapping function are determined using the reference value, the decoded magnitude of the update term, and the determined sign of the update term.

[0161] In one embodiment, the sign of an update term is determined to be positive if the reference value is greater than zero and the decoded syntax element representing the sign is 1; or if the reference value is less than or equal to zero and the decoded syntax element representing the sign is 0. Similarly, the sign of an update term is determined to be negative if the reference value is greater than zero and the decoded syntax element representing the sign is 0; or if the reference value is less than or equal to zero and the decoded syntax element representing the sign is 1.

[0162] In one embodiment, the sign of an update term is determined to be negative if the reference value is greater than zero and the decoded syntax element representing the sign is 1; or if the reference value is less than or equal to zero and the decoded syntax element representing the sign is 0. Similarly, the sign of an update term is determined to be positive if the reference value is greater than zero and the decoded syntax element representing the sign is 0; or if the reference value is less than or equal to zero and the decoded syntax element representing the sign is 1.

[0163] In one embodiment, the sign is determined such that the update term refines the reference value toward zero if the decoded binary syntax element representing the sign of the update term is 0, and refines the reference value toward zero if the decoded binary syntax element representing the sign of the update term is 1.

[0164] In one embodiment, the sign is determined such that the update term refines the reference value toward zero if the decoded binary syntax element representing the sign of the update term is 1, and refines the reference value toward zero if the decoded binary syntax element representing the sign of the update term is 0.

[0165] In one embodiment, the video or image encoder performs the following steps:

[0166] The reference values ​​for the parameters of the chroma-to-chroma mapping function are determined. The values ​​of the update terms for the reference values ​​are determined. The displayed values ​​of the update terms are determined such that, if the reference value is greater than zero, the sign of the displayed value is opposite to the sign of the determined value of the update term. The sign and magnitude of the displayed values ​​of the update terms are encoded within the video bitstream.

[0167] In one embodiment, the displayed value of the update term is determined by reversing the sign of the update term if the value of the reference parameter is greater than or equal to zero.

[0168] In one embodiment, the displayed value of the update term is determined by leaving the update term unchanged if the value of the reference parameter is less than or equal to zero.

[0169] In one embodiment, the displayed value of the update term is determined by leaving the update term unchanged if the value of the reference parameter is less than zero.

[0170] In one embodiment, rate-distortion is performed to determine the value of the update term, and the displayed update term is determined based on the value of the update term.

[0171] In one embodiment, rate-distortion is performed to determine the value of the displayed update term, and the update term is determined based on the value of the displayed update term.

[0172] Figure 5c shows the control point p outside the determined initial mapping line. r Select and perform gradient updates with respect to such points. In this case, the offset parameter b should be updated accordingly. For example, p r If b' is a specific amount of chroma value on the mapping line, then b' should also be increased by the same amount. Alternatively, b' is the updated gradient parameter a' and control point p r This can be calculated using the coordinates of the control point p. r The luma value y r and chroma value c r Using this method, it can be done as follows: b'=c r -((a'*y r )>>k')

[0173] Figure 5d shows the control point p outside the determined initial mapping line. r This demonstrates using a method that only updates the offset parameter b without changing the gradient. In this alternative, the gradient parameter is calculated based on two or more Luma-Chroma value pairs or control points. In addition, an additional control point p r However, this is calculated by using different or the same set of reference luma-chroma pairs. For example, the average luma value of a set of reference luma values ​​is calculated for control point p r The luma value y r It can be used as; the average chroma value of the set of reference chroma values, control point p r Chroma value c rIt can be used as follows. The set of reference chroma values ​​can, for example, include all the values ​​of the reconfigured boundary chroma samples immediately above and immediately to the left of a block. The set of reference luma values ​​can be the corresponding luma sample values ​​that can be obtained, for example, by interpolation, if the chroma resolution and luma resolution are different. Boundary chroma samples can include the samples directly above and to the left of a block, but can further include extensions of those arrays. For example, the set of boundary samples above a block with width w can include w samples adjacent to the block, or 2*w samples, which further include w additional samples in the same row as the first w samples immediately above the block. Then the updated offset value b' is used with the reference luma value y r And using the reference chroma value cr, it can be calculated as follows: b'=cr-((a*y r )>>k)

[0174] Advantageously, the set of reference luma and chroma values ​​for calculating the gradient parameter a is used to generate c for generating the offset parameter b'. r and y r The values ​​are chosen to be smaller than the set of reference luma and chroma values ​​used to calculate the parameter. This choice can keep the computational complexity of calculating the first-order gradient parameter a low, and c r and y r A relatively simple calculation of, for example, c r To produce the reference chroma values, average the reference chroma values ​​and y r By averaging the reference luma values ​​to produce c r And by determining cy, it can be performed more accurately.

[0175] The set of reference luma and chroma values ​​used to compute the bias term b' of a linear model can, favorably, be chosen to be a superset of the reference luma and chroma values ​​used to compute the gradient parameter a of the linear model. This reduces the computational complexity associated with determining the linear model, as already generated or fetched reference samples can be used in determining the bias term b', while improving the accuracy and stability of the offset parameter b', as a larger set of reference samples is made available for its computation.

[0176] Figures 6a to 6c illustrate various selections for the reference sample R above block B. Figure 6a shows the selected set of reference samples just above the block. An example in Figure 6b shows the set of reference samples extended to include additional samples to the upper right of the block, and Figure 6c shows the reference sample array extended to the left by just one sample. Of course, other selections may be made to include, for example, 2*w+1 samples in the set. Similar selections may be made to the left of block B, and further combinations of samples or sample arrays from the left and top of the block may be used.

[0177] In one embodiment, the video or image decoder performs the following steps in relation to the flowchart in Figure 7a:

[0178] The decoder decodes the sign and magnitude of the initial update term (702). The decoder determines the value of the initial update term based on the decoded sign and magnitude of the initial update term (704). The decoder determines the reference value of the parameter of the luma-to-chroma mapping function (706), and the decoder determines the value of the update term by inverting the sign of the initial update term if the value of the reference parameter is greater than zero (708). The decoder further determines the value of the parameter of the luma-to-chroma mapping function using the reference value and the value of the update term (710).

[0179] In one embodiment, the video or image decoder performs the following steps in relation to the flowchart in Figure 7b:

[0180] The decoder calculates at least a reference value for the parameter of the luma-to-chroma mapping function (722). The decoder then decodes the magnitude of the update term u for the parameter of the luma-to-chroma mapping function (724). The decoder also decodes the binary syntax element that should be used to determine the sign of the update term (726). The decoder determines the sign of the update term by interpreting the binary syntax element based on the reference value (728). The decoder also uses the reference value, the decoded magnitude of the update term, and the determined sign of the update term to determine the value of the parameter of the luma-to-chroma mapping function (730).

[0181] The calculation of parameters that define the mapping from the value of one color component to the value of another color component can be performed in various ways. For example, linear regression or other statistical methods may be used to minimize the error that the model may create for a set of reference samples. Reference samples may be obtained from the boundaries of the predicted block or using another set of data. There may be many candidate reference sets, from which the encoder selects one and indicates its selection to the decoder in a bitstream. A subset of boundary samples or other sample sets may also be used. In such a case, for example, there may be four sample pairs selected from the available reference samples, each sample pair consisting of a luminous sample and a corresponding chroma sample. The two pairs with the lowest luminous values ​​of the four pairs may form a first set, and the two pairs with the highest luminous values ​​may form a second set. The luminous values ​​of the first set may be averaged to form a first mean luminous value, and the chroma values ​​of the first set may be averaged to form a first mean chroma value. Similarly, the luma values ​​of a second set may be averaged to form a second mean luma value, and the chroma values ​​of a second set may be averaged to form a second mean chroma value. Linear model parameters may be calculated from two mean luma values ​​and two mean chroma values, as in the H.266 / VVC standard.

[0182] The update term for gradient values ​​can be determined in various ways. For example, a video encoder can use rate-distortion optimization techniques to determine an appropriate update term for gradient values. A video decoder can determine the update term for gradient values ​​by parsing it from the bitstream and predicting it, or by a combination of prediction and bitstream signaling, or by other means.

[0183] The update term for the gradient value can be scaled to the same basis as the calculated gradient value a, or the calculated gradient value a can be scaled to the same basis as the update term, or both values ​​can be scaled to a third basis, or scaling can be omitted. The third basis can be determined, for example, by the maximum allowable basis used for the update term and the gradient parameter.

[0184] The update term may be limited to a specific range. For example, the update term may be defined to have integer values ​​in the range of -3 to 3, or -4 to 4, or -N to N, or N to M, where N and M can be constants or determined by various means. For example, they may be determined based on the characteristics of the block being predicted, coded, or decoded, such as the size of the block.

[0185] In one embodiment, there exists a predetermined set of update terms that are signaled in a bitstream and can be decoded by a decoder.

[0186] In one embodiment, the set of update terms signaled in the bitstream and decoded by the decoder depends on the characteristics of the sample block, such as the size of the block.

[0187] In one embodiment, the offset term b' in linear mode is: b'=c r -((a*y r It is calculated as )>>k), where c r is the average of the boundary chroma values ​​above the block, and y r is the average of the boundary luma values ​​on the block, a is the gradient parameter, and k is the shifting parameter.

[0188] In one embodiment, the offset term b' in linear mode is: b'=c r -((a*y r It is calculated as )>>k), where c ris the average of the left boundary chroma values ​​of the block, and y r is the average of the left boundary luma values ​​of the block, a is the gradient parameter, and k is the shifting parameter.

[0189] In one embodiment, the offset term b' in linear mode is: b'=c r -((a*y r It is calculated as )>>k), where c r is the average of the left and top boundary chroma values ​​of the block, and y r is the average of the left and top boundary luma values ​​of the block, a is the gradient parameter, and k is the shifting parameter.

[0190] In one embodiment, to determine the offset parameter b', c r and y r The set of reference luma and chroma values ​​used to calculate the parameter is selected to be greater than the set of reference luma and chroma values ​​used to calculate the gradient parameter a.

[0191] In one embodiment, to determine the offset parameter b', c r and y r The set of reference luma and chroma values ​​for calculating the parameter is selected to be a superset of the set of reference luma and chroma values ​​for calculating the gradient parameter a.

[0192] The determination of a reference luma value or reference point can be carried out in various ways. For example, the reference luma value can be the average of two luma values ​​used when determining the gradient parameters of a linear model; or it can be a weighted average of such luma values; or it can be determined in other ways.

[0193] There can be multiple linear models that form a mapping from ruma values ​​to chroma values. For example, if the ruma value is below a threshold, a first linear model may be used, and if the ruma value is above the threshold, a second model may be used. In such cases, each of the linear models can receive its independent update terms based on bitstream signaling. Furthermore, only a specific subset of the linear models may be signaled to be updated using one or more signaled update terms. Moreover, the same update term may be decided to be used in multiple models based on bitstream signaling or other means. In such cases, the way in which the signaled update term is applied to the various models may be further determined. For example, the same update term may be used in two models, or a negative version of the update term may be used in one model and a positive version of the update term may be used in the other model.

[0194] The mapping and parameter update processes are described here assuming that the input has luma values ​​and the output has chroma values, but the input and output are not limited to such color components. For example, the same process may be applied between chroma channels, having one chroma channel such as a Cb channel as the input and another chroma channel such as a Cr channel as the output. As a further example, the input channel may be a luma channel, and the output channel may be an auxiliary information channel consisting of, for example, depth, distance, parallax, transparency, or other types of values.

[0195] According to one embodiment, the chroma block can also correspond to any of the red, green, or blue color components of the RGB color space.

[0196] An apparatus according to one embodiment comprises: means for calculating at least two parameters that define a mapping from a first color component to a second color component, wherein at least two parameters include at least a gradient parameter and an offset parameter; means for determining an update term for the gradient parameter; means for applying the update term to the gradient parameter by adding the update term to the value of the gradient parameter to generate an updated gradient parameter; means for determining a reference value for the first color component; and means for calculating an updated offset parameter based on the reference value for the first color component and the updated gradient parameter.

[0197] In a further embodiment, a device is provided comprising: at least one processor and at least one memory, wherein code is stored in the at least one memory, and the code, when executed by the at least one processor, causes the device to: calculate at least two parameters that define a mapping from a first color component to a second color component, wherein the at least two parameters include at least a gradient parameter and an offset parameter; determine an update term for the gradient parameter; apply the update term to the gradient parameter by adding the update term to the value of the gradient parameter to generate an updated gradient parameter; determine a reference value for the first color component; and calculate an updated offset parameter based on the reference value for the first color component and the updated gradient parameter.

[0198] Such a device may include, for example, a functional unit disclosed in any of Figures 1, 2, 4a, and 4b for carrying out an embodiment.

[0199] Such a device further includes code stored in the at least one memory, which, when executed by the at least one processor, causes the device to perform one or more of the embodiments disclosed herein.

[0200] Figure 8 is a graphical representation of an exemplary multimedia communication system in which various embodiments may be implemented. The data source 1510 provides a source signal in analog format, uncompressed digital format, compressed digital format, or any combination of these formats. The encoder 1520 may include, or be connected to, preprocessing of the source signal, such as data format conversion and / or filtering. The encoder 1520 encodes the source signal into a coded media bitstream. It should be noted that the bitstream to be decoded may be received directly or indirectly from a remote device located within virtually any type of network. Additionally, the bitstream may be received from local hardware or software. The encoder 1520 may be capable of encoding two or more media types, such as audio and video, or two or more encoders 1520 may be required to encode source signals of different media types. The encoder 1520 may also receive synthetically produced inputs, such as graphics and text, or produce coded bitstreams of synthetic media. Below, only the processing of one coded media bitstream of one media type is considered for the sake of simplicity. However, it should be noted that, generally, real-time broadcast services include several streams (typically at least one audio, video, and text subtitle stream). While a system can include many encoders, it should also be noted that, for the sake of simplicity and without loss of generality, only one encoder 1520 is shown in the diagram. While the text and examples contained herein may specifically illustrate the encoding process, it should be further understood that those skilled in the art will understand that the same concepts and principles apply to the corresponding decoding process, and vice versa.

[0201] The coded media bitstream can be transferred to storage 1530. Storage 1530 may include any type of mass memory for storing coded media bitstreams. The format of the coded media bitstream in storage 1530 can be a basic self-contained bitstream format, or one or more coded media bitstreams can be encapsulated in a container file, or coded media bitstreams can be encapsulated in a segment format suitable for DASH (or a similar streaming system) and stored as a sequence of segments. If one or more media bitstreams are encapsulated in a container file, a file generator (not shown) can be used to store one or more media bitstreams in a file and create file format metadata, which can also be stored in a file. The encoder 1520 or storage 1530 may include a file generator, or the file generator may be operably mounted on either the encoder 1520 or storage 1530. Some systems operate "live," meaning they omit storage and directly transfer the coded media bitstream from encoder 1520 to sender 1540. The coded media bitstream can then be transferred to sender 1540, also called server, as needed. The format used for transmission can be a basic self-contained bitstream format, a packet stream format, a segment format suitable for DASH (or a similar streaming system), or one or more coded media bitstreams can be encapsulated in a container file. Encoder 1520, storage 1530, and server 1540 may reside on the same physical device or be contained in separate devices.The encoder 1520 and server 1540 can operate with live real-time content, in which case the encoded media bitstream is generally buffered for a short period in the content encoder 1520 and / or server 1540 rather than being permanently stored, to smooth out processing delays, transfer delays, and fluctuations in the encoded media bitrate.

[0202] Server 1540 sends coded media bitstreams using a communication protocol stack. The stack may include, but is not limited to, one or more of the following: Real-time Transport Protocol (RTP), User Datagram Protocol (UDP), Hypertext Transfer Protocol (HTTP), Transmission Control Protocol (TCP), and Internet Protocol (IP). If the communication protocol stack is packet-oriented, Server 1540 encapsulates the coded media bitstream in packets. For example, when RTP is used, Server 1540 encapsulates the coded media bitstream in RTP packets according to the RTP payload format. Generally, each media type has its own dedicated RTP payload format. The system may include two or more Server 1540s, but for simplicity, please note again that the following description will consider only one Server 1540.

[0203] If media content is encapsulated in a container file for storage 1530 or for inputting data to sender 1540, sender 1540 may include, or operably attach to, a “sending file parser” (not shown). In particular, if the container file is not transmitted in that manner, but rather at least one of the contained coded media bitstreams is encapsulated for transport over a communication protocol, the sending file parser identifies the location of the appropriate portion of the coded media bitstream to be carried over the communication protocol. The sending file parser can also help create the correct format for the communication protocol, such as packet headers and payloads. A multimedia container file may include encapsulation instructions, such as hint tracks in ISOBMFF, to encapsulate at least one of the contained media bitstreams based on a communication protocol.

[0204] Server 1540 may or may not be connected to Gateway 1550 through a communication network that can be, for example, a CDN, the Internet, and / or a combination of one or more access networks. The gateway is sometimes referred to as a middlebox, or alternatively. In DASH, the gateway can be an edge server (of the CDN) or a web proxy. While the system can generally include any number of gateways, etc., for simplicity, please note that the following description will consider only one gateway 1550. Gateway 1550 can perform various types of functions, such as translating packet streams according to one communication protocol stack to another communication protocol stack, merging and forking data streams, and manipulating data streams according to downlink and / or receiver capabilities, such as controlling the bitrate of the forwarded stream according to the dominant downlink network state. Gateway 1550 can be a server entity in various embodiments.

[0205] The system generally includes one or more receivers 1560 that can receive, demodulate, and decapsulate transmitted signals into coded media bitstreams. The coded media bitstreams may be transferred to recording storage 1570. Recording storage 1570 may include any type of mass memory for storing coded media bitstreams. Recording storage 1570 may also include computation memory, such as random access memory, as an alternative or additional measure. The format of the coded media bitstreams in recording storage 1570 may be a basic self-contained bitstream format, or one or more coded media bitstreams may be encapsulated within a container file. When there are many coded media bitstreams, such as related audio and video streams, a container file is generally used, and the receiver 1560 may have or be fitted with a container file generator that creates a container file from the input streams. Some systems operate "live," i.e., omitting recording storage 1570 and transferring coded media bitstreams directly from receiver 1560 to decoder 1580. In some systems, only the most recent portion of a recorded stream, for example, only the most recent 10 minutes of the recorded stream, is retained in the recording storage 1570, while older recorded data is discarded from the recording storage 1570.

[0206] Encoded media bitstreams can be transferred from recording storage 1570 to decoder 1580. If there are many encoded media bitstreams, such as audio and video streams, that are related to each other and encapsulated within a container file, or if a single media bitstream is encapsulated in a container file for easier access, for example, a file parser (not shown) is used to decapsulate each encoded media bitstream from the container file. The recording storage 1570 or decoder 1580 may include a file parser, or the file parser may be attached to either the recording storage 1570 or decoder 1580. It should also be noted that while the system can include many decoders, only one decoder 1570 is discussed here for the sake of simplicity and without loss of generality.

[0207] The encoded media bitstream may be further processed by the decoder 1570, the decoder's output being one or more uncompressed media streams. Finally, the renderer 1590 can play the uncompressed media streams, for example, using a loudspeaker or display. The receiver 1560, recording storage 1570, decoder 1570, and renderer 1590 may reside in the same physical device or be contained in separate devices.

[0208] Sender 1540 and / or gateway 1550 may be configured to switch between different representations, for example, for switching between different viewports of 360-degree video content, view switching, bitrate matching, and / or for fast startup, and / or sender 1540 and / or gateway 1550 may be configured to select the representation to be transmitted. Switching between different representations may occur for many reasons, such as in response to a request from receiver 1560 or in response to a dominant condition such as the throughput of the network through which the bitstream is carried. In other words, receiver 1560 can initiate switching between representations. Requests from the receiver may be, for example, a request for a segment or subsegment with a different representation than before, a request for a change in the scalability layer and / or sublayer being transmitted, or a change in rendering device having different capabilities compared to the previous one. A segment request may be an HTTP GET request. A subsegment request may be an HTTP GET request containing a byte range. As an addition or alternative, bitrate adjustment or bitrate fitting may be used, for example, in a streaming service to provide so-called fast startup, in which case the bitrate of the transmitted stream is lower than the channel bitrate at the start of streaming or after random access, in order to immediately begin playback and to achieve a buffer occupancy level that allows for occasional packet delays and / or retransmissions. Bitrate fitting may include a number of representation or layer-up switching and representation or layer-down switching operations occurring in various orders.

[0209] The decoder 1580 may be configured to switch between different representations for, for example, switching between different viewports of 360-degree video content, view switching, bitrate adaptation, and / or for fast startup, and / or the decoder 1580 may be configured to select the transmitted representation. Switching between different representations may occur for many reasons, such as achieving faster decoding operation or adapting the transmitted bitstream to dominant conditions, such as bitrate, the throughput of the network through which the bitstream is carried. For example, faster decoding operation may be required if the device containing the decoder 1580 is multitasking and uses computing resources for purposes other than decoding the video bitstream. In another example, faster decoding operation may be required when content is played back at a faster pace than normal playback speed, for example, twice or three times faster than conventional real-time playback rates.

[0210] In the foregoing, several embodiments have been described with reference to and / or using the terms HEVC and / or VVC. It should be understood that these embodiments can similarly be implemented using any video encoder and / or video decoder.

[0211] In the foregoing, if an exemplary embodiment is described with reference to an encoder, it should be understood that the resulting bitstream and decoder may have corresponding elements within them. Similarly, if an exemplary embodiment is described with reference to a decoder, it should be understood that the encoder may have a structure and / or computer program for generating a bitstream to be decoded by the decoder. For example, some embodiments are described in relation to generating prediction blocks as part of encoding. Embodiments may be similarly realized by generating prediction blocks as part of decoding, but with the difference that coding parameters such as horizontal and vertical offsets are decoded from the bitstream rather than being determined by the encoder.

[0212] The embodiments of the present invention described above describe the codec in terms of separate encoder and decoder devices to aid in understanding the related processes. However, it will be understood that the device, structure, and operation may be implemented as a single encoder-decoder device / structure / operation. Furthermore, the coder and decoder may share some or all common elements.

[0213] While the above examples illustrate embodiments of the present invention operating within a codec in an electronic device, it will be understood that the present invention, as defined in the claims, can be implemented as part of any video codec. Therefore, for example, embodiments of the present invention can be implemented in a video codec capable of implementing video coding over a fixed or wired communication path.

[0214] Therefore, the user device may include a video codec such as those described in the embodiments of the present invention described above. It will be understood that the term "user device" is intended to encompass any suitable type of wireless user device, such as a mobile phone, portable data processing device, or mobile web browser.

[0215] Furthermore, elements of a public land mobile network (PLMN) can further include the video codec described above.

[0216] In general, various embodiments of the present invention can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. For example, one aspect can be implemented in hardware, while another aspect can be implemented in firmware or software executable by a controller, a microprocessor, or other computing device, but the present invention is not limited thereto. Various aspects of the present invention can be illustrated and described as block diagrams, flowcharts, or using other graphical representations, but these blocks, devices, systems, techniques, or methods described herein are, by way of non-limiting example, hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers, or other computing devices, or any combination thereof.

[0217] Embodiments of the present invention can be implemented by computer software executable by a data processor of a mobile device, such as within a processor entity, or by hardware, or by a combination of software and hardware. Further, in this regard, any block of the logical flow as shown in the figures can represent a program step, or interconnected logical circuits, blocks, and functions, or a combination of program steps and logical circuits, blocks and functions. Software can be stored in a physical medium, such as a memory chip, or a memory block implemented within a processor, a magnetic medium such as a hard disk or floppy disk, and an optical medium such as, for example, a DVD and its data variants of a CD.

[0218] The memory can be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The data processor can be of any type suitable for the local technical environment and can include, by way of non-limiting example, one or more of a general-purpose computer, a dedicated computer, a microprocessor, a digital signal processor (DSP), and a processor based on a multi-core processor architecture.

[0219] Embodiments of the present invention can be practiced with various components such as integrated circuit modules. The design of integrated circuits is generally a highly automated process. Complex and powerful software tools are available to convert a logic level design into a semiconductor circuit design ready to be etched and formed on a semiconductor substrate.

[0220] Programs such as those provided by Synopsys, Inc. of Mountain View, California and Cadence Design of San Jose, California use well-established design rules and a library of pre-stored design modules to automatically route conductors and place components on a semiconductor chip. Once the design of the semiconductor circuit is complete, the resulting design in a standard electronic format (e.g., Opus, GDSII, etc.) can be sent to a semiconductor manufacturing facility or "fab" for manufacturing.

[0221] The foregoing description has provided a complete and useful explanation of exemplary embodiments of the present disclosure by way of illustrative and non-limiting examples. However, various changes and modifications will become apparent to those skilled in the art upon consideration of the foregoing description in conjunction with the accompanying drawings and the appended claims. However, all such and similar changes to the teachings of the present disclosure will still fall within the scope of the present disclosure.

Claims

1. Determine the reference values ​​for the parameters of the mapping function from the first color component to the second color component. Decode the initial size of the update term, Decode the syntax elements that should be used when determining the sign of the update term. The sign of the update term is determined by interpreting the decoded syntax element based on the reference value. The values ​​of the mapping function parameters are determined using the reference value, the decoded size of the update term, and the determined sign of the update term. Equipped with means for, The means for determining the code is Compare the reference value with the threshold, The sign of the update term is determined to be positive or negative if the reference value is greater than the threshold and the syntax element contains a first display value, or if the reference value is less than or equal to the threshold and the syntax element contains a second display value different from the first display value. A device equipped with means for doing so.

2. The apparatus according to claim 1, wherein the first display value is 0 and the second display value is 1, or the first display value is 1 and the second display value is 0.

3. The apparatus according to claim 1 or 2, wherein the threshold is 0.

4. The means for determining the code is The sign of the update term is determined such that when the decoded binary syntax element representing the sign of the update term is 0, the reference value is refined toward zero, and when the decoded binary syntax element representing the sign of the update term is 1, the reference value is refined toward zero. The apparatus according to claim 1 or 2, comprising means for doing so.

5. The means for determining the code is The sign of the update term is determined such that when the decoded binary syntax element representing the sign of the update term is 1, the reference value is refined toward zero, and when the decoded binary syntax element representing the sign of the update term is 0, the reference value is refined toward zero. The apparatus according to claim 1 or 2, comprising means for doing so.

6. When the value of the reference parameter is greater than zero, the value of the update term is determined by reversing the sign of the initial value of the update term. The apparatus according to claim 1 or 2, further comprising means for doing so.

7. When the value of a reference parameter is less than or equal to zero, the value of the update term is determined by leaving the initial value of the update term unchanged. The apparatus according to claim 1 or 2, further comprising means for doing so.

8. The apparatus according to claim 1 or 2, wherein the first color component is a luma component and the second color component is a single chroma component.

9. Determining the reference values ​​of the parameters of the mapping function from the first color component to the second color component, Decode the initial size of the update term, Decoding the syntax elements that should be used when determining the sign of the update term, The sign of the update term is determined by interpreting the decoded syntax element based on the reference value, The values ​​of the mapping function parameters are determined using the reference value, the decoded size of the update term, and the determined sign of the update term. Includes, Determining the code is Comparing a reference value to a threshold, When the reference value is greater than the threshold and the syntax element contains a first display value, or when the reference value is less than or equal to the threshold and the syntax element contains a second display value different from the first display value, the sign of the update term is determined to be positive, or the sign of the update term is determined to be negative. Methods that include...

10. Determine the reference values ​​for the parameters of the mapping function from the first color component to the second color component. Determine the initial value of the reference value update term, If the reference value is greater than the threshold, the displayed value of the update term is determined such that the sign of the displayed value of the update term is opposite to the sign of the determined initial value of the update term. Encode the sign and magnitude of the displayed values ​​in the update item into the video bitstream. A device equipped with means for doing so.

11. When the value of the reference parameter is greater than zero, the sign of the update term is reversed. The apparatus according to claim 10, further comprising means for doing so.

12. When the value of a reference parameter is less than or equal to zero, the displayed value of the update term is determined by leaving the update term unchanged. The apparatus according to claim 10 or 11, further comprising means for doing so.

13. Rate distortion is determined to determine the value of the update term. The displayed update items are determined based on the values ​​of the update items. The apparatus according to claim 10 or 11, further comprising means for doing so.

14. Rate distortion is determined to determine the values ​​of the displayed update terms. Determine the update item based on the displayed update item value. The apparatus according to claim 10 or 11, further comprising means for doing so.

Images