Cross-component chroma linear transformation method and apparatus for video coding

Abstract

The invention discloses a method and a device for carrying out video coding and decoding by using cross-component chroma linear transformation (Crosscomponent Chroma Line Transform). The method and the device are used for carrying out video coding and decoding by using cross-component chroma linear transformation (Crosscomponent Chroma Line Transform). For the decoder side, input data associated with a current block in the current picture is received, the current block comprising a luma component and two or more second chroma components in a second chroma format. The input data includes encoded data associated with a current block to be decoded. In an encoding process, a forward CCCLT is applied to two or more first chroma components of a first chroma format to generate the two or more second chroma components of a second chroma format. In the reconstruction process, a reverse CCCLT is applied to the reconstructed two or more second chroma components of the second chroma format to generate reconstructed two or more first chroma components of the first chroma format. Comprising the reconstructed two or more first chroma components in a first chroma format is provided.

Description

[0001] Related citations This invention relates to and claims priority to U.S. Provisional Patent Application No. 63 / 515,848 (filed July 27, 2023) and U.S. Provisional Patent Application No. 63 / 595,383 (filed November 2, 2023), which are non-provisional applications. The aforementioned U.S. Provisional Patent Applications are incorporated herein by reference in their entirety. Technical Field

[0002] This invention relates to video encoding and decoding systems. Specifically, it discloses a cross-component chroma linear transform in a video encoding and decoding system for converting chroma components to another chroma format. Background Technology

[0003] Versatile Video Coding (VVC) is the latest international video coding standard developed by the Video Coding Experts Group (VCEG) of the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) and the Joint Video Experts Team (JVET) of the Moving Picture Experts Group (MPEG) of the International Organization for Standardization / International Electrotechnical Commission (ISO / IEC). This standard was published as an ISO standard in February 2021: ISO / IEC 23090-3:2021, Information technology – Codec representation of immersive media – Part 3: Versatile Video Coding. VVC was developed based on its predecessor, High Efficiency Video Coding (HEVC). It improves encoding and decoding efficiency and can handle various types of video sources, including three-dimensional (3D) video signals, by adding more encoding and decoding tools. Bross et al. published an overview of the VVC standard, titled "An Overview of the VVC Standard and Its Applications," in the IEEE Transactions on Video Technology Circuits and Systems, Vol. 31, No. 10 (October 2021), pp. 3736-3764.

[0004] VVC uses adaptive intra / inter-frame prediction and quantizes and transforms the prediction residuals for encoding and decoding, similar to previous video codec standards. However, VVC also incorporates various newer codec tools. Figure 1An exemplary block diagram of a VVC encoder is shown. VVC also uses block-based encoding and decoding, where each input image is divided into a Coding Tree Unit (CTU), typically 128x128 or 64x64 in size. Input video is typically in 4:2:0 YCbCr format, where the resolution of Cb and Cr is 1 / 4 of the luminance component (half horizontally and half vertically). A sequence of 4:2:0 YCbCr format input video can be converted from 4:4:4 RGB video through a predetermined conversion process. The human eye is not very sensitive to chroma information. Therefore, a spatial downsampling procedure is applied to the chroma channel, and 1 / 4 of the luminance sample is retained for chroma. After decoding, if the display device requires 4:4:4 RGB format to display frames, the decoded 4:2:0 YCbCr video can be converted to the format required by the display device for viewing.

[0005] like Figure 1 As shown, subtractor 112 subtracts prediction signal 111 from input video signal 102. The output of subtractor 112 corresponds to the residual signal. For VVC, luma mapping 110 is applied to input video signal 102 before prediction. In addition to intra-picture prediction 142 and inter-picture prediction 146, VVC introduces a new prediction type called combined inter-frame / intra-frame prediction (CIIP, 144). Inter-picture prediction 146 is based on reconstructed pictures stored in decoded picture buffer 170 and motion information from motion estimation 148. Since the input video signal passes through luma mapping 110, a corresponding processing (i.e., luma mapping 114) is applied to the output before it is selected and used as prediction signal 111. Selection mechanism 116 is used to adaptively select between intra-picture prediction 142, inter-picture prediction 146, and combined inter-frame / intra-frame prediction (CIIP, 144).

[0006] Similar to HEVC, the residual signal from subtractor 112 undergoes a transform, scaling, and quantization process 120 to generate quantized transform coefficients (labeled "B"). The encoder side also needs to generate a reconstructed image for subsequent image encoding. To generate the reconstructed image, scaling and inverse transform processing 130 is applied to the quantized transform coefficients (labeled "B") to generate the reconstructed residual. VVC also introduces chroma scaling 132 applied to the reconstructed residual. After chroma scaling, the reconstructed residual is added to the prediction signal 111 to form the reconstructed signal. For intra-frame prediction, the reconstructed signal can be used directly without loop filtering. For inter-frame prediction, the reconstructed residual undergoes inverse luma mapping, deblocking, Sample Adaptive Offset (SAO), and Adaptive Loop Filter (ALF) 160. Since luma mapping has already been applied to the input video signal, inverse luma mapping must be applied in the reconstruction path. VVC adaptively selects the filter parameters. To determine the filter parameters, the filter control analysis 150 determines the filter parameters based on the input video signal 102 and the reconstructed residual.

[0007] Within the encoder, various processing modules determine modes, parameters, or other control information. For example, the prediction mode selection 116 determines whether to use intra-frame, inter-frame, or CIIP modes based on mode selection information from the general codec control 190. Furthermore, the general codec control 190 provides control information to intra-picture estimation 140, inter-picture estimation 146, and scaling and inverse transform 130. The quantization transform coefficients (labeled "B") from the transform, scaling, and quantization processing 120 are encoded and decoded by header formatting and context-adaptive binary arithmetic coding (CABAC) 180 to be included in the encoded bitstream. In addition, various related codec information, such as general codec control information (labeled "A"), intra-prediction mode information from intra-picture estimation 140 (labeled "C"), inter-prediction mode information from inter-picture estimation 146 (labeled "E"), and filter control information from filter control analysis 150 (labeled "D"), will be included in the encoded bitstream formatted using header and CABAC 180.

[0008] Since the encoder also needs to generate the reconstructed signal and use it for prediction, the encoder includes some of the key processing units used by the decoder. Figure 1The data is displayed in gray. The decoder side receives the encoded bitstream. The module that performs header formatting and CABAC180 inversion functions is used to recover the quantization transform coefficients (labeled "B") and various side information or control information (i.e., Figure 1 (Indicated by "A", "C", "D", and "E"). The decoder operates on the quantized transform coefficients to generate the reconstructed video. At the decoder, intra-frame predictive mode information and inter-frame predictive motion information, along with the necessary auxiliary information (i.e., "C" and "F"), can be generated. However, for decoder-side Intra Mode Derivation (DIMD), the decoder derives the mode information without requiring a signal from the encoder.

[0009] Recently, P. Wennersten et al. published Motion Compensated Temporal Filtering (MCTF). (“AHG10: Encoder-only GOP-based temporal filtering,” Joint Video Experts Group (JVET) of ITU-TSG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11, 15th meeting: Gothenburg, Sweden, July 3-12, 2019, document: JVET-O0549). The filtering is performed as a preprocessing step on the encoder side, where motion estimation and motion compensation methods are applied to adjacent images on an 8x8 luma block. References Figure 1 The encoder system block diagram shows that MCTF processing is located before the input video signal 102.

[0010] In this invention, a method and apparatus for improving performance using cross-component chromaticity linear transformation are proposed, wherein chromaticity components of a first format are converted into chromaticity components of another format through a set of linear equations. [Summary of the Invention] A method and apparatus for video encoding and decoding using cross-component chroma linear transform (CCCLT) are proposed. On the decoder side, input data associated with a current block in a current image is received, the current block comprising a luma component and two or more second chroma components in a second chroma format, wherein the input data includes encoded / decoded data associated with the current block to be decoded, wherein, in the encoding process, a forward cross-component chroma linear transform (CCCLT) is applied to two or more first chroma components in a first chroma format to generate the two or more second chroma components in a second chroma format. In the reconstruction process, a reverse CCCLT is applied to two or more reconstructed second chroma components in the second chroma format to generate the two or more reconstructed first chroma components in the first chroma format. An output video including the two or more reconstructed first chroma components in the first chroma format is proposed.

[0012] In one embodiment, an inverse CCCLT is applied before deblocking, after deblocking and before Sample Adaptive Offset (SAO) processing, after SAO processing and before Adaptive Loop Filter (ALF) processing, or after ALF processing.

[0013] In one embodiment, the flag is parsed to determine whether the forward CCCLT is applied to the two or more first chroma components in the first chroma format and whether the reverse CCCLT is applied to the reconstructed two or more second chroma components.

[0014] In one embodiment, transform parameters for inverse CCCLT are determined for each sequence, each image, each coding tree unit (CTU), or each coding unit (CU). In another embodiment, one or more transform parameters for inverse CCCLT are selected from multiple sets of candidate parameters for the one or more transform parameters. In yet another embodiment, an index is received to indicate the target set among the multiple sets of candidate parameters selected for the one or more transform parameters.

[0015] In one embodiment, the transform parameters of the inverse CCCLT are received from a bitstream at the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Codec Tree Unit (CTU), or Codec Unit (CU) level. In another embodiment, one or more transform parameters of the inverse CCCLT are updated based on one or more incremental values.

[0016] In one embodiment, one or more transform parameters of the inverse CCCLT are inherited from a neighboring CTU or CU. In one embodiment, whether the inheritance of the one or more transform parameters of the inverse CCCLT is enabled is indicated by a flag. In one embodiment, the candidates from which the one or more transform parameters are inherited are indicated by an index.

[0017] In one embodiment, a forward CCCLT is applied to the two or more first chroma components in the first chroma format only if the current block is in an I-slice, or a P-slice, or a B-slice, and a reverse CCCLT is applied to the two or more second chroma components in the reconstruction.

[0018] A method for the encoder side is disclosed. On the encoder side, input data associated with a current block in a current image is received, the current block comprising a luma component and two or more first chroma components, wherein the input data includes pixel data to be encoded, and wherein the two or more first chroma components are in a first chroma format. A forward cross-component chroma linear transform (CCCLT) is applied to the two or more first chroma components to generate two or more second chroma components in a second chroma format, wherein the forward CCCLT is applied in the encoding process to convert the chroma components from the original format to another format. In the reconstruction process, a reverse CCCLT is applied to the reconstructed two or more second chroma components in the second chroma format to generate the reconstructed two or more first chroma components in the first chroma format. The reconstructed two or more first chroma components in the first chroma format and / or the reconstructed two or more second chroma components in the second chroma format are provided for processing one or more subsequent blocks of the current block.

[0019] In one embodiment, a positive CCCLT is applied before, after, or after the Motion Compensated Temporal Filtering (MCTF), or after the Joint Coding of Chroma Residual (JCCR) is determined, and before Low Compexity Rate Distortion Optimization (LC-RDO).

[0020] In one embodiment, a first reconstruction buffer is used for a first encoding process that includes the application forward CCCLT and the application reverse CCCLT, and a second reconstruction buffer is used for a second encoding process that does not include the application forward CCCLT and the application reverse CCCLT, and selects the application forward CCCLT and the application reverse CCCLT if better performance is achieved by using the application forward CCCLT and the application reverse CCCLT.

[0021] In one embodiment, a first pass is performed on a first encoding process that includes the applied forward CCCLT and the applied reverse CCCLT, a second pass is performed on a second encoding process that does not include the applied forward CCCLT and the applied reverse CCCLT, and the applied forward CCCLT and the applied reverse CCCLT are selected if the first pass achieves better performance. In one embodiment, the first and second passes are performed at the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Codec Tree Unit (CTU), or Codec Unit (CU) level.

[0022] In one embodiment, whether to use the application-forward CCCLT and the application-reverse CCCLT is implicitly determined by using one or more templates outside the current codec tree unit (CTU) or codec unit (CU). Attached Figure Description

[0023] Figure 1 An exemplary system block diagram of a VVC encoder is shown.

[0024] Figure 2A -B illustrates an example of using a neural network to derive parameters or selection indices for a cross-component chromaticity linear transformation according to an embodiment of the present invention.

[0025] Figure 3 A flowchart illustrating an exemplary video decoding system using cross-component chroma linear transform according to an embodiment of the present invention is shown.

[0026] Figure 4 A flowchart illustrating an exemplary video coding system using cross-component chroma linear transform according to an embodiment of the present invention is shown. Detailed Implementation

[0027] The components of the present invention, as generally described and illustrated in the figures, can be arranged and designed in a variety of different configurations. Therefore, the following more detailed description of embodiments of the systems and methods of the present invention, as illustrated in the figures, is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the invention. References to “one embodiment,” “an embodiment,” or similar language throughout this specification mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of the invention. Therefore, the phrases “in one embodiment” or “in one embodiment” appearing throughout this specification do not necessarily refer to the same embodiment.

[0028] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. However, those skilled in the art will recognize that the invention can be practiced without using one or more specific details, or using other methods, components, etc. In other instances, well-known structures or operations are not shown or described in detail to avoid obscuring aspects of the invention. Embodiments of the invention will be best understood by referring to the accompanying drawings, wherein like parts are designated by like numbers throughout. The following description is illustrative only and only illustrates embodiments of specific selected apparatus and methods consistent with the invention declared herein.

[0029] Chromaticity components Cb and Cr have been widely used in video encoding and decoding. In this invention, a cross-component linear chroma transform is proposed to convert chroma components into different chroma formats during encoding and / or decoding of coding entities at different levels, including sequence-level, picture-level, CTU-level, and CU-level entities. First, a formula for the cross-component linear chroma transform is proposed. Then, different methods for encoding and / or decoding video using the cross-component linear chroma transform at different levels are described.

[0030] Cross-Component Chroma Linear Transformation (CCCLT) For video encoding and decoding, color video is typically represented by luminance (e.g., Y) and chrominance (e.g., Cr and Cb) components. Furthermore, the color format is fixed across all video sequences. On the other hand, input video sequences are typically acquired using a camera that outputs video in RGB format. Therefore, RGB video is converted to YCbCr format before the encoder processes the video data. In typical video coding systems, video data remains in YCbCr format throughout the encoding or decoding process. While the YCbCr format matches human color vision sensitivity well, it may not always produce optimal encoding results in terms of bit rate or video quality. Therefore, this invention discloses a technique for Cross-Component Chromaticity Linear Transform (CCCLT).

[0031] According to the present invention, a cross-component chromaticity linear transformation is performed on the input chromaticity components (e.g., Cb and Cr, referred to in this disclosure as the original chromaticity format) to convert the chromaticity components (e.g., Cb and Cr, referred to in this disclosure as the original chromaticity format) into other chromaticity formats (e.g., Cp and Cq, referred to as optional chromaticity formats), as shown in the following formula: Forward transformation: (1) (2) Backward transformation: (3) (4) In the above equation, a, b, c, d, e, and f are called transform parameters. An example case of transform parameters a through f for encoding 10-bit video is: a = 1 / 2, b = 1 / 2, c = 0, d = 1 / 2, e = -1 / 2, f = 512. Therefore, the forward and backward transforms are as follows: , , , .

[0032] While the example transform parameters are shown above, other CCCLT transform parameters can also be used. A forward transform can be applied during the encoding process to generate an encoded bitstream. An inverse transform will be applied during the reconstruction process to convert the optional chroma format back to the original chroma format.

[0033] The CCCLT of this invention can be adaptively applied to the chroma components at different video data levels, such as sequence level, picture level, CTU level, and CU level. Furthermore, the CCCLT can be applied to various locations in the encoding / decoding process. Details are described in the proposed method below.

[0034] Proposed Method A: Sequence-level cross-component chromaticity linear transformation This section discloses the sequence-level cross-component chroma linear transform (CCCLT), where CCCLT is applied to the input video sequence. There are several scenarios where the cross-component chroma linear forward transform is applied to the input image of the video sequence, including: - Apply a forward CCCLT transform before Motion Compensated Temporal Filtering (MCTF) processing.

[0035] - Apply the forward CCCLT transform after MCTF processing and before determining the JCCR mode.

[0036] - Apply the forward CCCLT transformation after determining the JCCR mode and before LC-RDO. In the above text, JCCR stands for Chromatic Residual Joint Coding and Decoding, a coding and decoding tool introduced in VVC. JCCR utilizes the residual correlation in the quantized chroma residual signal, where only one residual block is sent and used to derive residual blocks for both chroma components. JCCR decision can be performed in the General Encoder Control 190. In the above text, LC-RDO stands for Low Complexity Rate Distortion Optimization. It uses prediction and raw pixels to calculate the cost (cost = estimated bits + lambda * distortion). LC-RDO can be performed in the General Encoder Control 190. Sometimes the Hadamard transform is used to simulate the real transform to approximate the transform coefficients to obtain the estimated bits. In the block diagram, the "LC-RDO" and "JCCR" stages can be considered as within the General Encoder Control 190. JCCR uses the cost of LC-RDO to determine the mode of High Complexity Rate Distortion Optimization (HC-RDO) (as described in the mode decision equation below). The cost calculated in the raw pixel domain or the cost calculated in the forward CCCLT domain can differ. According to one embodiment of the present invention, whether or not CCCLT is applied will affect the JCCR mode in the LC-RDO stage. The JCCR mode selected by LC-RDO should compete with other codec modes in the HC-RDO stage to select the final optimal codec mode.

[0037] There are several locations where cross-component chroma linear inverse transforms can be applied to images in video sequences, including: - Apply the inverse CCCLT transform before deblocking.

[0038] - Apply the inverse CCCLT transform after deblocking and before SAO processing.

[0039] - Apply the inverse CCCLT transform after SAO processing and before ALF processing.

[0040] - Apply the inverse CCCLT transform after ALF processing.

[0041] In some embodiments, the coefficients of the cross-component chroma linear transform can typically be a = 1 / 2, b = 1 / 2, c = 0, d = 1 / 2, e = -1 / 2, and f = 512, as described above for encoding 10-bit video sequences. In another embodiment, a table with a predetermined set of forward transform parameters is generated, and the index of the selected set is sent. Table 1 shows an example of the parameter sets.

[0042] Table 1. Predetermined forward transformation parameter set In some embodiments, all forward transform parameters are sent from the encoder to the decoder. For example, a = 1 / 2, b = 1 / 2, c = 0, d = 1 / 2, e = 1 / 4, and f = 128 can be converted to an appropriate representation and sent in the bitstream. In another embodiment, some forward transform parameters are fixed, and only some parameters are sent. For example, a = 1 / 2, b = 1 / 2, c = 0, d = 1 / 2, and e = 1 / 4 are fixed parameters, and an offset value “f” is sent. To update the offset value “f”, the value can be sent directly; otherwise, it can be sent via an index from a reference value table, as follows: Table 2. Offset Value Index In another embodiment, the neural network is used to estimate the forward transformation parameters or to select the index of an entry from a parameter set table, such as... Figure 2A and Figure 2B As shown, the neural network generates transformation parameters ( Figure 2A ) and the index for selecting transformation parameters ( Figure 2B ).

[0043] In another embodiment, a default forward transformation is pre-defined or specified in the SPS header. In yet another embodiment, the sequence header sends partial or complete increment values ​​for a, b, c, d, e, and f to update these parameters with these increment values. The increment values ​​can be values ​​involving mathematical calculations (addition, subtraction, multiplication, division, etc.). For example: new value = old value + increment value (where the increment value can be positive or negative). Alternatively, the increment values ​​can be used as an index table to obtain modifications for updating the parameters.

[0044] In some embodiments, the video sequence is encoded twice, once without CCCLT and once with CCCLT. The output bitstream with better rate distortion performance is the final encoded result. In some embodiments, detection processing can be used to adaptively enable / disable the general cross-component chroma linear transformation. Several embodiments of the detection method are proposed in the proposed method H. A single detection is performed on the entire sequence. Statistics from the entire sequence or a portion of the sequence can be used for detection. This detection reduces the two encoding runs to a single encoding run. Additional flags for sequence-level enabling of CCCLT are recorded in the Sequence Parameter Set (SPS) or other sequence-level coding entity. In some embodiments, the cross-component chroma linear transformation is applied to all types of segments. In some embodiments, the cross-component chroma linear transformation is applied to I-segments. In some embodiments, the cross-component chroma linear transformation is applied to B-segments and / or P-segments.

[0045] Proposed Method B: Image-level cross-component chromaticity linear transformation Several embodiments of applying a general cross-component chroma linear forward transform to images of video sequences have been proposed, including: - Apply a forward CCCLT transform before MCTF processing.

[0046] - Apply a forward CCCLT transform after MCTF processing and before determining the JCCR mode.

[0047] - Apply the forward CCCLT transformation after determining the JCCR mode and before LC-RDO. Some examples of applying a general cross-component chroma linear inverse transform to images of video sequences include: - Apply the inverse CCCLT transform before deblocking.

[0048] - Apply the inverse CCCLT transform after deblocking and before SAO processing.

[0049] - Apply the inverse CCCLT transform after SAO processing and before ALF processing.

[0050] - Apply the inverse CCCLT transform after ALF processing.

[0051] To achieve image-level cross-component chroma linear transformation, we propose maintaining two reconstruction buffers. If a forward CCCLT is applied to the current image, we store the ALF-processed reconstructed sample in one of the reconstruction buffers. Then, we apply the inverse CCCLT transformation to the reconstructed sample and store the result in the other reconstruction buffer. If no forward CCCLT is applied to the current image, we store the ALF-processed reconstructed sample in one of the reconstruction buffers. Then, we apply the forward CCCLT transformation to the reconstructed sample and store the result in the other reconstruction buffer. Before storing the reconstructed sample in the reconstruction buffer, we propose adding padding to the reconstructed sample for motion compensation processing in future frames. Examples of adding padding to the reconstructed sample include: - Add repeating padding during refactoring.

[0052] - After ALF processing, add motion-compensated padding to the reconstructed sample.

[0053] In some embodiments, the coefficients of the general cross-component chroma linear transform can be a=1 / 2, b=1 / 2, c=0, d=1 / 2, e=-1 / 2, and f=512, as described in the section on encoding 10-bit video sequences in CCCLT. In another embodiment, a table with predetermined forward transform parameter groups is generated, and the index of the selected group is sent as shown in Table 1. In some embodiments, all forward transform parameters are sent from the encoder to the decoder. In another embodiment, some forward transform parameters are fixed, and only some parameters are sent. For example, a=1 / 2, b=1 / 2, c=0, d=1 / 2, and e=1 / 4 are fixed parameters, and an offset value “f” is sent. To update the offset value “f”, the value can be sent directly; otherwise, it can be sent by referencing an index in a value table, as shown in Table 2. In another embodiment, a neural network is used to estimate the forward transform parameters or indexes to select entries from the parameter group table shown in Figure 2. In another embodiment, a default forward transform is predetermined or specified in the SPS header. The incremental values ​​for a, b, c, d, e, and f are sent partially or entirely from the image header or fragment header to update these parameters using the incremental values. The incremental values ​​can be values ​​involving mathematical calculations (addition, subtraction, multiplication, division, etc.). For example: new value = old value + incremental value (where the incremental value can be positive or negative). Alternatively, the incremental value can be used as an index in a reference table to retrieve the modification used to update the parameters.

[0054] In some embodiments, video frames are encoded twice, once without CCCLT and once with CCCLT. A bitstream with better rate distortion performance is output as the final encoded result for that frame. In some embodiments, a universal cross-component chroma linear transform can be self-adjusted to be enabled / disabled using a detection process. Detection is performed frame-by-frame while encoding the video sequence. Statistical data from the entire frame or a portion of a frame can be used for detection. Several embodiments of the detection method are provided in the proposed method H section. Using this detection process, two encoding runs can be reduced to one. Additional flags for enabling picture-level CCCLT are recorded in the PPS (Picture Parameter Set) or PH (Picture Header) or other picture-level coding entities. In some embodiments, the cross-component chroma linear transform is applied to all types of segments. In some embodiments, the cross-component chroma linear transform is applied to I-segments. In some embodiments, the cross-component chroma linear transform is applied to B-segments and / or P-segments.

[0055] The proposed method C: CTU-level cross-component chromaticity linearity In some embodiments, a general cross-component chroma linear forward transform can be applied to the CTU before encoding it. A cross-component chroma linear inverse transform is applied to the reconstructed chroma samples before loop filtering processes (e.g., deblocking, SAO, ALF). In some embodiments, the coefficients of the general cross-component chroma linear transform can be a = 1 / 2, b = 1 / 2, c = 0, d = 1 / 2, e = -1 / 2, and f = 512, as described in the section on encoding 10-bit video sequences in CCCLT. In another embodiment, a table with predetermined sets of forward transform parameters is generated, and an index of the selected set, as shown in Table 1, is issued. In some embodiments, all forward transform parameters are sent from the encoder to the decoder. In another embodiment, some forward transform parameters are fixed, and only some parameters are sent. For example, a = 1 / 2, b = 1 / 2, c = 0, d = 1 / 2, e = 1 / 4 are fixed parameters, and an offset value “f” is sent. To update the offset value "f", the value can be sent directly; otherwise, it can be represented by the index of the value table shown in Table 2. In another embodiment, the neural network is used to estimate the forward transformation parameters or index to select entries from the parameter set table, such as... Figure 2A and Figure 2BAs shown. In another embodiment, a default forward transform is pre-defined or specified in the SPS / image / fragment header. The incremental values ​​of a, b, c, d, e, and f are sent partially or entirely by CTU-level syntax elements to update these parameters using the incremental values. The incremental values ​​can be values ​​involving mathematical calculations (addition, subtraction, multiplication, division, etc.). For example: new value = old value + incremental value (where the incremental value can be positive or negative). Alternatively, the incremental value can be used as an index in a reference table to retrieve the modified item for the updated parameter.

[0056] In some embodiments, the CTU is encoded twice, once without CCCLT and once with CCCLT. The bitstream with better rate distortion performance is output as the final encoded result of the CTU. In some embodiments, the detection process can be adaptively turned on / off with general cross-component chroma linear transformation. Statistics from the entire CTU or a portion of the CTU can be used to perform the detection. Several embodiments of the detection method are disclosed in the proposed method H section. Using this detection process, two encoding runs can be reduced to one encoding process run. An additional flag for CTU-level enable for CCCLT is recorded in the CTU-level field. In some embodiments, the cross-component chroma linear transformation is applied to all types of segments. In some embodiments, the cross-component chroma linear transformation is applied to I segments. In some embodiments, the cross-component chroma linear transformation is applied to B segments and / or P segments.

[0057] The proposed method, D:CU-level cross-component chromaticity linear transformation In some embodiments, a general cross-component chroma linear forward transform can be applied to the chroma CU before encoding. A cross-component chroma linear inverse transform is applied to the reconstructed chroma samples before loop filtering processes (e.g., deblocking, SAO, ALF). In some embodiments, the coefficients of the general cross-component chroma linear transform can be a = 1 / 2, b = 1 / 2, c = 0, d = 1 / 2, e = -1 / 2, and f = 512, as described in the section on encoding 10-bit video sequences in CCCLT. In another embodiment, a table with predetermined sets of forward transform parameters is generated, and the index of the selected set, as shown in Table 1, is sent. In some embodiments, all forward transform parameters are sent from the encoder to the decoder. In another embodiment, some forward transform parameters are fixed, and only some parameters are sent. For example, a = 1 / 2, b = 1 / 2, c = 0, d = 1 / 2, e = 1 / 4 are fixed parameters, and an offset value “f” is sent. To update the offset value "f", the value can be sent directly; otherwise, it can be sent by referencing an index in the value table shown in Table 2. In another embodiment, a neural network is used to estimate the forward transformation parameters or indices to select entries from the parameter set table, such as... Figure 2A and Figure 2B As shown. In another embodiment, a default forward transform is pre-defined or specified in the SPS / image / fragment header or via a CTU-level syntax element. The CTU-level syntax element emits some or all of the incremental values ​​for a, b, c, d, e, and f to update these parameters using the incremental values. The incremental values ​​can be values ​​involving mathematical calculations (addition, subtraction, multiplication, division, etc.). For example: new value = old value + incremental value (where the incremental value can be positive or negative). Otherwise, the incremental value can be an index into a reference table to retrieve the modification used to update the parameters.

[0058] In some embodiments, the CU is encoded twice, once without CCCLT and once with CCCLT. The bitstream with better rate distortion performance is output as the final encoded result of the chroma CU. In some embodiments, a detection process can be used to adaptively turn general cross-component chroma linear transformation on / off. Statistics from the entire CU or a portion of the CU can be used for detection. Several embodiments of the detection method are proposed in the proposed method H section. Using this detection process, two encoding runs can be reduced to one encoding run. An additional flag for CU-level enable for CCCLT is recorded in the CU-level field. In some embodiments, the cross-component chroma linear transformation is applied to all types of segments. In some embodiments, the cross-component chroma linear transformation is applied to I segments. In some embodiments, the cross-component chroma linear transformation is applied to B segments and / or P segments.

[0059] Proposed method E: Spatial filtering of chromaticity samples The proposed methods A and D utilize forward and inverse transform pairs. In this section, we propose applying a spatial filter to the chroma, such as a 3x3 or 5x5 Gaussian filter. This spatial filtering has no inverse operation. An additional flag is introduced to enable this spatial filtering. This flag can be a sequence / image / CTU / CU level syntax element. In some embodiments, the encoding process is performed twice to select a better encoding mode between enabling or disabling this spatial filtering. In some embodiments, a detection process can be used to adaptively turn the spatial filtering on / off. Statistics from a portion of the sequence / image / CTU / CU or a portion of the sequence / image / CTU / CU can be used for detection. In the proposed method H section, several embodiments of the detection method are proposed. Using this detection process, two encoding runs can be reduced to one encoding run. In some embodiments, spatial filtering is applied to all types of segments. In some embodiments, spatial filtering is applied to I segments. In some embodiments, spatial filtering is applied to B segments and / or P segments.

[0060] The proposed method F: applies spatial filtering to other types of samples. The methods mentioned in the preceding sections are applied to the original samples before encoding. Spatial filtering can be applied to the predicted samples of the chroma components. An additional flag is introduced to enable this spatial filtering. This flag can be a sequence / image / CTU / CU level syntax element.

[0061] The proposed method G: CCCLT merging mode To reduce the overhead of sending forward transform parameters, these parameters can be inherited from neighboring regions. This is often called a merge mode, where the same codec parameters are propagated from a neighboring CTU or CU to the current CTU or CU. Additional flags are sent to enable inheritance for this CCCLT merge mode. Inheritance candidates can be the top, left, top-left, top-right, and bottom-left CTU / CUs of I, B, and P segments. For B and P segments, temporal candidates (in the reference frame) can be used, such as temporally co-located CTUs / CUs and / or adjacent CTUs / CUs of co-located CTUs / CUs. Sometimes, multiple inheritance candidates can exist. For example, both the top and left CTUs / CUs are encoded using the CCCLT mode. Assuming the top CTU / CU is the first candidate (index = 0) and the left CTU / CU is the second candidate (index = 1), the inheritance candidate can be specified by sending an index (the so-called merge index). In this case, 0 or 1 is sent to determine the final merge candidate.

[0062] Proposed Method H: CCCLT Detection Method In one embodiment, the difference between the mean variance of the chroma sub-blocks before and after applying CCCLT can be used to determine the CCCLT flag. For example, a target region (e.g., an image, frame, CTU, or CU) can be divided into several chroma sub-blocks, each containing M x N chroma samples, where M and N are positive integers greater than 1. The variance of each chroma sub-block is calculated, and then the mean variance of all chroma sub-blocks is calculated. If the difference between the mean variance before and after applying CCCLT is greater than a predetermined threshold, then CCCLT is enabled.

[0063] In another embodiment, the difference between the median variance of the chroma sub-blocks before and after applying CCCLT can be used to determine the CCCLT flag. For example, a target region (e.g., an image, frame, CTU, or CU) can be divided into several chroma sub-blocks, each containing M x N chroma samples, where M and N are positive integers greater than 1. The variance of each chroma sub-block is calculated, and then the median variance of all chroma sub-blocks is calculated. If the difference between the median variance before and after applying CCCLT is greater than a predetermined threshold, then CCCLT is enabled.

[0064] In another embodiment, principal component analysis (PCA) can be used to determine the CCCLT flag. For example, PCA is first applied to the two chromaticity components of the target region (e.g., image, frame, CTU, or CU), and then we can obtain the first principal component and the second principal component. If the CCCLT transform is similar to the first principal component or the second principal component, then CCCLT is enabled.

[0065] In another embodiment, the difference between the regression matching loss before and after applying CCCLT can be used to determine the CCCLT flag. For example, first, a cross-component linear model (e.g., a Cross-component linear model, CCLM) is derived for the target region (e.g., an image, frame, CTU, or CU). The input to this model is the luminance component of the target region, and the output is one of the chrominance components of the target region. During regression processing, the regression matching loss can be obtained. If the regression matching loss after applying CCCLT is less than the regression matching loss before applying CCCLT, then CCCLT is enabled. As another example, two cross-component models for the target region are derived. One model outputs the Cb component of the target region, and the other model outputs the Cr component of the target region. The total matching loss is the sum of the two regression matching losses, and this total matching loss is used to determine the CCCLT flag.

[0066] In one embodiment, the detection method described above can be used to determine the CCCLT flag on the encoder side to accelerate the encoding algorithm.

[0067] In another embodiment, the detection method described above can be used to implicitly determine the CCCLT flag on both the encoder and decoder sides. If this implicit method is used, the CCCLT flag is not sent. The detection method can be applied to reconstructed samples in adjacent template regions.

[0068] Any of the methods proposed above can be implemented in the encoder and / or decoder. For example, any of the proposed methods can be implemented in the inter-frame / intra-frame / prediction module of the encoder and / or the inter-frame / intra-frame / prediction module of the decoder. Alternatively, any of the proposed methods can be implemented as circuitry coupled to the inter-frame / intra-frame / prediction module of the encoder and / or the inter-frame / intra-frame / prediction module of the decoder to provide the information required by the inter-frame / intra-frame / prediction module.

[0069] Figure 3 A flowchart illustrating an exemplary video decoding system using cross-component chroma linear transform (CCCLT) according to an embodiment of the present invention is provided. The steps shown in the flowchart can be implemented as program code executable on one or more processors (e.g., one or more CPUs) on the decoder side. The steps shown in the flowchart can also be implemented in hardware, such as one or more electronic devices or processors arranged to perform the steps in the flowchart. According to the method, on the decoder side, in step 310, input data associated with a current block in a current image is received, the current image including a luma component and two or more second chroma components in a second chroma format, wherein the input data includes encoded data associated with the current block to be decoded, and wherein a forward cross-component chroma linear transform (CCCLT) is applied in the encoding process to two or more first chroma components of a first chroma format to generate the two or more second chroma components of the second chroma format. In step 320, during the reconstruction process, a reverse CCCLT is applied to two or more second chroma components in the reconstructed second chroma format to generate two or more first chroma components in the reconstructed first chroma format. In step 330, an output video comprising two or more first chroma components in the reconstructed first chroma format is provided.

[0070] Figure 4A flowchart illustrating an exemplary video coding system using cross-component chroma linear transform (CCCLT) according to an embodiment of the present invention is shown. On the encoder side, in step 410, input data associated with the current block in the current image is received, the input data including a luma component and two or more first chroma components, wherein the input data includes pixel data to be encoded, and wherein the two or more first chroma components are in a first chroma format. In step 420, a forward cross-component chroma linear transform (CCCLT) is applied to the two or more first chroma components to generate two or more second chroma components in a second chroma format, wherein the forward CCCLT is applied during the encoding process to convert the chroma components from the original format to another format before applying transform coding. In step 430, during the reconstruction process, a reverse CCCLT is applied to the two or more second chroma components reconstructed in the second chroma format to generate two or more first chroma components reconstructed in the first chroma format. In step 440, two or more first chroma components of the reconstructed first chroma format and / or two or more second chroma components of the reconstructed second chroma format are provided for processing one or more subsequent blocks of the current block.

[0071] The flowchart shown is intended to illustrate examples of video encoding and decoding according to the present invention. Those skilled in the art can modify each step, rearrange the steps, split the steps, or combine the steps to practice the invention without departing from its spirit. In this disclosure, specific syntax and semantics are used to illustrate examples of embodiments of the invention. Those skilled in the art can practice the invention by substituting equivalent syntax and semantics without departing from its spirit.

[0072] The above description is intended to enable those skilled in the art to practice the invention in the context of specific applications and their requirements. Various modifications to the described embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments. Therefore, the invention is not intended to be limited to the specific embodiments shown and described, but rather to be given the broadest scope consistent with the principles and novel features disclosed herein. In the above detailed description, various specific details have been set forth to provide a thorough understanding of the invention. However, the invention can be practiced by those skilled in the art.

[0073] As described above, embodiments of the present invention can be implemented through various hardware, software code, or a combination of both. For example, embodiments of the present invention may be one or more circuits integrated into a video compression chip, or program code integrated into video compression software to perform the processes described herein. Embodiments of the present invention may also be program code to be executed on a digital signal processor (DSP) to perform the processes described herein. The present invention may also relate to multiple functions performed by a computer processor, digital signal processor, microprocessor, or field-programmable gate array (FPGA). These processors can be configured to perform specific tasks according to the invention, defining the specific methods embodied in the invention by executing machine-readable software code or firmware code. The software code or firmware code can be developed in different programming languages ​​and different formats or styles. The software code can also be compiled for different target platforms. However, different code formats, styles, and languages ​​of the software code, as well as other means of configuring the code to perform tasks according to the invention, do not depart from the spirit and scope of the invention.

[0074] This invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The examples described are to be considered illustrative in all respects, not limiting. Therefore, the scope of the invention should be indicated by the appended claims rather than the foregoing description. All variations within the meaning and equivalence of the claims should be included within its scope.

Claims

1. A method of video decoding, comprising: receiving input data associated with a current block in a current picture, the current picture comprising a luma component and two or more second chroma components of a second chroma format, wherein the input data comprises coded data associated with the current block to be decoded, wherein, in an encoding process, a forward cross-component chroma linear transform is applied to two or more first chroma components of a first chroma format to generate the two or more second chroma components of the second chroma format; in a reconstruction process, applying an inverse cross-component chroma linear transform to reconstructed two or more second chroma components of the second chroma format to generate reconstructed two or more first chroma components of the first chroma format; and providing an output video comprising the reconstructed two or more first chroma components of the first chroma format.

2. The video decoding method of claim 1, wherein, the inverse cross-component chroma linear transform is applied before a deblocking process, after a deblocking process, and before a sample adaptive offset process, after a sample adaptive offset process, and before an adaptive loop filtering process, or after an adaptive loop filtering process.

3. The video decoding method of claim 1, wherein, a flag is parsed to decide whether the forward cross-component chroma linear transform is applied to the two or more first chroma components of the first chroma format and whether the inverse cross-component chroma linear transform is applied to the reconstructed two or more second chroma components.

4. The video decoding method of claim 1, wherein, a plurality of transform parameters of the inverse cross-component chroma linear transform is decided per sequence, per picture, per coding tree unit, or per coding unit.

5. The video decoding method of claim 4, wherein, one or more transform parameters of the plurality of transform parameters of the inverse cross-component chroma linear transform is selected from a plurality of candidate parameters of the one or more transform parameters.

6. The video decoding method of claim 4, wherein, the plurality of transform parameters of the inverse cross-component chroma linear transform is received from a bitstream at a sequence parameter set, a picture parameter set, a coding tree unit, or a coding unit level.

7. The video decoding method of claim 4, wherein, one or more transform parameters of the inverse cross-component chroma linear transform is inherited from a neighboring coding tree unit or coding unit.

8. The video decoding method of claim 7, wherein, whether to allow inheritance of the one or more transform parameters for the inverse cross-component chroma linear transform is indicated by a flag.

9. The video decoding method of claim 7, wherein, the one or more transform parameters is inherited from a candidate, the candidate being represented by an index.

10. The video decoding method of claim 1, wherein, the forward cross-component chroma linear transform is applied to the two or more first chroma components in the first chroma format and the inverse cross-component chroma linear transform is applied to the reconstructed two or more second chroma components only when the current block is located in an I slice, a P slice, or a B slice.

11. The video decoding method of claim 1, wherein, whether to use the application of the forward cross-component chroma linear transform and the application of the inverse cross-component chroma linear transform is decided implicitly using one or more templates outside a current coding tree unit or coding unit.

12. A video decoding apparatus, the apparatus comprising one or more electronics or processors configured to: receiving input data associated with a current block in a current picture, the current picture comprising a luma component and two or more second chroma components of a second chroma format, wherein, The input data comprises coded data associated with the current block to be decoded, wherein, in an encoding process, a forward cross-component linear color transform is applied to two or more first chroma components of a first chroma format to generate the two or more second chroma components of the second chroma format; in a reconstruction process, a reverse cross-component linear color transform is applied to the reconstructed two or more second chroma components of the second chroma format to generate reconstructed two or more first chroma components of the first chroma format; and an output video is provided comprising the reconstructed two or more first chroma components of the first chroma format.

13. The video decoding apparatus of claim 12, wherein, Whether to use the application of the forward cross-component linear color transform and the application of the reverse cross-component linear color transform is implicitly decided by using one or more templates outside a current coding tree unit or coding unit.

14. A video encoding method, comprising: receiving input data associated with a current block in a current picture, the input data comprising a luma component and two or more first chroma components, wherein the input data comprises pixel data to be encoded, and wherein the two or more first chroma components are in a first chroma format; applying a forward cross-component linear color transform to the two or more first chroma components to generate two or more second chroma components in a second chroma format, wherein the forward cross-component linear color transform is applied in an encoding process to convert the chroma components from an original format to another format; in a reconstruction process, a reverse cross-component linear color transform is applied to the reconstructed two or more second chroma components of the second chroma format to generate reconstructed two or more first chroma components of the first chroma format; and providing the reconstructed two or more first chroma components of the first chroma format and / or the reconstructed two or more second chroma components of the second chroma format for processing one or more subsequent blocks of the current block.

15. The video coding method of claim 14, wherein, The forward cross-component linear color transform is applied before motion compensated temporal filtering, after motion compensated temporal filtering, or after chroma residual joint coding decision and before low complexity rate-distortion optimization.

16. The video coding method of claim 15, wherein, A first reconstruction buffer is used for a first encoding process comprising the application of the forward cross-component linear color transform and the application of the reverse cross-component linear color transform, a second reconstruction buffer is used for a second encoding process not comprising the application of the forward cross-component linear color transform and not comprising the application of the reverse cross-component linear color transform, and the application of the forward cross-component linear color transform and the application of the reverse cross-component linear color transform are selected if better performance is achieved by using the application of the forward cross-component linear color transform and the application of the reverse cross-component linear color transform.

17. The video coding method of claim 14, wherein, A first pass is performed for a first encoding process including the application of the forward cross-component linear color transform and the application of the inverse cross-component linear color transform, a second pass is performed for a second encoding process not including the application of the forward cross-component linear color transform and not including the application of the inverse cross-component linear color transform, and the application of the forward cross-component linear color transform and the application of the inverse cross-component linear color transform are selected if the first pass achieves better performance.

18. The video coding method of claim 17, wherein, The first pass and the second pass are performed at a sequence parameter set, a picture parameter set, a coding tree unit, or a coding unit level.

19. The video coding method of claim 14, wherein, Whether to use the application of the forward cross-component linear color transform and the application of the inverse cross-component linear color transform is implicitly decided by using one or more templates outside a current coding tree unit or coding unit.

20. A video encoding apparatus comprising one or more electronic devices or processors configured to: receiving input data associated with a current block in a current picture, the input data including a luma component and two or more first chroma components, wherein, The input data comprises pixel data to be encoded, and wherein the two or more first chroma components are in a first chroma format; applying a forward cross-component linear color transform to the two or more first chroma components to generate two or more second chroma components in a second chroma format, wherein the forward cross-component linear color transform is applied in an encoding process to convert the chroma components from an original format to another format; in a reconstruction process, applying an inverse cross-component linear color transform to the reconstructed two or more second chroma components in the second chroma format to generate reconstructed two or more first chroma components in the first chroma format; and providing the reconstructed two or more first chroma components in the first chroma format and / or the reconstructed two or more second chroma components in the second chroma format for processing one or more subsequent blocks of the current block.

Images