Secondary conversion of joint components

Abstract

To provide a method for decoding an encoded video bitstream using at least one processor.SOLUTION: A method for decoding an encoded video bitstream includes: obtaining an encoded video bitstream including encoded color components; entropy-parsing the encoded color components; dequantizing the color components and obtaining transform coefficients of the color components; applying a joint component secondary transform (JCST) on the transform coefficients of the color components, thereby generating JCST outputs; performing a backward transform on the JCST outputs, thereby obtaining residual components of the color components; and decoding the encoded video bitstream on the basis of the residual components of the color components.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 020,280, filed May 5, 2020, and U.S. Patent Application No. 17 / 072,606, filed October 16, 2020, the entire contents of which are incorporated herein by reference.

[0002] This disclosure generally relates to the field of data processing, and more specifically to video encoding and decoding. Even more specifically, embodiments of this disclosure relate to a joint components secondary transform (JCST) technique for encoding residuals from multiple color components, e.g., residuals from two chroma components.

Background Art

[0003] AOMedia Video 1 (AV1) is an open video coding format designed for video transmission over the Internet. As a successor to VP9, AV1 was established in 2015 and developed by the Alliance for Open Media (AOMedia), which consists of semiconductor companies, video - on - demand providers, video content producers, software development companies, and web browser vendors. Many components of the AV1 project were supplied from previous research activities by AOMedia members. And individual contributors had started experimental technology platforms years ago. For example, Xiph’s / Mozilla’s Daala made its code public in 2010. Google's experimental VP9 evolution project VP10 was announced on September 12, 2014. Cisco's Thor was made public on August 11, 2015.

[0004] Built on the VP9 codebase, AV1 incorporates other technologies, some of which were developed in these experimental forms. The first version of the AV1 reference codec (0.1.0) was released on April 7, 2016. AOMedia announced the release of the AV1 bitstream specification on March 28, 2018, along with a reference software-based encoder and decoder. Verified version 1.0.0 of the AV1 specification was released on June 25, 2018. Verified version 1.0.0 of the AV1 specification was released on January 8, 2019, along with Errata 1. The AV1 bitstream specification includes the reference video codec. [Overview of the Initiative] [Means for solving the problem]

[0005] In AV1, the predicted residual signals generated for the chroma channel, such as Cb and Cr, are highly correlated with each other. Therefore, it is expected that the coding of residuals can be further improved by reducing the statistical redundancy between the predicted residuals of Cb and Cr.

[0006] Embodiments of this disclosure provide solutions to the aforementioned problems.

[0007] For example, a method for decoding an encoded video bitstream using at least one processor, comprising the steps of: obtaining an encoded video bitstream containing encoded color components; performing entropy analysis on the encoded color components; dequantizing the color components to obtain conversion coefficients for the color components; generating a JCST output by applying a Joint Component Quadratic Transform (JCST) to the conversion coefficients for the color components; obtaining residual components for the color components by performing an inverse transform on the JCST output; and decoding the encoded video bitstream based on the residual components for the color components. [Brief explanation of the drawing]

[0008] [Figure 1] This is a schematic diagram of the coding coefficients covered by the local template. [Figure 2] This is a block diagram of a communication system according to an embodiment. [Figure 3] This figure shows the arrangement of G-PCC (graph-based point cloud compression), G-PCC compressors, and G-PCC decompressors in an environment according to an embodiment. [Figure 4] This is a schematic diagram of an encoder / decoder scheme according to an embodiment. [Figure 5] This is a schematic diagram of an encoder / decoder scheme according to an embodiment. [Figure 6] This is a schematic diagram of pairs of Cb and Cr conversion coefficients derived from two 4x2 size blocks according to an embodiment. [Figure 7] This is a schematic diagram of JCST applied to two 4x2 size Cb and Cr blocks according to an embodiment. [Figure 8] This is a schematic diagram of JCST utilizing the four-point transformation according to the embodiment. [Figure 9] This is a flowchart showing the decoding method according to the embodiment. [Figure 10] This figure shows a computer system suitable for implementing each embodiment. [Modes for carrying out the invention]

[0009] Embodiments described herein provide methods and apparatus for encoding and / or decoding image data.

[0010] Residual coding using AV1

[0011] For each defined conversion unit, the AV1 coefficient encoder starts with a skip symbol, followed by the conversion kernel type and, if conversion coding is not skipped, the end-of-block (EOB) position of all non-zero coefficients. Next, each coefficient value can be mapped to multiple hierarchical diagrams and symbols. Here, the symbol layer covers the coefficient symbols and three hierarchical levels, and each coefficient value corresponds to a different hierarchical range of the coefficient, namely the lower, middle, and upper layers. The lower layer corresponds to the range of 0 to 2, the middle layer to the range of 3 to 14, and the upper layer to the range of 15 and above.

[0012] After the EOB position is encoded, the lower layer, which indicates whether the magnitude of the coefficient is between 0 and 2, and the middle layer, which indicates whether the magnitude is between 3 and 14, are encoded together in reverse scan order. Next, the symbol layer and the upper layer, which indicates residual values ​​of 14 or greater, are encoded together in forward scan order. The remaining part is entropy coded using the Exp-Golomb code. AV1 employs the conventional zigzag scan order.

[0013] Such separation allows for the assignment of rich context models to the lower layers, considering transformation directions such as bidirectional, horizontal, and vertical, transformation size, and up to five neighboring coefficients for improved compression efficiency with a reasonable context model size. In the middle layers, a context model similar to that of the lower layers is used, but the number of context neighboring coefficients is reduced from five to two. In the upper layers, encoding is performed using Exp-Golomb codes without using context models. In the symbol layer, symbols other than DC symbols are encoded using the DC symbols of the neighboring transformation units as context information. Other symbol bits are encoded directly without using context models.

[0014] In Versatile Video Coding (VVC), a coding block is first divided into 4x4 subblocks, and the subblocks within the coding block, along with the conversion coefficients within those subblocks, are encoded according to a predefined scan order. For subblocks with at least one non-zero conversion coefficient, the coding of the conversion coefficient is divided into four scans.

[0015] For example, absLevel is assumed to be the absolute value of the current conversion coefficient. In the first pass, the syntactic elements sig_coeff_flag (indicating that absLevel is greater than 0), par_level_flag (indicating the parity of absLevel), and rem_abs_gt1_flag (indicating that (absLevel-1)>> 1 is greater than 0) are encoded; in the second pass, the syntactic element rem_abs_gt2_flag (indicating that absLevel is greater than 4) is encoded; in the third pass, the remaining value of the coefficient level (called abs_remainder) is invoked; and if necessary, in the fourth pass, the symbolic information is encoded.

[0016] To utilize the correlation between transformation coefficients, the encoded coefficients covered by the local template shown in Figure 1 are used for context selection of the current coefficient. In the figure, the position (101) shown in black indicates the position of the current transformation coefficient. The position (102) shown in shaded area indicates its five neighboring coefficients. Here, absLevel1[x][y] represents the partially reconstructed absolute level of the coefficient at position (x, y) after the first pass, d represents the diagonal position of the current coefficient (d=x+y), numSig represents the number of non-zero coefficients in the local template, and sumAbs1 represents the sum of the partially reconstructed absolute levels absLevel1[x][y] of the coefficients covered by the local template.

[0017] When encoding the current coefficient sig_coeff_flag, the context model index is selected according to sumAbs1 and the diagonal position d. More specifically, for the Luma component, the context model index is determined according to Equation 1 below: ctxSig = 18 * max(0, state - 1) + min(sumAbs1, 5 + (d < 2? 12 : (d < 5? 6 : 0))). This is equivalent to the following Equation 2 and Equation 3. Equation 2: ctxIdBase = 18 * max(0, state - 1 + (d < 2? 12 : (d < 5? 6 : 0)). Equation 3: ctxSig = ctxIdSigTable[min(sumAbs1, 5)] + ctxIdBase.

[0018] For Chroma, the context model index is determined according to Equation 4 below: ctxSig = 12 * max(0, state - 1 + min(sumAbs1, 5 + (d < 2? 6 : 0)). This is equivalent to the following Equation 5 and Equation 6. Equation 5: ctxIdBase = 12 * max(0, state - 1 + (d < 2? 6 : 0). Equation 6: ctxSig = ctxIdSigTable[min(sumAbs1, 5)] + ctxIdBase.

[0019] Here, when dependent quantization is used and the state is derived using the state conversion process, a scalar quantizer is used. The table ctxIdSigTable stores the context model index offsets, ctxIdSigTable[0~5] = {0, 1, 2, 3, 4, 5}.

[0020] When encoding the par_level_flag which is the current coefficient, the context model index is selected according to sumAbs1, numSig, and the diagonal position d. More specifically, for the Luma component, the context model index is determined according to the following formula 7: ctxPar = 1 + min(sumAbs1 - numSig, 4) + (d == 0? 15 : (d < 3? 10 : (d < 10? 5 : 0))). This is equivalent to the following formulas 8 and 9. Formula 8: ctxIdBase = (d == 0? 15 : (d < 3? 10 : (d < 10? 5 : 0))). Formula 9: ctxPar = 1 + ctxIdTable[min(sumAbs1 - numSig, 4)] + ctxIdBase. For chroma, the context model index is determined according to the following formula 10: ctxPar = 1 + min(sumAbs1 - numSig, 4) + (d == 0? 5 : 0). This is equivalent to the following formulas 11 and 12. Formula 11: ctxIdBase = (d == 0? 5 : 0). Formula 12: ctxPar = 1 + ctxIdTable[min(sumAbs1 - numSig, 4)] + ctxIdBase.

[0021] Here, the table ctxIdTable stores the context model index offsets, ctxIdTable[0~4] = {0, 1, 2, 3, 4}. When encoding the rem_abs_gt1_flag and rem_abs_gt2_flag which are the current coefficients, the context model index is determined in the same way as in the case of par_level_flag. ctxGt1 = ctxPar and ctxGt2 = ctxPar (Formula 13).

[0022] Different context model sets are used for rem_abs_gt1_flag and rem_abs_gt2_flag. Therefore, even when ctxGt1 is equal to ctxGt2, the context model used for rem_abs_gt1_flag is different from the context model of rem_abs_gt2_flag.

[0023] Residual coding for Transform Skip Mode (TSM) and Differential Pulse-Code Modulation (DPCM)

[0024] In order to adapt the residual coding to the statistical and signal characteristics of the residual levels of Block Differential Pulse-Code Modulation (BDPCM) with transform skips representing quantized predicted residuals (spatial domain), it is proposed to modify the residual coding process as follows, in addition to the residual coding scheme described in the residual coding section of AV1 above, and apply it to TSM and BDPCM modes.

[0025] The following describes the three encoding paths. In the first encoding path, sig_coeff_flag, coeff_sign_flag, abs_level_gt1_flag, and par_level_flag are encoded in a single pass. In the second encoding path, abs_level_gtX_flag is encoded, where X can be 3, 5, 7, ... In the third pass, the remaining coefficient levels are encoded. The encoding paths are operated at the coefficient group (CG) level, meaning that for each CG, three encoding paths are executed.

[0026] There are no valid scan positions. The residual signal reflects the predicted spatial residual, and since no energy compression by transformation is performed on the TS, there is no high probability of a trailing zero or meaningless level occurring in the lower right corner of the transformation block. Therefore, in this case, the last valid scan position signaling is omitted. Instead, the first subblock to be processed is the subblock located in the lower right corner of the transformation block.

[0027] Next, we will discuss the Coding Block Flag (CBF). If there is no last valid scan position signaling, the TS's subblock CBF signaling with coded_sub_block_flag must be modified as follows:

[0028] Due to quantization, the aforementioned meaningless sequences may partially occur within the transformation block. Therefore, the last valid scan position is removed as described above, and the coded_sub_block_flag is encoded for all subblocks.

[0029] The coded_sub_block_flag for the subblock covering the DC frequency position (the upper left subblock) is considered a special case. In VVC Draft 3, the coded_sub_block_flag for this subblock is not signaled and is presumed to be equal to 1. If the last valid scan position is in another subblock, there is at least one valid level other than the DC subblock. Therefore, the coded_sub_block_flag for this subblock is presumed to be equal to 1, although the DC subblock may contain only zero / invalid levels. Since the TS does not have information on the last scan position, the coded_sub_block_flag for each subblock is signaled. This includes the coded_sub_block_flag for the DC subblock, unless all other coded_sub_block_flag syntactic elements are already equal to 0. In this case, the DC coded_sub_block_flag is presumed to be equal to 1 (inferDcSbCbf=1). Since this DC subblock has at least one valid level, the sig_coeff_flag syntax element at the first position at (0,0) is not signaled, and if all other sig_coeff_flag syntax elements in the DC subblock are equal to 0, then it is derived to be equal to 1 (inferSbDcSigCoeffFlag=1).

[0030] The context modeling of coded_sub_block_flag can be changed. The context model index can be calculated as the sum of the coded_sub_block_flag on the right and the coded_sub_block_flag below the current subblock, but not as a logical OR of the two.

[0031] The following describes the sig_coeff_flag context modeling. The local template within the sig_coeff_flag context modeling can be modified to include only the neighbor to the right of the current scan position (NB0) and its neighbor below it (NB1). The offset of the context model represents the number of valid adjacencies, sig_coeff_flag[NB0] + sig_coeff_flag[NB1]. Thus, the selection of different contexts is set according to the diagonal d in the current transformation block (d is removed). This results in three context models and a single set of context models for coding the sig_coeff_flag flag.

[0032] The context modeling for abs_level_gt1_flag and par_level_flag is described below. A single context model is employed for both abs_level_gt1_flag and par_level_flag.

[0033] The abs_remainder coding is described below. The empirical distribution of the absolute level residuals of a transformation skip typically fits a Laplacian or geometric distribution, but there may be greater instability than at the absolute level of the transformation coefficients. In particular, the variance within the window of continuous realization is high with respect to the absolute level residuals. This allows for the binarization and context modeling of the abs_remainder syntax to be modified as follows:

[0034] Use a higher cutoff value for binarization. That is, using a dedicated context model for the transition point from coding to the rice code of abs_remainder using sig_coeff_flag, abs_level_gt1_flag, par_level_flag, and abs_level_gt3_flag, and for each bin position, results in higher compression efficiency. A larger cutoff generates more "greater than X" flags, such as introducing abs_level_gt5_flag, abs_level_gt7_flag, etc., until the cutoff is reached. The cutoff itself is fixed at 5 (numGtFlags=5).

[0035] Similar to the local template for the sig_coeff_flag modeling context, the template for rice parameter derivation can be modified. This means that only neighbors to the left and below the current scan position will be considered.

[0036] The following describes the coeff_sign_flag context modeling. Due to the instability within the code sequence and the fact that prediction residuals are often biased, a context model can be used to encode symbols even when the global empirical distribution is nearly uniform. A single dedicated context model can be used to encode symbols, and the symbols can be parsed after sig_coeff_flag to aggregate all the binary numbers encoded in the context.

[0037] The following describes the limitations on the number of context-encoded binaries. The total number of context-encoded binaries for each Transform Unit (TU) is limited to the TU area size multiplied by 2. For example, for 16 × 8 TUs, the maximum number of context-encoded binaries is 16 × 8 × 2 = 256. The budget for the number of context-encoded binaries is consumed at the TU level. That is, rather than individual budgets for the number of context-encoded binaries per CG, all CGs in the current TU share a single budget for the number of context-encoded binaries.

[0038] Joint coding of chroma residuals

[0039] VVC Draft 6 supports a mode for joint coding of chroma residuals. The use (activation) of the joint chroma coding mode is indicated by the TU level flag tu_joint_cbcr_residual_flag, and the selected mode is implicitly indicated by the chroma CBF. The flag tu_joint_cbcr_residual_flag is considered to be present if either or both of the chroma CBFs of the TU are equal to 1. In the PPS and slice header, the chroma quantization parameter (QP) offset value is signaled for the joint chroma residual coding mode to distinguish it from the chroma QP offset value signaled for the normal chroma residual coding mode. These chroma QP offset values ​​are used to derive the chroma QP value of the block coded by the joint chroma residual coding mode. When the corresponding joint chroma coding mode (mode 2 in Table 1) is active in the TU, the chroma QP offset is added to the applied luminance during the quantization and decoding period of that TU to obtain the chroma QP. For other modes (modes 1 and 3 in Table 1), the chroma QP is derived in the same way as for conventional Cb or Cr blocks. The reconstruction process of the chroma residuals (resCb and resCr) from the transmitted transformation block is shown in Table 1. When this mode is active, a signal is sent to a single-joint chroma residual block (resJointC [x] [y] Table 1), and the Cb residual block (resCb) and the Cr residual block (resCr) are derived considering information such as tu_cbf_cb, tu_cbf_cr, and CSign (sign value specified in the slice header).

[0040] The three joint chroma coding modes described above are supported only in intra-coded coding units (CUs). In intercoded CUs, only mode 2 is supported. Therefore, in the case of intercoded CUs, the syntax element tu_joint_cbcr_residual_flag exists only when both chroma cbfs are equal to 1.

[0041] [Table 1]

[0042] Here, the value CSign is the symbol value (+1 or -1) specified in the slice header, and resJointC[][] is the residual sent.

[0043] Referring now to Figure 2, which is a block diagram of a communication system 200 according to an embodiment, the communication system 200 may include at least two terminals 210 and 220 interconnected via a network 250. When transmitting data in one direction, the first terminal 210 can encode data at its local location for transmission to the second terminal 220 via the network 250. The second terminal 220 can receive the encoded data from the first terminal 210 via the network 250, decode the encoded data, and display the decoded data. One-way transmission of data may be common in media serving applications, etc.

[0044] Figure 2 further shows terminals 230 and 240 provided to support, for example, the bidirectional transmission of encoded data that may occur during a video conference. In the case of bidirectional data transmission, each terminal 230 or 240 can encode data captured at its local location and transmit it to other terminals over the network 250. Each terminal 230 or 240 can also receive encoded data transmitted by other terminals, decode the encoded data, and display the decoded data on a local display device.

[0045] In Figure 2, terminals 210-240 may be shown as servers, personal computers, and smartphones, but the principles of the embodiment are not limited to these. The embodiment is suitable for laptop computers, tablet computers, media players, and / or dedicated video conferencing devices. Network 250 represents any number of networks that transmit encoded data between terminals 210-240, including, for example, wired and / or wireless communication networks. Communication network 250 can exchange data over circuit-switched and / or packet-switched channels. Typical networks include telecommunications networks, local area networks, wide area networks, and / or the Internet. For the purposes of this discussion, the architecture and topology of network 250 may not be important to the operation of the embodiment unless described below herein.

[0046] Figure 3 shows the arrangement of the G-PCC compressor 303 and G-PCC decompressor 310 in an environment according to an embodiment. The disclosed subject matter is equally applicable to other mountable applications, such as video conferencing, digital TV, and storage of compressed data on digital media such as CDs, DVDs, and memory sticks.

[0047] The streaming system 300 may include a source 310, for example, a digital camera, and a capture subsystem 313 capable of creating, for example, uncompressed data 302. Larger data volumes of data 302 may be processed by a G-PCC compressor 303 coupled to the source 301. The G-PCC compressor 303 may include hardware, software, or a combination thereof, to enable the realization or implementation of embodiments of the disclosed subject matter, which are described in more detail below. Smaller encoded data 304 can be stored in the streaming server 305 for subsequent use. One or more streaming clients 306 and 308 can access the streaming server 305 to retrieve copies 307 and 309 of the encoded data 304. A client terminal 306 may include a G-PCC decompressor 310 that decodes an incoming copy 307 of the encoded data and creates outgoing data 311 that can be displayed on a display 312 or other display device (not shown). In some streaming systems, encoded data 304, 307, and 309 can be encoded according to video coding / compression standards. Examples of these standards include those developed by MPEG for G-PCC.

[0048] Embodiments of this disclosure allow for the joint application of a quadratic transformation to the transformation coefficients of multiple color components. The joint transformation scheme proposed herein may be called a Joint Component Quadratic Transform (JCST). Figure 4 shows an encoder scheme in which JCST is applied to two color components, where JCST is performed after the forward transformation and before quantization.

[0049] Embodiments of this disclosure, as shown in Figure 5, allow JCST to be performed after the inverse quantization transform and before the inverse transform.

[0050] Referring to Figure 9, LRRX is located in the first block 901. Method 900 includes the step of obtaining an encoded video bitstream containing the encoded color components.

[0051] In the second block 902, method 900 includes the step of performing entropy analysis on the encoded color components.

[0052] In the third block 903, method 900 includes the step of dequantizing the color components and obtaining conversion coefficients for the color components.

[0053] In the fourth block 904, method 900 includes the step of generating a JCST output by applying a Joint Component Quadratic Transform (JCST) to the conversion coefficients of the color components.

[0054] According to the embodiment, a fifth block 905 may be provided. In the fifth block 905, method 900 may include the step of determining the residual components of the color components by performing an inverse conversion on the JCST output.

[0055] According to the embodiment, a sixth block 906 may be provided. In the sixth block 906, method 900 may include the step of decoding the encoded video bitstream based on the residual components of the color components.

[0056] According to embodiments, this method can be reversed as an encoding method. In practice, this specification may refer to specific encoding or decoding schemes, but these descriptions are not limited to specific encoding or decoding schemes. That is, they are equally applicable to both encoding and decoding schemes.

[0057] In one embodiment, the input to JCST may be Cb and Cr conversion coefficients.

[0058] In another embodiment, the input to the JCST may be Y, Cb, and Cr conversion coefficients.

[0059] In one embodiment, JCST may be performed element by element so that JCST is performed for each pair of Cb and Cr conversion coefficients located at the same coordinates. Figure 6 illustrates pairs of Cb and Cr conversion coefficients derived from two 4x2 size blocks.

[0060] In one embodiment, JCST can be a two-point transformation, and the input can be a pair of Cb and Cr transformation coefficients located at the same coordinates.

[0061] In one embodiment, JCST may be a two-point transformation, and the output may be a pair of quadratic transformation coefficients that replace a pair of Cb and Cr transformation coefficients.

[0062] In one embodiment, the output pairs of Cb and Cr conversion coefficients may be placed in the same position as the pairs of Cb and Cr conversion coefficients used as inputs to the JCST. Figure 7 illustrates a JCST applied to two 4x2 sized Cb and Cr blocks. The outputs of the JCST, which construct yet another two 4x2 sized Cb and Cr blocks, are further quantized / dequantized and entropy coded / analyzed by an encoder / decoder.

[0063] In one embodiment, the output of the JCST may be less than the input. For example, the input may be a pair of Cb and Cr coefficients, while the output may be just a single quadratic transformation coefficient.

[0064] In one embodiment, the transformations applied to JCST may include, but are not limited to, the Hadamard transform, discrete cosine / sine transform, and data-driven transforms such as KLT and LGT (line graph transform).

[0065] In one embodiment, the input to the JCST can be derived from one or more pairs (e.g., three pairs) of different color components located at different coordinates.

[0066] In one embodiment, JCST may be a four-point transformation, and the input may be two pairs of Cb and Cr transformation coefficients located at the same coordinates. An example is shown in Figure 8.

[0067] In one embodiment, the output may be one or more quadratic conversion coefficients that replace one or more pairs of Cb and Cr conversion coefficients.

[0068] In one embodiment, the output of JCST may be less than the input. For example, the input may be one or more pairs of Cb and Cr coefficients, while the output may be only two quadratic transformation coefficients.

[0069] In one embodiment, the transformations applied to JCST may include, but are not limited to, the Hadamard transform, discrete cosine / sine transform, and data-driven transforms such as KLT and LGT (line graph transform).

[0070] In one embodiment, JCST can be applied to block sizes within a limited range.

[0071] In one example, JCST may be applied to block sizes below a predetermined threshold, where block size can refer to block width, block height, block area size, block width and height, and the maximum (or minimum) values ​​of block width and height.

[0072] In one example, JCST may be applied to block sizes above a predetermined threshold, where block size can refer to block width, block height, block area size, block width and height, and the maximum (or minimum) values ​​of block width and height.

[0073] In one embodiment, whether JCST is applied may be indicated at the translation block level by a JCST flag.

[0074] In one embodiment, the JCST flag may be signaled after the conversion coefficient.

[0075] In one embodiment, the JCST flag may be signaled only if at least one color component to which JCST is applied has at least one non-zero coefficient.

[0076] In one embodiment, the JCST flag is indicated only if each color component to which JCST is applied has at least one non-zero coefficient.

[0077] In one embodiment, the JCST flag is signaled only if the total number of non-zero coefficients of the color components to which JCST is applied is greater than a predetermined threshold, for example, 0, 1, 2, 3, or 4.

[0078] In one embodiment, the JCST flag is indicated only if the last non-zero coefficient of the color component to which JCST is applied is located at a point along the scan order that is greater than a predetermined threshold, for example, 0, 1, 2, 3, or 4.

[0079] In one embodiment, whether JCST applies is indicated by a JCST flag at the CU level (or CB level).

[0080] In one embodiment, whether JCST can be applied is indicated by a signal via high-level syntax. High-level syntax includes, but is not limited to, the Video Parameter Set (VPS), Sequence Parameter Setting (SPS), Picture Parameter Set (PPS), slice header, image header, tile, tile group, or Coding Tree Unit (CTU) levels.

[0081] In one embodiment, when JCST is applied, the primary transformation is of a predetermined transformation type.

[0082] In one example, the given conversion type is DCT-2. In another example, the given conversion type is DST-7.

[0083] In one embodiment, the selection of the transformation to be applied to JCST is determined by the encoding information, where the encoding information includes, but is not limited to, the primary transformation type, intra-prediction mode such as intra-prediction direction / angle, inter-prediction mode, whether intra-block copy (IntraBC) is applied, whether palette mode is applied, whether DPCM mode is applied, motion vector information (direction, magnitude), whether sub-block motion is applied, and whether warp motion (affine motion) is applied.

[0084] In one embodiment, the context used for entropy coding of the flag indicating whether JCST applies depends on neighboring block information from adjacent blocks, where neighboring block information includes, but is not limited to, the information listed above.

[0085] The above technology can be implemented by a video encoder and / or decoder adapted for compression / decompression. The encoder and / or decoder can be implemented in hardware, software, or any combination thereof, and if software is present, it may be stored in one or more non-temporary computer-readable media. For example, each of the method (or embodiment), encoder, and decoder may be implemented by a processing circuit system (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-temporary computer-readable media.

[0086] The above technologies can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media. For example, Figure 10 shows a computer system 900 suitable for carrying out some embodiments of the present disclosure.

[0087] Computer software can be coded using any suitable machine code or computer language, and mechanisms such as assembly, compilation, and linking can be used to create code that includes instructions that can be executed directly by a computer's central processing unit (CPU), graphics processing unit (GPU), etc., or through interpretation, microcode execution, etc., for any suitable machine code or computer language.

[0088] Instructions can be executed on various types of computers or components, including, for example, personal computers, tablet computers, servers, smartphones, game consoles, and Internet of Things devices.

[0089] The components for the computer system 900 shown in Figure 10 are illustrative and are not intended to limit the scope of use or functionality of computer software implementing embodiments of this disclosure. The arrangement of components should not be construed as having any dependency or requirement relating to any one or combination of components shown in non-limiting embodiments of the computer system 900.

[0090] The computer system 900 may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users through tactile input (e.g., keystrokes, swipes, data glove movements, etc.), voice input (e.g., voice, applause, etc.), visual input (e.g., gestures, etc.), and olfactory input (not shown). The human interface devices may also be used to capture several media that are not necessarily directly related to conscious human input, such as audio (voices, music, ambient sounds, etc.), images (scanned images, photographs taken from still image cameras, etc.), and video (2D video, 3D video including stereoscopic video, etc.).

[0091] Input human interface devices may include one or more of the following (only one of each item is depicted): keyboard 901, mouse 902, trackpad 903, touchscreen 910, data glove, joystick 905, microphone 906, scanner 907, and camera 908.

[0092] The computer system 900 may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through tactile output, sound, light, and smell / taste. Such human interface output devices may include tactile output devices (e.g., tactile feedback via a touchscreen 910, data glove, or joystick 905, but there may also be tactile feedback devices that do not function as input devices). For example, such devices may include audio output devices (speakers 909, headphones (not shown), etc.), visual output devices (screens 910 including cathode ray tube (CRT) screens, liquid crystal display (LCD) screens, plasma screens, organic light-emitting diode (OLED) screens, etc., each with or without touchscreen input functionality, each with or without haptic feedback functionality, some of which may be able to output two-dimensional visual output or three-dimensional or more output by a stereographic output method), virtual reality glass (not shown), holographic displays and smoke tanks (not shown), and printers (not shown).

[0093] The computer system 900 may also include human-accessible storage devices and associated media, such as optical media including ROM / RW 920 with media 921 such as CD / DVD, a thumb drive 922, a removable hard drive or solid-state drive 923, legacy magnetic media such as tape or floppy disk (not shown), and dedicated ROM / ASIC / PLD-based devices such as security dongles (not shown).

[0094] Furthermore, it should be understood by those skilled in the art that the term “computer-readable medium” as used in relation to the subject matter currently disclosed does not include transmission media, carrier waves, or other transient signals.

[0095] The computer system 900 may also include interfaces to one or more communication networks. These networks may be, for example, wireless, wired, or optical. They may further be local, wide-area, metropolitan, vehicle and industrial, real-time, or latency-tolerant. Examples of networks include local area networks such as Ethernet, cellular networks including wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., wired or wireless wide-area digital television networks including cable television, satellite television, and terrestrial broadcast television, and vehicle and industrial networks including CANBus, etc. Certain networks typically require an external network interface adapter connected to a specific general-purpose data port or peripheral bus 949 (for example, a USB port on the computer system 900). Other networks are generally integrated into the core of the computer system 900 by connecting to the system bus described below (for example, an Ethernet interface or cellular network interface to a PC computer system). Using any of these networks, the computer system 900 can communicate with other entities. Such communications may be unidirectional (e.g., broadcast television), unidirectional (e.g., to the CANbus of a specific CANbus device), or bidirectional (e.g., to other computer systems using a local or wide-area digital network). Such communications include communications to cloud computing environments 955. Specific protocols and protocol stacks may be used for each of the networks or network interfaces described above.

[0096] The aforementioned human interface device, human-accessible storage device, and network interface 954 can be connected to the core 940 of the computer system 900.

[0097] The core 940 may include one or more central processing units (CPUs) 941, graphics processing units (GPUs) 942, dedicated programmable processing units in the form of field-programmable gate areas (FPGAs) 943, hardware accelerators 944 for specific tasks, and the like. These devices may be connected via a system bus 948, along with read-only memory (ROM) 945, random access memory 946, and internal mass storage 947 such as internal hard drives and SSDs that are not accessible to the user. In some computer systems, the system bus 948 can be accessed in the form of one or more physical plugs, allowing for expansion with additional CPUs, GPUs, etc. Peripherals can be connected to the core's system bus 948 directly or via a peripheral bus 949. Peripheral bus architectures include PCI, USB, etc. A graphics adapter 950 may be included in the core 940.

[0098] The CPU 941, GPU 942, FPGA 943, and accelerator 944 can execute specific instructions that, when combined, constitute the aforementioned computer code. This computer code may be stored in ROM 945 or RAM 946. Transition data may also be stored in RAM 946. Permanent data may be stored, for example, in internal mass storage device 947. By using cache memory that can be closely associated with one or more CPUs 941, GPUs 942, mass storage devices 947, ROMs 945, RAM 946, etc., high-speed storage and retrieval to any memory device can be enabled.

[0099] Computer-readable media may contain computer code for performing various computer operations. The media and computer code may be specifically designed and constructed for the purposes of this disclosure, or may be of a type that is well known and available to vendors in the computer software field.

[0100] As an example, but not limited to, a computer system 900 having an architecture, in particular a core 940, can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) that runs software embodied in one or more tangible computer-readable media. Such computer-readable media may be user-accessible mass storage devices as described above, as well as media related to certain non-transient storage devices such as the core 940, for example, the mass storage device 947 or ROM 945 inside the core. Software that implements various embodiments of this disclosure may be stored in such devices and run by the core 940. The computer-readable media may include one or more storage devices or chips, depending on the specific needs. The software may cause the core 940, in particular the processor (including a CPU, GPU, FPGA, etc.) within it, to perform certain operations or certain parts of certain operations as described herein, including limitations to data structures stored in RAM 946 and modifications to these data structures in accordance with the operations limited by the software. Furthermore, or alternatively, a computer system may provide functionality as a result of logic incorporated into wired logic or circuitry (e.g., accelerator 944), the logic of which may, in place of or in conjunction with software, perform specific processes or specific parts of specific processes described herein. Where necessary, such software may encompass logic, and conversely, such logic may encompass software. Where necessary, such computer-readable medium may encompass circuitry containing executable software (such as an integrated circuit (IC)), circuitry embodying executable logic, or both. This disclosure includes any suitable combination of hardware and software.

[0101] While this disclosure describes several non-limiting embodiments, there are many modifications, substitutions, and alternative equivalents that fall within the scope of this disclosure. It should be understood that, although not expressly shown or described herein, those skilled in the art can devise many systems and methods that embody the principles of the disclosure and thus fall within its spirit and scope.

[0102] ALF: Adaptive Loop Filter APS: Adaptive Parameter Set AV1: AOMedia Video 1 AV2: AOMedia Video 2 CB: Encoding Block CC-ALF: Cross-Component Adaptive Loop Filter CDEF: Constrained Directional Enhancement Filter CU: Encoding Unit, CTU: Coding Tree Unit DPCM: Differential Pulse Coding Modulation DPS: Decryption parameter set HDR: High Dynamic Range HEVC: High Efficiency Video Coding ISP: Intra Subpartition JCCT: Joint Chroma Component Conversion JVET: Joint Video Exploration Team LR: Loop Restoration Filter PDPC: Location-dependent prediction combination PPS: Image Parameter Set PU: Prediction Unit SDR: Standard Dynamic Range SPS: Sequence Parameter Settings TSM: Conversion Skip Mode TU: Conversion Unit VVC: General-Purpose Video Encoding WAIP: Wide-angle intra-prediction VPS: Video Parameter Set [Explanation of Symbols]

[0103] 944 Accelerator 948 System Bus 950 Graphics Adapter 954 Network Interface

Claims

1. A method for decoding a video bitstream using at least one processor, The steps include obtaining an encoded video bitstream that includes encoded Cb and Cr color components, The steps include: performing entropy analysis on the encoded Cb and Cr color components; A step of dequantizing the Cb color component and the Cr color component and obtaining conversion coefficients for the Cb color component and the Cr color component, wherein the conversion components for the Cb color component and the Cr color component each include Cb and Cr conversion coefficients located at the same coordinates in the Cb and Cr conversion block, A step of generating a JCST output by applying a Joint Component Quadratic Transform (JCST) to the conversion coefficients of the Cb and Cr color components for each pair of Cb and Cr conversion coefficients located at the same coordinates in the Cb and Cr conversion block, wherein the JCST is a two-point transformation performed on the Cb and Cr conversion values ​​of each pair located at the same coordinates in the Cb and Cr conversion block, and for each pair of Cb and Cr conversion values ​​located at the same coordinates, when the Cr conversion value is in the Cr conversion block, the Cb conversion value is located at the same coordinates as the Cb conversion block. The steps include: performing an inverse transform on the JCST output to obtain the residual components of the Cb and Cr color components; A method comprising the step of decoding the encoded video bitstream based on the residual components of the Cb color component and the Cr color component.

2. The method according to claim 1, wherein the JCST is applied to a block size within a limited range.

3. A step of obtaining an encoded video sequence (CVS) from the encoded video bitstream, which includes an image unit corresponding to an encoded image, The steps include obtaining an image header (PH) network abstraction layer (NAL) unit included in the aforementioned image unit, The steps include obtaining at least one video coding layer (VCL) NAL unit included in the image unit, The method according to claim 1, comprising the step of analyzing a JCST flag that signals at the conversion block level when the JCST will be applied.

4. A step of obtaining an encoded video sequence (CVS) from the encoded video bitstream, which includes an image unit corresponding to an encoded image, The steps include obtaining an image header (PH) network abstraction layer (NAL) unit included in the aforementioned image unit, The steps include obtaining at least one video coding layer (VCL) NAL unit included in the image unit, The method according to claim 1, comprising the step of analyzing a JCST flag that signals at the CU or CB level when the JCST will be applied.

5. A step of obtaining an encoded video sequence (CVS) from the encoded video bitstream, which includes an image unit corresponding to an encoded image, The steps include obtaining an image header (PH) network abstraction layer (NAL) unit included in the aforementioned image unit, The steps include obtaining at least one video coding layer (VCL) NAL unit included in the image unit, The method according to claim 1, comprising the step of analyzing a JCST flag that signals when the JCST is applied via high-level syntax.

6. The method according to claim 1, wherein the JCST includes a second transformation determined by the encoded information.

7. An apparatus for decoding an encoded video bitstream, An apparatus for performing a method for decoding an encoded video bitstream according to any one of claims 1 to 6.

8. A computer program stored in a non-temporary computer-readable storage medium, A computer program, when executed on at least one processor, that implements a method for decoding an encoded video bitstream as described in any one of claims 1 to 6.

9. A method for storing a video bitstream using at least one processor, A step of generating an encoded video bitstream, A step of acquiring video data comprising multiple video frames, including the current video frame which contains Cb and Cr color components, A step of obtaining residual blocks for each of the Cb color component and the Cr color component, The steps include obtaining Cb and Cr transformation blocks by performing a linear transformation in each of the residual blocks, A step of generating a JCST output by applying a joint component secondary transform (JCST) to the Cb and Cr transform block for each element, wherein the JCST is a two-point transform performed on each pair of Cb and Cr transform values ​​located at the same coordinates in the Cb and Cr transform block, and for each pair of Cb and Cr transform values ​​located at the same coordinates, when the Cr transform value is in the Cr transform block, the Cb transform value is located at the same coordinates as the Cb transform block, The steps include obtaining conversion coefficients by performing quantization on the JCST output, The steps include encoding the conversion coefficients, The step of transmitting the encoded conversion coefficients within a video bitstream is included, Steps and A method comprising storing the encoded video bitstream in a non-temporary computer-readable medium.

Images