Interaction between size transformation and encoding tools

Abstract

This invention provides a method and apparatus for implementing discrete trigonometric transformations based on the maximum transform size. In one embodiment, matrix-based intra-frame prediction is enabled for coding unit sizes up to a specific size, regardless of the maximum transform size. In another embodiment, low-frequency inseparable transforms are used to improve coding gain. The syntax in the bitstream can be used to indicate the coding tools used.

Description

Technical Field

[0001] At least one embodiment of this implementation typically relates to a method or apparatus for video encoding or decoding, compression or decompression. Background Technology

[0002] To achieve high compression efficiency, image and video coding schemes typically employ prediction and transformation, including motion dimension prediction, to utilize spatial and temporal redundancy in video content. Generally, intra-frame or inter-frame prediction is used to leverage intra-frame or inter-frame correlations, and then the difference between the original image and the predicted image (usually represented as prediction error or prediction residual) is transformed, quantized, and entropy-coded. To reconstruct the video, the compressed data is decoded through the inverse process corresponding to entropy coding, quantization, transformation, and prediction.

[0003] In the development of the Universal Video Coding (VVC) standard, the maximum transform size could be varied between 32 and 64. The maximum transform size interacts with other transform coding tools. Summary of the Invention

[0004] At least one embodiment of this implementation generally relates to a method or apparatus for video encoding or decoding, and more specifically to a method or apparatus for interacting between a maximum transform size and a transform encoding tool in a video encoder or video decoder.

[0005] According to a first aspect, a method is provided. The method includes the steps of enabling an encoding tool based on a maximum transform size; performing at least a portion of a discrete trigonometric transform on a subset of samples comprising the block; and encoding the block using the enabled encoding tool.

[0006] According to a second aspect, a method is provided. The method includes the steps of enabling an encoding tool based on a maximum transform size; performing at least a portion of an inverse discrete trigonometric transform on a subset of samples comprising the block; and decoding the block using the enabled encoding tool.

[0007] According to another aspect, an apparatus is provided. The apparatus includes a processor. The processor can be configured to encode video blocks or decode bitstreams by performing any of the methods described above.

[0008] According to another general aspect of at least one embodiment, an apparatus is provided comprising: means according to any embodiment of the decoding embodiment; and at least one of: (i) an antenna configured to receive a signal including a video block; (ii) a band limiter configured to limit the received signal to a band including the video block; and (iii) a display configured to display an output representing the video block.

[0009] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided, the non-transitory computer-readable medium containing data content generated according to any embodiment or variant of the described encoding embodiment or variant.

[0010] According to another general aspect of at least one embodiment, a signal is provided that includes video data generated according to any embodiment or variant of the described encoding embodiment or variant.

[0011] According to another general aspect of at least one implementation scheme, the bitstream is formatted to include data content generated according to any of the described encoding implementation schemes or variations.

[0012] According to another general aspect of at least one embodiment, a computer program product is provided, the computer program product including instructions that, when executed by a computer, cause the computer to perform any of the described coded embodiments or variations.

[0013] These and other aspects, features, and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments read in conjunction with the accompanying drawings. Attached Figure Description

[0014] Figure 1 This demonstrates a standard, general-purpose video compression scheme.

[0015] Figure 2 This demonstrates a standard, general-purpose video decompression scheme.

[0016] Figure 3 An exemplary processor-based subsystem for implementing the aspects described in general is shown.

[0017] Figure 4 An implementation of the method in the described aspect is shown.

[0018] Figure 5 Another implementation of the method in the described aspect is shown.

[0019] Figure 6An exemplary device is shown in the case described. Detailed Implementation

[0020] At least one embodiment of this implementation generally relates to a method or apparatus for video encoding or decoding, and more specifically to a method or apparatus for interacting between a maximum transform size and a transform encoding tool in a video encoder or video decoder.

[0021] The general aspect described here is in the field of video compression. This general aspect is the interaction between the maximum transform size and other transform coding tools, where the most recently adopted VVC maximum transform size has become capable of changing between 32 and 64. The value of this maximum transform size is calculated as follows:

[0022] MinTbLog2SizeY=2 (7-27)

[0023] MaxTbLog2SizeY=sps_max_luma_transform_size_64_flag? 6:5 (7-28)

[0024] MinTbSizeY=1< <MinTbLog2SizeY

[0025] (7-29)

[0026] MaxTbSizeY = 1 <MaxTbLog2SizeY

[0027] (7-30)

[0028] The maximum transform size interacts with the following tools:

[0029] 1-Clearing Process: Initially, VVC performs a clear to reduce the complexity of large transform sizes. For 2D DCT2 transforms, only the top-left 32×32 coefficients are preserved, while the rest are set to zero. That is, for 64×64, 64×32, and 32×32, DCT2 is performed to compute the first 32 coefficients in both the horizontal and vertical directions. Similarly, non-DCT2 transforms (DST7 and DCT8) perform a clear to preserve the top-left 16×16 coefficients. With the adoption of JVET-00545, it is not possible to resolve how to perform a clear when the maximum transform size is 32.

[0030] The following shows a capture of the draft text, where the shaded areas are cleared:

[0031]

[0032]

[0033] This shows that if tu_mts_idx is greater than zero, which means that MTS transformation (DST7,DST7) is used, the width and height of the clear are set to 16, while when tu_mts_idx is zero, the clear is set to 32 (DCT2).

[0034] 2-MTS-Size: MTS or multiple transform selections are used as transform tools adapted to VVC, allowing selection of additional transform pairs between DST7 and DCT8 pairs as enhancements to the DCT2 transform pair. MTS is performed for block sizes from 4×4 to 32×32. That is, half the DCT2 size. With the adoption of JVET-00545, it is not possible to address how to account for the MTS size when the maximum transform size is 32.

[0035] The following shows a capture of the draft text, where the shaded area represents the MTS size:

[0036]

[0037]

[0038]

[0039] This shows that if both the width and height are less than 32, the MTS is signaled, regardless of MaxTbSizeY.

[0040] 3. Chroma Transform Size: In VVC, the chroma size is half the luma size. That is, chroma transform blocks are allowed from 2×2 to 32×32, while luma sizes are from 4×4 to 64×64. With the adoption of JVET-00545, it is not possible to fix the chroma size when the maximum transform size is 32.

[0041] In the VVC specification, chromaticity is calculated as follows:

[0042] Table 6-1 - SubWidthC and derived from chroma_format_idc and separate_colour_plane_flag SubHeightC value

[0043] chroma_format_idc separate_colour_plane_flag Color format SubWidthC SubHeightC 0 0 monochrome 1 1 1 0 4:2:0 2 2 2 0 4:2:2 2 1 3 0 4:4:4 1 1 3 1 4:4:4 1 1

[0044] maxTbWidth=(cIdx==0)? MaxTbSizeY:MaxTbSizeY / SubWidthC (8-41)

[0045] maxTbHeight=(cIdx==0)? MaxTbSizeY:MaxTbSizeY / SubHeightC (8-42)

[0046] cIdx is the color index (0 represents luminance and 1 represents chromaticity).

[0047] In Common Test Conditions (CTC), a 4:2:0 chroma format is used, and the maximum transformation size (MaxTbSizeY) is 64. Therefore, for chroma in CTC, the maximum transformation size is 32. However, if MaxTbSizeY is 32, then according to the current specification, the maximum chroma size is 16.

[0048] 4-Transform Skip Size: In VVC, transform skipping is performed for block sizes with the same range as DCT2. In other words, transform skipping is performed for block sizes from 4×4 to 64×64. With the adoption of JVET-00545, it is not possible to fix the transform skip size when the maximum transform size is 32.

[0049] In the VVC specification, the maximum transform skip size is defined as:

[0050] Log2_transform_skip_max_size_minus2 specifies the maximum block size to be skipped during transformation, and it should be in the range of 0 to 3.

[0051] When it does not exist, it indicates that the value of log2_transform_skip_max_size_minus2 is equal to 0.

[0052] The variable MaxTsSize is set to be equal to 1 << (log2_transform_skip_max_size_minus2+2).

[0053] In other words, the maximum value of MaxTsSize can be between 4 and 32, regardless of the value of MaxTbSizeY.

[0054] 5-BDCM Size: BD-PCM is a block-based incremental pulse code modulation. It is a coding tool used for transform skip residuals. Currently, it allows for the same size conditions as transform skips, that is, up to MaxTsSize. The following text illustrates the conditions (shaded) for coding BDPCM.

[0055]

[0056]

[0057] 6-MIP

[0058] In VVC draft 6, MIP (Matrix-Based Intra-Prediction) is an intra-prediction mode where a prediction signal is generated by multiplying a reference sample by some trained constant-shift prediction matrix. This mode is signaled when the CU size is less than or equal to the maximum allowed transform size. This limitation is necessary to limit storage requirements and coding complexity. This is because MIP is a matrix-based approach, where the prediction matrix becomes larger for larger blocks.

[0059] Initially, the maximum transform size (MaxTbSizeY) remained constant at 64 in VTM 5.0. However, in VTM 6.0, this value can be either 64 or 32. Below is a sample of draft text in VTM 6.0 (the shaded area indicates the MIP portion):

[0060]

[0061]

[0062] The MaxTbSizeY value is fixed at 64. However, with the adoption of JVET-00545, MaxTbSizeY can be either 64 or 32.

[0063] Intuitively, if MaxTbSizeY is 32, then MIP is signaled for a CU size up to 32×32. This prevents the use of MIP for larger CU sizes and thus limits coding efficiency. The aspect described here proposes enabling MIP for CU sizes up to 64×64, regardless of the maximum transform size. This is accomplished by enabling TU splicing when the CU size is larger than MaxTbSizeY.

[0064] Initially, the maximum transform size remained at 64 in VTM 5.0. However, with the recent adoption of JVET-00545, the maximum transform size (MaxTbSizeY) can be either 64 or 32, controlled by the SPS flag (sps_sbt_max_size_64_flag). When this is the case, the clear process, MTS-size, chroma transform size, transform skip size, and BDCM size need to be adapted to this change.

[0065] Generally, it is proposed that the signaling of the following tools be adjusted based on the maximum transform size: clear process, MTS-size, chroma transform size, transform skip size, and BDCM. The affected codec modules are... Figure 1 and Figure 2 Intra-frame coding design (160) and 260.

[0066] Implementation Plan 1: Clearing Process

[0067] In this implementation, the clearing process depends on the maximum transform size. In this way, if the maximum size is 32 instead of 64, the clearing size is reduced by half. This is indicated in the text below (in italics).

[0068]

[0069] This can also be done independently for the DCT2 transform and other MTS transforms (DST7 and DST8). That is, if you only want to operate on DCT2, then:

[0070]

[0071] Otherwise, only for DST7 / DCT8

[0072]

[0073]

[0074] Implementation Plan 2: MTS - Size

[0075] MTS signaling is allowed up to a size of 32×32. This is independent of MaxTbSizeY, whether it is 64 or 32. To connect with MaxTbSizeY, MTS signaling can be allowed up to MaxTbSizeY / 2 x MaxTbSizeY / 2. This is shown in italics in the specification below:

[0076]

[0077]

[0078]

[0079] It should be noted that this directly affects the Sub-Block Transform (SBT) tool. SBT is a transform unit partitioning tool used for inter-frame blocks, which implies the selection of transform from DCT2, DST7, and DCT8. According to the specification:

[0080] The variable implicitMtsEnabled is exported as follows:

[0081] - Set implicitMtsEnabled to 1 if sps_mts_enabled_flag equals 1 and one of the following conditions is true:

[0082] -IntraSubPartitionsSplitType is not equal to ISP_NO_SPLIT

[0083] -cu_sbt_flag is equal to 1 and Max(nTbW,nTbH) is less than or equal to 32

[0084] -sps_explicit_mts_intra_enabled_flag equals 0 and CuprdMode[0][xTbY][yTbY] equals MODE_INTRA, and lfnst_idx[x0][y0] equals 0 and intra_mip_flag[x0][y0] equals 0.

[0085] Otherwise, set implicitMtsEnabled to 0.

[0086] The variables trTypeHor, which specify the horizontal transformation kernel, and trTypeVer, which specify the vertical transformation kernel, are exported as follows:

[0087] - If cIdx is greater than 0, then set trTypeHor and trTypeVer to 0.

[0088] - Otherwise, if implicitMtsEnabled equals 1, the following applies:

[0089] - If IntraSubPartitionsSplitType is not equal to ISP_NO_SPLIT or sps_explicit_mts_intra_enabled_flag is equal to 0 and CuPredMode[0][xTbY][yTbY] is equal to MODE_INTRA, then trTypeHor and trTypeVer are exported as follows:

[0090] trTypeHor=(nTbW>=4&&nTbW<=16)? 1:0 (8-975)

[0091] trTypeVer=(nTbH>=4&&nTbH<=16)? 1:0 (8-976)

[0092] - Otherwise (cu_sbt_flag equals 1), specify trTypeHor and trTypeVer in Table 8-15 according to cu_sbt_horizontal_flag and cu_sbt_pos_flag.

[0093] - Otherwise, specify trTypeHo and trTypeVer in Table 8-14 according to tu_mts_idx[xTbY][yTbY].

[0094] Table 8-15 - Specifying trTypeHor and cu_sbt_horizontal_flag and cu_sbt_pos_flag trTypeVer

[0095] cu_sbt_horizontal_flag cu_sbt_pos_flag trTypeHor trTypeVer 0 0 2 1 0 1 1 1 1 0 1 2 1 1 1 1

[0096] Transformation type 2 means DCT8 and transformation type 1 means DST7.

[0097] In other words, when the MTS is limited to MaxTbSizeY / 2 and MaxTbSizeY is 32, the above table cannot be used because DST7 and DCT8, which are 32×32 in size, are not supported. Instead, DCT2 should be used. The corresponding specification changes are as follows:

[0098] The variable implicitMtsEnabled is exported as follows:

[0099] - Set implicitMtsEnabled to 1 if sps_mts_enabled_flag equals 1 and one of the following conditions is true:

[0100] -IntraSubPartitionsSplitType is not equal to ISP_NO_SPLIT

[0101] -cu_sbt_flag is not equal to 1 and Max(nTbW,nTbH) is less than or equal to MaxTbSizeY / 2

[0102] -sps_explicit_mts_intra_enabled_flag equals 0 and CuprdMode[0][xTbY][yTbY] equals MODE_INTRA, and lfnst_idx[x0][y0] equals 0 and intra_mip_flag[x0][y0] equals 0.

[0103] Implementation Scheme 3: Magnitude of Chromaticity Transformation

[0104] According to the VVC specification, the maximum chroma transformation width and height can be half of the maximum luminance transformation width and height. However, when the maximum luminance transformation size is 32, the maximum chroma size can be 16. This seems like a small number, but it's not practical. Therefore, this implementation fixes the minimum chroma size at 32. The specification can be changed as follows (in italics):

[0105] maxTbWidth=(cIdx==0)? MaxTbSizeY:max(MaxTbSizeY / SubWidthC,32) (8-41)

[0106] maxTbHeight=(cIdx==0)? MaxTbSizeY:max(MaxTbSizeY / SubHeightC,32) (8-42)

[0107] Implementation Plan 4: Change the skip size

[0108] Transform skip flags can be signaled for sizes up to 32x32. This is independent of the maximum transform block size, whether it's 64 or 32. To connect with the maximum transform size, the text is modified as follows:

[0109] log2_transform_skip_max_size_minus2 specifies the maximum block size to be skipped during transformation, and it should be in the range of 0 to maxtblog 2sizey-3.

[0110] The MaxTbLog2SizeY is calculated as follows (based on VVC specifications):

[0111] MaxTbLog2SizeY=sps_max_luma_transform_size_64_flag? 6:5 (7-28)

[0112] Implementation Plan 5: BDCM

[0113] BDPCM uses the same conditions as the transition skipping when the signal is sent. Therefore, the above implementation scheme 4 is also applicable to BDPCM.

[0114] Implementation Plan 6: MIP

[0115] The general aspect described proposes enabling MIP for CU sizes up to 64×64, regardless of the maximum transform size. This is accomplished by enabling TU concatenation when the CU size is greater than MaxTbSizeY. This improves coding efficiency when MaxTbSizeY is 32.

[0116] The basic idea of ​​this invention is to allow MIP for CU sizes up to 64×64, regardless of MaxTbSizeY. This is to improve coding performance by enabling MIP when the CU size is greater than 32×32 while setting MaxTbSizeY to 32. Experimentally, it has been shown that MIP performs better for sequences with large sizes. The following results were generated using VTM software as an anchor, and the tests were conducted using VTM instead of MIP:

[0117]

[0118]

[0119] Clearly, MIP imparts a 0.6% gain to class A1 (large size) and 0.3 to class C (small size). Therefore, enabling MIP for a 64×64CU size with MaxTbSizeY of 32 provides encoding benefits for sequences with large sizes. Furthermore, no additional tools are required because the same MIP structure is maintained.

[0120] Compared to the current design, even with a smaller transform size, it allows the encoder to have more flexibility to perform MIP for CU sizes up to 64×64.

[0121] This can be achieved through TU splicing. That is, the CU is divided into multiple TUs and each MIP is executed independently. There are two ways to perform this operation:

[0122] 1- First perform TU splicing, then perform MIP prediction.

[0123] 2- First perform MIP prediction, then perform TU stitching

[0124] In other words, considering a CU size of 64×64, 32×64, or 64×32 and a MaxTbSizeY of 32, the first option is to divide the CU into 32×32 TUs, perform MIP on the 32×32 blocks to generate the prediction signal and encode the residuals. When doing so, reference samples from the reconstructed 32×32 blocks can be used to generate the prediction signal. The second option is to perform MIP using a large CU (64×64, 32×64, or 64×32), then divide the CU into 32×32 TUs and encode the residuals. The second option is more in line with the current design of VTM. This is because for regular intra-frame prediction (angle, DC, or plane), the prediction signal is generated at the same size as the TU, allowing reference samples from the reconstructed blocks to be used to improve predictions of neighboring blocks.

[0125] The corresponding specification text is shown in italics below:

[0126]

[0127]

[0128]

[0129] This specification text supports TU splicing when the transformation unit size is larger than the maximum transform size, as indicated in the text below (shaded).

[0130]

[0131]

[0132] Implementation Plan 7: LFNST Index

[0133] The specification text allows LFNST (Low Frequency Inseparable Transform) up to a maximum of MaxTbSizeY. The initial motivation was to avoid latency issues when decoding a large CU size of 128×128, where the LFNST index is decoded after the TU residual. Therefore, it was decided to allow LFNST up to a maximum transform size initially set to 64. With the adoption of JVET-00545, latency issues are not critical when the CU size is 64×64 and MaxTbSizeY is 32. Therefore, when MaxTbSizeY is set to 32, the LFNST index can be allowed to improve coding gain in this case.

[0134] The corresponding specification changes are in italics:

[0135]

[0136]

[0137] Furthermore, since LFNST is only allowed when the primary transformation is DCT2 (condition: tu_mts_idx[x0][y0] == 0), this condition needs to be checked for multiple TU sizes. The changes are as follows:

[0138]

[0139]

[0140]

[0141] In other words, a variable MTS_notDCT2 is defined, which checks whether any of the TU sizes does not use DCT2. If so, LFNST is not allowed.

[0142] Figure 4 The diagram illustrates one embodiment of method 400 according to the general aspects described herein. The method begins at a start box 401 and control advances to box 410 to enable an encoding tool based on a maximum transform size. Control advances from box 410 to box 420 to perform at least a portion of a discrete trigonometric transformation on a subset of samples comprising the block. Control advances from box 420 to box 430 to encode the block using the enabled encoding tool. The transform coefficients of the block are determined using the transformed subset of samples.

[0143] Figure 5The diagram illustrates one embodiment of method 500 according to the general aspects described herein. The method begins at a start box 501 and control advances to box 510 to enable an encoding tool based on a maximum transform size. Control advances from box 510 to box 520 to perform at least a portion of an inverse discrete trigonometric transformation on a subset of samples comprising the block. Control advances from box 520 to box 530 to decode the block using the enabled encoding tool.

[0144] Figure 6 An embodiment of an apparatus 600 for compressing, encoding, or decoding video using various encoding tools according to the maximum transform size is shown. The apparatus includes a processor 1410 and can be interconnected to a memory 1420 via at least one port. Both the processor 1410 and the memory 1420 may also have one or more additional interconnects for external connection.

[0145] The processor 610 is also configured to insert or receive information in the bitstream and to use any of the aspects described to perform compression, encoding, or decoding.

[0146] This document describes multiple aspects, including tools, features, implementation schemes, models, methods, etc. Many of these aspects are described in detail, and often in a way that may sound limited, at least to illustrate individual characteristics. However, this is for clarity and does not limit the application or scope of these aspects. In fact, all the different aspects can be combined and interchanged to provide further aspects. Furthermore, these aspects can also be combined and interchanged with those described in previous submissions.

[0147] The aspects described and envisioned in this document can be implemented in many different forms. The following... Figure 1 , Figure 2 and Figure 3 Some implementation schemes are provided, but other implementation schemes are envisioned, and Figure 1 , Figure 2 and Figure 3 The discussion does not limit the breadth of specific implementations. At least one of these aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a bitstream generated or encoded. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having instructions stored thereon for encoding or decoding video data according to any of the methods, and / or computer-readable storage media having a bitstream generated according to any of the methods stored thereon.

[0148] In this application, the terms "reconstruction" and "decoding" are used interchangeably, as are the terms "pixel" and "sample," and the terms "image," "picture," and "frame." Generally, but not necessarily, the term "reconstruction" is used at the encoding end, while "decoding" is used at the decoding end.

[0149] This document describes various methods, and each method includes one or more steps or actions for implementing the method. Unless the correct operation of the method requires a specific order of steps or actions, the order and / or purpose of a particular step and / or action may be modified or combined.

[0150] The various methods and other aspects described in this document can be used to modify modules of the video encoder 100 and decoder 200 (e.g., intra-frame prediction, entropy coding, and / or decoding modules (160, 360, 145, 330)), such as Figure 1 and Figure 2 As shown. Furthermore, aspects of the invention are not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations (whether pre-existing or developed in the future) and any extensions of such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this document may be used alone or in combination.

[0151] Various numerical values ​​are used in this document, such as {{1,0},{3,1},{1,1}}. Specific values ​​are for illustrative purposes only, and the aspects described are not limited to these specific values.

[0152] Figure 1 Encoder 100 is illustrated. Variations of this encoder 100 are envisioned, but for clarity, encoder 100 is described below without describing all anticipated variations.

[0153] Before encoding, the video sequence may undergo pre-coding (101), for example, by applying color transformations to the input color image (e.g., a conversion from RGB 4:4:4 to YCbCr 4:2:0), or by performing remapping of the input image components to obtain a more resilient signal distribution to compression (e.g., histogram equalization using one of the color components). Metadata may be associated with the pre-processing and attached to the bitstream.

[0154] In encoder 100, the image is encoded by encoder elements as described below. The image to be encoded is partitioned (102) and processed in units such as CUs. For example, each unit is encoded using either an intra-frame mode or an inter-frame mode. When a unit is encoded in intra-frame mode, it performs intra-frame prediction (160). In inter-frame mode, motion estimation (175) and compensation (170) are performed. The encoder determines (105) which of the intra-frame mode or inter-frame mode is used to encode the unit and indicates the intra-frame / inter-frame decision by, for example, a prediction mode flag. For example, the prediction residual is calculated by subtracting (110) the prediction block from the original image block.

[0155] The predicted residual is then transformed (125) and quantized (130). The quantized transform coefficients, motion vectors, and other syntax elements are entropy encoded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., encode the residual directly without applying the transform or quantization process.

[0156] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) to reconstruct the image block. A loop filter (165) is applied to the reconstructed image to perform, for example, deblocking / SAO (sample adaptive offset) filtering to reduce coded artifacts. The filtered image is stored in a reference image buffer (180).

[0157] Figure 2 A block diagram of a video decoder 200 is illustrated. In decoder 200, the bitstream is decoded by decoder elements, as described below. Video decoder 200 typically performs operations similar to... Figure 1 The encoding process is the opposite of the decoding process. Encoder 100 typically also performs video decoding as part of the video data encoding.

[0158] The decoder's input includes a video bitstream, which can be generated by the video encoder 100. First, entropy decoding (230) is performed on the bitstream to obtain transform coefficients, motion vectors, and other encoded information. Image partitioning information indicates how the image should be partitioned. Therefore, the decoder can partition (235) the image based on the decoded image partitioning information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residuals. The decoded prediction residuals and prediction blocks are combined (255) to reconstruct the image blocks. Prediction blocks (270) can be obtained from intra-frame prediction (260) or motion-compensated prediction (i.e., inter-frame prediction) (275). A loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference image buffer (280).

[0159] The decoded image can also undergo post-decoding processing (285), such as inverse color transformation (e.g., a transformation from YCbCr 4:2:0 to RGB 4:4:4) or inverse remapping of the remapping process performed in the pre-encoding process (101). Post-decoding processing can utilize metadata derived in the pre-encoding process and signaled in the bitstream.

[0160] Figure 3 A block diagram illustrating examples of systems implementing various aspects and embodiments therein is provided. System 1000 may be embodied as a device including the various components described below and configured to perform one or more aspects described in this document. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. Elements of system 1000 may be embodied individually or in combination in a single integrated circuit (IC), multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to other similar systems or other electronic devices via, for example, a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described in this document.

[0161] System 1000 includes at least one processor 1010 configured to execute instructions loaded thereon for implementing various aspects, such as those described in this document. Processor 1010 may include embedded memory, input / output interfaces, and various other circuitry known in the art. System 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). System 1000 includes a storage device 1040 that may include non-volatile memory and / or volatile memory, including but not limited to EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash memory, disk drives, and / or optical disk drives. As a non-limiting example, storage device 1040 may include internal storage devices, attached storage devices, and / or network-accessible storage devices.

[0162] System 1000 includes an encoder / decoder module 1030 configured to, for example, process data to provide encoded or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 represents a module that can be included in a device to perform encoding and / or decoding functions. It is well known that a device may include one or both of an encoding module and a decoding module. Furthermore, the encoder / decoder module 1030 may be implemented as a standalone element of system 1000, or may be incorporated within processor 1010 as a combination of hardware and software known to those skilled in the art.

[0163] Program code to be loaded onto processor 1010 or encoder / decoder 1030 to execute the various aspects described in this document may be stored in storage device 1040 and subsequently loaded onto memory 1020 for execution by processor 1010. According to various embodiments, one or more of processor 1010, memory 1020, storage device 1040, and encoder / decoder module 1030 may store one or more items from various projects during the execution of the processes described in this document. Such stored items may include, but are not limited to, input video, decoded or partially decoded video, bitstreams, matrices, variables, and intermediate or final results of processing equations, formulas, operations, and operational logic.

[0164] In several embodiments, the memory within processor 1010 and / or encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, external memory (e.g., the processing device may be processor 1010 or encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as volatile memory and / or non-volatile flash memory. In several embodiments, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, fast external volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as for MPEG-2, HEVC, or VVC (Video Coding).

[0165] Inputs to the components of system 1000 may be provided through various input devices as shown in block 1130. Such input devices include, but are not limited to: (i) receiving inputs, for example, via radio from a broadcaster.

[0166] The RF section of the transmitted RF signal, (ii) a composite input terminal, (iii) a USB input terminal, and / or (iv) an HDMI input terminal.

[0167] In various embodiments, the input device of block 1130 has associated corresponding input processing elements as known in the art. For example, the RF section may be associated with elements necessary to: (i) select a desired frequency (also known as select a signal, or limit the signal band to a band), (ii) down-convert the selected signal, (iii) re-band limit the signal to a narrower band to select (e.g.,) a signal band that may be referred to as a channel in some embodiments), (iv) demodulate the down-converted and band-limited signal, (v) perform error correction, and (vi) demultiplex to select the desired data packet stream. The RF section of various embodiments includes one or more elements for performing these functions, such as frequency selectors, signal selectors, band limiters, channel selectors, filters, downconverters, demodulators, error correctors, and demultiplexers. The RF section may include tuners that perform various functions among these functions, including, for example, down-converting received signals to a lower frequency (e.g., intermediate frequency or near-baseband frequency) or to baseband. In one set-top box implementation, the RF section and its associated input processing elements receive RF signals transmitted via a wired (e.g., cable) medium and perform frequency selection by filtering, down-converting, and re-filtering to the desired frequency band. Various implementations rearrange the order of the aforementioned (and other) components, remove some of these components, and / or add other components that perform similar or different functions. Adding components may include inserting components between existing components, such as inserting amplifiers and analog-to-digital converters. In various implementations, the RF section includes an antenna.

[0168] Furthermore, USB and / or HDMI terminals may include corresponding interface processors for connecting system 1000 to other electronic devices across USB and / or HDMI connections. It should be understood that various aspects of input processing (e.g., Reed-Solomon error correction) may be implemented as needed, for example, within a separate input processing IC or within processor 1010. Similarly, aspects of USB or HDMI interface processing may be implemented as needed, either within a separate interface IC or within processor 1010. Demodulated streams, error-corrected streams, and demultiplexed streams are provided to various processing elements, including, for example, processor 1010 and encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on the output device.

[0169] Various components of system 1000 can be housed within an integrated housing. Within the integrated housing, various components can be interconnected and transmit data therebetween using a suitable connection arrangement 1140 (e.g., internal buses as known in the art, including inter-IC (I2C) buses, wiring, and printed circuit boards).

[0170] System 1000 includes a communication interface 1050 capable of communicating with other devices via a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data via the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or network interface card (NIC), and the communication channel 1060 may be implemented, for example, in a wired and / or wireless medium.

[0171] In various implementations, data is streamed to system 1000 using a wireless network such as IEEE 802.11. For example, the wireless signals in these implementations are received via a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 in these implementations is typically connected to an access point or router that provides access to external networks, including the Internet, to allow streaming applications and other cloud-based communications. Other implementations use a set-top box to provide streaming data to system 1000, delivering data via an HDMI connection to input block 1130. Still other implementations use an RF connection to input block 1130 to provide streaming data to system 1000.

[0172] System 1000 can provide output signals to various output devices, including display 1100, speaker 1110, and other peripheral devices 1120. In various embodiments, the other peripheral devices 1120 include one or more of the following: a standalone DVR, a disk player, a stereo system, a lighting system, and other devices that provide functionality based on the output of system 1000. In various embodiments, control signals are transmitted between system 1000 and display 1100, speaker 1110, or other peripheral devices 1120 using signaling (such as AV.Link, CEC, or other communication protocols), which enable device-to-device control with or without user intervention. Output devices can be communicatively coupled to system 1000 via dedicated connections through corresponding interfaces 1070, 1080, and 1090. Alternatively, output devices can be connected to system 1000 via communication interface 1050 using communication channel 1060. Display 1100 and speaker 1110 can be integrated into a single unit with other components of system 1000 in electronic devices (e.g., television sets). In various implementations, the display interface 1070 includes a display driver, such as a timing controller (T Con) chip.

[0173] Alternatively, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 may be separated from one or more other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0174] These implementations can be executed by processor 1010 or by computer software implemented in hardware or a combination of hardware and software. As a non-limiting example, these implementations can be implemented by one or more integrated circuits. As a non-limiting example, memory 1020 can be of any type suitable for the technical environment and can be implemented using any suitable data storage technology.

[0175] Such as optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. As a non-limiting example, processor 1010 can be any type suitable for the technical environment and can encompass one or more of microprocessors, general-purpose computers, special-purpose computers, and processors based on multi-core architectures.

[0176] Various specific implementations involve decoding. As used in this application, "decoding" may include all or part of a process, for example, performing on a received coded sequence to produce a final output suitable for display. In various embodiments, such a process includes one or more processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such a process also or alternatively includes processes performed by a decoder of various embodiments described in this application, such as extracting weight indices to be used for various intra-frame prediction reference arrays.

[0177] As a further example, in one embodiment, "decoding" refers only to entropy decoding; in another embodiment, "decoding" refers only to differential decoding; and in yet another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" specifically refers to a subset of operations or broadly refers to a wider decoding process will be clear based on the specific context of the description and is believed to be well understood by those skilled in the art.

[0178] Various specific implementations involve encoding. In a manner similar to the discussion above regarding “decoding,” the term “encoding,” as used herein, can encompass all or part of a process performed, for example, on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as partitioning, differential coding, transform, quantization, and entropy coding. In various embodiments, such processes also or alternatively include processes performed by an encoder of the various embodiments described herein, such as weighting an intra-frame prediction reference array.

[0179] As a further example, in one implementation, "encoding" refers only to entropy encoding; in another implementation, "encoding" refers only to differential encoding; and in yet another implementation, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" specifically refers to a subset of operations or broadly refers to a wider encoding process will be clear based on the specific context of the description and is believed to be well understood by those skilled in the art.

[0180] Note that the grammatical elements used in this article are descriptive terms. Therefore, the use of other grammatical element names is not excluded.

[0181] When the accompanying drawings are presented as flowcharts, it should be understood that block diagrams of the corresponding devices are also provided. Similarly, when the accompanying drawings are presented as block diagrams, it should be understood that flowcharts of the corresponding methods / processes are also provided.

[0182] Various implementation schemes refer to rate-distortion computation or rate-distortion optimization. During the encoding process, a balance or trade-off between rate and distortion is typically considered, often taking into account computational complexity constraints. Rate-distortion optimization is generally formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. Different approaches exist to solve the rate-distortion optimization problem. For example, these methods may be based on extensive testing of all encoding options (including all considered modes or encoding parameter values) and a complete evaluation of their encoding costs and the associated distortion of the reconstructed signal after encoding and decoding. Faster methods can also be used to reduce encoding complexity, particularly for the computation of approximate distortion based on prediction or prediction of the residual signal rather than the reconstructed residual signal. A hybrid of these two approaches can also be used, such as by using approximate distortion for only some of the possible encoding options and full distortion for others. Other methods evaluate only a subset of the possible encoding options. More generally, many methods employ any of a variety of techniques to perform optimization, but optimization is not necessarily a complete evaluation of both encoding costs and associated distortion.

[0183] The specific embodiments and aspects described herein may be implemented, for example, in methods or processes, apparatus, software programs, data streams, or signals. Even if discussed only in the context of a single form of specific embodiment (e.g., discussed only as a method), specific embodiments of the discussed features may be implemented in other forms (e.g., apparatus or program). Apparatus may be implemented, for example, in suitable hardware, software, and firmware. Methods may be implemented, for example, in a processor, which generally refers to a processing device.

[0184] The processing device includes, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication devices, such as computers, mobile phones, portable / personal digital assistants (“PDAs”), and other devices that facilitate information communication between end users.

[0185] The reference to “an implementation plan” or “implementation plan” or “a specific implementation” or “specific implementation”, and their other variations, means that the specific features, structures, characteristics, etc., described in connection with the implementation plan are included in at least one implementation plan. Therefore, the appearance of the phrase “in an implementation plan” or “in an implementation plan” or “in a specific implementation” or “in a specific implementation”, and any other variations appearing throughout this document, do not necessarily refer to the same implementation plan.

[0186] In addition, this document may refer to the various pieces of information marked "determined". Determined information may include one or more of the following: estimated information, calculated information, predicted information, or information retrieved from memory.

[0187] Furthermore, this document may refer to "accessing" each piece of information. Accessing information may include, for example, receiving information, retrieving information (e.g., from memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information, or more of these.

[0188] Furthermore, this document may refer to "receiving" individual pieces of information. Like "access," "receiving" is intended as a broad term. Receiving information may include, for example, accessing information or retrieving information (e.g., from memory). Moreover, "receiving" typically participates in one or more of the following operations: such as storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0189] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one” is intended to cover selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As a further example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” such phrases are intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or selecting all three options (A, B, and C). As will be apparent to those skilled in the art and related fields, this can be extended to as many items as possible listed.

[0190] Moreover, as used herein, the term "signaling" refers to (among other things) instructing the corresponding decoder to do something. For example, in some implementations, the encoder signals a specific weight among several weights to be used in the intra-prediction reference array. Thus, in one implementation, the same parameters are used on both the encoder and decoder sides. Therefore, for example, the encoder can transmit (explicit signaling) specific parameters to the decoder so that the decoder can use the same specific parameters. Conversely, if the decoder already has specific parameters and others, signaling can be used without transmission (implicit signaling) to simply allow the decoder to know and select specific parameters. Bit savings are achieved in various implementations by avoiding the transmission of any actual functionality. It should be understood that signaling can be implemented in various ways. For example, in various implementations, information is signaled to the corresponding decoder using one or more syntax elements, flags, etc. Although the verb form of the term "signal" has been used above, the term "signal" can also be used as a noun herein.

[0191] It will be apparent to those skilled in the art that embodiments can generate various signals formatted to carry, for example, storable or transmissible information. The information may include, for example, instructions for performing a method or data generated by one of the embodiments. For example, a signal may be formatted to carry a bitstream of the embodiment. Such signals may be formatted as, for example, electromagnetic waves (e.g., using the radio frequency portion of a spectrum) or baseband signals. Formatting may include, for example, encoding a data stream and modulating the carrier with the encoded data stream. The information carried by the signal may be, for example, analog or digital information. It is known that signals can be transmitted via various wired or wireless links. Signals may be stored on a processor-readable medium.

[0192] Implementations may include one or more of the following features or entities, individually or in combination, across a variety of different claim classes and types:

[0193] • For coding units up to a certain size, enable matrix-based intra-frame prediction, regardless of the maximum transform size.

[0194] • When the size of the encoding unit is greater than the maximum transform size, transform unit splicing is enabled.

[0195] • Enable the Low Frequency Non-Separable Transform (LFNST) up to a defined size.

[0196] • Include a syntax element in the bitstream or check the bitstream for a syntax element that indicates whether any transform unit does not use DCT2 as a transform, and if so, LFNST is not allowed.

[0197] • Includes a bitstream or signal of one or more syntax elements or their variants from the described syntax elements.

[0198] • To create and / or transmit and / or receive and / or decode bit streams or signals that include one or more of the described syntax elements or their variants.

[0199] • A television, set-top box, mobile phone, tablet computer or other electronic device that performs loop filtering according to any of the embodiments described in the implementation plan.

[0200] • A television, set-top box, mobile phone, tablet computer, or other electronic device that performs loop filtering according to any of the described implementation schemes and displays the resulting image (e.g., using a monitor, screen, or other type of display).

[0201] A television, set-top box, mobile phone, tablet computer, or other electronic device that tunes (e.g., uses a tuner) a channel to receive signals including encoded images and performs loop filtering according to any of the described embodiments.

[0202] A television, set-top box, mobile phone, tablet computer, or other electronic device that receives signals including encoded images via radio (e.g., using an antenna) and performs loop filtering according to any of the described embodiments.

[0203] Various other general and specific inventions and claims are also supported and envisioned throughout this disclosure.

Claims

1. A method for video encoding, the method comprising: For blocks up to the first size, enable matrix-based intra-frame prediction MIP coding tools, regardless of the maximum transform size; When the size of the block is greater than the maximum transform size, the block is divided into multiple transform blocks with the maximum transform size; Perform matrix-based intra-frame prediction for each block of the maximum transform size to obtain the prediction; as well as The block is encoded based on the prediction.

2. The method according to claim 1, wherein the maximum transformation size is 32×32 and the first size is 64×64.

3. The method according to claim 1, wherein the size of the block is 64×64, 64×32 or 32×64.

4. The method of claim 1, wherein for blocks up to a first size, enabling matrix-based intra-frame prediction (MIP) regardless of the maximum transform size includes: If the size of the block is greater than the maximum transform size, matrix-based intra-frame prediction is enabled for blocks up to the first size.

5. An apparatus, the apparatus comprising: Processor, the processor being configured to execute: For blocks up to the first size, enable matrix-based intra-frame prediction MIP coding tools, regardless of the maximum transform size; When the size of the block is greater than the maximum transform size, the block is divided into multiple transform blocks with the maximum transform size; Perform matrix-based intra-frame prediction for each block of the maximum transform size to obtain the prediction; as well as The block is encoded based on the prediction.

6. The apparatus of claim 5, wherein the maximum transformation size is 32×32 and the first size is 64×64.

7. The apparatus according to claim 5, wherein the size of the block is 64×64, 64×32, or 32×64.

8. The apparatus of claim 5, wherein for blocks up to a first size, matrix-based intra-frame prediction (MIP) is enabled, regardless of the maximum transform size, including: If the size of the block is greater than the maximum transform size, matrix-based intra-frame prediction is enabled for blocks up to the first size.

9. A method for video decoding, the method comprising: For blocks up to the first size, enable matrix-based intra-frame prediction MIP coding tools, regardless of the maximum transform size; When the size of the block is greater than the maximum transform size, the block is divided into multiple transform blocks with the maximum transform size; Perform matrix-based intra-frame prediction for each block of the maximum transform size to obtain the prediction; as well as Based on the prediction, the block is decoded.

10. The method of claim 9, wherein the maximum transformation size is 32×32 and the first size is 64×64.

11. The method according to claim 9, wherein the size of the block is 64×64, 64×32, or 32×64.

12. The method of claim 9, wherein for blocks up to a first size, enabling matrix-based intra-frame prediction (MIP) regardless of the maximum transform size includes: If the size of the block is greater than the maximum transform size, matrix-based intra-frame prediction is enabled for blocks up to the first size.

13. An apparatus comprising: Processor, the processor being configured to execute: For blocks up to the first size, enable matrix-based intra-frame prediction MIP coding tools, regardless of the maximum transform size; When the size of the block is greater than the maximum transform size, the block is divided into multiple transform blocks with the maximum transform size; Perform matrix-based intra-frame prediction for each block of the maximum transform size to obtain the prediction; as well as Based on the prediction, the block is decoded.

14. The apparatus of claim 13, wherein the maximum transformation size is 32×32 and the first size is 64×64.

15. The apparatus of claim 13, wherein the size of the block is 64×64, 64×32, or 32×64.

16. The apparatus of claim 13, wherein for blocks up to a first size, matrix-based intra-frame prediction (MIP) is enabled, regardless of the maximum transform size, including: If the size of the block is greater than the maximum transform size, matrix-based intra-frame prediction is enabled for blocks up to the first size.

17. An apparatus comprising: The apparatus according to claim 13; as well as At least one of the following: (i) an antenna configured to receive a signal comprising a video block; (ii) a bandwidth limiter configured to limit the received signal to a bandwidth comprising the video block; and (iii) a display configured to display an output representing a video block.

18. A non-transitory computer-readable medium comprising data content for playback using a processor, the data content being generated according to the method of claim 1.

Images