Transform size interaction with coding tools

Abstract

To provide a method or a device for realizing a coding or a decoding, a compression or an extension of a video.SOLUTION: A method and a device for mounting a discreteness trigger metric conversion are based on the maximum conversion size. According to an embodiment, an intra prediction of a matrix base is not related to the maximum conversion size, and validates a code unit size until a size to be designated. In the other embodiment, low frequency non-separable conversion is used for improve a code gain. By using a construction in a bit stream, a code tool to be used can be indicated.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] At least one of the present embodiments relates generally to a method or apparatus for video encoding or decoding, compression or decompression. [Background technology]

[0002] To achieve high compression efficiency, image and video coding schemes typically employ prediction, including motion vector prediction, and transformation that exploits spatial and temporal redundancy in the video content. Typically, intra- or inter-prediction is used to exploit correlation within or between frames, and then the difference between the original and predicted image, often called the prediction error or prediction residual, is transformed, quantized, and entropy coded. To reconstruct the video, the compressed data is decoded by the inverse processes corresponding to entropy coding, quantization, transformation, and prediction.

[0003] In the development of the Versatile Video Coding (VVC) standard, the maximum transform size is variable between 32 and 64. The maximum transform size interacts with other transform coding tools. Summary of the Invention

[0004] At least one of the present embodiments relates generally to a method or apparatus for video encoding or decoding, and more particularly to a method or apparatus for interaction between a maximum transform size and a transform coding tool in a video encoder or video decoder.

[0005] According to a first aspect, there is provided a method, the method comprising the steps of enabling a coding tool based on a maximum transform size, performing at least a portion of a discrete triangular transform on a subset of samples comprising a block, and encoding the block using the enabled coding tool.

[0006] According to a second aspect, there is provided a method, the method including the steps of enabling a coding tool based on a maximum transform size, performing at least a portion of an inverse discrete triangular transform on a subset of samples comprising a block, and decoding the block using the enabled coding tool.

[0007] According to another aspect, there is provided an apparatus, comprising: a processor, the processor may be configured to encode a block of video or decode a bitstream by performing any of the methods described above.

[0008] According to another general aspect of at least one embodiment, there is provided a device comprising an apparatus according to any of the decoding embodiments and at least one of: (i) an antenna configured to receive a signal, the signal including a video block; (ii) a band limiter configured to limit the received signal to a frequency band including the video block; or (iii) a display configured to display an output representing the video block.

[0009] According to another general aspect of at least one embodiment, there is provided a non-transitory computer-readable medium that includes data content generated in accordance with any of the described encoding embodiments or variations.

[0010] According to another general aspect of at least one embodiment, there is provided a signal including video data generated in accordance with any of the described encoding embodiments or variations.

[0011] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variations.

[0012] According to another general aspect of at least one embodiment, there is provided a computer program product including instructions that, when executed by a computer, cause the computer to perform any of the described decoding embodiments or variations.

[0013] These and other aspects, features, and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings. [Brief explanation of the drawings]

[0014] [Figure 1] Indicates a standard common video compression method. [Figure 2] This shows a standard common video decompression method. [Figure 3] 1 illustrates an exemplary processor-based subsystem for implementing the generally described aspects. [Figure 4] 1 illustrates one embodiment of a method under the described aspect. [Figure 5] 1 illustrates another embodiment of the method according to the described embodiment. [Figure 6] 1 illustrates an exemplary apparatus under the described embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] At least one of the present embodiments relates generally to a method or apparatus for video encoding or decoding, and more particularly to a method or apparatus for interaction between a maximum transform size and a transform coding tool in a video encoder or video decoder.

[0016] A general aspect described herein is the field of video compression, where in recent adoption of VVC the maximum transform size is variable between 32 and 64, and the interaction between the maximum transform size and other transform coding tools. The value is calculated as follows:

[0017]

number

[0018] Maximum transform size interacts with the following tools: 1-Zeroing Process: First, VVC performs zeroing to reduce the complexity of large transform sizes. For 2D DCT2 transforms, only the top-left 32x32 coefficients are kept while the rest are set to zero. That is, for 64x64, 64x32, and 32x32, DCT2 calculates the first 32 coefficients in both the horizontal and vertical directions, while non-DCT2 transforms (DST7 and DCT8) perform zeroing to keep the top-left 16x16 coefficients. The adoption of JVET-00545 does not resolve how zeroing is performed when the maximum transform size is 32.

[0019] A capture of the draft text is shown below, with the zero settings shaded.

[0020] [Table 1]

[0021] This means that if tu_mts_idx is greater than zero, meaning that an MTS transform is used (DST7, DST7), the zeroing width and height are set to 16, while if tu_mts_idx is zero (DCT2), the zeroing is set to 32. 2-MTS Size: MTS or Multiple Transform Selection is a transform tool employed in VVC that allows the selection between DCT7 and DCT8 pairs as other transform pairs that enhance the DCT2 transform pair. MTS is performed for block sizes from 4x4 to 32x32, i.e., half size DCT2. With the adoption of JVET-00545, it is not resolved how the MTS size is considered when the maximum transform size is 32.

[0022] A capture of the draft text is shown below, with the mts size shaded.

[0023] [Table 2-1]

[0024] [Table 2-2]

[0025] This indicates that if both width and height are less than 32, an MTS is signaled, regardless of MaxTbSizeY. 3- Saturation transform size: In VVC, the saturation size is half of the luma size. That is, the saturation transform block is allowed to be 2x2 to 32x32, while the luma size is 4x4 to 64x64. With the adoption of JVET-00545, if the maximum transform size is 32, it is not resolved how the saturation size is fixed.

[0026] In the VVC spec, the chroma size is calculated as follows:

[0027] [Table 3] maxTbWidth=(cIdx==0)?MaxTbSizeY:MaxTbSizeY / SubWidthC(8-41) maxTbHeight=(cIdx==0)?MaxTbSizeY:MaxTbSizeY / SubHeightC(8-42) cIdx is the color index (0 for luminance, 1 for saturation).

[0028] In common testing condition (CTC), a 4:2:0 chroma format is used and the maximum transform size (MaxTbSizeY) is 64. Therefore, the maximum transform size is 32 for chroma in CTC. However, if MaxTbSizeY is 32, the chroma maximum size is 16 according to the current SPEC. 4-Transform Skip Size: In VVC, transform skip is performed for block sizes with the same range as DCT2. In other words, transform skip is performed for block sizes from 4x4 to 64x64. With the adoption of JVET-00545, it is not resolved how the transform skip size is fixed when the maximum transform size is 32.

[0029] In the VVC specification, the maximum transform skip size is defined as follows: log2_transform_skip_max_size_minus2 specifies the maximum block size used for transform skipping and shall be in the range 0 to 3.

[0030] If not present, the value of log2_transform_skip_max_size_minus2 is inferred to be equal to 0.

[0031] The variable MaxTsSize is set equal to 1<<(log2_transform_skip_max_size_minus2+2).

[0032] That is, the maximum MaxTsSize can take a value between 4 and 32, regardless of the MaxTbSizeY value. 5 - BDCM Size: BD-PCM is a block-based delta pulse code modulation. It is a coding tool for the transform skip residual. It is currently allowed with the same size conditions as the transform skip, i.e., a maximum of MaxTsSize. The text below shows the conditions (shaded) for coding the BDPCM flags.

[0033] [Table 4]

[0034] 6-MIP In VVC Draft 6, MIP (Matrix-Based Intra Prediction) is an intra prediction mode in which a prediction signal is generated by multiplying a reference sample by several trained prediction matrices with a fixed shift. The mode is signaled when the CU size is less than or equal to the maximum allowed transform size dimension. This restriction was necessary to limit memory requirements and coding complexity because MIP is a matrix-based method and the prediction matrix is ​​larger for larger blocks.

[0035] First, the maximum transform size (MaxTbSizeY) is always kept as 64 in VTM5.0. However, in VTM6.0, this value can be 64 or 32. A sample of the draft text of VTM6.0 is provided below (the shaded area indicates the MIP part):

[0036] [Table 5-1]

[0037] [Table 5-2]

[0038] The MaxTbSizeY value was fixed at 64. However, in the adoption of JVET-00545, MaxTbSizeY can be either 64 or 32.

[0039] Intuitively, if MaxTbSizeY is 32, MIPs are signaled up to a CU size of 32x32. This prevents larger MIPs from using CUs, thus limiting coding efficiency. The currently described aspects propose enabling MIPs for CUs up to 64x64, regardless of the maximum transform size. This is done by enabling TU tiling when the CU size is larger than MaxTbSizeY.

[0040] Initially, the maximum transform size is always kept as 64 in VTM5.0. However, with the recent adoption of JVET-00545, the maximum transform size (MaxTbSizeY) can be either 64 or 32, controlled by the SPS flag (sps_sbt_max_size_64_flag). When this happens, the zeroing process, MTS size, chroma transform size, transform skip size, and BDCM size need to be adapted to this change.

[0041] A general aspect proposes to adapt the signaling of the following tools according to the maximum transform size: zeroing process, MTS size, chroma transform size, transform skip size and BDCM. The affected codec modules are the intra-coding designs (160) and 260 of Figures 1 and 2.

[0042] Embodiment 1: Zeroing Process In this embodiment, the zeroing process depends on the maximum transform size. Thus, if the maximum size is 32 instead of 64, the zeroing size is reduced by half. This is shown in the following text (in italics):

[0043] [Table 6]

[0044] This can also be done independently for the DCT2 transform and the other MTS transforms (DST7 and DCT8), i.e. if you want to do it only for the DCT2:

[0045] [Table 7]

[0046] Otherwise, for DST7 / DCT8 only

[0047] [Table 8]

[0048] Embodiment 2: MTS size MTS signaling is allowed up to a size of 32 x 32. This is independent of MaxTbSizeY, whether it is 64 or 32. To accommodate MaxTbSizeY, MTS signaling can be resized up to MaxTbSizeY / 2 x MaxTbSizeY / 2. This is shown in italics in the spec below:

[0049] [Table 9-1]

[0050] [Table 9-2]

[0051] Note that this has a direct impact on the subblock transform (SBT) tool. SBT is a transform unit partitioning tool for inter-blocks that semantically selects between DCT2, DST7, and DCT8 transforms. According to this specification: The variable implicitMtsEnabled is derived as follows: - implicitMtsEnabled is set equal to 1 if sps_mts_enabled_flag is equal to 1 and one of the following conditions is true: -IntraSubPartitionsSplitType is not equal to ISP_NO_SPLIT -cu_sbt_flag is equal to 1 and Max(nTbW, nTbH) is less than or equal to 32 -sps_explicit_mts_intra_enabled_flag is equal to 0, CuPredMode[0][xTbY][yTbY] is equal to MODE_INTRA, lfnst_idx[x0][y0] is equal to 0, and intra_mip_flag[x0][y0] is equal to 0 - Otherwise, implicitMtsEnabled is equal to 0.

[0052] The variable trTypeHor that specifies the horizontal transformation kernel and the variable trTypeVer that specifies the vertical transformation kernel are derived as follows. If cIdx is greater than 0, trTypeHor and trTypeVer are equal to 0. - Otherwise, if ImplicitMtsEnabled is equal to 1, the following applies: - If IntraSubPartitionsSplitType is not equal to ISP_NO_SPLIT or sps_explicit_mts_intra_enabled_flag is equal to 0 and CuPredMode[0][xTbY][yTbY] is equal to MODE_INTRA, then trTypeHor and trTypeVer are derived as follows: trTypeHor=(nTbW>=4&&nTbW<=16)?1:0(8-975) trTypeVer=(nTbH>=4&&nTbH<=16)?1:0(8-976) Otherwise (cu_sbt_flag is equal to 1), trTypeHor and trTypeVer are specified in Tables 8-15 depending on cu_sbt_horizontal_flag and cu_sbt_pos_flag. Otherwise, trTypeHor and trTypeVer are specified in Table 8-14 according to tu_mts_idx[xTbY][yTbY].

[0053] [Table 10]

[0054] Transform type 2 means DCT8, 1 means DST7.

[0055] That is, if MTS is limited to MaxTbSizeY / 2 and MaxTbSizeY is 32, then the above table cannot be used because DST7 and DCT8 of size 32x32 are not supported. Instead, DCT2 must be used. The corresponding specification changes are as follows:

[0056] The variable implicitMtsEnabled is derived as follows: - inclipitMtsEnabled is set equal to 1 if sps_mts_enabled_flag is equal to 1 and one of the following conditions is true: -IntraSubPartitionsSplitType is not equal to ISP_NO_SPLIT -cu_sbt_flag is equal to 1 and Max(nTbW, nTbH) is less than or equal to MaxTbSizeY / 2 -sps_explicit_mts_intra_enabled_flag is equal to 0, CuPredMode[0][xTbY][yTbY] is equal to MODE_INTRA, lfnst_idx[x0][y0] is equal to 0, and intra_mip_flag[x0][y0] is equal to 0

[0057] Embodiment 3: Saturation conversion size According to the VVC spec, the saturation maximum transform width and height can be half of the maximum luma 1. However, since the maximum luma transform size can be 32, the maximum size of saturation can be 16. This is a small number and is not considered useful in practice. Therefore, this embodiment fixes the minimum of the saturation size to 32. The spec can be modified as follows (italics): maxTbWidth=(cIdx==0)?MaxTbSizeY:max(MaxTbSizeY / SubWidthC,32)(8-41) maxTbHeight=(cIdx==0)?MaxTbSizeY:max(MaxTbSizeY / SubHeightC,32)(8-42)

[0058] Embodiment 4: Transform Skip Size Transform skip flags can be signaled up to a size of 32x32. This is independent of the maximum transform block size, whether it is 64 or 32. To make the connection with the maximum transform size, the text is modified as follows: log2_transform_skip_max_size_minus2 specifies the maximum block size used for the transformation and shall be in the range 0 to MaxTbLog2SizeY-3.

[0059] MaxTbLog2SizeY is calculated as follows (per the VVC spec): MaxTbLog2SizeY=sps_max_luma_transform_size_64_flag?6:5(7-28)

[0060] Embodiment 5: BDCM BDPCM uses the same signaling conditions as transform skip, so the above embodiment 4 is also applicable to BDPCM.

[0061] Embodiment 6: MIP The described general aspect proposes to enable MIP for CUs up to 64x64, regardless of the maximum transform size. This is done by enabling TU tiling when the CU size is larger than MaxTbSizeY. This is to improve coding efficiency when MaxTbSizeY is 32.

[0062] The basic idea of ​​the present invention is to allow MIP for CUs with a maximum size of 64x64, regardless of MaxTbSizeY. This improves coding performance when MaxTbSizeY is set to 32 by enabling MIP when the CU size is larger than 32x32. Experiments have shown that MIP performs better for arrays with large dimensions. The following results are generated by taking the VTM software as the anchor and test is a VTM that does not include MIP.

[0063] [Table 11]

[0064] Clearly, MIP provides a gain of 0.6% for class A1 (large dimensions) and a gain of 0.3% for class C (small dimensions). Therefore, when MaxTbSizeY is 32, enabling MIP for 64x64 CUs provides coding gain for arrays with large dimensions. Furthermore, since the same structure of MIP is maintained, no additional tools are required.

[0065] Compared to the current design, more flexibility to the encoder allows it to perform MIPs of up to 64x64 CUs, even for smaller transform sizes.

[0066] This can be achieved by TU tiling, i.e. distributing a CU across multiple TUs and running MIPs independently. There are two ways to do this: 1-TU tiling, then MIP prediction TU tiling after 2-MIP prediction

[0067] That is, considering that the size of a CU is 64x64, 32x64, or 64x32 and MaxTbSizeY is 32, the first option is to divide the CU into 32x32 TUs, perform MIP on the 32x32 blocks, generate a prediction signal, and code the residual. In doing so, a reference sample from the reconstructed 32x32 block can be used to generate a prediction signal. The second option is to perform MIP on a large CU (64x64, 32x64, or 64x32), then divide the CU into 32x32 TUs and code the residual. This second option is more consistent with the current design of VTM. This is because for traditional intra prediction (angular, DC, or planar), a prediction signal is generated with the same size as the TU, and as a result, the block from which the reference sample is reconstructed can be used to improve the prediction of neighboring blocks.

[0068] The corresponding specifications are shown below in italics.

[0069] [Table 12-1]

[0070] [Table 12-2]

[0071] The specification text already supports TU tiling when the transform unit is larger than the maximum transform size, which is indicated by the text below (shaded):

[0072] [Table 13]

[0073] Embodiment 7: LFNST Index In the specification text, LFNST (Low Frequency Non-Separable Transform) is allowed up to MaxTbSizeY. The initial motivation was to avoid latency issues when decoding large CUs of size 128x128, where LFNST indices are decoded after decoding the TU residual. Therefore, it was decided to allow LFNST up to the maximum transform size, which was originally 64. With the adoption of JVET-00545, the latency issue is not significant when the CU size is 64x64 and MaxTbSizeY is 32. Therefore, when MaxTbSIzeY is set to 32, it can be possible to allow LFNST indices in this case to improve coding gain.

[0074] The corresponding specification changes are shown in italics.

[0075] [Table 14]

[0076] Furthermore, since LFNST is only allowed if the primary transform is DCT2 (condition: tu_mts_idx[x0][y0]==0), this condition needs to be checked for multiple TU's. The changes are as follows:

[0077] [Table 15]

[0078] [Table 16]

[0079] That is, we define a variable MTS_notDCT2 that checks whether any of the TUs do not use DCT2. If so, LFNST is not allowed.

[0080] One embodiment of a method 400 under the general aspects described herein is shown in Figure 4. The method begins at start block 401, with control passing to block 410 to enable a coding tool based on a maximum transform size. Control passes from block 410 to block 420 to perform at least a portion of a discrete triangular transform on a subset of samples comprising the block. Control passes from block 420 to block 430 to encode the block using the enabled coding tool. The transformed subset of samples is used to determine transform coefficients for the block.

[0081] One embodiment of a method 500 under the general aspects described herein is shown in Figure 5. The method begins at start block 501, with control passing to block 510 to enable a coding tool based on a maximum transform size. Control passes from block 510 to block 520 to perform at least a portion of an inverse discrete triangular transform on a subset of samples that comprise the block. Control passes from block 520 to block 530 to decode the block using the enabled coding tool.

[0082] 6 shows one embodiment of an apparatus 600 for compressing, encoding, or decoding video using various coding tools depending on a maximum transform size. The apparatus includes a processor 1410, which may be interconnected through at least one port to a memory 1420. Both the processor 1410 and the memory 1420 may also have one or more additional interconnections to external connections.

[0083] Additionally, processor 610 is configured to insert or receive information in a bitstream and compress, encode, or decode using any of the described aspects.

[0084] This document describes various aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are specifically described and, at least to illustrate individual features, may often be read as limiting the invention. However, this is for clarity of description only and does not limit the applicability or scope of the aspects. Indeed, all different aspects can be combined or substituted to provide further aspects. Furthermore, these aspects can be combined or substituted with aspects described in previous applications.

[0085] The aspects described and contemplated in this document can be implemented in many different forms. Figures 1, 2, and 3 below provide some embodiments, but other embodiments are contemplated, and the descriptions of Figures 1, 2, and 3 do not limit the scope of implementations. At least one of the aspects relates generally to encoding and decoding video, and at least one other aspect relates generally to transmitting generated or encoded bitstreams. These and other aspects may be implemented as a method, an apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0086] In this application, the terms "reconstructed" and "decoded" can be used interchangeably, the terms "pixel" and "sample" can be used interchangeably, and the terms "image," "picture," and "frame" can be used interchangeably. Typically, but not necessarily, the term "reconstructed" is used on the encoder side, and the term "decoded" is used on the decoder side.

[0087] Various methods are described herein, each method including one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for the method to function properly, the order and / or use of specific steps and / or actions may be varied or combined.

[0088] Various methods and other aspects described in this document can be used to modify modules, such as intra-prediction modules, entropy encoding modules, and / or decoding modules (160, 360, 145, 330), of video encoder 100 and video decoder 200, as shown in Figures 1 and 2. Furthermore, aspects of this disclosure are not limited to VVC or HEVC, but can be applied, for example, to other standards and recommendations, whether existing or developed in the future, and extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically impossible, aspects described in this document can be used individually or in combination.

[0089] Various numerical values ​​are used in this document, for example, {{1,0}, {3,1}, {1,1}}, etc. The specific values ​​are for illustrative purposes, and the described aspects are not limited to these specific values.

[0090] 1 shows an encoder 100. Variations of this encoder 100 are possible, but for the sake of clarity, the following will describe the encoder 100 without listing all possible variations.

[0091] Before being encoded, a video sequence may undergo pre-encoding processing (101), such as applying a color transform to the input color picture (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0) or performing a remapping of the input picture components to obtain a signal distribution that is more resistant to compression (e.g., using histogram equalization of one of the color components). Metadata can be associated with the pre-processing and added to the bitstream.

[0092] In encoder 100, a picture is encoded by the encoder elements described below. The picture to be encoded is divided (102) into units, e.g., CUs, and processed. Each unit is encoded, e.g., using either intra mode or inter mode. If the unit is encoded in intra mode, intra prediction (160) is performed. In inter mode, motion estimation (175) and motion compensation (170) are performed. The encoder determines (105) whether to use one of intra mode or inter mode to encode the unit, and indicates the intra / inter decision, e.g., by a prediction mode flag. A prediction residual is calculated, e.g., by subtracting (110) the predicted block from the original image block.

[0093] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with the motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can also bypass both the transform and quantization, i.e., the residual is coded directly without applying a transform or quantization process.

[0094] The encoder decodes the coded block to provide reference data for further prediction. To decode the prediction residual, the quantized transform coefficients are dequantized (140) and inverse transformed (150). The decoded prediction residual and the predicted block are combined (155) to reconstruct an image block. An in-loop filter (165) is applied to the reconstructed picture, for example, to perform deblocking / SAO (Sample Adaptive Offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0095] 2 shows a block diagram of a video decoder 200. In the decoder 200, the bitstream is decoded by the decoder elements described below. The video decoder 200 generally performs a decoding pass that is the reverse of the encoding pass as described in FIG. Encoder 100 typically also performs video decoding as part of encoding the video data.

[0096] The decoder's input includes a video bitstream, which may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other coded information. Picture partition information indicates how the picture is divided. The decoder can then partition the picture according to the decoded picture partition information (235). To decode the prediction residual, the transform coefficients are inverse quantized (240) and inverse transformed (250). The decoded prediction residual and the predicted block are combined (255) to reconstruct an image block. The predicted block may result from intra prediction (260) or motion-compensated prediction (i.e., inter prediction) (275) (270). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0097] The decoded picture may further undergo post-decoding processing (285), such as an inverse color transformation (e.g., YCbCr 4:2:0 to RGB 4:4:4) or an inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process (101). The post-decoding process may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0098] FIG. 3 illustrates a block diagram of an example system in which various aspects and embodiments may be implemented. System 1000 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances, and servers. The elements of system 1000, singly or in combination, may be embodied in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to other similar systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more of the aspects described herein.

[0099] The system 1000 includes at least one processor 1010 configured to execute instructions loaded thereon, for example, to implement various aspects described herein. The processor 1010 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include non-volatile and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drives, and / or optical disk drives. The storage device 1040 may include, by way of non-limiting example, an internal storage device, an external storage device, and / or a network-accessible storage device.

[0100] The system 1000 includes, for example, an encoder / decoder module 1030 configured to process data to provide encoded or decoded video, which may include its own processor and memory. The encoder / decoder module 1030 represents a module that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Furthermore, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000 or may be incorporated within the processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0101] Program code loaded into the processor 1010 or the encoder / decoder 1030 to perform various aspects described herein may be stored in the storage device 1040 and then loaded into the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during the execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and arithmetic logic.

[0102] In some embodiments, memory internal to the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (which may be, for example, either the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be the memory 1020 and / or the storage device 1040, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, external non-volatile flash memory is used to store the television's operating system. In at least one embodiment, high-speed external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as MPEG-2, HEVC, or VVC (Versatile Video Coding).

[0103] Input to the elements of system 1000 can be provided through various input devices, as shown in block 1130. Such input devices include, but are not limited to, (i) an RF section that receives RF signals transmitted over the air by, for example, a broadcast station, (ii) a composite input, (iii) a USB input, and / or (iv) an HDMI input.

[0104] In various embodiments, the input devices of block 1130 have associated respective input processing elements, as is known in the art. For example, the RF section may be associated with elements necessary to (i) select a desired frequency (also referred to as selecting a signal or band-limiting a signal to a particular frequency band), (ii) downconvert the selected signal, (iii) band-limit again to a narrower frequency band to select a signal frequency band, which in certain embodiments may be referred to as a channel (for example), (iv) demodulate the downconverted, band-limited signal, (v) perform error correction, and (vi) demultiplex to select a desired stream of data packets. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a downconverter, a demodulator, an error corrector, and a demultiplexer. The RF section may include, for example, a tuner that performs various of these functions, including downconverting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency near baseband) or to baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements receive RF signals transmitted over a wired (e.g., cable) medium and perform frequency selection by filtering, downconverting, and re-filtering to a desired frequency band. Various embodiments rearrange the order of the above-mentioned (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements can include inserting elements between existing elements, such as inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0105] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or within processor 1010, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented, for example, in a separate interface IC or within processor 1010, as desired. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, an encoder / decoder 1030 operating in combination with processor 1010 and memory and storage elements, to process the data stream as needed for presentation on an output device.

[0106] The various elements of the system 1000 may be provided within an integrated housing in which the various elements may be interconnected and transmit data between each other using suitable connection arrangements 1140, e.g., internal buses known in the art, including an I2C bus, wiring, and printed circuit boards.

[0107] System 1000 includes a communication interface 1050 that enables communication with other devices over a communication channel 1060. Communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over communication channel 1060. Communication interface 1050 may include, but is not limited to, a modem or a network card, and communication channel 1060 may be implemented in a wired and / or wireless medium, for example.

[0108] In various embodiments, data is streamed to system 1000 using a wireless network such as IEEE 802.11. The wireless signal in these embodiments is received, for example, via communication channel 1060 and communication interface 1050 adapted for Wi-Fi communication. Communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. In another embodiment, streaming data is provided to system 1000 using a set-top box that delivers data via an HDMI connection in input block 1130. In yet another embodiment, streaming data is provided to system 1000 using an RF connection in input block 1130.

[0109] System 1000 can provide output signals to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. The other peripheral devices 1120, in various example embodiments, include one or more of a standalone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of system 1000. In various embodiments, control signals are communicated between system 1000 and display 1100, speakers 1110, or other peripheral devices 1120 using signaling such as AV.Link, CEC, or other communication protocols that enable inter-device control with or without user intervention. Output devices can be communicatively coupled to system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, output devices can be connected to system 1000 using communication channel 1060 via communication interface 1050. The display 1100 and speakers 1110 may be integrated into a single unit along with other components of the system 1000, for example, in an electronic device such as a television. In various embodiments, the display interface 1070 includes a display driver, for example, a timing controller (T Con) chip.

[0110] Alternatively, the display 1100 and speakers 1110 can be separate from one or more of the other components, for example, if the RF portion of the input 1130 is part of a separate set-top box. In various embodiments where the display 1100 and speakers 1110 are external components, the output signal can be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0111] The embodiments may be performed by computer software implemented by the processor 1010, by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technology environment and may be implemented using any suitable data storage technology, such as, by way of non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 1010 may be of any type suitable for the technology environment and may include, by way of non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0112] Various implementations include decoding. As used herein, "decoding" can encompass all or part of the processing performed on a received encoded sequence, e.g., to generate a final output suitable for display. In various embodiments, such processing includes one or more of the processing typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processing additionally or alternatively includes processing performed by decoders in various implementations described herein, such as, for example, extracting indices for weights used for various intra prediction reference arrays.

[0113] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to refer generally to a broader decoding process will be clear based on the context of the particular description and is believed to be well understood by those skilled in the art.

[0114] Various implementations include encoding. As used herein, "encoding," similar to the above description of "decoding," can encompass, for example, all or part of the processing performed on an input video sequence to generate an encoded bitstream. In various embodiments, such processing includes one or more of the processing typically performed by an encoder, such as, for example, partitioning, differential encoding, transforming, quantizing, and entropy encoding. In various embodiments, such processing additionally or alternatively includes processing performed by the encoders of various implementations described herein, such as, for example, weighting intra-prediction reference arrays.

[0115] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to refer generally to a broader encoding process will be clear based on the context of a particular description and is believed to be well understood by those skilled in the art.

[0116] It should be noted that the syntax elements used herein are descriptive terms and therefore do not exclude the use of other syntax element names.

[0117] Where a figure is presented as a flowchart, it should be understood that the figure also provides a block diagram of the corresponding apparatus. Similarly, where a figure is presented as a block diagram, it should be understood that the figure also provides a flowchart of the corresponding method / process.

[0118] Various embodiments refer to rate-distortion calculation or rate-distortion optimization. During the encoding process, a balance or trade-off between rate and distortion is typically considered, often subject to computational complexity constraints. Rate-distortion optimization is typically formulated to minimize a rate-distortion function, which is a weighted sum of rate and distortion. There are various approaches to solving the rate-distortion optimization problem. For example, these approaches may be based on extensive testing of all encoding options, including all considered modes or coding parameter values, with a thorough evaluation of their encoding costs and the associated distortion of the reconstructed signal after encoding and decoding. Also, to reduce encoding complexity, faster approaches may be used, particularly calculation of approximate distortion based on a prediction or prediction residual signal rather than a reconstructed signal. These two approaches may also be used in combination, e.g., using approximate distortion for only some of the possible encoding options and full distortion for others. Another approach evaluates only a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform the optimization, but the optimization does not necessarily involve a thorough evaluation of both the encoding cost and the associated distortion.

[0119] Implementations and aspects described herein may be implemented in, for example, a method or process, an apparatus, a software program, a data stream, or a signal. Even if described in the context of only one type of implementation (e.g., described only as a method), the described implementation of the features may also be implemented in other forms (e.g., an apparatus or a program). An apparatus may be implemented in, for example, appropriate hardware, software, and firmware. A method may be implemented in, for example, a processor, where a processor refers to a general processing device including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Furthermore, a processor may include, for example, a communication device such as a computer, a mobile phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate communication of information between end users.

[0120] References to "one embodiment" or "embodiment" or "one implementation" or "implementation," as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with an embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation," and other variations thereof, in various places throughout this document do not necessarily all refer to the same embodiment.

[0121] Additionally, this document may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0122] Additionally, this document may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0123] Additionally, this document may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information can include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" generally involves in some way, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0124] Use of any of " / ", "and / or", "at least one of", e.g., "A / B", "A and / or B", "at least one of A and B" should be understood to be intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, "A, B, and / or C" and "at least one of A, B, and C" are intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, B, and C). This can be expanded as many times as the number of items listed, as would be apparent to one of ordinary skill in the art.

[0125] Also, the term "signaling" as used herein specifically means indicating something to a corresponding decoder. For example, in a particular embodiment, an encoder signals a particular one of multiple weights to be used for an intra-prediction reference array. Thus, in some embodiments, the same parameters are used on both the encoder and decoder sides. Thus, for example, the encoder may transmit a particular parameter to the decoder (explicit signaling) so that the decoder can use the same particular parameter. Conversely, if the decoder already has that particular parameter and other parameters, signaling without transmission (implicit signaling) may be used to simply allow the decoder to recognize and select that particular parameter. By avoiding transmitting actual functions, bit savings are realized in various embodiments. It should be understood that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder in various embodiments. While the above description refers to the verb form of the word "signal," the word "signal" may also be used as a noun herein.

[0126] As will be apparent to those skilled in the art, implementations can generate a variety of signals formatted to carry information that can be stored or transmitted, for example. Such information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal can be formatted to carry a bitstream of a described embodiment. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0127] Embodiments may include one or more of the following features or entities, alone or in combination, across a variety of different claim categories and types. Enabling matrix-based intra prediction for coding units up to a determined size, regardless of the maximum transform size. · Enabling transform unit tiling when the coding unit size is larger than the maximum transform size. Enable the low-frequency non-separable transform (LFNST) up to a determined size. Indicate whether any transform units do not use DCT2 as a transform, and if so, include in the bitstream or check the bitstream for syntax elements that do not allow LFNST. A bitstream or signal containing one or more of the described syntax elements or variations thereof. Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal that includes one or more of the described syntax elements or variations thereof. A television, set-top box, mobile phone, tablet, or other electronic device that performs in-loop filtering according to any of the described embodiments. A television, set-top box, mobile phone, tablet, or other electronic device that performs in-loop filtering according to any of the described embodiments and displays the resulting image (e.g., using a monitor, screen, or other type of display). A television, set-top box, mobile phone, tablet, or other electronic device that tunes to a channel (e.g., using a tuner) to receive a signal containing the encoded image and performs in-loop filtering according to any of the described embodiments. A television, set-top box, mobile phone, tablet, or other electronic device that receives a signal containing an encoded image wirelessly (e.g., using an antenna) and performs in-loop filtering according to any of the described embodiments.

[0128] Various other generalized and specific inventions and claims are also supported and contemplated throughout this disclosure.

Claims

1. 1. A method for video encoding, comprising: determining whether matrix-based intra prediction (MIP) is enabled for the block; obtaining a maximum transform size; determining whether a block size of the block is greater than the maximum transform size; In response to determining that matrix-based intra prediction (MIP) is enabled for the block and determining that the block size of the block is greater than the maximum transform size, dividing the block into a plurality of transform blocks of the maximum transform size; performing matrix-based intra prediction on each of a plurality of blocks of the maximum transform size to obtain a prediction; encoding the block based on the prediction; A method for providing the above.

2. The method of claim 1, wherein the maximum transform size is 32 x 32.

3. The method of claim 1, wherein the block size is 64x64, 64x32 or 32x64.

4. 1. An apparatus comprising: one or more processors; at least one memory coupled to the one or more processors; wherein the one or more processors: determining whether matrix-based intra prediction (MIP) is enabled for the block; Getting the maximum conversion size, determining whether a block size of the block is greater than the maximum transform size; In response to determining that matrix-based intra prediction (MIP) is enabled for the block and determining that the block size of the block is greater than the maximum transform size, dividing the block into a plurality of transform blocks of the maximum transform size; performing matrix-based intra prediction on each of a plurality of blocks of the maximum transform size to obtain a prediction; and encoding the block based on the prediction.

2. A device configured to perform the steps of:

5. The device of claim 4, wherein the maximum transform size is 32 x 32.

6. The device of claim 4, wherein the block size is 64x64, 64x32 or 32x64.

7. A method for video decoding, comprising: determining whether matrix-based intra prediction (MIP) is enabled for the block; obtaining a maximum transform size; determining that a block size of the block is greater than the maximum transform size; In response to determining that matrix-based intra prediction (MIP) is enabled for the block and determining that the block size of the block is greater than the maximum transform size, dividing the block into a plurality of transform blocks of the maximum transform size; performing matrix-based intra prediction on each of a plurality of blocks of the maximum transform size to obtain a prediction; decoding the block based on the prediction; A method for providing the above.

8. The method of claim 7, wherein the maximum transform size is 32x32.

9. The method of claim 7, wherein the block size is 64x64, 64x32 or 32x64.

10. 1. An apparatus comprising: one or more processors; at least one memory coupled to the one or more processors; wherein the one or more processors: determining whether matrix-based intra prediction (MIP) is enabled for the block; Getting the maximum conversion size, determining that a block size of the block is greater than the maximum transform size; In response to determining that matrix-based intra prediction (MIP) is enabled for the block and determining that the block size of the block is greater than the maximum transform size, dividing the block into a plurality of transform blocks of the maximum transform size; performing matrix-based intra prediction on each of a plurality of blocks of the maximum transform size to obtain a prediction; and decoding the block based on the prediction; 2. A device configured to perform the steps of:

11. The device of claim 10, wherein the maximum transform size is 32x32.

12. The device of claim 10, wherein the block size is 64x64, 64x32, or 32x64.

13. A device, An apparatus according to claim 10; at least one of (i) an antenna configured to receive a signal including the block; (ii) a band limiter configured to limit the received signal to a frequency band including the block; and (iii) a display configured to display an output representing the video block; A device with.

14. A non-transitory computer readable medium containing a computer program that causes a processor to perform the method of any one of claims 1 to 3 or 7 to 9.

Images