Transform size interaction with coding tools

Abstract

The method and apparatus for implementing a discrete triangular transform is based on a maximum transform size. In one embodiment, matrix-based intra prediction enables coding unit sizes up to a specified size, regardless of the maximum transform size. In another embodiment, a low-frequency non-separable transform is used to improve coding gain. Syntax in the bitstream can be used to indicate the coding tool used.

Description

[Technical field]

[0001] At least one of the present embodiments relates generally to a method or apparatus for video encoding or decoding, compression or decompression. [Background technology]

[0002] To achieve high compression efficiency, image and video coding schemes usually employ prediction, including motion vector prediction, and transformation that exploits spatial and temporal redundancy in the video content. In general, intra- or inter-prediction is used to exploit correlation within or between frames, and then the difference between the original and predicted image, often called prediction error or prediction residual, is transformed, quantized, and entropy coded. To reconstruct the video, the compressed data is decoded by the inverse processes corresponding to entropy coding, quantization, transformation, and prediction.

[0003] In the development of the Versatile Video Coding (VVC) standard, the maximum transform size is variable between 32 and 64. The maximum transform size interacts with other transform coding tools. Summary of the Invention

[0004] At least one of the present embodiments relates generally to a method or apparatus for video encoding or decoding, and more specifically to a method or apparatus for interaction between a maximum transform size and a transform coding tool in a video encoder or video decoder.

[0005] According to a first aspect, there is provided a method, comprising the steps of enabling a coding tool based on a maximum transform size, performing at least a portion of a discrete triangular transform on a subset of samples that comprise a block, and encoding the block using the enabled coding tool.

[0006] According to a second aspect, a method is provided, the method including the steps of enabling a coding tool based on a maximum transform size, performing at least a portion of an inverse discrete triangular transform on a subset of samples that comprise a block, and decoding the block using the enabled coding tool.

[0007] According to another aspect, there is provided an apparatus, comprising a processor, the processor may be configured to encode a block of a video or decode a bitstream by performing any of the methods described above.

[0008] According to another general aspect of at least one embodiment, there is provided a device comprising an apparatus according to any of the decoding embodiments and at least one of: (i) an antenna configured to receive a signal, the signal including a video block; (ii) a band limiter configured to limit the received signal to a frequency band including the video block; or (iii) a display configured to display an output representative of the video block.

[0009] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that includes data content generated in accordance with any of the described encoding embodiments or variations.

[0010] According to another general aspect of at least one embodiment there is provided a signal including video data generated in accordance with any of the described encoding embodiments or variations.

[0011] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described encoding embodiments or variations.

[0012] According to another general aspect of at least one embodiment, there is provided a computer program product including instructions that, when executed by a computer, cause the computer to perform any of the described decoding embodiments or variations.

[0013] These and other aspects, features, and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments, which is to be read in conjunction with the accompanying drawings. [Brief description of the drawings]

[0014] [Figure 1] Denotes standard and common video compression methods. [Diagram 2] 1 shows a standard common video decompression method. [Diagram 3] 1 illustrates an exemplary processor-based subsystem for implementing general described aspects. [Figure 4] 1 illustrates one embodiment of a method according to the described embodiment. [Diagram 5] 1 illustrates another embodiment of the method according to the described embodiment. [Figure 6] 1 illustrates an exemplary apparatus under the described embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] At least one of the present embodiments relates generally to a method or apparatus for video encoding or decoding, and more specifically to a method or apparatus for interaction between a maximum transform size and a transform coding tool in a video encoder or video decoder.

[0016] A general aspect described herein is the field of video compression, where in the recent adoption of VVC, the maximum transform size is variable from 32 to 64, and the interaction between the maximum transform size and other transform coding tools. The value is calculated as follows:

[0017]

number

[0018] Maximum transform size interacts with the following tools: 1-Zero-setting process: First, VVC performs zero-setting to reduce the complexity for large transform sizes. For 2-D DCT2 transforms, only the top-left 32x32 coefficients are kept while the rest are set to zero. That is, for 64x64, 64x32, and 32x32, DCT2 calculates the first 32 coefficients in both horizontal and vertical directions, and non-DCT2 transforms (DST7 and DCT8) perform zero-setting to keep the top-left 16x16 coefficients. With the adoption of JVET-00545, it is not resolved how zero-setting is performed when the maximum transform size is 32.

[0019] A capture of the draft text is shown below, with the zero settings shaded.

[0020] [Table 1]

[0021] This means that if tu_mts_idx is greater than zero, meaning that an MTS transform is used (DST7, DST7), the zero set width and height are set to 16, while if tu_mts_idx is zero (DCT2), the zero set is set to 32. 2-MTS size: MTS or multiple transform selection is a transform tool adopted in VVC where the choice between DCT7 and DCT8 pairs is allowed as other transform pairs strengthen the DCT2 transform pair. MTS is implemented for block sizes from 4x4 to 32x32; i.e. half size DCT2. With the adoption of JVET-00545, it is not resolved how the MTS size is taken into account when the maximum transform size is 32.

[0022] A capture of the draft text is shown below, with the mts size shaded.

[0023] [Table 2-1]

[0024] [Table 2-2]

[0025] This indicates that if both width and height are less than 32, MTS will be signaled, regardless of MaxTbSizeY. 3- Chroma transform size: In VVC, the chroma size is half of the luma size. That is, the chroma transform block is allowed to be 2x2~32x32, while the luma size is 4x4~64x64. With the adoption of JVET-00545, it is not resolved how the chroma size is fixed when the maximum transform size is 32.

[0026] In the VVC spec, the chroma size is calculated as follows:

[0027] [Table 3] maxTbWidth=(cIdx==0)?MaxTbSizeY:MaxTbSizeY / SubWidthC(8-41) maxTbHeight=(cIdx==0)?MaxTbSizeY:MaxTbSizeY / SubHeightC(8-42) cIdx is the color index (0 for luminance, 1 for saturation).

[0028] In common testing condition (CTC), 4:2:0 chroma format is used and the maximum transform size (MaxTbSizeY) is 64. Therefore, the maximum transform size is 32 for chroma in CTC. However, when MaxTbSizeY is 32, the chroma maximum size is 16 according to the current SPEC. 4- Transform skip size: In VVC, transform skip is performed for block sizes that have the same range as DCT2. In other words, transform skip is performed for block sizes from 4x4 to 64x64. With the adoption of JVET-00545, it is not resolved how the transform skip size is fixed when the maximum transform size is 32.

[0029] In the VVC specification, the maximum transform skip size is defined as follows: log2_transform_skip_max_size_minus2 specifies the maximum block size used for transform skipping and shall be in the range 0 to 3.

[0030] If not present, the value of log2_transform_skip_max_size_minus2 is inferred to be equal to 0.

[0031] The variable MaxTsSize is set equal to 1<<(log2_transform_skip_max_size_minus2+2).

[0032] That is, the maximum MaxTsSize can take a value between 4 and 32, regardless of the MaxTbSizeY value. 5-BDCM Size: BD-PCM is a block-based delta pulse code modulation. It is a coding tool for the transform skip residual. It is currently allowed with the same size conditions as the transform skip, i.e. a maximum of MaxTsSize. The text below shows the conditions (shaded) for coding the BDPCM flags.

[0033] [Table 4]

[0034] 6-MIP In VVC Draft 6, MIP (Matrix-Based Intra Prediction) is an intra prediction mode in which the prediction signal is generated by multiplying the reference sample with some trained prediction matrix with a fixed shift. The mode is signaled when the CU size is less than or equal to the maximum allowed transform size dimension. This restriction was necessary to limit memory requirements and coding complexity. This is because MIP is a matrix-based method and the prediction matrix is ​​larger for larger blocks.

[0035] First, the maximum transform size (MaxTbSizeY) is always kept as 64 in VTM5.0. However, in VTM6.0, this value can be 64 or 32. A sample of the draft text of VTM6.0 is provided below (the shaded area indicates the MIP part):

[0036] [Table 5-1]

[0037] [Table 5-2]

[0038] The MaxTbSizeY value was fixed to 64. However, in the adoption of JVET-00545, MaxTbSizeY can be either 64 or 32.

[0039] Intuitively, if MaxTbSizeY is 32, MIP is signaled up to CU size of 32x32. This prevents larger MIPs from using CUs, thus limiting coding efficiency. The present described embodiment proposes to enable MIP for CUs up to 64x64, regardless of the maximum transform size. This is done by enabling TU tiling when the CU size is larger than MaxTbSizeY.

[0040] Initially, the maximum transform size is always kept as 64 in VTM5.0. However, with the recent adoption of JVET-00545, the maximum transform size (MaxTbSizeY) can be either 64 or 32, controlled by the SPS flag (sps_sbt_max_size_64_flag). When this happens, the zeroing process, MTS size, chroma transform size, transform skip size, and BDCM size need to be adapted to this change.

[0041] A general aspect proposes to adapt the signaling of the following tools according to the maximum transform size: zero set process, MTS size, chroma transform size, transform skip size and BDCM. The affected codec modules are the intra-coding designs (160) and 260 of Figures 1 and 2.

[0042] Embodiment 1: Zero Setting Process In this embodiment, the zeroing process depends on the maximum transform size. Thus, if the maximum size is 32 instead of 64, the zeroing size is reduced by half. This is shown in the text below (in italics):

[0043] [Table 6]

[0044] This can also be done independently for the DCT2 transform and the other MTS transforms (DST7 and DCT8). I.e. if you want to do it only for the DCT2:

[0045] [Table 7]

[0046] Otherwise, for DST7 / DCT8 only

[0047] [Table 8]

[0048] Example 2: MTS size MTS signaling is allowed up to a size of 32x32. This is independent of MaxTbSizeY, whether it is 64 or 32. To accommodate a connection with MaxTbSizeY, MTS signaling can be resized up to MaxTbSizeY / 2xMaxTbSizeY / 2. This is shown in italics in the spec below:

[0049] [Table 9-1]

[0050] [Table 9-2]

[0051] Note that this has a direct impact on the subblock transform (SBT) tool. SBT is a transform unit partitioning tool for inter-blocks that semantically selects between DCT2, DST7, and DCT8 transforms. According to this specification, The variable implicitMtsEnabled is derived as follows: -If sps_mts_enabled_flag is equal to 1 and one of the following conditions is true, then implicitMtsEnabled is set equal to 1: -IntraSubPartitionsSplitType is not equal to ISP_NO_SPLIT -cu_sbt_flag is equal to 1 and Max(nTbW, nTbH) is less than or equal to 32 -sps_explicit_mts_intra_enabled_flag is equal to 0, CuPredMode[0][xTbY][yTbY] is equal to MODE_INTRA, lfnst_idx[x0][y0] is equal to 0, and intra_mip_flag[x0][y0] is equal to 0 - Otherwise, implicitMtsEnabled is equal to 0.

[0052] A variable trTypeHor that specifies the horizontal transformation kernel and a variable trTypeVer that specifies the vertical transformation kernel are derived as follows. - If cIdx is greater than 0, trTypeHor and trTypeVer are equal to 0. - Otherwise, if ImplicitMtsEnabled is equal to 1, the following applies: - If IntraSubPartitionsSplitType is not equal to ISP_NO_SPLIT or sps_explicit_mts_intra_enabled_flag is equal to 0 and CuPredMode[0][xTbY][yTbY] is equal to MODE_INTRA, trTypeHor and trTypeVer are derived as follows: trTypeHor=(nTbW>=4&&nTbW<=16)?1:0(8-975) trTypeVer=(nTbH>=4&&nTbH<=16)?1:0(8-976) Otherwise (cu_sbt_flag is equal to 1), trTypeHor and trTypeVer are specified in Tables 8-15 depending on cu_sbt_horizontal_flag and cu_sbt_pos_flag. - Otherwise, trTypeHor and trTypeVer are as specified in Table 8-14 according to tu_mts_idx[xTbY][yTbY].

[0053] [Table 10]

[0054] Transform type 2 means DCT8, 1 means DST7.

[0055] That is, if MTS is limited to MaxTbSizeY / 2 and MaxTbSizeY is 32, then the above table cannot be used because DST7 and DCT8 of size 32x32 are not supported. Instead, DCT2 must be used. The corresponding specification changes are as follows:

[0056] The variable implicitMtsEnabled is derived as follows: - if sps_mts_enabled_flag is equal to 1 and one of the following conditions is true, inclipitMtsEnabled is set equal to 1: -IntraSubPartitionsSplitType is not equal to ISP_NO_SPLIT -cu_sbt_flag is equal to 1 and Max(nTbW, nTbH) is less than or equal to MaxTbSizeY / 2 -sps_explicit_mts_intra_enabled_flag is equal to 0, CuPredMode[0][xTbY][yTbY] is equal to MODE_INTRA, lfnst_idx[x0][y0] is equal to 0, and intra_mip_flag[x0][y0] is equal to 0

[0057] Example 3: Saturation conversion size According to the VVC spec, the chroma maximum transform width and height can be half of the maximum luma 1. However, since the maximum luma transform size can be 32, the maximum size of chroma can be 16. This is a small number and is not considered to be useful in practice. Therefore, this embodiment fixes the minimum of chroma size to 32. The specification can be modified as follows (italics): maxTbWidth=(cIdx==0)?MaxTbSizeY:max(MaxTbSizeY / SubWidthC,32)(8-41) maxTbHeight=(cIdx==0)?MaxTbSizeY:max(MaxTbSizeY / SubHeightC,32)(8-42)

[0058] Example 4: Transform Skip Size Transform skip flags can be signaled up to a size of 32x32. This is independent of the maximum transform block size, whether it is 64 or 32. To make the connection with the maximum transform size, the text is modified as follows: log2_transform_skip_max_size_minus2 specifies the maximum block size used for the transformation and shall be in the range of 0 to MaxTbLog2SizeY-3.

[0059] MaxTbLog2SizeY is calculated (according to the VVC spec) as follows: MaxTbLog2SizeY=sps_max_luma_transform_size_64_flag?6:5(7-28)

[0060] Embodiment 5: BDCM BDPCM uses the same signaling conditions as transform skip, so the above embodiment 4 is also applicable to BDPCM.

[0061] Embodiment 6: MIP The described general aspect proposes to enable MIP for CUs up to 64x64, regardless of the maximum transform size. This is done by enabling TU tiling when the CU size is larger than MaxTbSizeY. This is to improve coding efficiency when MaxTbSizeY is 32.

[0062] The basic idea of ​​the present invention is to allow MIP for CUs with a maximum size of 64x64, regardless of MaxTbSizeY. This is to improve the coding performance when MaxTbSizeY is set to 32 by enabling MIP when the CU size is larger than 32x32. Experiments have shown that MIP works better for arrays with large dimensions. The following results are generated by taking the VTM software as an anchor and test is a VTM without MIP.

[0063] [Table 11]

[0064] Apparently, MIP gives a gain of 0.6% for class A1 (large dimensions) and a gain of 0.3% for class C (small dimensions). Thus, enabling MIP on a 64x64 CU provides coding gain for arrays with large dimensions when MaxTbSizeY is 32. Moreover, since the same structure of MIP is maintained, no additional tools are required.

[0065] Compared to current designs, more flexibility to the encoder allows it to perform MIPs of up to 64x64 CUs, even for smaller transform sizes.

[0066] This can be achieved by TU tiling, i.e. distributing a CU across multiple TUs and running MIPs independently. There are two ways to do this: 1-TU tiling, followed by MIP prediction TU tiling after 2-MIP prediction

[0067] That is, considering that the CU of size 64x64, 32x64 or 64x32 and MaxTbSizeY is 32, the first option is to split the CU into TUs of 32x32, perform MIP on the 32x32 block, generate a prediction signal, and code the residual. In that case, the reference sample from the reconstructed 32x32 block can be used to generate the prediction signal. The second option is to perform MIP on a large CU (64x64, 32x64 or 64x32), then split the CU into TUs of size 32x32, and code the residual. The second option is more consistent with the current design of VTM. This is because for traditional intra prediction (angular, DC or planar), the prediction signal is generated with the same size as the TU, so that the block with the reference sample reconstructed can be used to improve the prediction of the neighboring block.

[0068] The corresponding specifications are shown below in italics.

[0069] [Table 12-1]

[0070] [Table 12-2]

[0071] The specification text already supports TU tiling when the transform unit is larger than the maximum transform size, which is indicated in the text below (shaded).

[0072] [Table 13]

[0073] Example 7: LFNST index In the specification text, LFNST (Low Frequency Non-separable Transform) is allowed up to MaxTbSizeY. The original motivation was to avoid latency issues when decoding large CUs of size 128x128, where LFNST indices are decoded after decoding the TU residual. Therefore, it was decided to allow LFNST up to the maximum transform size, which was initially 64. With the adoption of JVET-00545, the latency issue is not significant when the CU size is 64x64 and MaxTbSizeY is 32. Thus, it can be allowed to allow LFNST indices in this case to improve coding gain when MaxTbSIzeY is set to 32.

[0074] The corresponding specification changes are shown in italics.

[0075] [Table 14]

[0076] Furthermore, LFNST is only allowed if the primary transform is DCT2 (condition: tu_mts_idx[x0][y0]==0), so this condition needs to be checked for multiple TU's. The changes are as follows:

[0077] [Table 15]

[0078] [Table 16]

[0079] That is, we define a variable MTS_notDCT2 that checks whether any of the TUs do not use DCT2. If so, LFNST is not allowed.

[0080] One embodiment of a method 400 under the general aspects described herein is shown in Figure 4. The method begins at start block 401, where control passes to block 410 for enabling a coding tool based on a maximum transform size. Control passes from block 410 to block 420, where at least a portion of a discrete triangular transform is performed on a subset of samples that comprise the block. Control passes from block 420 to block 430, where the enabled coding tool is used to code the block. The transformed subset of samples is used to determine transform coefficients for the block.

[0081] One embodiment of a method 500 under the general aspects described herein is shown in Figure 5. The method begins at start block 501, where control passes to block 510 for enabling a coding tool based on a maximum transform size. Control passes from block 510 to block 520, where at least a portion of an inverse discrete triangular metric transform on a subset of samples that comprise the block. Control passes from block 520 to block 530, where the enabled coding tool is used to decode the block.

[0082] 6 shows an embodiment of an apparatus 600 for compressing, encoding, or decoding video using various coding tools depending on a maximum transform size. The apparatus includes a processor 1410 and may be interconnected through at least one port to a memory 1420. Both the processor 1410 and the memory 1420 may also have one or more additional interconnections to external connections.

[0083] Additionally, processor 610 is configured to insert or receive information in a bitstream and compress, encode, or decode using any of the described aspects.

[0084] Various aspects are described in this document, including tools, features, embodiments, models, methods, and the like. Many of these aspects are specifically described and may often be read as limiting the invention, at least to illustrate individual features. However, this is for purposes of clarity of description and not to limit the applicability or scope of the aspects. Indeed, all different aspects can be combined or substituted to provide further aspects. Additionally, these aspects can be combined or substituted with aspects described in previous applications.

[0085] The aspects described and contemplated in this document can be implemented in many different forms. Figures 1, 2, and 3 below provide some embodiments, but other embodiments are contemplated, and the descriptions of Figures 1, 2, and 3 do not limit the scope of implementations. At least one of the aspects generally relates to encoding and decoding video, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as a method, an apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0086] In this application, the terms "reconstructed" and "decoded" can be used interchangeably, the terms "pixel" and "sample" can be used interchangeably, and the terms "image", "picture", and "frame" can be used interchangeably. Usually, but not necessarily, the term "reconstructed" is used on the encoder side, and the term "decoded" is used on the decoder side.

[0087] Various methods are described herein, each of which includes one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for the method to operate properly, the order and / or use of specific steps and / or actions may be varied or combined.

[0088] Various methods and other aspects described herein may be used to modify modules of the video encoder 100 and video decoder 200, such as the intra prediction module, the entropy encoding module, and / or the decoding module (160, 360, 145, 330) as shown in Figures 1 and 2. Furthermore, aspects of the present disclosure are not limited to VVC or HEVC, but may be applied to other standards and recommendations, such as existing or developed in the future, and extensions of any such standards and recommendations, including VVC and HEVC. Unless otherwise indicated or technically impossible, aspects described herein may be used individually or in combination.

[0089] Various numerical values ​​are used in this document, for example, {{1,0}, {3,1}, {1,1}}, etc. The specific values ​​are for illustrative purposes, and the described aspects are not limited to these specific values.

[0090] 1 shows an encoder 100. Variations of this encoder 100 are possible, but for the sake of clarity, the following describes the encoder 100 without listing all possible variations.

[0091] Before being encoded, a video sequence may undergo pre-encoding processing (101), such as applying a color transformation to the input color picture (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0) or performing a remapping of the input picture components to obtain a signal distribution that is more resistant to compression (e.g., using histogram equalization of one of the color components). Metadata can be associated with the pre-processing and added to the bitstream.

[0092] In the encoder 100, a picture is encoded by the encoder elements described below. The picture to be encoded is divided (102) into units, e.g., CUs, and processed. Each unit is encoded, e.g., using either intra mode or inter mode. If the unit is encoded in intra mode, intra prediction (160) is performed. In inter mode, motion estimation (175) and motion compensation (170) are performed. The encoder decides (105) whether to use one of intra mode or inter mode to encode the unit, and indicates the intra / inter decision, e.g., by a prediction mode flag. A prediction residual is calculated, e.g., by subtracting (110) the predicted block from the original image block.

[0093] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, as well as the motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., the residual is coded directly without applying any transform or quantization processes.

[0094] The encoder decodes the coded block to provide reference data for further prediction. To decode the prediction residual, the quantized transform coefficients are dequantized (140) and inverse transformed (150). The decoded prediction residual and the predicted block are combined (155) to reconstruct an image block. An in-loop filter (165) is applied to the reconstructed picture, for example to perform deblocking / SAO (Sample Adaptive Offset) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0095] Figure 2 shows a block diagram of a video decoder 200, where the bitstream is decoded by the decoder elements described below. The video decoder 200 generally performs a decoding pass that is the reverse of the encoding pass as described in Figure 1. The encoder 100 also generally performs video decoding as part of the encoding of the video data.

[0096] The input of the decoder includes a video bitstream, which may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other coded information. Picture partition information indicates how the picture is partitioned. The decoder can then partition the picture according to the decoded picture partition information (235). To decode the prediction residual, the transform coefficients are inverse quantized (240) and inverse transformed (250). The decoded prediction residual and the predicted block are combined (255) to reconstruct an image block. The predicted block may result from intra prediction (260) or motion compensated prediction (or inter prediction) (275) (270). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0097] The decoded picture may further undergo post-decoding processing (285), such as an inverse color transformation (e.g., from YCbCr 4:2:0 to RGB 4:4:4), or an inverse remapping that performs the inverse of the remapping process performed in the pre-encoding process (101). The post-decoding process may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0098] FIG. 3 illustrates a block diagram of an example of a system in which various aspects and embodiments may be implemented. System 1000 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected appliances, and servers. The elements of system 1000 may be embodied, alone or in combination, in a single integrated circuit, multiple ICs, and / or discrete components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or discrete components. In various embodiments, system 1000 is communicatively coupled to other similar systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more of the aspects described herein.

[0099] The system 1000 includes at least one processor 1010 configured to execute instructions loaded thereon, for example to implement various aspects described herein. The processor 1010 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include non-volatile and / or volatile memory, including, but not limited to, EEPROM, ROM, PROM, RAM, DRAM, SRAM, flash, magnetic disk drives, and / or optical disk drives. The storage device 1040 may include, by way of non-limiting example, an internal storage device, an external storage device, and / or a network-accessible storage device.

[0100] The system 1000 includes, for example, an encoder / decoder module 1030 configured to process data to provide encoded or decoded video, which may include its own processor and memory. The encoder / decoder module 1030 represents a module that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Furthermore, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000 or may be incorporated within the processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0101] Program code loaded into the processor 1010 or the encoder / decoder 1030 to perform various aspects described herein may be stored in the storage device 1040 and then loaded into the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during the execution of the operations described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, as well as intermediate or final results from the processing of equations, formulas, operations, and arithmetic logic.

[0102] In some embodiments, memory internal to the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in another embodiment, memory external to the processing device (which may be, for example, either the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be the memory 1020 and / or the storage device 1040, for example, dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store the TV's operating system. In at least one embodiment, a high-speed external dynamic volatile memory, such as RAM, is used as working memory for video encoding and decoding operations, such as MPEG-2, HEVC, or VVC (Versatile Video Coding).

[0103] Input to the elements of system 1000 may be provided through various input devices, as shown in block 1130. Such input devices may include, but are not limited to, (i) an RF section that receives, for example, an RF signal transmitted over the air by a broadcast station, (ii) a composite input, (iii) a USB input, and / or (iv) an HDMI input.

[0104] In various embodiments, the input devices of block 1130 have associated respective input processing elements as known in the art. For example, the RF section may have associated elements necessary to (i) select a desired frequency (also referred to as selecting a signal or band-limiting a signal to a particular frequency band), (ii) down-convert the selected signal, (iii) band-limit again to a narrower frequency band to select a signal frequency band, which may be referred to as a channel (for example) in a particular embodiment, (iv) demodulate the down-converted band-limited signal, (v) perform error correction, and (vi) demultiplex to select a desired stream of data packets. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include, for example, a tuner that performs various of these functions, including down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a frequency close to baseband) or to baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements receive RF signals transmitted over a wired (e.g., cable) medium and perform frequency selection by filtering, downconverting, and refiltering to a desired frequency band. Various embodiments rearrange the order of the above-mentioned (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements can include inserting elements between existing elements, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0105] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It should be appreciated that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or implemented within the processor 1010, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented, for example, in a separate interface IC or within the processor 1010, as desired. The demodulated, error corrected, and further demultiplexed stream is provided to various processing elements, including, for example, an encoder / decoder 1030 operating in combination with the processor 1010 and memory and storage elements, to process the data stream as desired for presentation on an output device.

[0106] The various elements of the system 1000 can be provided within a unified housing in which the various elements can be interconnected and transmit data between each other using suitable connection arrangements 1140, e.g., internal buses known in the art, including an I2C bus, hardwired, and printed circuit boards.

[0107] The system 1000 includes a communication interface 1050 that enables communication with other devices over a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented in a wired and / or wireless medium, for example.

[0108] Data is streamed to the system 1000 in various embodiments using a wireless network such as IEEE 802.11. The wireless signals in these embodiments are received, for example, via a communication channel 1060 and communication interface 1050 adapted for Wi-Fi communication. The communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. In another embodiment, streaming data is provided to the system 1000 using a set-top box that delivers data via an HDMI connection in the input block 1130. In yet another embodiment, streaming data is provided to the system 1000 using an RF connection in the input block 1130.

[0109] The system 1000 can provide output signals to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. The other peripheral devices 1120, in various example embodiments, include one or more of a standalone DVR, a disc player, a stereo system, a lighting system, and other devices that provide functionality based on the output of the system 1000. In various embodiments, control signals are communicated between the system 1000 and the display 1100, speakers 1110, or other peripheral devices 1120 using signaling such as AV.Link, CEC, or other communication protocols that allow inter-device control with or without user intervention. The output devices can be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, the output devices can be connected to the system 1000 using a communication channel 1060 via the communication interface 1050. The display 1100 and speakers 1110 may be integrated into a single unit along with other components of the system 1000, for example in an electronic device such as a television. In various embodiments, the display interface 1070 includes a display driver, for example a timing controller (T Con) chip.

[0110] Alternatively, the display 1100 and speakers 1110 can be separate from one or more of the other components, for example if the RF portion of the input 1130 is part of a separate set-top box. In various embodiments where the display 1100 and speakers 1110 are external components, the output signal can be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0111] The embodiments may be performed by computer software implemented by the processor 1010, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technology environment, and may be implemented using any suitable data storage technology, such as, as non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memories, and removable memories. The processor 1010 may be of any type suitable for the technology environment, and may include, as non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0112] Various implementations include decoding. As used herein, "decoding" can encompass all or part of the processing performed on a received encoded sequence to generate a final output suitable for display, for example. In various embodiments, such processing includes one or more of the processing typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processing may additionally or alternatively include processing performed by the decoders of the various implementations described herein, such as extracting indices of weights used for various intra prediction reference arrays.

[0113] As a further example, in one embodiment "decoding" refers only to entropy decoding, in another embodiment "decoding" refers only to differential decoding, and in another embodiment "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or to refer generally to a broader decoding process will be clear based on the context of a particular description and will be well understood by one of ordinary skill in the art.

[0114] Various implementations include encoding. As used herein, "encoding" can encompass all or part of the processing performed on an input video sequence to generate an encoded bitstream, for example, in a manner similar to the above description of "decoding." In various embodiments, such processing includes one or more of the processing typically performed by an encoder, such as, for example, partitioning, differential encoding, transforming, quantizing, and entropy encoding. In various embodiments, such processing includes, in addition to or in place of, the processing performed by the encoder of the various implementations described herein, such as, for example, weighting of intra-prediction reference arrays.

[0115] As a further example, in one embodiment, "encoding" refers to only entropy encoding, in another embodiment, "encoding" refers to only differential encoding, and in another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or to a broader encoding process generally will be clear based on the context of a particular description and will be well understood by one of ordinary skill in the art.

[0116] It should be noted that the syntax elements used herein are descriptive terms, and therefore do not exclude the use of other syntax element names.

[0117] Where a figure is presented as a flow chart, it should be understood that the figure also provides a block diagram of the corresponding apparatus. Similarly, where a figure is presented as a block diagram, it should be understood that the figure also provides a flow chart of the corresponding method / process.

[0118] In various embodiments, reference is made to rate-distortion calculation or rate-distortion optimization. During the encoding process, a balance or trade-off between rate and distortion is usually considered, often given a computational complexity constraint. Rate-distortion optimization is usually formulated to minimize a rate-distortion function, which is a weighted sum of rate and distortion. There are various approaches to solving the rate-distortion optimization problem. For example, these approaches may be based on extensive testing of all encoding options, including all considered modes or coding parameter values, with a full evaluation of their coding costs and associated distortion of the reconstructed signal after encoding and decoding. Also, faster approaches may be used to reduce the coding complexity, in particular the calculation of approximate distortion based on a prediction or prediction residual signal, rather than on the reconstructed signal. These two approaches may also be used in combination, for example using approximate distortion for only some of the possible encoding options and full distortion for others. Another approach evaluates only a subset of the possible encoding options. More generally, many approaches employ any of a variety of techniques to perform the optimization, but the optimization is not necessarily a full evaluation of both the coding cost and the associated distortion.

[0119] The implementations and aspects described herein may be implemented, for example, in a method or process, an apparatus, a software program, a data stream, or a signal. Even if only described in the context of a single type of implementation (e.g., only described as a method), the implementation of the described features may also be implemented in other forms (e.g., an apparatus or a program). An apparatus may be implemented, for example, in appropriate hardware, software, and firmware. A method may be implemented, for example, in a processor, where a processor refers to a general processing device including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Additionally, a processor may include, for example, a communication device such as a computer, a mobile phone, a portable / personal digital assistant ("PDA"), and other devices that facilitate communication of information between end users.

[0120] References to "one embodiment" or "embodiment" or "one implementation" or "implementation," as well as other variations thereof, mean that a particular feature, structure, characteristic, etc. described in connection with an embodiment is included in at least one embodiment. Thus, the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" appearing in various places throughout this document, as well as other variations thereof, do not necessarily all refer to the same embodiment.

[0121] Additionally, this document may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from memory.

[0122] Additionally, this document may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0123] Additionally, this document may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing the information or retrieving the information (e.g., from a memory). Additionally, "receiving" generally involves in some way, an operation such as, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0124] Use of any of " / ", "and / or", "at least one of", e.g., "A / B", "A and / or B", "at least one of A and B" is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of both alternatives (A and B). As a further example, "A, B, and / or C" and "at least one of A, B, and C" are intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A, B, and C). This may be expanded as many times as the number of items listed, as would be apparent to one of ordinary skill in the art and related fields.

[0125] Also, the term "signaling" as used herein means to indicate something to a corresponding decoder, among others. For example, in a particular embodiment, an encoder signals a particular one of a plurality of weights used for an intra-prediction reference array. Thus, in an embodiment, the same parameters are used at both the encoder and decoder sides. Thus, for example, an encoder can transmit a particular parameter to a decoder (explicit signaling) so that the decoder can use the same particular parameter. Conversely, if the decoder already has the particular parameter and other parameters, a non-transmitting signaling (implicit signaling) can be used to simply allow the decoder to recognize and select the particular parameter. By avoiding the transmission of actual features, bit savings are realized in various embodiments. It should be understood that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder in various embodiments. Although the above description is about the verb form of the word "signal", the word "signal" can also be used as a noun in this specification.

[0126] As will be apparent to one skilled in the art, implementations can generate a variety of signals formatted to carry information that can be stored or transmitted, for example. Such information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal can be formatted to carry a bit stream of the described embodiments. Such a signal can be formatted, for example, as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. The signal can be transmitted over a variety of different wired or wireless links, as is known. The signal can be stored on a processor-readable medium.

[0127] Embodiments can include one or more of the following features or entities, alone or in combination, across a variety of different claim categories and types. · Enabling matrix-based intra prediction for coding units up to a determined size, regardless of the maximum transform size. · Enabling transform unit tiling when the coding unit size is larger than the maximum transform size. Enable the low-frequency non-separable transform (LFNST) up to a determined size. Indicate whether any transform units do not use DCT2 as a transform, and if so, include in the bitstream or check the bitstream for syntax elements that do not allow LFNST. A bitstream or signal containing one or more of the described syntax elements or variations thereof. Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal that includes one or more of the described syntax elements or variations thereof. A television, set-top box, mobile phone, tablet, or other electronic device that performs in-loop filtering according to any of the described embodiments. A television, set-top box, mobile phone, tablet, or other electronic device that performs in-loop filtering in accordance with any of the described embodiments and displays the resulting image (e.g., using a monitor, screen, or other type of display). A television, set-top box, mobile phone, tablet, or other electronic device that tunes to a channel (e.g., using a tuner) to receive a signal including the encoded image and performs in-loop filtering according to any of the described embodiments. A television, set-top box, mobile phone, tablet, or other electronic device that receives a signal containing the encoded image wirelessly (e.g., using an antenna) and performs in-loop filtering according to any of the described embodiments.

[0128] Various other generalized and specific inventions and claims are also supported and contemplated throughout this disclosure.

Claims

1. 1. A method for video encoding, comprising: enabling matrix-based intra prediction (MIP) for the block, regardless of the maximum transform block size; if a block size of the block is greater than the maximum transform block size, dividing the block into a plurality of transform units of the maximum transform block size; performing matrix-based intra prediction for each block of the maximum transform block size to obtain a prediction; encoding the block based on the prediction; A method for providing the above.

2. The method of claim 1, wherein the maximum transform block size is 32 x 32.

3. The method of claim 1, wherein the block size is 64x64, 64x32 or 32x64.

4. An apparatus comprising: one or more processors; at least one memory coupled to the one or more processors; The one or more processors: Enabling matrix-based intra prediction (MIP) for blocks regardless of maximum transform block size; If a block size of the block is greater than the maximum transform block size, dividing the block into a plurality of transform units of the maximum transform block size; performing matrix-based intra prediction for each block of the maximum transform block size to obtain a prediction; and encoding said block based on said prediction. configured to carry out Device.

5. The apparatus of claim 4, wherein the maximum transform block size is 32x32.

6. The device of claim 4, wherein the block size is 64x64, 64x32 or 32x64.

7. 1. A method for video decoding, comprising: enabling matrix-based intra prediction (MIP) for the block, regardless of the maximum transform block size; if a block size of the block is greater than the maximum transform block size, dividing the block into a plurality of transform units of the maximum transform block size; performing matrix-based intra prediction for each block of the maximum transform block size to obtain a prediction; decoding the block based on the prediction; A method for providing the above.

8. The method of claim 7, wherein the maximum transform block size is 32×32.

9. The method of claim 7, wherein the block size is 64x64, 64x32 or 32x64.

10. An apparatus comprising: one or more processors; at least one memory coupled to the one or more processors; The one or more processors: Enabling matrix-based intra prediction (MIP) for blocks regardless of maximum transform block size; If a block size of the block is greater than the maximum transform block size, dividing the block into a plurality of transform units of the maximum transform block size; performing matrix-based intra prediction for each block of the maximum transform block size to obtain a prediction; and Decoding the block based on the prediction. configured to carry out Device.

11. The apparatus of claim 10, wherein the maximum transform block size is 32x32.

12. The apparatus of claim 10, wherein the block size is 64x64, 64x32 or 32x64.

13. A device, comprising: An apparatus according to claim 4 or 10, at least one of: (i) an antenna configured to receive a signal including the block; (ii) a band limiter configured to limit the received signal to a frequency band including the block; and (iii) a display configured to display an output representative of the video block; A device with.

14. A non-transitory computer-readable medium comprising a computer program causing a processor to perform a method according to any one of claims 1 to 3 or 7 to 9.

Images