Image encoding and decoding method and transmission method and non-transient computer readable storage medium

Abstract

Image encoding and decoding method and transmission method and non-transitory computer-readable storage medium. An image decoding method performed by a decoding device according to the present invention includes the following steps: deriving quantized transform coefficients about a target block from a bit stream; performing inverse quantization on the quantized transform coefficients about the target block and deriving transform coefficients; deriving residual samples about the target block based on a simplified inverse transform of the transform coefficients; and generating a reconstructed picture based on the residual samples about the target block and the predicted samples about the target block. The simplified inverse transform is performed based on a simplified inverse transform matrix. The simplified inverse transform matrix is ​​a non-square matrix with a smaller number of columns than rows.

Description

[0001] This application is a divisional application of the invention patent application with the original application number 201880086783.6 (International application number: PCT / KR2018 / 015815, application date: December 13, 2018, invention name: Transform-based image coding method and device for the method). Technical Field

[0002] The present disclosure relates to an image coding technology, and more particularly, to an image coding method and apparatus based on transformation of an image coding system. Background Art

[0003] The demand for high-resolution, high-quality images such as HD (high-definition) images and UHD (ultra-high-definition) images is increasing in various fields. Because image data has high resolution and high quality, the amount of information or bit volume to be transmitted increases relative to conventional image data. Therefore, when image data is transmitted using a medium such as a conventional wired / wireless broadband line or when image data is stored using an existing storage medium, its transmission cost and storage cost increase.

[0004] Therefore, there is a need for an efficient image compression technology for effectively transmitting, storing, and reproducing information of high-resolution and high-quality images. Summary of the invention

[0005] Technical tasks

[0006] An object of the present disclosure is to provide a method and apparatus for enhancing image coding efficiency.

[0007] Another object of the present disclosure is to provide a method and apparatus for enhancing conversion efficiency.

[0008] Yet another object of the present disclosure is to provide a method and apparatus for enhancing the efficiency of residual coding through transformation.

[0009] Yet another object of the present disclosure is to provide an image encoding method and device based on simplified transformation.

[0010] Solution

[0011] An exemplary embodiment of the present disclosure provides an image decoding method performed by a decoding device. The method includes the following steps: deriving quantized transform coefficients of a target block from a bitstream; deriving transform coefficients by performing inverse quantization on the quantized transform coefficients of the target block; deriving residual samples of the target block based on a simplified inverse transform of the transform coefficients; and generating a reconstructed picture based on the residual samples of the target block and the predicted samples of the target block, wherein the simplified inverse transform is performed based on a simplified inverse transform matrix, and the simplified inverse transform matrix is ​​a non-square matrix, and the number of columns of the non-square matrix is ​​less than the number of rows of the non-square matrix.

[0012] Another exemplary embodiment of the present disclosure provides an image encoding method performed by an encoding device. The method includes the following steps: deriving residual samples of a target block; deriving transform coefficients of the target block based on a simplified transform of the residual samples; deriving quantized transform coefficients by performing quantization based on the transform coefficients of the target block; and encoding information about the quantized transform coefficients, wherein the simplified transform is performed based on a simplified transform matrix, and the simplified transform matrix is ​​a non-square matrix, and the number of rows of the non-square matrix is ​​less than the number of columns of the non-square matrix.

[0013] Another exemplary embodiment of the present disclosure provides a decoding device for performing image decoding. The decoding device includes: an entropy decoder that derives quantized transform coefficients of a target block from a bitstream; a dequantizer that derives transform coefficients by performing dequantization on the quantized transform coefficients of the target block; an inverse transformer that derives residual samples of the target block based on a simplified inverse transform of the transform coefficients; and an adder that generates a reconstructed picture based on the residual samples of the target block and the predicted samples of the target block, and performs the simplified inverse transform based on a simplified inverse transform matrix, and the simplified inverse transform matrix is ​​a non-square matrix, and the number of columns of the non-square matrix is ​​less than the number of rows of the non-square matrix.

[0014] Another exemplary embodiment of the present disclosure provides an encoding device for performing image encoding. The encoding device includes: a subtractor that derives residual samples of a target block; a transformer that derives transform coefficients of the target block based on a simplified transform of the residual samples; a quantizer that derives quantized transform coefficients by performing quantization based on the transform coefficients of the target block; and an entropy encoder that encodes information about the quantized transform coefficients, and the simplified transform is performed by a simplified transform matrix, and the simplified transform matrix is ​​a non-square matrix, and the number of rows of the non-square matrix is ​​less than the number of columns of the non-square matrix.

[0015] Technical Effects

[0016] According to the present disclosure, the overall image / video compression efficiency can be enhanced.

[0017] According to the present disclosure, the amount of data to be transmitted for residual processing can be reduced through efficient transformation, and residual encoding efficiency can be enhanced.

[0018] According to the present disclosure, it is possible to concentrate non-zero transform coefficients on low-frequency components through secondary transform in the frequency domain.

[0019] According to the present disclosure, it is possible to perform image encoding based on simplified transformation, thereby enhancing image encoding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 : is a schematic diagram illustrating the configuration of a video / image encoding device to which the present disclosure is applicable.

[0021] Figure 2 FIG. 1 is a schematic diagram illustrating a configuration of a video / image decoding device to which the present disclosure is applicable.

[0022] Figure 3 is a schematic diagram illustrating a multiple transformation technique according to an exemplary embodiment.

[0023] Figure 4 is a diagram exemplarily illustrating 65 intra directional modes of prediction directions.

[0024] Figure 5a to Figure 5c is a flowchart illustrating a non-separable secondary transformation process according to an exemplary embodiment.

[0025] Figure 6 is a diagram for explaining a simplified transformation according to an exemplary embodiment of the present disclosure.

[0026] Figure 7 is a flow chart illustrating a simplified transformation process according to an exemplary embodiment of the present disclosure.

[0027] Figure 8 is a flowchart illustrating a simplified transformation process according to another exemplary embodiment of the present disclosure.

[0028] Fig. 9 is a flowchart illustrating a simplified transformation process based on a non-separable secondary transformation according to an exemplary embodiment of the present disclosure.

[0029] Fig.10 is a diagram illustrating a block to which a simplified transformation according to an exemplary embodiment of the present disclosure is applied.

[0030] Fig.11 is a flowchart illustrating the operation of a video encoding apparatus according to an exemplary embodiment of the present disclosure.

[0031] Fig.12 is a flowchart illustrating the operation of a video decoding apparatus according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] The present disclosure can be modified in various forms, and its specific embodiments will be described and illustrated in the accompanying drawings. However, these embodiments are not intended to limit the present disclosure. The terms used in the following description are only used to describe specific embodiments and are not intended to limit the present disclosure. The singular expression includes the plural expression as long as it is clearly read out differently. Terms such as "including" and "having" are intended to indicate the presence of features, numbers, steps, operations, elements, components or combinations thereof used in the following description, so it should be understood that there is no possibility of excluding the presence or addition of one or more different features, numbers, steps, operations, elements, components or combinations thereof.

[0033] Mode for Carrying Out the Invention

[0034] In addition, for the convenience of explaining different specific functions, the elements in the drawings described in this embodiment are drawn independently, which does not mean that these elements are implemented by independent hardware or independent software. For example, two or more of the elements can be combined to form a single element, or an element can be divided into multiple elements. Without departing from the concept of this embodiment, the embodiment in which the elements are combined and / or divided belongs to this embodiment.

[0035] The following description may be applied to the technical field of processing a video, one or more images. For example, the methods or exemplary embodiments disclosed in the following description may be associated with the disclosure of the Versatile Video Coding (VVC) standard (ITU-T H.266 Recommendation), the next generation video / image coding standard after VVC, or the standard before VVC (e.g., the High Efficiency Video Coding (HEVC) standard (ITU-T H.265 Recommendation), etc.).

[0036] Hereinafter, an example of the present embodiment will be described in detail with reference to the accompanying drawings. In addition, throughout the drawings, like reference numerals are used to indicate like elements, and the same description of the like elements will be omitted.

[0037] In the present disclosure, a video may mean a collection of a series of images according to the passage of time. Generally, a picture means a unit representing an image at a specific time, and a slice is a unit constituting a part of a picture. A picture may be composed of multiple slices, and the terms picture and slice may be mixed with each other as occasion requires.

[0038] A pixel or a pel may refer to the smallest unit constituting a picture (or image). In addition, a "sample" may be used as a term corresponding to a pixel. A sample may generally represent a pixel or a value of a pixel, may represent only a pixel (pixel value) of a luminance component, and may represent only a pixel (pixel value) of a chrominance component.

[0039] A unit indicates a basic unit of image processing. The unit may include at least one of a specific region and information related to the region. Optionally, the unit may be mixed with terms such as block, region, etc. In a typical case, an M×N block may represent a set of samples or transform coefficients arranged in M ​​columns and N rows.

[0040] Figure 1 The structure of the video encoding device to which the present embodiment is applicable is briefly illustrated. Hereinafter, the encoding device may include a video encoding device and / or an image encoding device, and the video encoding device may be used as a concept including an image encoding device.

[0041] Reference Figure 1 , the video encoding apparatus 100 may include a picture segmenter 105, a predictor 110, a residual processor 120, an entropy encoder 130, an adder 140, a filter 150, and a memory 160. The residual processor 120 may include a subtractor 121, a transformer 122, a quantizer 123, a rearranger 124, a dequantizer 125, and an inverse transformer 126.

[0042] The picture divider 105 may separate the input picture into at least one processing unit.

[0043] In the example, the processing unit may be referred to as a coding unit (CU). In this case, the coding unit may be recursively separated from the maximum coding unit (LCU) according to a quadtree binary tree (QTBT) structure. For example, a coding unit may be separated into multiple coding units of a deeper depth based on a quadtree structure and / or a binary tree structure. In this case, for example, the quadtree structure may be applied first, and the binary tree structure may be applied later. Alternatively, the binary tree structure may be applied first. The encoding process according to the present embodiment may be performed based on a final coding unit that is no longer further separated. In this case, the maximum coding unit may be used as the final coding unit based on coding efficiency, etc., depending on image characteristics, or the coding unit may be recursively separated into coding units of a lower depth as needed and a coding unit with an optimal size may be used as the final coding unit. Here, the encoding process may include processes such as prediction, transformation, and reconstruction, which will be described later.

[0044] In another example, the processing unit may include a coding unit (CU), a prediction unit (PU), or a transformer (TU). The coding unit may be separated from the maximum coding unit (LCU) into coding units with a deeper depth according to a quadtree structure. In this case, the maximum coding unit may be directly used as the final coding unit based on coding efficiency, etc. according to image characteristics, or the coding unit may be recursively separated into coding units with a deeper depth when necessary, and a coding unit with an optimal size may be used as the final coding unit. When the minimum coding unit (SCU) is set, the coding unit may not be separated into coding units smaller than the minimum coding unit. Here, the final coding unit refers to a coding unit that is partitioned or separated into a prediction unit or a transformer. A prediction unit is a unit partitioned from a coding unit and may be a unit of sample prediction. Here, the prediction unit may be divided into sub-blocks. The transformer may be divided from the coding unit according to the quadtree structure, and the transformer may be a unit that derives a transform coefficient and / or a unit that derives a residual signal from a transform coefficient. Hereinafter, a coding unit may be referred to as a coding block (CB), a prediction unit may be referred to as a prediction block (PB), and a transformer may be referred to as a transform block (TB). A prediction block or prediction unit may refer to a specific area in the form of a block in a picture and include an array of prediction samples. In addition, a transform block or transform unit may refer to a specific area in the form of a block in a picture and include an array of transform coefficients or residual samples.

[0045] The predictor 110 may perform prediction on a processing target block (hereinafter, it may represent a current block or a residual block), and may generate a prediction block including a prediction sample for the current block. The unit of prediction performed in the predictor 110 may be a coding block, or may be a transform block, or may be a prediction block.

[0046] The predictor 110 may determine whether to apply intra prediction or inter prediction to the current block. For example, the predictor 110 may determine whether to apply intra prediction or inter prediction in units of CUs.

[0047] In the case of intra-frame prediction, the predictor 110 can derive the prediction sample of the current block based on the reference sample outside the current block in the picture to which the current block belongs (hereinafter, the current picture). In this case, the predictor 110 can derive the prediction sample based on the average value or interpolation of the neighboring reference samples of the current block (case (i)), or can derive the prediction sample based on the reference sample existing in the specific (prediction) direction of the prediction sample in the neighboring reference samples of the current block (case (ii)). Case (i) can be called a non-directional mode or a non-angle mode, and case (ii) can be called a directional mode or an angle mode. In intra-frame prediction, the prediction mode may include 33 directional modes and at least two non-directional modes as examples. The non-directional mode may include a DC mode and a plane mode. The predictor 110 may determine the prediction mode to be applied to the current block by using the prediction mode applied to the neighboring block.

[0048] In the case of inter-frame prediction, the predictor 110 can derive the prediction sample of the current block based on the sample specified by the motion vector on the reference picture. The predictor 110 can derive the prediction sample of the current block by applying any one of the skip mode, merge mode and motion vector prediction (MVP) mode. In the case of skip mode and merge mode, the predictor 110 can use the motion information of the neighboring block as the motion information of the current block. In the case of skip mode, unlike merge mode, the difference (residual) between the prediction sample and the original sample is not sent. In the case of MVP mode, the motion vector of the neighboring block is used as a motion vector predictor, and is therefore used as a motion vector predictor of the current block to derive the motion vector of the current block.

[0049] In the case of inter-frame prediction, the neighboring blocks may include spatial neighboring blocks present in the current picture and temporal neighboring blocks present in the reference picture. The reference picture including the temporal neighboring blocks may also be referred to as a collocated picture (colPic). Motion information may include motion vectors and reference picture indexes. Information such as prediction mode information and motion information may be (entropy) encoded and then output as a bitstream.

[0050] When motion information of temporally adjacent blocks is used in skip mode and merge mode, the highest picture in the reference picture list may be used as a reference picture. Reference pictures included in the reference picture list may be aligned based on a picture order number (POC) difference between the current picture and the corresponding reference picture. POC corresponds to a display order and may be distinguished from a coding order.

[0051] The subtractor 121 generates a residual sample which is a difference between the original sample and the predicted sample. If the skip mode is applied, the residual sample may not be generated as described above.

[0052] The transformer 122 transforms the residual samples in units of transform blocks to generate transform coefficients. The transformer 122 may perform the transform based on the size of the corresponding transform block and the prediction mode applied to the coding block or prediction block that spatially overlaps the transform block. For example, if intra prediction is applied to the coding block or prediction block that overlaps the transform block and the transform block is a 4×4 residual array, the residual samples may be transformed using a discrete sine transform (DST) transform kernel, and in other cases, the residual samples may be transformed using a discrete cosine transform (DCT) transform kernel.

[0053] The quantizer 123 may quantize the transform coefficient to generate a quantized transform coefficient.

[0054] The rearranger 124 rearranges the quantized transform coefficients. The rearranger 124 may rearrange the quantized transform coefficients in a block form into a one-dimensional vector by a coefficient scanning method. Although the rearranger 124 is described as a separate component, the rearranger 124 may be a part of the quantizer 123.

[0055] The entropy encoder 130 may perform entropy encoding on the quantized transform coefficients. Entropy encoding may include encoding methods such as exponential Golomb, context adaptive variable length coding (CAVLC), context adaptive binary arithmetic coding (CABAC), etc. In addition to the quantized transform coefficients, the entropy encoder 130 may also perform entropy encoding or a predetermined method on information required for video reconstruction (e.g., syntax element values, etc.) together or separately. Entropy encoding information may be transmitted or stored in units of a network abstraction layer (NAL) in the form of a bitstream.

[0056] The dequantizer 125 dequantizes the value (transform coefficient) quantized by the quantizer 123 , and the inverse transformer 126 inversely transforms the value dequantized by the dequantizer 125 to generate residual samples.

[0057] The adder 140 adds the residual samples to the prediction samples to reconstruct the picture. The residual samples may be added to the prediction samples in units of blocks to generate a reconstructed block. Although the adder 140 is described as a separate component, the adder 140 may be part of the predictor 110. In addition, the adder 140 may be referred to as a reconstructor or a reconstructed block generator.

[0058] The filter 150 may apply deblocking filtering and / or sample adaptive offset to the reconstructed picture. Artifacts at block boundaries in the reconstructed picture or distortion during quantization may be corrected by deblocking filtering and / or sample adaptive offset. After deblocking filtering is completed, sample adaptive offset may be applied in units of samples. The filter 150 may apply an adaptive loop filter (ALF) to the reconstructed picture. ALF may be applied to the reconstructed picture to which deblocking filtering and / or sample adaptive offset have been applied.

[0059] The memory 160 may store a reconstructed picture (decoded picture) or information required for encoding / decoding. Here, the reconstructed picture may be a reconstructed picture filtered by the filter 150. The stored reconstructed picture may be used as a reference picture for (inter) prediction of other pictures. For example, the memory 160 may store a (reference) picture for inter prediction. Here, the picture for inter prediction may be specified according to a reference picture set or a reference picture list.

[0060] Figure 2 The structure of the video / image decoding device to which the present disclosure is applicable is briefly illustrated. Hereinafter, the video decoding device may include an image decoding device.

[0061] Reference Figure 2 , the video decoding apparatus 200 may include an entropy decoder 210, a residual processor 220, a predictor 230, an adder 240, a filter 250, and a memory 260. The residual processor 220 may include a rearranger 221, an inverse quantizer 222, and an inverse transformer 223.

[0062] In addition, although not depicted, the video decoding apparatus 200 may include a receiver for receiving a bitstream including video information. The receiver may be configured as a separate module, or may be included in the entropy decoder 210.

[0063] When a bitstream including video / image information is input, the video decoding apparatus 200 may reconstruct the video / image / picture in association with a process of processing the video information in the video encoding apparatus.

[0064] For example, the video decoding device 200 may perform video decoding using a processing unit applied in a video encoding device. Thus, the processing unit block of video decoding may be, for example, a coding unit, and in another example, a coding unit, a prediction unit, or a transformer. The coding unit may be separated from the maximum coding unit according to a quadtree structure and / or a binary tree structure and / or a ternary tree structure.

[0065] In some cases, a prediction unit and a transform unit may be further used, and in this case, a prediction block is a block derived or partitioned from a coding unit and may be a unit of sample prediction. Here, a prediction unit may be divided into sub-blocks. A transform unit may be separated from a coding unit according to a quadtree structure and may be a unit for deriving a transform coefficient or a unit for deriving a residual signal from a transform coefficient.

[0066] The entropy decoder 210 can parse the bitstream to output information required for video reconstruction or picture reconstruction. For example, the entropy decoder 210 can decode the information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, CABAC, etc., and can output the values ​​of syntax elements required for video reconstruction and the quantized values ​​of the transform coefficients about the residual.

[0067] More specifically, the CABAC entropy decoding method may receive a bin (binary bit) corresponding to each syntax element in a bitstream, determine a context model using decoding target syntax element information and decoding information of neighbors and decoding target blocks or information of symbols / bins decoded in a previous step, predict bin generation probabilities based on the determined context model, and perform arithmetic decoding of the bins to generate symbols corresponding to each syntax element value. Here, the CABAC entropy decoding method may update the context model using information of symbols / bins decoded by the context model for the next symbol / bin after determining the context model.

[0068] Information about prediction among the information decoded in the entropy decoder 210 may be provided to the predictor 230 , and a residual value, that is, a quantized transform coefficient, which has been entropy-decoded by the entropy decoder 210 may be input to the rearranger 221 .

[0069] The rearranger 221 may rearrange the quantized transform coefficients into a two-dimensional block form. The rearranger 221 may perform rearrangement corresponding to the coefficient scanning performed by the encoding device. Although the rearranger 221 is described as a separate component, the rearranger 221 may be a part of the inverse quantizer 222.

[0070] The inverse quantizer 222 may inverse quantize the quantized transform coefficient based on the (inverse) quantization parameter to output the transform coefficient. In this case, information for deriving the quantization parameter may be signaled from the encoding device.

[0071] The inverse transformer 223 may inversely transform the transform coefficients to derive residual samples.

[0072] The predictor 230 may perform prediction on the current block, and may generate a prediction block including a prediction sample of the current block. A unit of prediction performed in the predictor 230 may be a coding block, or may be a transform block, or may be a prediction block.

[0073] The predictor 230 may determine whether to apply intra prediction or inter prediction based on the information about the prediction. In this case, the unit for determining which one to use between intra prediction and inter prediction may be different from the unit for generating prediction samples. In addition, the unit for generating prediction samples may also be different in inter prediction and intra prediction. For example, it may be determined in units of CUs which one to apply between inter prediction and intra prediction. In addition, for example, in inter prediction, prediction samples may be generated by determining a prediction mode in units of PUs, and in intra prediction, prediction samples may be generated in units of TUs by determining a prediction mode in units of PUs.

[0074] In the case of intra prediction, the predictor 230 may derive the prediction samples of the current block based on the neighboring reference samples in the current picture. The predictor 230 may derive the prediction samples of the current block by applying a directional mode or a non-directional mode based on the neighboring reference samples of the current block. In this case, the prediction mode to be applied to the current block may be determined by using the intra prediction mode of the neighboring blocks.

[0075] In the case of inter-frame prediction, the predictor 230 may derive a prediction sample of the current block based on a sample specified in a reference picture according to a motion vector. The predictor 230 may derive a prediction sample of the current block using one of a skip mode, a merge mode, and an MVP mode. Here, motion information (e.g., a motion vector and information about a reference picture index) required for inter-frame prediction of the current block provided by the video encoding device may be acquired or derived based on information about prediction.

[0076] In the skip mode and the merge mode, the motion information of the neighboring blocks may be used as the motion information of the current block. Here, the neighboring blocks may include spatial neighboring blocks and temporal neighboring blocks.

[0077] The predictor 230 can use the motion information of the available neighboring blocks to construct a merge candidate list, and use the information indicated by the merge index on the merge candidate list as the motion vector of the current block. The merge index can be signaled by the encoding device. The motion information may include a motion vector and a reference picture. When the motion information of the temporal neighboring blocks is used in the skip mode and the merge mode, the highest picture in the reference picture list can be used as a reference picture.

[0078] In the case of the skip mode, unlike the merge mode, the difference (residual) between the predicted sample and the original sample is not transmitted.

[0079] In the case of the MVP mode, the motion vector of the current block may be derived using the motion vector of the neighboring block as a motion vector predictor. Here, the neighboring block may include a spatial neighboring block and a temporal neighboring block.

[0080] When the merge mode is applied, for example, the motion vector of the reconstructed spatial neighboring block and / or the motion vector corresponding to the Col block as the temporal neighboring block may be used to generate a merge candidate list. The motion vector of the candidate block selected from the merge candidate list is used as the motion vector of the current block in the merge mode. The above-mentioned information about the prediction may include a merge index indicating a candidate block with the best motion vector selected from the candidate blocks included in the merge candidate list. Here, the predictor 230 may derive the motion vector of the current block using the merge index.

[0081] When the MVP (motion vector prediction) mode is applied as another example, the motion vector of the reconstructed spatial neighboring block and / or the motion vector corresponding to the Col block as the temporal neighboring block may be used to generate a motion vector predictor candidate list. That is, the motion vector of the reconstructed spatial neighboring block and / or the motion vector corresponding to the Col block as the temporal neighboring block may be used as a motion vector candidate. The above-mentioned information about prediction may include a predicted motion vector index indicating the best motion vector selected from the motion vector candidates included in the list. Here, the predictor 230 may select the predicted motion vector of the current block from the motion vector candidates included in the motion vector candidate list using the motion vector index. The predictor of the encoding device may obtain a motion vector difference (MVD) between the motion vector of the current block and the motion vector predictor, encode the MVD and output the encoded MVD in the form of a bitstream. That is, the MVD may be obtained by subtracting the motion vector predictor from the motion vector of the current block. Here, the predictor 230 may obtain the motion vector included in the information about prediction, and derive the motion vector of the current block by adding the motion vector difference to the motion vector predictor. In addition, the predictor may obtain or derive a reference picture index indicating a reference picture from the above-mentioned information about prediction.

[0082] The adder 240 may add the residual sample to the prediction sample to reconstruct the current block or the current picture. The adder 240 may reconstruct the current picture by adding the residual sample to the prediction sample in units of blocks. When the skip mode is applied, the residual is not sent, and thus the prediction sample may become a reconstructed sample. Although the adder 240 is described as a separate component, the adder 240 may be part of the predictor 230. In addition, the adder 240 may be referred to as a reconstructor or a reconstructed block generator.

[0083] The filter 250 may apply deblocking filtering, sample adaptive offset and / or ALF to the reconstructed picture. Here, sample adaptive offset may be applied in units of samples after deblocking filtering. ALF may be applied after deblocking filtering and / or applying sample adaptive offset.

[0084] The memory 260 may store a reconstructed picture (decoded picture) or information required for decoding. Here, the reconstructed picture may be a reconstructed picture filtered by the filter 250. For example, the memory 260 may store pictures for inter-frame prediction. Here, pictures for inter-frame prediction may be specified according to a reference picture set or a reference picture list. The reconstructed picture may be used as a reference picture for other pictures. The memory 260 may output the reconstructed picture in an output order.

[0085] In addition, as described above, when performing video encoding, prediction is performed to enhance compression efficiency. Therefore, a prediction block including prediction samples for the current block, that is, a coding target block, can be generated. In this case, the prediction block includes prediction samples in the spatial domain (or pixel domain). The prediction block is derived in the same manner in the encoding device and the decoding device. The encoding device can improve the image coding efficiency by signaling the residual information about the residual between the original block and the prediction block instead of the original sample value of the original block itself to the decoding device. The decoding device can derive a residual block including residual samples based on the residual information, can generate a reconstructed block including reconstructed samples by adding the residual block to the prediction block, and can generate a reconstructed picture including the reconstructed block.

[0086] Residual information can be generated through a transformation and quantization process. For example, the encoding device can derive a residual block between the original block and the prediction block, can derive a transform coefficient by performing a transformation process on the residual samples (residual sample array) included in the residual block, can derive a quantized transform coefficient by performing a quantization process on the transform coefficient, and can signal the relevant residual information to the decoding device (through a bitstream). In this case, the residual information may include information such as value information, position information, a transformation scheme, a transform kernel, and a quantization parameter of the quantized transform coefficient. The decoding device can perform an inverse quantization / inverse transformation process based on the residual information, and can derive residual samples (or residual blocks). The decoding device can generate a reconstructed picture based on the prediction block and the residual block. In addition, the encoding device can also derive a residual block for inter-frame prediction reference of subsequent pictures by performing an inverse quantization / inverse transformation on the quantized transform coefficient, and can generate a reconstructed picture based on the residual block.

[0087] Figure 3 The multiple transformation technology according to the present disclosure is schematically illustrated.

[0088] Reference Figure 3 , the converter can correspond to Figure 1 The transformer in the above-mentioned encoding device, and the inverse transformer may correspond to Figure 1 The above-mentioned inverse transformer in the encoding device or Figure 2 The above-mentioned inverse transformer in the decoding device.

[0089] The transformer may derive (primary) transform coefficients by performing a primary transform based on the residual samples (residual sample array) in the residual block (S310). Here, the primary transform may include a multiple transform set (MTS). In some cases, the multiple transform set may also be referred to as an adaptive multi-core transform.

[0090] Adaptive multi-core transform may refer to a method of performing transform using discrete cosine transform (DCT) type 2, discrete sine transform (DST) type 7, DCT type 8, and / or DST type 1 in addition. That is, multi-core transform may refer to a transform method of transforming a residual signal (or residual block) in a spatial domain into a transform coefficient (or primary transform coefficient) in a frequency domain based on a plurality of transform kernels selected from DCT type 2, DST type 7, DCT type 8, and DST type 1. Here, from the perspective of the transformer, the primary transform coefficient may be referred to as a time transform coefficient.

[0091] That is, if an existing transform method is applied, a transform from the spatial domain to the frequency domain for the residual signal (or residual block) may be applied based on DCT type 2 to generate transform coefficients. On the other hand, if an adaptive multi-core transform is applied, a transform from the spatial domain to the frequency domain for the residual signal (or residual block) may be applied based on DCT type 2, DST type 7, DST type 8, and / or DST type 1, etc. to generate transform coefficients (or primary transform coefficients). Here, DCT type 2, DST type 7, DCT type 8, DST type 1, etc. may be referred to as transform types, transform kernels, or transform cores.

[0092] For reference, a DCT / DST transform type may be defined based on a basis function, and the basis function may be represented as shown in the following table.

[0093] Table 1

[0094]

[0095] If an adaptive multi-core transform is performed, a vertical transform core and a horizontal transform core for the target block may be selected from the transform cores, and a vertical transform may be performed on the target block based on the vertical transform core, and a horizontal transform may be performed on the target block based on the horizontal transform core. Here, the horizontal transform may refer to a transform for the horizontal component of the target block, and the vertical transform may refer to a transform for the vertical component of the target block. The vertical transform core / horizontal transform core may be adaptively determined based on the prediction mode of the target block (CU or subblock) containing the residual block and / or a transform index indicating a transform subset.

[0096] The transformer may derive (secondary) transform coefficients (S320) by performing a secondary transform based on the (primary) transform coefficients. If the primary transform is a transform from the spatial domain to the frequency domain, the secondary transform may be regarded as a transform from the frequency domain to the frequency domain. The secondary transform may include an inseparable transform. In this case, the secondary transform may be referred to as a non-separable secondary transform (NSST) or a mode-dependent non-separable secondary transform (MDNSST). The non-separable secondary transform may represent a transform that generates transform coefficients (or secondary transform coefficients) for a residual signal by performing a secondary transform on the (primary) transform coefficients derived by the primary transform based on an inseparable transform matrix. Here, the transform may be applied at one time based on the inseparable transform matrix without applying the vertical transform and the horizontal transform separately (or not applying the horizontal and vertical transforms independently) to the (primary) transform coefficients. That is, the non-separable secondary transform may represent a transform method that generates transform coefficients (or secondary transform coefficients) by transforming the vertical component and the horizontal component of the (primary) transform coefficient together based on the inseparable transform matrix. An inseparable secondary transform may be applied to the upper left region of a block consisting of (primary) transform coefficients (hereinafter referred to as a transform coefficient block). For example, if both the width (W) and the height (H) of the transform coefficient block are 8 or greater, an 8×8 inseparable secondary transform may be applied to the upper left 8×8 region of the transform coefficient block. In addition, if both the width (W) and the height (H) of the transform coefficient block are 4 or greater and the width (W) or the height (H) of the transform coefficient block is less than 8, a 4×4 inseparable secondary transform may be applied to the upper left min(8,W)×min(8,H) region of the transform coefficient block. However, exemplary embodiments are not limited thereto, and for example, if only the condition that the width (W) or the height (H) of the transform coefficient block is less than 8 is satisfied, a 4×4 inseparable secondary transform may also be applied to the upper left min(8,W)×min(8,H) of the transform coefficient block.

[0097] Specifically, for example, when a 4×4 input block is used, NSST may be performed as follows.

[0098] A 4×4 input block X can be represented as follows.

[0099] [Formula 1]

[0100]

[0101] When X is expressed in vector form, the vector can be expressed as follows

[0102] [Formula 2]

[0103]

[0104] In this case, the NSST can be calculated as follows.

[0105] [Formula 3]

[0106]

[0107] here, denotes a transform coefficient vector, and T denotes a 16×16 (non-separable) transform matrix.

[0108] By using Equation 3, the 16×1 transform coefficient vector can be derived And can be scanned in order (horizontally, vertically, diagonally, etc.) Reorganize into 4×4 blocks. However, the above calculation is an example, and Hypercube-Givens Transform (HyGT) or the like may be used for calculation of the secondary transform to reduce the calculation complexity of NSST.

[0109] Furthermore, in NSST, a transform kernel (or transform core, transform type) may be selected depending on a mode. Here, the mode may include an intra prediction mode and / or an inter prediction mode.

[0110] As described above, NSST can be performed based on an 8×8 transform or a 4×4 transform determined based on the width (W) and height (H) of the transform coefficient block. That is, NSST can be performed based on an 8×8 sub-block size or a 4×4 sub-block size. For example, in order to select a transform kernel based on a pattern, 35 groups of three NSST kernels can be configured for NSST for both the 8×8 sub-block size and the 4×4 sub-block size. That is, 35 transform sets can be configured for the 8×8 sub-block size, and 35 transform sets can be configured for the 4×4 sub-block size. In this case, the 35 transform sets for the 8×8 sub-block size can contain 3 8×8 transform kernels, respectively, and in this case, the 35 transform sets for the 4×4 sub-block size can include 3 4×4 transform kernels. However, the size of the transform sub-block, the number of sets, and the number of transform kernels in the set are examples, and sizes other than 8×8 or 4×4 can be used or n sets can be configured, and k transform kernels can be included in each set.

[0111] The transform set may also be referred to as an NSST set, and the transform core in the NSST set may be referred to as an NSST core.For example, the selection of a specific one in the transform set may be performed based on the intra prediction mode of the target block (CU or subblock).

[0112] For reference, for example, the intra prediction mode may include two non-directional or non-angular intra prediction modes and 65 directional or angular intra prediction modes. The non-directional intra prediction mode may include plane intra prediction mode #0 and DC intra prediction mode #1, and the directional intra prediction mode may include 65 intra prediction modes #2 to #66. However, this is an example, and the present disclosure may be applied even when the number of intra prediction modes is different. In addition, intra prediction mode #67 may be further used in some cases, and intra prediction mode #67 may represent a linear model (LM) mode.

[0113] Figure 4 An intra-frame directional mode of 65 prediction directions is exemplarily shown.

[0114] Reference Figure 4 , based on the intra prediction mode #34 having the upper left diagonal prediction direction, the intra prediction mode having the horizontal directionality and the intra prediction mode having the vertical directionality can be divided. Figure 3 The H and V in refer to horizontal and vertical directivities, respectively, and the numbers -32 to 32 indicate displacements of 1 / 32 units on the sample grid position. Intra-prediction modes #2 to #33 have horizontal directivities, and intra-prediction modes #34 to #66 have vertical directivities. Intra-prediction mode #18 and intra-prediction mode #50 represent horizontal intra-prediction modes and vertical intra-prediction modes, respectively. Intra-prediction mode #2 may be referred to as a lower-left diagonal intra-prediction mode, intra-prediction mode #34 may be referred to as an upper-left diagonal intra-prediction mode, and intra-prediction mode #66 may be referred to as an upper-right diagonal intra-prediction mode.

[0115] In this case, for example, as shown in the following table, mapping between 35 transform sets and intra prediction modes may be represented. For reference, when the LM mode is applied to a target block, a secondary transform may not be applied to the target block.

[0116] Table 2

[0117]

[0118] In addition, when it is determined to use a specific set, one of the k transform cores in the specific set can be selected by the NSST index. The encoding device can derive an NSST index indicating a specific transform core based on a rate-distortion (RD) check, and can signal the NSST index to the decoding device. The decoding device can select one of the k transform cores in a specific set based on the NSST index. For example, an NSST index value of 0 can indicate a first NSST core, an NSST index value of 1 can indicate a second NSST core, and an NSST index value of 2 can indicate a third NSST core. Alternatively, an NSST index value of 0 can indicate that the first NSST is not applied to the target block, and NSST index values ​​1 to 3 can indicate three transform cores.

[0119] Return to reference Figure 3 , the transformer can perform NSST based on the selected transform kernel and obtain (secondary) transform coefficients. The transform coefficients can be derived as quantized transform coefficients by a quantizer as described above, and encoded and signaled to a decoding device and transmitted to an inverse quantizer / inverse transformer in an encoding device.

[0120] In addition, when the secondary transform is omitted as described above, the (primary) transform coefficients as the output of the primary (separable) transform can be derived as quantized transform coefficients through the quantizer as described above, and can be encoded and signaled to the decoding device and transmitted to the inverse quantizer / inverse transformer in the encoding device.

[0121] The inverse transformer may perform a series of processes in the reverse order of the processes performed by the above-mentioned transformer. The inverse transformer may receive (inverse quantized) transform coefficients, perform a secondary (inverse) transform to derive (primary) transform coefficients (S350), and perform a primary (inverse) transform on the (primary) transform coefficients to obtain a residual block (residual sample). Here, from the perspective of the inverse transformer, the primary transform coefficients may be referred to as modified transform coefficients. As described above, the encoding device and the decoding device may generate a reconstructed block based on the residual block and the prediction block, and may generate a reconstructed picture based on the reconstructed block.

[0122] In addition, as described above, when the secondary (inverse) transform is omitted, the (inverse quantized) transform coefficients may be received and the primary (separable) transform may be performed thereon to obtain a residual block (residual sample). As described above, the encoding device and the decoding device may generate a reconstructed block based on the residual block and the prediction block, and may generate a reconstructed picture based on the reconstructed block.

[0123] Figure 5a to Figure 5c is a diagram for explaining a simplified transformation according to an exemplary embodiment of the present disclosure.

[0124] As above Figure 3As described in the non-separable secondary transform (hereinafter, referred to as "NSST"), block data of transform coefficients obtained by applying the primary transform is divided into M×M blocks, and then M×M blocks may be performed for each M×M block. 2 xM 2 NSST. M can be, for example, 4 or 8, but is not limited thereto.

[0125] M 2 xM 2 NSST can also be applied in the form of matrix products, but Figure 3 The hypercube-Givens transform (HyGT) described in can be used to calculate NSST in order to reduce the amount of computation and the required storage capacity. HyGT is an orthogonal transform, and HyGT can include a Givens rotation G defined by an orthogonal matrix G(m,n,θ) i,j (m,n) as the basic component. The Givens rotation G can be expressed by the following formula 4 i,j (m, n).

[0126] [Formula 4]

[0127]

[0128] Can be Figure 5a The Givens rotation based on equation 4 is exemplified in FIG. Referring to equation 4 and Figure 5a , it can be confirmed that a Givens rotation is described by only one angle (θ).

[0129] Figure 5b An example of constructing a round of 16×16 NSST is illustrated. More specifically, HyGT can be performed by combining Givens rotations in a hypercube arrangement, and as Figure 5b As illustrated in , the process for HyGT with 16 elements can be illustrated in butterfly form. Figure 5b As shown in the example, one round consists of 4 Givens rotation layers, each Givens rotation layer consists of 8 Givens rotations, and each Givens rotation can be Figure 5b The connection configuration illustrated in is formed as a structure that selects 2 input data to apply the rotation transformation, and then outputs the two input data after the rotation transformation as is at the corresponding selected position. 16×16 NSST can apply two rounds and one permutation layer in sequence, and 16 data can be arbitrarily mixed through the corresponding permutation layer. Figure 5b All two wheels are connected as shown in , but all Givens rotation layers of these two wheels may be different.

[0130] The 64×64 NSST consists of a Givens rotation layer with 64 inputs and outputs, and like the 16×16 NSST, at least one round can be applied, and one round can consist of 6 Givens rotation layers that are arranged in a similar way to Figure 5b For example, four rounds may be applied to a 64×64 NSST, and then a permutation layer for randomly mixing the 64 data may be applied. The Givens rotation layer for each of the four rounds may be different from each other.

[0131] Figure 5b The rounds applied to the forward transform are illustrated. When applying the inverse transform, the backward permutation layer is applied first, and then the layers can be applied in order from the last round to the first round, along Figure 5b The corresponding Givens rotation is applied in the direction from bottom to top in . The angle corresponding to each Givens rotation of the backward NSST may be a value obtained by applying a negative ("-") sign to the corresponding positive angle.

[0132] To enhance the coding efficiency, one or more rounds of HyGT can be used. Figure 5c As illustrated in , NSST can be composed of R HyGT rounds and can additionally include a sorting pass. The sorting pass can also be interpreted as an optional permutation pass, and the transform coefficients can be sorted based on the variance. As an example, two rounds of HyGT can be applied to 16×16 NSST, and four rounds of HyGT can be applied to 64×64 NSST.

[0133] Figure 6 is a diagram for explaining a simplified transformation according to an exemplary embodiment of the present disclosure.

[0134] In this specification, a 'target block' may mean a current block or a residual block to be encoded.

[0135] In this specification, "simplified transform" may mean a transform performed on the residual samples of the target block based on a transform matrix whose size is reduced according to a simplification factor. If a simplified transform is performed, the amount of computation required for the transform can be reduced due to the reduction in the size of the transform matrix. That is, the simplified transform can be used to eliminate the computational complexity problem that occurs when a large block is transformed or an inseparable transform is performed. The simplified transform may be referred to as a primary transform (or core transform). The simplified transform may also be used for any type of transform such as a primary transform (e.g., DCT, DST, etc.) and a secondary transform (e.g., NSST).

[0136] The simplified transform may be referred to by various terms such as simplified transform, reduced transform, reduced transform, simplified secondary transform, downsized transform, simplification transform, simple transform, RTS, RST, etc., and the names that may refer to the simplified transform are not limited to the listed examples.

[0137] In a simplified transformation according to an exemplary embodiment, a simplified transformation matrix may be determined by mapping an N-dimensional vector to an R-dimensional vector located in a different space, wherein R is less than N. N may refer to the square of the length of one side of a block to be transformed or the total number of transform coefficients corresponding to the block to be transformed, and a simplification factor may refer to an R / N value. The simplification factor may be referred to as various terms such as a simplified factor, a reduction factor, a reduction factor, a simplified factor, a simple factor, etc. In addition, R may be referred to as a simplification coefficient, but in some cases, the simplification factor may also refer to R. In addition, in some cases, the simplification factor may also refer to an N / R value.

[0138] In an exemplary embodiment, the simplification factor or simplification coefficient may be signaled by a bitstream, but the exemplary embodiment is not limited thereto. For example, a predetermined value of the simplification factor or simplification coefficient may be stored in each of the encoding device 100 and the decoding device 200, in which case the simplification factor or simplification coefficient may not be signaled separately.

[0139] The size of the simplified transformation matrix according to an exemplary embodiment may be R×N, which is smaller than the size of a normal transformation matrix N×N, and may be defined as expressed in Equation 5 below.

[0140] [Formula 5]

[0141]

[0142] Figure 6 The matrix (T) in the simplified transform block illustrated in (a) may refer to the matrix (T) of Formula 5. RxN ). If Figure 6 In (a), the simplified transformation matrix (T RxN ) is multiplied with the residual samples of the target block, the transform coefficients of the target block can be derived.

[0143] In an exemplary embodiment, if the size of the transformed block is 8×8, R=16 (ie, R / N=16 / 64=1 / 4) and the size of the target block is 64×64, the following matrix operation can be used to express the following equation 6: Figure 6 A simplified transformation of (a).

[0144] [Formula 6]

[0145]

[0146] In formula 6, r1 to r 64 It can represent the residual sample of the target block. As the result of the operation of formula 6, the transform coefficient (c i ), and c can be shown in Equation 7i The derivation process.

[0147] [Formula 7]

[0148]

[0149] As a result of the operation of Equation 7, the transform coefficients (c1 to c R That is, if R=16, the transform coefficients (c1 to c 16 ). If a conventional transform is applied instead of a simplified transform and a transform matrix of size 64×64 (N×N) is multiplied by a matrix including residual samples of size 64×1 (N×1), 64 (N) transform coefficients of the target block are derived, but since a simplified transform is applied, only 16 (R) transform coefficients of the target block are derived. The total number of transform coefficients of the target block is reduced from N to R, so the amount of data sent from the encoding device 100 to the decoding device 200 can be reduced, thereby enhancing the transmission efficiency between the encoding device 100 and the decoding device 200.

[0150] Looking at the size of the transformation matrix, the size of the conventional transformation matrix is ​​64×64 (N×N), but the size of the simplified transformation matrix is ​​reduced to 16×64 (R×N), so that the use of memory when performing the simplified transformation can be reduced at a ratio of R / N compared to when performing the conventional transformation. In addition, compared to the number of multiplication operations N×N when using the conventional transformation matrix, when using the simplified transformation matrix, the number of multiplication operations can be reduced at a ratio of R / N.

[0151] In an exemplary embodiment, the transform coefficient of the target block may be derived by transforming the residual sample of the target block by the transformer 122 of the encoding device 100; the transform coefficient of the target block may be transmitted to the inverse transformer of the decoding device 200; and the inverse transformer 223 of the decoding device 200 may inversely transform the transform coefficient of the target block. The residual sample of the target block may be derived based on the inverse transform performed on the transform coefficient of the target block. That is, the detailed operation according to the (simplified) inverse transform is only in the opposite order to the detailed operation according to the (simplified) transform, and the detailed operation according to the (simplified) inverse transform is substantially similar to the detailed operation according to the (simplified) transform.

[0152] According to an exemplary embodiment, the simplified inverse transformation matrix T NxR The size of is N×R which is smaller than the size of the conventional inverse transformation matrix N×N, and the simplified inverse transformation matrix T NxR The simplified transformation matrix T expressed in Equation 5 RxN Has a transposition relationship.

[0153] Figure 6 The matrix T in the simplified inverse transform block illustrated in (b) ist It can be referred to as the simplified inverse transformation matrix T NxR If Figure 6 As shown in (b), the transform coefficients of the target block are combined with the simplified inverse transform matrix T NxR By multiplying, the primary transform coefficients of the target block or the residual samples of the target block can be derived.

[0154] More specifically, if a simplified inverse transform is applied based on the quadratic inverse transform, when the transform coefficients are compared with the simplified inverse transform matrix T NxR When multiplied, the primary transform coefficients of the target block can be derived. On the other hand, if a simplified inverse transform is applied based on the primary inverse transform, when the transform coefficients of the target block are multiplied by the simplified inverse transform matrix T NxR When multiplied, the residual samples of the target block can be derived.

[0155] In an exemplary embodiment, if the size of the block to be inversely transformed is 8×8, R=16 (ie, R / N=16 / 64=1 / 4) and the size of the target block is 64×64, the following matrix operation can be expressed according to Figure 6 The simplified inverse transform of (b).

[0156] [Formula 8]

[0157]

[0158] In formula 8, c1 to c 16 As a result of the operation of equation 8, r representing the primary transform coefficient of the target block or the residual sample of the target block can be derived. j , and can be shown in Equation 9 as j The derivation process.

[0159] [Formula 9]

[0160]

[0161] As a result of the operation of equation 9, r1 to r2 representing the primary transform coefficients of the target block or the residual samples of the target block can be derived. N . Looking at the size of the inverse transformation matrix, the size of the conventional inverse transformation matrix is ​​64×64 (N×N), but the size of the simplified inverse transformation matrix is ​​reduced to 64×16 (N×R), so that the use of memory when performing the simplified inverse transformation can be reduced at a ratio of R / N compared to when performing the conventional inverse transformation. In addition, compared to the number of multiplication operations N×N when using the conventional inverse transformation matrix, when using the simplified inverse transformation matrix, the number of multiplication operations can be reduced to (N×R) at a ratio of R / N.

[0162] Figure 7is a flow chart illustrating a simplified transformation process according to an exemplary embodiment of the present disclosure.

[0163] Figure 7 Each step illustrated in can be represented by Figure 2 More specifically, S700 may be performed by the decoding device 200 illustrated in FIG. Figure 2 The dequantizer 222 illustrated in FIG. 1 is executed, and S710 and S720 may be performed by Figure 2 Therefore, the inverse transformer 223 illustrated in FIG. Figure 2 The above mentioned contents are repeated in detailed description.

[0164] In the above reference Figure 6 In the described exemplary embodiment, the detailed operations according to the (simplified) transformation are in the opposite order to the detailed operations according to the (simplified) inverse transformation, and the detailed operations according to the (simplified) transformation and the detailed operations according to the (simplified) inverse transformation can be substantially similar. Therefore, it will be easily understood by those skilled in the art that the description of S700 to S720 of the simplified inverse transformation described below can also be applied to the simplified transformation in the same or similar manner.

[0165] The decoding apparatus 200 according to an exemplary embodiment may induce a transform coefficient by performing inverse quantization on a quantized transform coefficient of a target block ( S700 ).

[0166] The decoding device 200 according to the exemplary embodiment may select a transform kernel (S710). More specifically, the decoding device 200 may select the transform kernel based on at least one of a transform index, a width and a height of a region to be transformed, an intra prediction mode used in image decoding, and information about a color component of a target block. However, the exemplary embodiment is not limited thereto, and for example, the transform kernel is predefined, and separate information for selecting the transform kernel may not be signaled.

[0167] In an example, information about the color components of the target block may be signaled by CIdx. If the target block is a luma block, CIdx may indicate 0, and if the target block is a chroma block (eg, C b Block or C r block), CIdx can indicate a non-zero value (e.g., 1).

[0168] The decoding apparatus 200 according to the exemplary embodiment may apply a simplified inverse transform to the transform coefficient based on the selected transform kernel and the simplification factor ( S720 ).

[0169] Figure 8 is a flowchart illustrating a simplified transformation process according to another exemplary embodiment of the present disclosure.

[0170] Figure 8 Each step illustrated in can be represented by Figure 2 More specifically, S800 may be performed by the decoding device 200 illustrated in FIG. Figure 2 The dequantizer 222 illustrated in FIG. 1 is executed, and S810 to S860 may be performed by Figure 2 Therefore, the inverse transformer 223 illustrated in FIG. Figure 2 The above mentioned contents are repeated in detailed description.

[0171] In the above reference Figure 6 In the described exemplary embodiment, the detailed operations according to the (simplified) transformation are only in the opposite order to the detailed operations according to the (simplified) inverse transformation, and the detailed operations according to the (simplified) transformation and the detailed operations according to the (simplified) inverse transformation can be substantially similar. Therefore, it will be easily understood by those skilled in the art that the description of S800 to S860 of the simplified inverse transformation described below can also be applied to the simplified transformation in the same or similar manner.

[0172] The decoding device 200 according to the exemplary embodiment may perform inverse quantization on the quantized coefficients of the target block (S800). If the transform is performed in the encoding device 100, the decoding device 200 may derive the transform coefficients of the target block by inverse quantizing the quantized transform coefficients of the target block in S800. On the contrary, if the transform is not performed in the encoding device 100, the decoding device 200 may derive the residual samples of the target block by inverse quantizing the quantized residual samples of the target block in S800.

[0173] The decoding device 200 according to the exemplary embodiment may determine whether the transform has been performed on the residual sample of the target block in the encoding device 100 (S810), and when it is determined that the transform has been performed, the decoding device 200 may parse (or decode from the bitstream) the transform index (S820). The transform index may include a horizontal transform index for horizontal transform and a vertical transform index for vertical transform.

[0174] In an example, the transform index may include a primary transform index, a core transform index, an NSST index, etc. For example, the transform index may be expressed as Transform_idx, and the NSST index may be expressed as NSST_idx. In addition, the horizontal transform index may be expressed as Transform_idx_h, and the vertical transform index may be expressed as Transform_idx_v.

[0175] When it is determined in S810 that transform is not performed on the residual sample of the target block in the encoding apparatus 100, the decoding apparatus 200 according to the exemplary embodiment may omit operations according to S820 to S860.

[0176] The decoding apparatus 200 according to an exemplary embodiment may select a transform kernel based on at least one of a transform index, a width and a height of a region to which the transform is applied, an intra prediction mode used in image decoding, and information about a color component of a target block ( S830 ).

[0177] The decoding apparatus 200 according to an exemplary embodiment may determine whether a condition for performing simplified inverse transform on a transform coefficient of a target block is satisfied ( S840 ).

[0178] In an example, if both the width and the height of the region to which the simplified inverse transform is applied are greater than the first coefficient, the decoding device 200 may determine that a condition for performing the simplified inverse transform on the transform coefficient of the target block is satisfied.

[0179] In another example, if the product of the width and height of the area to which the simplified inverse transform is applied is greater than the second coefficient and the smaller of the width and height of the area to which the simplified inverse transform is applied is greater than the third coefficient, the decoding device 200 can determine that the conditions for performing a simplified inverse transform on the transform coefficients of the target block are met.

[0180] In another example, if the width and height of the region to which the simplified inverse transform is applied are both less than or equal to the fourth coefficient, the decoding device 200 may determine that the condition for performing the simplified inverse transform on the transform coefficient of the target block is satisfied.

[0181] In another example, if the product of the width and height of the area to which the simplified inverse transform is applied is less than or equal to the fifth coefficient and the smaller of the width and height of the area to which the simplified inverse transform is applied is less than or equal to the sixth coefficient, the decoding device 200 can determine that the conditions for performing a simplified inverse transform on the transform coefficients of the target block are met.

[0182] In another example, if at least one of the conditions that the width and height of the area to which the simplified inverse transform is applied are both greater than the first coefficient, the product of the width and height of the area to which the simplified inverse transform is applied is greater than the second coefficient and the smaller of the width and height of the area to which the simplified inverse transform is applied is greater than the third coefficient, the width and height of the area to which the simplified inverse transform is applied are both less than or equal to the fourth coefficient, and the product of the width and height of the area to which the simplified inverse transform is applied is less than or equal to the fifth coefficient and the smaller of the width and height of the area to which the simplified inverse transform is applied is less than or equal to the sixth coefficient, the decoding device 200 can determine that the conditions for performing a simplified inverse transform on the transform coefficients of the target block are met.

[0183] In the above examples, the first to sixth coefficients may be any predetermined positive integers. For example, the first to sixth coefficients may be 4, 8, 16 or 32.

[0184] The simplified inverse transform according to the exemplary embodiment may be applied to a square area included in the target block (i.e., if the width and height of the area to which the simplified inverse transform is applied are the same), and in some cases, the width and height of the area to which the simplified inverse transform is applied may be fixed to a value of a predetermined coefficient (e.g., 4, 8, 16, 32, etc.). In addition, the area to which the simplified inverse transform is applied is not limited to a square area, and the simplified inverse transform may be applied to a rectangular area or a non-rectangular area. Fig.10 A more detailed description of the region where the simplified inverse transform is applied is described below.

[0185] In an example, whether the condition for performing a simplified inverse transform is satisfied may be determined based on a transform index. That is, the transform index may indicate which transform has been performed on the target block.

[0186] When it is determined in S840 that the condition for performing the simplified inverse transform is not satisfied, the decoding apparatus 200 according to the exemplary embodiment may perform a (conventional) inverse transform on the transform coefficients of the target block. Figure 3 As described, the (inverse) transform may include, for example, DCT2, DCT4, DCT5, DCT7, DCT8, DST1, DST4, DST7, NSST, JEM-NSST (HyGT), etc., but is not limited thereto.

[0187] When it is determined in S840 that the condition for performing the simplified inverse transform is satisfied, the decoding apparatus 200 according to the exemplary embodiment may perform the simplified inverse transform on the transform coefficient of the target block ( S860 ).

[0188] Fig. 9 is a flowchart illustrating a simplified transformation process based on a non-separable secondary transformation according to an exemplary embodiment of the present disclosure.

[0189] Fig. 9 Each step illustrated in can be represented by Figure 2 The decoding device 200 illustrated in FIG. 1 is executed, and more specifically, S900 may be performed by Figure 2 The dequantizer 222 illustrated in FIG. 1 is executed, and S910 to S980 may be performed by Figure 2 In addition, Fig. 9 The S900 can correspond to Figure 8 S800, Fig. 9 The S940 can correspond to Figure 8 S830, and Fig. 9 The S950 can correspond to Figure 8 Therefore, the above reference will be omitted or simplified. Figure 2 and Figure 8 The content described is repeated in detail.

[0190] In the above reference Figure 6 In the described exemplary embodiment, the detailed operations according to the (simplified) transformation are only in the opposite order to the detailed operations according to the (simplified) inverse transformation, and the detailed operations according to the (simplified) transformation and the detailed operations according to the (simplified) inverse transformation can be substantially similar. Therefore, it will be easily understood by those skilled in the art that the description of S900 to S980 of the simplified inverse transformation described below can also be applied to the simplified transformation in the same or similar manner.

[0191] The decoding apparatus 200 according to the exemplary embodiment may perform inverse quantization on a quantization coefficient of a target block ( S900 ).

[0192] The decoding device 200 according to an exemplary embodiment may determine whether NSST has been performed on the residual sample of the target block in the encoding device 100 (S910), and when it is determined that NSST has been performed, the decoding device 200 may parse (or decode from the bitstream) an NSST index (S920).

[0193] According to an exemplary embodiment, the decoding device 200 can determine whether the NSST index is greater than 0 (S930), and when it is determined that the NSST index is greater than 0, the decoding device 200 can select a transform kernel based on the NSST index, the width and height of the area to which the NSST index is applied, the intra-frame prediction mode, and at least one of information about the color component of the target block (S940).

[0194] The decoding apparatus 200 according to an exemplary embodiment may determine whether a condition for performing simplified inverse transform on a transform coefficient of a target block is satisfied ( S950 ).

[0195] If it is determined in S950 that the condition for performing the simplified inverse transform is not satisfied, the decoding apparatus 200 according to the exemplary embodiment may perform a (normal) inverse transform not based on the simplified inverse transform on the transform coefficient of the target block.

[0196] If it is determined in S950 that the condition for performing the simplified inverse transform is satisfied, the decoding apparatus 200 according to the exemplary embodiment may perform inverse NSST on the transform coefficient of the target block based on the simplified inverse transform.

[0197] When it is determined in S910 that NSST is not performed on the residual sample of the target block in the encoding apparatus 100, the decoding apparatus 200 according to the exemplary embodiment may omit operations according to S920 to S970.

[0198] If it is determined in S930 that the NSST index is not greater than 0, the decoding apparatus 200 according to the exemplary embodiment may omit operations according to S940 to S970 .

[0199] The decoding apparatus 200 according to an exemplary embodiment may perform a primary inverse transform on a primary transform coefficient of a target block derived by applying the inverse NSST. When the primary inverse transform is performed on the primary transform coefficient, a residual sample of the target block may be derived.

[0200] Fig.10 is a block diagram illustrating a block for applying a simplified transformation according to an exemplary embodiment of the present disclosure.

[0201] As mentioned above Figure 8 As described, the region in the target block to which the simplified (inverse) transform is applied is not limited to a square region, and the simplified transform may be applied to a rectangular region or a non-rectangular region.

[0202] Fig.10 An example of applying a simplified transformation to a non-rectangular region in a target block 1000 having a size of 16×16 is illustrated. Fig.10 The ten shaded blocks 1010 in the middle represent the areas where the simplified transformation is applied in the target block 1000. Since the size of the corresponding minimum unit block is 4×4, according to Fig.10 For example, the simplified transformation is applied to 10 4×4 pixels (ie, the simplified transformation is applied to 160 pixels). When R=16, the size of the simplified transformation matrix may be 16×160.

[0203] In addition, those skilled in the art can easily understand that the application Fig.10 The arrangement of the minimum unit blocks 1010 included in the simplified transformed region illustrated in is only one of many examples. For example, the minimum unit blocks included in the region to which the simplified transformation is applied may not be adjacent to each other, or they may have a relationship of only sharing one vertex.

[0204] Fig.11 is a flowchart illustrating the operation of a video encoding apparatus according to an exemplary embodiment of the present disclosure.

[0205] Fig.11 Each step illustrated in can be represented by Figure 1 More specifically, the encoding device 100 illustrated in FIG. Figure 1 The subtractor 121 illustrated in the example performs S1100, which can be performed by Figure 1 The converter 122 illustrated in the example performs S1110, which can be performed by Figure 1 The quantizer 123 illustrated in FIG. 1 performs S1120 and can be Figure 1 The entropy encoder 130 illustrated in FIG. 1 performs S1130. In addition, the operations according to S1100 to S1130 are based on the above reference Figures 6 to 10 Therefore, the contents described above will be omitted or simplified. Figure 1 and Figures 6 to 10 The content described is repeated in detail.

[0206] The encoding apparatus 100 according to an exemplary embodiment may induce a residual sample of a target block ( S1100 ).

[0207] The encoding apparatus 100 according to an exemplary embodiment may derive a transform coefficient of a target block based on a simplified transform of residual samples (S1110). In an example, the simplified transform may be performed based on a simplified transform matrix, and the simplified transform matrix may be a non-square matrix having a smaller number of rows than a number of columns.

[0208] In an exemplary embodiment, S1110 may include determining whether a condition for applying a simplified transform is met, generating and encoding a transform index based on the determination, selecting a transform kernel, and applying a simplified transform to the residual sample based on the selected transform kernel and a simplification factor when the condition for applying a simplified transform is met. At this time, the size of the simplified transform matrix may be determined based on the simplification factor.

[0209] If the simplified transform according to S1110 is based on the primary transform, when the simplified transform is performed on the residual sample of the target block, the primary transform coefficient of the target block can be derived. The decoding device 200 can perform NSST on the primary transform coefficient of the target block, and at this time, NSST can be performed based on the simplified transform or not based on the simplified transform. When NSST is performed based on the simplified transform, this can correspond to the operation according to S1110.

[0210] The encoding apparatus 100 according to an exemplary embodiment may induce a quantized transform coefficient by performing quantization based on a transform coefficient of a target block ( S1120 ).

[0211] The encoding apparatus 100 according to an exemplary embodiment may encode information about the quantized transform coefficient (S1130). More specifically, the encoding apparatus 100 may generate information about the quantized transform coefficient and encode the generated information about the quantized transform coefficient. The information about the quantized transform coefficient may include residual information.

[0212] In an example, the information about the quantized transform coefficients may include at least one of information about whether to apply a simplified transform, information about a simplification factor, information about a minimum transform size to which the simplified transform is applied, and information about a maximum transform size to which the simplified transform is applied. Fig.12 A more detailed description of information about quantized transform coefficients is described in .

[0213] Referring to S1110, it can be confirmed that the transform coefficients of the target block are derived based on the simplified transform of the residual samples. Looking at the size of the transform matrix, the size of the conventional transform matrix is ​​N×N, but the size of the simplified transform matrix is ​​reduced to R×N, so that the use of memory when performing the simplified transform can be reduced at a ratio of R / N compared to when performing the conventional transform. In addition, compared with the number of multiplication operations N×N when using the conventional transform matrix, when the simplified transform matrix is ​​used, the number of multiplication operations can be reduced to (R×N) at a ratio of R / N. In addition, since only R transform coefficients are derived when the simplified transform is applied, the total number of transform coefficients of the target block is reduced from N to R compared to the case where N transform coefficients are derived when the conventional transform is applied, so that the amount of data sent by the encoding device 100 to the decoding device 200 can be reduced. In summary, the transformation efficiency and encoding efficiency of the encoding device 100 can be enhanced by the simplified transform according to S1110.

[0214] Fig.12 is a flowchart illustrating the operation of a video decoding apparatus according to an exemplary embodiment of the present disclosure.

[0215] Fig.12 Each step illustrated in can be represented by Figure 2 More specifically, the decoding device 200 illustrated in FIG. Figure 2 The entropy decoder 210 illustrated in FIG. 1 performs S1200, which may be performed by Figure 2 The dequantizer 222 illustrated in the example performs S1210, which can be performed by Figure 2 The inverse transformer 223 illustrated in FIG. 1 performs S1220 and can be Figure 2 The adder 240 illustrated in FIG. 1 performs S1230. In addition, the operations according to S1200 to S1230 are based on the above reference Figures 6 to 10 Therefore, the contents described above will be omitted or simplified. Figure 2 and Figures 6 to 10 The content described is repeated in detail.

[0216] The decoding device 200 according to the exemplary embodiment can derive the quantized transform coefficient of the target block from the bitstream (S1200). More specifically, the decoding device 200 can decode the information about the quantized transform coefficient of the target block from the bitstream, and derive the quantized transform coefficient of the target block based on the information about the quantized transform coefficient of the target block. The information about the quantized transform coefficient of the target block may be included in a sequence parameter set (SPS) or a slice header, and may include information about whether to apply a simplified transform, information about a simplification factor, information about a minimum transform size for applying a simplified transform, information about a maximum transform size for applying a simplified transform, and at least one of information about a simplified inverse transform size.

[0217] More specifically, information about whether to apply a simplified transform may be represented by an enable flag, information about a simplification factor may be represented by a simplification factor value, information about a minimum transform size for applying a simplified inverse transform may be represented by a minimum transform size value, information about a maximum transform size for applying a simplified inverse transform may be represented by a maximum transform size value, and information about a simplified inverse transform size may be represented by a size value of the simplified inverse transform. At this time, the enable flag may be signaled by a first syntax element, the simplification factor value may be signaled by a second syntax element, the minimum transform size value may be signaled by a third syntax element, the maximum transform size value may be signaled by a fourth syntax element, and the simplified inverse transform size value may be signaled by a fifth syntax element.

[0218] In an example, the first syntax element may be represented as a syntax element Reduced_transform_enabled_flag. If a simplified transform is applied, the syntax element Reduced_transform_enabled_flag may indicate 1, and if a simplified transform is not applied, the syntax element Reduced_transform_enabled_flag may indicate 0. If the syntax element Reduced_transform_enabled_flag is not signaled, the value of the syntax element Reduced_transform_enabled_flag may be estimated to be 0.

[0219] In addition, the second syntax element may be represented as a syntax element Reduced_transform_factor. The syntax element Reduced_transform_factor may indicate a value of R / N, where N may mean the square of the length of one side of the block to which the transform is applied or the total number of transform coefficients corresponding to the block to which the transform is applied. R may mean a simplified coefficient less than N. However, the example is not limited thereto, and for example, Reduced_transform_factor may also indicate R instead of R / N. In terms of a simplified inverse transform matrix, R means the number of columns of the simplified inverse transform matrix, and N means the number of rows of the simplified inverse transform matrix, and at this time, the number of columns of the simplified inverse transform matrix needs to be less than the number of rows. R may be, for example, 8, 16, 32, etc., but is not limited thereto. If the syntax element Reduced_transform_factor is not signaled, the value of Reduced_transform_factor may be estimated as R / N (or R).

[0220] In addition, the third syntax element may be expressed as a syntax element min_reduced_transform_size. If the syntax element min_reduced_transform_size is not signaled, the value of min_reduced_transform_size may be estimated to be 0.

[0221] In addition, the fourth syntax element may be expressed as a syntax element max_reduced_transform_size. If the syntax element max_reduced_transform_size is not signaled, the value of max_reduced_transform_size may be estimated to be 0.

[0222] In addition, the fifth syntax element may be expressed as a syntax element reduced_transform_size. The size value of the simplified inverse transform included in the syntax element reduced_transform_size and signaled may indicate the size of a region to which the simplified inverse transform is applied or the size of a simplified transform matrix, but is not limited thereto. If the syntax element reduced_transform_size is not signaled, the value of reduced_transform_size may be estimated to be 0.

[0223] Table 3 below shows an example of including and signaling information about a quantized transform coefficient of a target block in an SPS.

[0224] [Table 3]

[0225]

[0226] The decoding apparatus 200 according to an exemplary embodiment may induce a transform coefficient by performing inverse quantization on a quantized transform coefficient of a target block ( S1210 ).

[0227] The decoding device 200 according to an exemplary embodiment may derive residual samples of the target block based on the simplified inverse transform of the transform coefficients (S1220). In an example, the simplified inverse transform may be performed based on a simplified inverse transform matrix, and the simplified inverse transform matrix may be a non-square matrix having a smaller number of columns than a number of rows.

[0228] In an exemplary embodiment, S1220 may include decoding the transform index, determining whether a condition for applying a simplified inverse transform is satisfied based on the transform index, selecting a transform kernel, and applying a simplified inverse transform to the transform coefficient based on the selected transform kernel and a simplification factor when the condition for applying a simplified inverse transform is satisfied. At this time, the size of the simplified inverse transform matrix may be determined based on the simplification factor.

[0229] If the simplified inverse transform according to S1220 is based on the inverse NSST, when the simplified inverse transform is performed on the transform coefficient of the target block, the primary transform coefficient of the target block can be derived. The decoding device 200 can perform a primary inverse transform on the primary transform coefficient of the target block, and at this time, the primary inverse transform can be performed based on the simplified inverse transform or not based on the simplified inverse transform.

[0230] Alternatively, when the simplified inverse transform according to S1220 is based on the primary inverse transform, residual samples of the target block may be directly derived by performing the simplified inverse transform on transform coefficients of the target block.

[0231] The decoding apparatus 200 according to the exemplary embodiment may generate a reconstructed picture based on the residual sample of the target block and the prediction sample of the target block ( S1230 ).

[0232] Referring to S1220, it can be confirmed that the residual samples of the target block are derived based on the simplified inverse transform of the transform coefficients of the target block. Looking at the size of the inverse transform matrix, the size of the conventional inverse transform matrix is ​​N×N, but the size of the simplified inverse transform matrix is ​​reduced to N×R, so that the use of memory when performing the simplified transform can be reduced at a ratio of R / N compared to when performing the conventional transform. In addition, compared with the number of multiplication operations N×N when using the conventional inverse transform matrix, when using the simplified inverse transform matrix, the number of multiplication operations can be reduced to (N×R) at a ratio of R / N. In addition, since only R transform coefficients need to be decoded when the simplified inverse transform is applied, the total number of transform coefficients of the target block can be reduced from N to R compared to the case where N transform coefficients need to be decoded when the conventional inverse transform is applied, thereby enhancing the decoding efficiency. In summary, the (inverse) transform efficiency and encoding efficiency of the decoding device 200 can be enhanced by the simplified inverse transform according to S1220.

[0233] The internal components of the above-mentioned device may be a processor that executes continuous processing stored in a memory, or may be a hardware component composed of other hardware. These may be located inside / outside the device.

[0234] According to exemplary embodiments, the above-mentioned modules may be omitted or replaced with other modules performing similar / same operations.

[0235] The above-mentioned method according to the present disclosure may be implemented in the form of software, and the encoding device and / or decoding device according to the present disclosure may be included in a device that performs image processing such as a TV, a computer, a smart phone, a set-top box, a display device, etc.

[0236] In the exemplary embodiments mentioned above, although the method is described based on a flowchart shown as a series of steps or boxes, the present disclosure is not limited to the order of the steps, and a certain step may occur in a different order than described above, or occur simultaneously with other steps described above. In addition, those skilled in the art will understand that the steps shown in the flowchart are not exclusive, and other steps may be included or one or more steps in the flowchart may be deleted without affecting the scope of the present disclosure.

[0237] When the embodiments of the present disclosure are implemented by software, the above methods can be implemented by modules (processes, functions, etc.) that perform the above functions. These modules can be stored in a memory and driven by a processor. The memory can be inside or outside the processor, and various well-known means can be used to connect the memory to the processor. The processor may include an application specific integrated circuit (ASIC), other chipsets, logic circuits and / or data processing devices. The memory may include a ROM (read-only memory), a RAM (random access memory), a flash memory, a memory card, a storage medium and / or other storage devices.

Claims

1. An image decoding method performed by a decoding device, the image decoding method comprising the following steps: deriving quantized transform coefficients of a target block from a bitstream; deriving transform coefficients by performing inverse quantization on the quantized transform coefficients of the target block; Deriving residual samples of the target block based on a second inverse transform of the transform coefficients; as well as generating a reconstructed picture based on the residual samples of the target block and the predicted samples of the target block, wherein the secondary inverse transform is based on a non-separable transform and is performed based on an inverse transform matrix, wherein the inverse transform matrix is ​​a non-square matrix, The output region of the secondary inverse transform is included in a region of size M×M. The output area of ​​the secondary inverse transform includes an upper left sub-block, an upper right sub-block, and a lower left sub-block of the M×M area and does not include a lower right sub-block of the M×M area, and Wherein, the size of the inverse transformation matrix is ​​N×R, R is equal to the number of elements in the input area of ​​the secondary inverse transformation based on the inseparable transformation, R is less than N, and each of N and R is a positive integer.

2. An image encoding method performed by an encoding device, the image encoding method comprising the following steps: Derive residual samples of the target block; deriving transform coefficients of the target block by performing a secondary transform based on the residual samples; deriving quantized transform coefficients by applying quantization based on the transform coefficients of the target block; as well as encoding information about the quantized transform coefficients to output a bitstream, wherein the secondary transform is based on a non-separable transform and is performed based on a transform matrix, wherein the transform matrix is ​​a non-square matrix, The input region of the secondary transformation is included in an M×M region. The input area of ​​the secondary transformation includes the upper left sub-block, the upper right sub-block, and the lower left sub-block of the M×M area and does not include the lower right sub-block of the M×M area, and The size of the transformation matrix is ​​R×N, N is equal to the number of elements in the input region of the secondary transformation based on the inseparable transformation, R is less than N, and each of N and R is a positive integer.

3. A method for sending image data, the sending method comprising the following steps: obtaining a bitstream of the image, wherein the bitstream is generated based on: deriving residual samples of a target block, deriving transform coefficients of the target block by performing a secondary transform based on the residual samples, deriving quantized transform coefficients by applying quantization based on the transform coefficients of the target block, and encoding information about the quantized transform coefficients; and sending said data comprising said bit stream, wherein the secondary transform is based on a non-separable transform and is performed based on a transform matrix, wherein the transform matrix is ​​a non-square matrix, The input region of the secondary transformation is included in an M×M region. The input area of ​​the secondary transformation includes the upper left sub-block, the upper right sub-block, and the lower left sub-block of the M×M area and does not include the lower right sub-block of the M×M area, and The size of the transformation matrix is ​​R×N, N is equal to the number of elements in the input region of the secondary transformation based on the inseparable transformation, R is less than N, and each of N and R is a positive integer.

Images