Method for decoding and encoding image and apparatus for transmitting compressed video data
Through multi-transformation technology, the video signal is encoded and decoded, which solves the problem of high-resolution video signal transmission and storage costs, and achieves efficient encoding and decoding effects, which are suitable for high-resolution and stereoscopic image content.
Patent Information
- Application Number
- CN202411335788.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2018-06-08
- Filing Date
- 2019-06-07
- Publication Date
- 2024-11-12
AI Technical Summary
High resolution and high-quality video signal data transmission and storage cost is high, and existing image compression technology is difficult to effectively process stereoscopic image content.
Multi-transformation technology is used to encode and decode the video signal. By performing secondary transformation on blocks that have performed primary transformation, inverse transformation and quantization using transformation matrix and one-dimensional matrix, the encoding and decoding process is optimized.
It improves the encoding and decoding efficiency of residual data, reduces transmission and storage costs, and is suitable for video signal processing of high-resolution and stereoscopic images.
Smart Images

Figure CN118945334A_ABST
Abstract
Description
[0001] This application is a divisional application of invention patent application No. 201980035163.4, whose application date is June 7, 2019, international application number is PCT / KR2019 / 006905, and which entered the Chinese national phase on November 25, 2020, and whose invention name is “Method and device for processing video signals”. Technical Field
[0002] The invention relates to a method and an apparatus for processing a video signal. Background Art
[0003] Recently, the demand for high-resolution and high-quality images, such as high-definition (HD) images and ultra-high-definition (UHD) images, has increased in various application fields. However, compared with conventional image data, higher resolution and quality image data has an increased data volume. Therefore, when the image data is transmitted by using a medium such as a conventional wired and wireless broadband network, or when the image data is stored by using a conventional storage medium, the cost of transmission and storage increases. In order to solve these problems arising as the resolution and quality of image data increase, efficient image encoding / decoding technology can be used.
[0004] The image compression technology includes various technologies, including: an inter-frame prediction technology that predicts a pixel value included in a current picture based on a previous picture or a subsequent picture of the current picture; an intra-frame prediction technology that predicts a pixel value included in the current picture by using pixel information in the current picture; an entropy coding technology that assigns a short code to a value with a high frequency of occurrence and a long code to a value with a low frequency of occurrence, etc. Image data can be effectively compressed by using such an image compression technology and can be transmitted or stored.
[0005] Meanwhile, along with the demand for high-resolution images, the demand for stereoscopic image content as a new image service is also increasing. Video compression technology for efficiently providing stereoscopic image content with high resolution and ultra-high resolution is being studied. Summary of the invention
[0006] Technical issues
[0007] An object of the present invention is to provide a method and apparatus for encoding / decoding residual data based on multiple transforms.
[0008] An object of the present invention is to provide a method and apparatus for performing a secondary transform on a partial area of a block on which a primary transform has been performed.
[0009] The technical objectives to be achieved by the present invention are not limited to the above-mentioned technical problems. Moreover, those skilled in the art will clearly understand other technical problems not mentioned according to the following description.
[0010] Technical Solution
[0011] The video signal decoding method and device according to the present invention can obtain the residual coefficient of the current block, inverse quantize the residual coefficient, perform a secondary inverse transform on the inversely quantized residual coefficient, and perform a primary inverse transform on the result of performing the secondary inverse transform. In this case, the secondary inverse transform is performed on a partial area of the current block.
[0012] The video signal encoding method and apparatus according to the present invention can perform a primary transform on the residual sample, perform a secondary transform on the primary transform coefficients generated due to the primary transform, quantize the secondary transform coefficients generated due to the secondary transform, and encode the quantized transform coefficients. In this case, the secondary transform is performed on a partial area of the current block.
[0013] In a method and apparatus for encoding / decoding a video signal according to the present invention, secondary transform / inverse transform is performed using a transform matrix and a one-dimensional matrix, and the one-dimensional matrix is generated by arranging inverse quantized residual coefficients included in a partial region in one dimension.
[0014] In the method and apparatus for encoding / decoding a video signal according to the present invention, the number of rows or columns of the transform matrix is smaller than the number of samples included in the partial domain.
[0015] In the method and apparatus for encoding / decoding a video signal according to the present invention, the size or shape of the partial partition is determined based on the size or shape of the current block.
[0016] In the method and apparatus for encoding / decoding a video signal according to the present invention, a transform / inverse transform type of a primary inverse transform is determined based on index information signaled through a bitstream.
[0017] In a method and apparatus for encoding / decoding a video signal according to the present invention, index information specifies any one of a plurality of transform sets, a first transform type candidate included in the transform set is determined as a horizontal direction transform type of a current block, and a second transform type candidate included in the transform set is determined as a vertical direction transform type of the current block.
[0018] In the method and apparatus for encoding / decoding a video signal according to the present invention, a transform type of a primary transform / inverse transform is determined based on a comparison result of a width of a current block and a height of the current block.
[0019] According to the present invention, a method for decoding an image includes: dividing a coding block into a plurality of child node blocks, a current block being one of the child node blocks; obtaining residual coefficients of the current block; performing inverse quantization on the residual coefficients; performing a secondary inverse transform on the current block; and performing a primary inverse transform on a result of performing the secondary inverse transform, wherein the secondary inverse transform is performed on the inverse quantized residual coefficients obtained from inverse quantization included in a partial region of the current block, wherein the secondary inverse transform is performed by using an inverse transform matrix and an input matrix, the input matrix being generated by arranging the inverse quantized residual coefficients in a two-dimensional form in the partial region into a one-dimensional form, and wherein the number of rows or columns of the inverse transform matrix is greater than the number of inverse quantized residual coefficients included in the partial region.
[0020] According to the present invention, the method for encoding an image includes: dividing a coding block into multiple child node blocks, the current block is one of the child node blocks; performing a primary transform on the residual samples of the current block; performing a secondary transform on the current block; quantizing the transform coefficients generated due to the secondary transform; and encoding the quantized transform coefficients, wherein the secondary transform is performed on the intermediate transform coefficients obtained from the primary transform and included in the partial area of the current block, wherein the secondary transform is performed by using a transform matrix and an input matrix, the input matrix is generated by arranging the intermediate transform coefficients in a two-dimensional form into a one-dimensional form, and wherein the number of rows or columns of the transform matrix is less than the number of the intermediate transform coefficients included in the partial area.
[0021] According to the present invention, an apparatus for sending compressed video data includes: a processor configured to obtain the compressed video data; and a sending unit configured to send the compressed video data, wherein obtaining the compressed video data includes: performing a primary transform on the residual samples of the current block; performing a secondary transform on the current block; quantizing the transform coefficients generated due to the secondary transform; and encoding the quantized transform coefficients, wherein the secondary transform is performed on the intermediate transform coefficients obtained from the primary transform included in a partial area of the current block, wherein the secondary transform is performed by using a transform matrix and an input matrix, the input matrix is generated by arranging the intermediate transform coefficients in a two-dimensional form into a one-dimensional form, and wherein the number of rows or columns of the transform matrix is less than the number of the intermediate transform coefficients included in the partial area.
[0022] The features briefly summarized above are merely illustrative aspects of the detailed description of the invention that follows and do not limit the scope of the invention.
[0023] Beneficial Effects
[0024] According to the present invention, the encoding / decoding efficiency of residual data can be improved by performing multiple transformations on the residual data.
[0025] According to the present invention, encoding / decoding efficiency can be improved by performing secondary transformation on a partial area of a block on which primary transformation has been performed.
[0026] Effects that can be obtained by the present invention are not limited to the above-mentioned effects, and other effects that are not mentioned can be clearly understood by those skilled in the art from the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 is a block diagram showing an apparatus for encoding a video according to an embodiment of the present invention.
[0028] Figure 2 is a block diagram showing an apparatus for decoding a video according to an embodiment of the present invention.
[0029] Figure 3 is a diagram showing partition mode candidates that can be applied to a coding block when the coding block is encoded by inter prediction.
[0030] Figure 4 2 is a diagram showing an example of hierarchical division of coding blocks based on a tree structure according to an embodiment of the present invention.
[0031] Figure 5 is a diagram showing partition types that allow binary tree-based partitioning according to an embodiment of the present invention.
[0032] Figure 6 is a diagram showing a ternary tree based partition type.
[0033] Figure 7 is a diagram showing an example in which only certain types of binary tree-based partitioning are allowed.
[0034] Figure 8 : is a diagram showing an example in which information related to the number of allowed binary tree divisions is encoded / decoded as an embodiment to which the present invention is applied.
[0035] Fig. 9 : is a flowchart showing a residual sample encoding process as an embodiment to which the present invention is applied.
[0036] Fig.10 is a flowchart showing a process of decoding residual samples according to an embodiment to which the present invention is applied.
[0037] Fig.11 It is a diagram showing whether the transform set in the vertical direction and the transform set in the horizontal direction are the same for 33 intra prediction modes.
[0038] Fig.12 is a diagram showing an execution area of a secondary transformation. DETAILED DESCRIPTION
[0039] Various modifications can be made to the present invention, and there are various embodiments of the present invention, examples of which will now be provided with reference to the accompanying drawings, and examples of which will be described in detail. However, the present invention is not limited thereto, and the exemplary embodiments may be interpreted as including all modifications, equivalents or alternatives within the technical concept and technical scope of the present invention. In the accompanying drawings described, similar reference numerals refer to similar elements.
[0040] The terms 'first', 'second', etc. used in the specification may be used to describe various components, but these components should not be interpreted as being limited to these terms. These terms are only used to distinguish one component from other components. For example, without departing from the scope of the present invention, a 'first' component may be referred to as a 'second' component, and a 'second' component may also be similarly referred to as a 'first' component. The term 'and / or' includes a combination of a plurality of items or any one of a plurality of items.
[0041] It will be understood that in this specification, when an element is simply referred to as being “connected to” or “coupled to” another element rather than being “directly connected to” or “directly coupled to” another element, the element may be “directly connected to” or “directly coupled to” another element, or the element may be connected to or coupled to another element with other elements interposed therebetween. In contrast, it should be understood that when an element is referred to as being “directly coupled to” or “directly connected to” another element, there are no intervening elements.
[0042] The terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention. Expressions used in the singular include expressions in the plural form unless the expression has a significantly different meaning in the context. In this specification, it should be understood that terms such as "including", "having" etc. are intended to indicate the existence of features, numbers, steps, actions, elements, parts or combinations thereof disclosed in this specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, actions, elements, parts or combinations thereof may exist or may be added.
[0043] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Hereinafter, the same constituent elements in the drawings are denoted by the same reference numerals, and repeated description of the same elements will be omitted.
[0044] Figure 1 is a block diagram showing an apparatus for encoding a video according to an embodiment of the present invention.
[0045] Reference Figure 1 , the device 100 for encoding a video may include: a picture division module 110, prediction modules 120 and 125, a transformation module 130, a quantization module 135, a rearrangement module 160, an entropy coding module 165, an inverse quantization module 140, an inverse transformation module 145, a filter module 150 and a memory 155.
[0046] Figure 1 The components shown in are shown independently to represent the different characteristic functions in the device for encoding the video. Therefore, this does not mean that each component is composed of a separate hardware or software component unit. In other words, for convenience, each component includes each of the listed components. Therefore, at least two components in each component can be combined to form a component, or a component can be divided into multiple components to perform each function. Without departing from the essence of the present invention, the embodiment of combining each component and the embodiment of dividing a component are also included in the scope of the present invention.
[0047] In addition, some of the components may not be indispensable components for performing the basic functions of the present invention, but selective components that only improve the performance of the present invention. The present invention can be implemented by excluding components for improving performance and only including indispensable components for implementing the essence of the present invention. Structures that exclude selective components that only improve performance and only include indispensable components are also included in the scope of the present invention.
[0048] The picture partitioning module 110 may partition an input picture into one or more processing units. Here, a processing unit may be a prediction unit (PU), a transform unit (TU), or a coding unit (CU). The picture partitioning module 110 may partition a picture into a combination of a plurality of coding units, prediction units, and transform units, and may encode a picture by selecting a combination of coding units, prediction units, and transform units using a predetermined criterion (e.g., a cost function).
[0049] For example, a picture may be divided into a plurality of coding units. A recursive tree structure such as a quadtree structure may be used to divide a picture into coding units. A coding unit divided into other coding units with a picture or a maximum coding unit as a root may be divided into child nodes corresponding to the number of divided coding units. Coding units that cannot be further divided according to predetermined restrictions are used as leaf nodes. That is, when it is assumed that only square division is feasible for a coding unit, a coding unit may be divided into up to four other coding units.
[0050] Hereinafter, in an embodiment of the present invention, a coding unit may mean a unit that performs encoding or a unit that performs decoding.
[0051] The prediction unit may be one of partitions split into a square shape or a rectangular shape having the same size in a single coding unit, or the prediction unit may be one of partitions split so as to have different shapes / sizes in a single coding unit.
[0052] When a prediction unit subject to intra prediction is generated based on a coding unit and the coding unit is not a minimum coding unit, intra prediction may be performed without splitting the coding unit into a plurality of prediction units NxN.
[0053] The prediction modules 120 and 125 may include an inter prediction module 120 that performs inter prediction and an intra prediction module 125 that performs intra prediction. It may be determined whether inter prediction or intra prediction is performed for a prediction unit, and detailed information (e.g., intra prediction mode, motion vector, reference picture, etc.) according to each prediction method may be determined. Here, the processing unit subjected to prediction may be different from the processing unit for which the prediction method and detailed content are determined. For example, the prediction method, prediction mode, etc. may be determined by the prediction unit, and the prediction may be performed by the transform unit. The residual value (residual block) between the generated prediction block and the original block may be input to the transform module 130. In addition, the prediction mode information, motion vector information, etc. used for prediction may be encoded together with the residual value by the entropy encoding module 165, and may be transmitted to a device for decoding a video. When a specific encoding mode is used, it may be transmitted to a device for decoding a video by encoding the original block as it is without generating a prediction block through the prediction modules 120 and 125.
[0054] The inter-frame prediction module 120 may predict a prediction unit based on information of at least one of a previous picture or a subsequent picture of the current picture, or in some cases, may predict a prediction unit based on information of some coding regions in the current picture. The inter-frame prediction module 120 may include a reference picture interpolation module, a motion prediction module, and a motion compensation module.
[0055] The reference picture interpolation module may receive reference picture information from the memory 155, and may generate pixel information of integer pixels or less than integer pixels according to the reference picture. In the case of luma pixels, an 8-tap interpolation filter based on DCT with different filter coefficients may be used to generate pixel information of integer pixels or less than integer pixels in units of 1 / 4 pixels. In the case of chrominance signals, a 4-tap interpolation filter based on DCT with different filter coefficients may be used to generate pixel information of integer pixels or less than integer pixels in units of 1 / 8 pixels.
[0056] The motion prediction module can perform motion prediction based on the reference picture interpolated by the reference picture interpolation module. Various methods such as full search based block matching algorithm (FBMA), three-step search (TSS), new three-step search algorithm (NTS), etc. can be used as methods for calculating motion vectors. Based on the interpolated pixels, the motion vector can have a motion vector value in units of 1 / 2 pixel or 1 / 4 pixel. The motion prediction module can predict the current prediction unit by changing the motion prediction method. Various methods such as skip method, merge method, AMVP (Advanced Motion Vector Prediction) method, intra-frame block copy method, etc. can be used as motion prediction methods.
[0057] The intra prediction module 125 may generate a prediction unit based on reference pixel information adjacent to the current block, which is pixel information in the current picture. When the adjacent block of the current prediction unit is a block subject to inter-frame prediction and thus the reference pixel is a pixel subject to inter-frame prediction, the reference pixel information of the adjacent block subject to intra-frame prediction may be used to replace the reference pixel included in the block subject to inter-frame prediction. That is, when the reference pixel is unavailable, at least one reference pixel among the available reference pixels may be used to replace the unavailable reference pixel information.
[0058] The prediction mode of the intra prediction may include a directional prediction mode using reference pixel information according to a prediction direction and a non-directional prediction mode that does not use directional information when performing prediction. The mode for predicting luminance information may be different from the mode for predicting chrominance information, and in order to predict chrominance information, the intra prediction mode information for predicting luminance information or the predicted luminance signal information may be used.
[0059] When performing intra prediction, when the size of the prediction unit is the same as the size of the transform unit, intra prediction can be performed on the prediction unit based on pixels located on the left, upper left, and upper sides of the prediction unit. However, when performing intra prediction, when the size of the prediction unit is different from the size of the transform unit, intra prediction can be performed using reference pixels based on the transform unit. In addition, intra prediction using NxN partitioning can be used only for the minimum coding unit.
[0060] In the intra prediction method, according to the prediction mode, a prediction block can be generated after applying an AIS (Adaptive Intra Smoothing) filter to a reference pixel. The type of the AIS filter applied to the reference pixel may be different. In order to perform the intra prediction method, the intra prediction mode of the current prediction unit may be predicted according to the intra prediction mode of a prediction unit adjacent to the current prediction unit. When the prediction mode of the current prediction unit is predicted by using the mode information predicted according to the adjacent prediction unit, when the intra prediction mode of the current prediction unit is the same as the intra prediction mode of the adjacent prediction unit, predetermined flag information may be used to transmit information indicating that the prediction mode of the current prediction unit and the prediction mode of the adjacent prediction unit are the same as each other. When the prediction mode of the current prediction unit is different from the prediction mode of the adjacent prediction unit, entropy coding may be performed to encode the prediction mode information of the current block.
[0061] In addition, a residual block including information on a residual value, which is a difference between the prediction unit subjected to prediction and the original block of the prediction unit, may be generated based on the prediction unit generated by the prediction modules 120 and 125. The generated residual block may be input to the transform module 130.
[0062] The transform module 130 may transform the residual block including information on the residual value between the original block and the prediction unit generated by the prediction modules 120 and 125 by using a transform method such as discrete cosine transform (DCT), discrete sine transform (DST), and KLT. Whether to apply DCT, DST, or KLT in order to transform the residual block may be determined based on intra prediction mode information of the prediction unit used to generate the residual block.
[0063] The quantization module 135 may quantize the value transformed to the frequency domain by the transform module 130. The quantization coefficient may be different according to the block or importance of the picture. The value calculated by the quantization module 135 may be provided to the inverse quantization module 140 and the rearrangement module 160.
[0064] The rearrangement module 160 may rearrange coefficients of the quantized residual value.
[0065] The rearrangement module 160 can change the coefficient of the two-dimensional block form into the coefficient of the one-dimensional vector form by the coefficient scanning method. For example, the rearrangement module 160 can use a zigzag scanning method to scan from the DC coefficient to the coefficient of the high frequency domain so that the coefficient is changed into a one-dimensional vector form. According to the size of the transform unit and the intra-frame prediction mode, the zigzag scanning can be replaced by the vertical scanning of the coefficient of the two-dimensional block form in the column direction or the horizontal scanning of the coefficient of the two-dimensional block form in the row direction. That is to say, it can be determined according to the size of the transform unit and the intra-frame prediction mode which scanning method to use among the zigzag scanning, the vertical scanning and the horizontal scanning.
[0066] The entropy encoding module 165 may perform entropy encoding based on the value calculated by the rearrangement module 160. The entropy encoding may use various encoding methods such as exponential Golomb coding, context adaptive variable length coding (CAVLC), and context adaptive binary arithmetic coding (CABAC).
[0067] The entropy coding module 165 can encode various information from the rearrangement module 160 and the prediction modules 120 and 125, such as residual value coefficient information and block type information of the coding unit, prediction mode information, partition unit information, prediction unit information, transformation unit information, motion vector information, reference frame information, block interpolation information, filtering information, etc.
[0068] The entropy encoding module 165 may entropy encode coefficients of the coding unit input from the rearrangement module 160 .
[0069] The inverse quantization module 140 may inversely quantize the value quantized by the quantization module 135, and the inverse transform module 145 may inversely transform the value transformed by the transform module 130. The residual values generated by the inverse quantization module 140 and the inverse transform module 145 may be combined with the prediction units predicted by the motion estimation module, the motion compensation module, and the intra prediction module of the prediction modules 120 and 125, so that a reconstructed block may be generated.
[0070] The filter module 150 may include at least one of a deblocking filter, an offset correction unit, and an adaptive loop filter (ALF).
[0071] The deblocking filter can remove block distortion that occurs due to the boundaries between blocks in the reconstructed picture. In order to determine whether to perform deblocking, the pixels in several rows or columns included in the block can be the basis for determining whether to apply the deblocking filter to the current block. When the deblocking filter is applied to the block, a strong filter or a weak filter can be applied according to the required deblocking filter strength. In addition, when applying the deblocking filter, horizontal filtering and vertical filtering can be processed in parallel.
[0072] The offset correction module can correct the offset from the original picture in units of pixels in the picture subjected to deblocking. In order to perform offset correction on a specific picture, a method of applying an offset in consideration of edge information of each pixel or a method of dividing the pixels of the picture into a predetermined number of regions, determining the region to be subjected to performing the offset, and applying the offset to the determined region can be used.
[0073] Adaptive loop filtering (ALF) can be performed based on a value obtained by comparing a filtered reconstructed picture with an original picture. The pixels included in the picture can be divided into predetermined groups, a filter to be applied to each group in the group can be determined, and filtering can be performed separately for each group. Information on whether ALF is applied and a luminance signal can be transmitted by a coding unit (CU). The shape and filter coefficients of the filter used for ALF can be different for each block. In addition, regardless of the characteristics of the application target block, a filter for ALF of the same shape (fixed shape) can be applied.
[0074] The memory 155 may store the reconstructed block or the reconstructed picture calculated by the filter module 150. The stored reconstructed block or the reconstructed picture may be provided to the prediction modules 120 and 125 when performing inter prediction.
[0075] Figure 2 is a block diagram showing an apparatus for decoding a video according to an embodiment of the present invention.
[0076] Reference Figure 2 The apparatus 200 for decoding a video may include an entropy decoding module 210 , a rearrangement module 215 , an inverse quantization module 220 , an inverse transform module 225 , prediction modules 230 and 235 , a filter module 240 , and a memory 245 .
[0077] When a video bitstream is input from the apparatus for encoding a video, the input bitstream may be decoded according to an inverse process of the apparatus for encoding a video.
[0078] The entropy decoding module 210 may perform entropy decoding according to an inverse process of entropy encoding performed by an entropy encoding module of a device for encoding a video. For example, corresponding to a method performed by a device for encoding a video, various methods such as exponential Golomb coding, context adaptive variable length coding (CAVLC), and context adaptive binary arithmetic coding (CABAC) may be applied.
[0079] The entropy decoding module 210 may decode information about intra prediction and inter prediction performed by the apparatus for encoding a video.
[0080] The rearrangement module 215 may perform rearrangement on the bitstream entropy-decoded by the entropy decoding module 210 based on the rearrangement method used in the apparatus for encoding a video. The rearrangement module may reconstruct and rearrange coefficients in the form of a one-dimensional vector into coefficients in the form of a two-dimensional block. The rearrangement module 215 may receive information related to coefficient scanning performed in the apparatus for encoding a video, and may perform rearrangement via a method of inversely scanning the coefficients based on the scanning order performed in the apparatus for encoding a video.
[0081] The inverse quantization module 220 may perform inverse quantization based on a quantization parameter received from the apparatus for encoding a video and coefficients of the rearranged block.
[0082] The inverse transform module 225 may perform an inverse transform, i.e., inverse DCT, inverse DST, and inverse KLT, on the quantization result of the apparatus for encoding the video, which is an inverse process of the transform, i.e., DCT, DST, and KLT, performed by the transform module. The inverse transform may be performed based on a transmission unit determined by the apparatus for encoding the video. The inverse transform module 225 of the apparatus for decoding the video may selectively perform a transform scheme (e.g., DCT, DST, and KLT) according to multiple pieces of information such as a prediction method, a size of a current block, a prediction direction, and the like.
[0083] The prediction modules 230 and 235 may generate a prediction block based on the information on prediction block generation received from the entropy decoding module 210 and previously decoded block or picture information received from the memory 245 .
[0084] As described above, similar to the operation of the apparatus for encoding a video, when performing intra prediction, when the size of the prediction unit is the same as the size of the transform unit, intra prediction can be performed on the prediction unit based on pixels located on the left, upper left, and upper sides of the prediction unit. When performing intra prediction, when the size of the prediction unit is different from the size of the transform unit, intra prediction can be performed using reference pixels based on the transform unit. In addition, intra prediction using NxN partitioning can be used only for the minimum coding unit.
[0085] The prediction modules 230 and 235 may include a prediction unit determination module, an inter-frame prediction module, and an intra-frame prediction module. The prediction unit determination module may receive various information such as prediction unit information, prediction mode information of an intra-frame prediction method, information about motion prediction of an inter-frame prediction method, etc. from the entropy decoding module 210, may divide the current coding unit into prediction units, and may determine whether to perform inter-frame prediction or intra-frame prediction on the prediction unit. By using the information required in the inter-frame prediction of the current prediction unit received from the device for encoding a video, the inter-frame prediction module 230 may perform inter-frame prediction on the current prediction unit based on information of at least one of a previous picture or a subsequent picture of the current picture including the current prediction unit. Alternatively, inter-frame prediction may be performed based on information of some pre-reconstructed areas in the current picture including the current prediction unit.
[0086] In order to perform inter prediction, which mode of the skip mode, the merge mode, the AMVP mode, and the inter block copy mode to use as a motion prediction method of a prediction unit included in the coding unit may be determined for the coding unit.
[0087] The intra prediction module 235 can generate a prediction block based on pixel information in the current picture. When the prediction unit is a prediction unit subjected to intra prediction, intra prediction can be performed based on intra prediction mode information of the prediction unit received from the device for encoding the video. The intra prediction module 235 may include an adaptive intra smoothing (AIS) filter, a reference pixel interpolation module, and a DC filter. The AIS filter performs filtering on the reference pixels of the current block, and it can be determined whether to apply the filter according to the prediction mode of the current prediction unit. AIS filtering can be performed on the reference pixels of the current block by using the AIS filter information received from the device for encoding the video and the prediction mode of the prediction unit. When the prediction mode of the current block is a mode in which AIS filtering is not performed, the AIS filter may not be applied.
[0088] When the prediction mode of the prediction unit is a prediction mode in which intra prediction is performed based on pixel values obtained by interpolating reference pixels, the reference pixel interpolation module may interpolate the reference pixels to generate reference pixels that are integer pixels or smaller than integer pixels. When the prediction mode of the current prediction unit is a prediction mode in which a prediction block is generated without interpolating reference pixels, the reference pixels may not be interpolated. When the prediction mode of the current block is a DC mode, a DC filter may generate a prediction block by filtering.
[0089] The reconstructed block or picture may be provided to the filter module 240. The filter module 240 may include a deblocking filter, an offset correction module, and an ALF.
[0090] Information about whether to apply a deblocking filter to a corresponding block or picture and information about which filter of a strong filter and a weak filter to apply when applying the deblocking filter can be received from a device for encoding a video. The deblocking filter of a device for decoding a video can receive information about the deblocking filter from a device for encoding a video, and can perform deblocking filtering on the corresponding block.
[0091] The offset correction module may perform offset correction on the reconstructed picture based on the type of offset correction applied to the picture when encoding is performed and offset value information.
[0092] The ALF may be applied to the coding unit based on information on whether the ALF is applied, ALF coefficient information, etc. received from the apparatus for encoding a video. The ALF information may be provided to be included in a specific parameter set.
[0093] The memory 245 may store the reconstructed picture or block to be used as a reference picture or block, and may provide the reconstructed picture to the output module.
[0094] As described above, in the embodiments of the present invention, for convenience of explanation, a coding unit is used as a term indicating a unit for encoding, but the coding unit may be used as a unit that performs decoding as well as encoding.
[0095] In addition, the current block may represent a target block to be encoded / decoded. And, according to the encoding / decoding step, the current block may represent a coding tree block (or coding tree unit), a coding block (or coding unit), a transform block (or transform unit), a prediction block (or prediction unit), etc. In this specification, a "unit" means a basic unit for performing a specific encoding / decoding process, and a "block" may represent a sample array of a predetermined size. If there is no difference between them, the terms "block" and "unit" may be used interchangeably. For example, in the embodiments described below, it can be understood that coding blocks and coding units have mutually equivalent meanings.
[0096] A picture can be encoded / decoded by being divided into basic blocks having a square shape or a non-square shape. At this time, the basic block can be referred to as a coding tree unit. The coding tree unit can be defined as a coding unit of the maximum size allowed within a sequence or slice. Information about whether the coding tree unit has a square shape or a non-square shape or information about the size of the coding tree unit can be signaled through a sequence parameter set, a picture parameter set, or a slice header. The coding tree unit can be divided into smaller-sized partitions. At this time, if it is assumed that the depth of the partition generated by splitting the coding tree unit is 1, the depth of the partition generated by splitting the partition with a depth of 1 can be defined as 2. That is, the partition generated by splitting the partition with a depth of k in the coding tree unit can be defined as a depth of k+1.
[0097] A partition of any size generated by splitting a coding tree unit may be defined as a coding unit. A coding unit may be recursively split or divided into a basic unit for performing prediction, quantization, transformation, or loop filtering, etc. For example, a partition of any size generated by splitting a coding unit may be defined as a coding unit, or may be defined as a transformation unit or a prediction unit, which is a basic unit for performing prediction, quantization, transformation, or loop filtering, etc.
[0098] Alternatively, a prediction block having the same size as or smaller than the coding block can be determined by predictive partitioning of the coding block. For predictive partitioning of the coding block, one of the partition mode (Part_mode) candidates can be specified. Information for determining a partition index indicating one of the partition mode candidates can be sent via a bitstream signal. Alternatively, the partition index of the coding block can be determined based on at least one of the size, shape, or coding mode of the current block. The size or shape of the prediction block can be determined based on the partition mode specified by the partition index. The partition mode candidates may include asymmetric partition types (e.g., nLx2N, nRx2N, 2NxnU, 2NxnD). The number or type of asymmetric partition mode candidates that can be used for the coding block can be determined based on the size, shape, or coding mode of the current block.
[0099] Figure 3 Partition mode candidates that can be applied to a coding block when the coding block is encoded by inter-frame prediction are shown.
[0100] When encoding a coding block using inter prediction, Figure 3 Any one of the eight partition pattern candidates shown in is applied to the coding block.
[0101] In contrast, when the coding block is encoded using intra prediction, only the partitioning of the square shape partition is applied to the coding block. That is, when the coding block is encoded using intra prediction, the partitioning mode PART_2Nx2N or PART_NxN is applied to the coding block.
[0102] When the coding block is at the minimum size, the partitioning mode PART_NxN is applied. Here, the minimum size of the coding block can be predefined in the encoder and decoder. Alternatively, information about the minimum size of the coding block can be signaled through the bitstream. For example, the minimum size of the coding block can be signaled through the slice header. Therefore, different minimum sizes of coding blocks can be determined for each slice.
[0103] As another example, the number or type of partition mode candidates that can be used by the coding block can be set to be different according to at least one of the size and shape of the coding block. For example, the number or type of partition mode candidates that can be used by the coding block can be set to be different according to at least one of the size and shape of the coding block.
[0104] Alternatively, based on the size or shape of the coding block, the type or number of asymmetric partitioning pattern candidates that the coding block can use can be determined. The number or type of asymmetric partitioning pattern candidates that the coding block can use can be set to be different according to at least one of the size and shape of the coding block. For example, when the coding block is in a non-square shape with a width greater than the height, at least one of the partitioning modes PART_2NxN, PART_2NxnU, or PART_2NxnD is not used as a partitioning pattern candidate for the coding block. When the coding block is in a non-square shape with a height greater than the width, at least one of the partitioning modes PART_Nx2N, PART_nLx2N, or PART_nRx2N is not used as a partitioning pattern candidate for the coding block.
[0105] Generally, the size of the prediction block has a range from a size of 64x64 to a size of 4x4. However, in the case of encoding a coding block using inter-frame prediction, in order to reduce a memory bandwidth when performing motion compensation, the prediction block is set not to be in a 4x4 size.
[0106] Based on the partition mode, the coding block can be partitioned in a recursive manner. That is, based on the partition mode determined by the partition index, the coding block can be partitioned, and each partition generated by partitioning the coding block can be defined as a coding block.
[0107] Hereinafter, the method of dividing the coding unit will be described in more detail. In the embodiments described later, the coding unit may refer to a coding tree unit or a coding unit included in a coding tree unit. In addition, the "division" generated by dividing the coding block may refer to a "coding block". The division method described below may be applied to divide a coding block into a plurality of prediction blocks or a plurality of transform blocks.
[0108] The coding unit may be divided by at least one line. Herein, the angle of the line dividing the coding unit has a value in the range from 0 to 360 degrees. For example, the angle of the horizontal line may be 0 degrees, the angle of the vertical line may be 90 degrees, the angle of the diagonal line in the right-up direction may be 45 degrees, and the angle of the diagonal line in the left-up direction may be 135 degrees.
[0109] When a coding unit is divided by multiple lines, the multiple lines are at the same angle. Alternatively, at least one of the multiple lines may be at a different angle from another line. Alternatively, the multiple lines that divide the coding tree unit or the coding unit may have a predefined angle difference (e.g., a 90 degree angle).
[0110] Information about the lines dividing the coding unit may be determined by a division pattern. Alternatively, information about at least one of the direction, angle, and position of the lines within the block, and the number of lines may be encoded.
[0111] For convenience of description, in an embodiment described later, it is assumed that a coding unit is split into a plurality of coding units by using at least one of a vertical line and a horizontal line.
[0112] The number of vertical lines or horizontal lines that divide the coding unit may be at least one. For example, the coding unit may be divided into two partitions by using one vertical line or one horizontal line. Alternatively, the coding unit may be divided into three partitions by using two vertical lines or two horizontal lines. Alternatively, by using one vertical line and one horizontal line, the coding unit may be divided into four partitions, wherein the width and height of each of the four partitions are 1 / 2 of the width and height of the coding unit.
[0113] When a coding unit is divided into multiple partitions by using at least one vertical line or at least one horizontal line, the partitions have a uniform size. Alternatively, any one partition may have a different size from the other partitions, or the partitions may have different sizes respectively. For example, when the coding unit is divided using two horizontal lines or two vertical lines, the coding unit is divided into three partitions. In this article, the width ratio or height ratio of the three partitions may be n:2n:n, 2n:n:n, or n:n:2n.
[0114] In an embodiment described later, dividing the coding unit into four partitions is referred to as a quadtree-based partition. In addition, dividing the coding unit into two partitions is referred to as a binary tree-based partition. In addition, dividing the coding unit into three partitions is referred to as a ternary tree-based partition.
[0115] In the figures described later, the coding unit is shown to be divided using one vertical line and / or one horizontal line. However, it is also within the scope of the present invention to divide the coding unit into a greater number or less number of divisions than the divisions shown in the figures by using a greater number of vertical lines and / or horizontal lines than the lines shown in the figures.
[0116] Figure 4 2 is a diagram showing an example of hierarchical division of coding blocks based on a tree structure according to an embodiment of the present invention.
[0117] The input video signal is decoded in predetermined block units. Such a default unit for decoding the input video signal is a coding block. A coding block may be a unit that performs intra / inter prediction, transformation, and quantization. In addition, a prediction mode (e.g., an intra prediction mode or an inter prediction mode) is determined in units of coding blocks, and the prediction blocks included in the coding block may share the determined prediction mode. A coding block may be a square block or a non-square block of any size in the range of 8x8 to 64x64, or may be a square block or a non-square block of a size of 128x128, 256x256, or larger.
[0118] Specifically, the coding block may be hierarchically divided based on at least one of a quadtree partitioning method, a binary tree partitioning method, or a ternary tree partitioning method. Quadtree-based partitioning may mean dividing a 2N×2N coding block into four N×N coding blocks. Binary tree-based partitioning may mean dividing one coding block into two coding blocks. Ternary tree-based partitioning may mean dividing one coding block into three coding blocks. Even if binary tree-based partitioning or ternary tree-based partitioning is performed, square coding blocks may exist at a lower depth.
[0119] The partitions generated by the binary tree-based partitioning may be in a symmetrical shape or an asymmetrical shape. In addition, the coding blocks generated by the binary tree-based partitioning may be square blocks or non-square blocks (eg, rectangular blocks).
[0120] Figure 52 is a diagram showing the division type of a coding block based on binary tree division. The division shape of the coding block based on binary tree division may include symmetric types such as 2NxN (horizontal non-square coding unit), Nx2N (vertical non-square coding unit), etc., or may include asymmetric types such as nLx2N, nRx2N, 2NxnU, 2NxnD, etc. Any of the symmetric type or the asymmetric type is allowed as the division type of the coding block.
[0121] The ternary tree-based partition type may include at least one of a type in which the coding block is partitioned by two vertical lines or a type in which the coding block is partitioned by two horizontal lines. Three non-square partitions may be generated through the ternary tree-based partition.
[0122] Figure 6 is a diagram showing a ternary tree based partition type.
[0123] The ternary tree-based partitioning may include a type of partitioning the coding block using two horizontal lines or a type of partitioning the coding block using two vertical lines. The width ratio or height ratio of the partition generated as a result of partitioning the coding block may be n:2n:n, 2n:n:n, or n:n:2n.
[0124] The position of the partition with the largest width or height among the three partitions may be predefined in the encoder and the decoder. Alternatively, information indicating the partition with the largest width or height among the three partitions may be signaled through the bitstream.
[0125] For coding units, only splitting in square shape or non-square symmetric shape is allowed. In this case, splitting the coding unit into square splits may correspond to quadtree CU splits, and splitting the coding unit into symmetric non-square splits may correspond to binary tree splits. Splitting the coding tree unit into square splits and symmetric non-square splits may correspond to quadtree and binary tree CU splits (QTBT).
[0126] Binary tree-based partitioning or ternary tree-based partitioning may be performed on a coding block for which quadtree-based partitioning is no longer performed. A coding block generated due to binary tree-based partitioning or ternary tree-based partitioning may be divided into smaller coding blocks. In this case, the coding block may be set so that at least one of quadtree-based partitioning, ternary tree-based partitioning, or binary tree-based partitioning is not applied to the coding block. Alternatively, binary tree-based partitioning in a predetermined direction or ternary tree-based partitioning in a predetermined direction may not be allowed in the coding block. For example, in a coding block generated due to binary tree-based partitioning or ternary tree-based partitioning, quadtree-based partitioning and ternary tree-based partitioning may not be allowed. Only binary tree-based partitioning may be allowed in a coding block.
[0127] Alternatively, only the coding block with the largest size among the three coding blocks generated due to the ternary tree-based partitioning may be partitioned into coding blocks with smaller sizes. Alternatively, binary tree-based partitioning or ternary tree-based partitioning may be allowed only for the coding block with the largest size among the three coding blocks generated due to the ternary tree-based partitioning.
[0128] In addition, the type of division of the sub-depth partition can be determined according to the partition type of the parent depth partition. For example, when the parent partition and the sub-partition are divided based on a binary tree, only the binary tree-based partition of the same type as the binary tree partition type of the parent depth partition is allowed in the sub-depth partition. For example, when the binary tree partition type of the parent depth partition is a 2NxN type, the binary tree partition type of the sub-depth partition is also set to a 2NxN type. Alternatively, when the binary tree partition type of the parent depth partition is an Nx2N type, the partition type of the sub-depth partition is also set to an Nx2N type.
[0129] Alternatively, the partition with the largest size among the partitions generated due to the ternary tree based partitioning can be configured to not allow binary tree based partitioning in the same direction as the partitioning direction of the parent depth partitioning or ternary tree based partitioning in the same direction as the partitioning direction of the parent depth partitioning.
[0130] Alternatively, the partition type of the sub-depth partition may be determined by considering the partition type of the parent depth partition and the partition type of the adjacent sub-depth partition. Specifically, when the parent depth partition is partitioned based on a binary tree, the partition type of the sub-depth partition is determined in a manner that does not generate the same result as the result of partitioning the parent depth partition based on a quadtree. For example, when the partition type of the parent depth partition is 2NxN and the partition type of the adjacent sub-depth partition is Nx2N, the partition type of the current sub-depth partition is not set to Nx2N. This is because, when the partition type of the current sub-depth partition is Nx2N, it results in the same result as the result of performing an NxN type quadtree partition on the parent depth partition. When the partition type of the parent depth partition is Nx2N and the partition type of the adjacent sub-depth partition is 2NxN, the partition type of the current sub-depth partition is not set to 2NxN. That is, when the binary tree partition type of the parent depth partition is different from the binary tree partition type of the adjacent sub-depth partition, the binary tree partition type of the current sub-depth partition is set to be the same as the binary tree partition type of the parent depth partition.
[0131] Alternatively, the binary tree partition type of the child depth partition may be set to be different from the binary tree partition type of the parent depth partition.
[0132] The allowed binary tree partition types may be determined on a per-sequence basis, on a per-slice basis, or on a per-coding unit basis. For example, the allowed binary tree partition types for a coding tree unit may be limited to 2NxN or Nx2N types. The allowed partition types may be predefined in an encoder or decoder. Alternatively, information about allowed partition types or disallowed partition types may be encoded and signaled via a bitstream.
[0133] Figure 7 is a diagram showing an example in which only certain types of binary tree-based partitioning are allowed.
[0134] Figure 7 (a) shows an example of a binary tree-based partitioning that only allows Nx2N types, and Figure 7 (b) shows an example of binary tree-based partitioning that only allows 2NxN type.
[0135] In order to represent various types of partitions, information about quadtree-based partitioning, information about binary tree-based partitioning, or information about ternary tree-based partitioning may be used. The information about quadtree-based partitioning may include at least one of information indicating whether quadtree-based partitioning is performed or information about the size / depth of the coding block for which quadtree-based partitioning is allowed. The information about binary tree-based partitioning may include at least one of the following: information indicating whether binary tree-based partitioning is performed; information indicating whether binary tree-based partitioning is performed in a vertical direction or a horizontal direction; information about the size / depth of the coding block for which binary tree-based partitioning is allowed; or information about the size / depth of the coding block for which binary tree-based partitioning is not allowed. The information about ternary tree-based partitioning may include at least one of the following: information indicating whether ternary tree-based partitioning is performed; information indicating whether ternary tree-based partitioning is performed in a vertical direction or a horizontal direction; information about the size / depth of the coding block for which ternary tree-based partitioning is allowed; or information about the size / depth of the coding block for which ternary tree-based partitioning is not allowed. The information about the size of the coding block may indicate a minimum value or a maximum value of at least one of a width, a height, a product of the width and the height, or a ratio of the width and the height of the coding block.
[0136] For example, when the width or height of the coding block is smaller than the minimum size allowed by the binary tree-based partitioning, or when the partition depth of the coding block is greater than the maximum depth allowed by the binary tree-based partitioning, the coding block may not allow binary tree-based partitioning.
[0137] For example, when the width or height of the coding block is smaller than the minimum size allowed by the ternary tree based partitioning, or when the partition depth of the coding block is greater than the maximum depth allowed by the ternary tree based partitioning, the coding block may not allow the ternary tree based partitioning.
[0138] Information about the allowable conditions of binary tree-based or ternary tree-based partitioning may be signaled through a bitstream. The information may be encoded in units of a sequence, a picture, or a fragment image. A fragment image may refer to at least one of a slice, a tile group, a tile, a block, a coding block, a prediction block, or a transform block.
[0139] For example, through the bitstream, the syntax "max_mtt_depth_idx_minus1" indicating the maximum depth allowed for binary / ternary tree partitioning may be encoded / decoded through the bitstream. In this case, max_mtt_depth_idx_minus1+1 may indicate the maximum depth allowed for binary / ternary tree partitioning.
[0140] For example, at least one of the following items is signaled at the sequence or slice level: the number of times binary / ternary tree partitions are allowed; the maximum depth at which binary / ternary tree partitions are allowed; or the number of depths allowed for binary / ternary tree partitions. Therefore, at least one of the number of times binary / ternary tree partitions are allowed, the maximum depth allowed for binary / ternary tree partitions, or the number of depths allowed for binary / ternary tree partitions may be different for the first slice and the second slice. For example, in the first slice, binary / ternary tree partitions may be allowed only at one depth, while in the second slice, binary / ternary tree partitions may be allowed at two depths.
[0141] Reference Figure 8 The example shown in Figure 8 , binary tree partitioning has been performed on the coding unit having a depth of 2 and the coding unit having a depth of 3. Therefore, at least one of the following items may be encoded / decoded through the bitstream: information indicating the number of times binary tree partitioning in the coding tree unit has been performed (i.e., 2 times); information indicating the maximum depth of partitions generated by binary tree partitioning in the coding tree unit (i.e., depth 3) or the number of partition depths to which binary tree partitioning has been applied in the coding tree unit (i.e., 2 (depth 2 and depth 3)).
[0142] Alternatively, the following items may be predefined in the encoder and decoder: the number of times binary / ternary tree partitioning is allowed; the depth at which binary / ternary tree partitioning is allowed; or the number of depths at which binary / ternary tree partitioning is allowed. Alternatively, based on at least one of the index of the sequence or slice or the size / shape of the coding unit, the following items may be determined: the number of times binary / ternary tree partitioning is allowed; the depth at which binary / ternary tree partitioning is allowed; or the number of depths at which binary / ternary tree partitioning is allowed. For example, in a first slice, binary / ternary tree partitioning may be allowed for only one depth, while in a second slice, binary / ternary tree partitioning may be allowed for two depths.
[0143] As another example, the number of times binary tree partitioning is allowed, the depth at which binary tree partitioning is allowed, or the number of depths at which binary tree partitioning is allowed may be set differently according to a temporal level identifier (TemporalID) of a slice or picture. Here, the temporal level identifier (TemporalID) is used to identify each of a plurality of layers of a video having scalability of at least one of view, space, time, or quality.
[0144] like Figure 4 As shown in , the first coding block 300 with a division depth (split depth) of k can be divided into multiple second coding blocks based on a quadtree. For example, the second coding blocks 310 to 340 can be square blocks with half the width and half the height of the first coding block, and the division depth of the second coding block can be increased to k+1.
[0145] The second coding block 310 with a division depth of k+1 may be divided into a plurality of third coding blocks with a division depth of k+2. The division of the second coding block 310 may be performed by selectively using one of a quadtree and a binary tree according to a division method. Here, the division method may be determined based on at least one of information indicating a quadtree-based division or information indicating a binary tree-based division.
[0146] When the second coding block 310 is divided based on a quadtree, the second coding block 310 may be divided into four third coding blocks 310a having half the width and half the height of the second coding block, and the division depth of the third coding block 310a may be increased to k+2. In contrast, when the second coding block 310 is divided based on a binary tree, the second coding block 310 may be divided into two third coding blocks. Here, each of the two third coding blocks may be a non-square block having half the width and half the height of the second coding block, and the division depth may be increased to k+2. The second coding block may be determined as a non-square block in the horizontal direction or the vertical direction according to the division direction, and the division direction may be determined based on information about whether the binary tree-based division is performed in the vertical direction or in the horizontal direction.
[0147] Meanwhile, the second coding block 310 may be determined as a leaf coding block that is no longer divided based on a quadtree or a binary tree. In this case, the leaf coding block may be used as a prediction block or a transform block.
[0148] Similar to the division of the second coding block 310, the third coding block 310a may be determined as a leaf coding block, or may be further divided based on a quadtree or a binary tree.
[0149] At the same time, the third coding block 310b based on the binary tree division can be further divided into a coding block 310b-2 in the vertical direction or a coding block 310b-3 in the horizontal direction based on the binary tree, and the division depth of the relevant coding block can be increased to k+3. Alternatively, the third coding block 310b can be determined as a leaf coding block 310b-1 that is no longer divided based on the binary tree. In this case, the coding block 310b-1 can be used as a prediction block or a transform block. However, the above division processing can be performed restrictively based on at least one of the following items: information about the size / depth of the coding block for which quadtree-based division is allowed; information about the size / depth of the coding block for which binary tree-based division is allowed; or information about the size / depth of the coding block for which binary tree-based division is not allowed.
[0150] The number of candidate sizes may be limited to a predetermined number, or the size of the coding block in a predetermined unit may have a fixed value. As an example, the size of the coding block in a sequence or in a picture may be limited to have one of 256x256, 128x128, or 32x32. Information indicating the size of the coding block in the sequence or in the picture may be signaled through a sequence header or a picture header.
[0151] As a result of quadtree and binary tree based partitioning, the coding unit may be represented in a square or rectangular shape of an arbitrary size.
[0152] like Figure 4 As shown in , the first coding block 300 with a division depth (split depth) of k can be divided into multiple second coding blocks based on a quadtree. For example, the second coding blocks 310 to 340 can be square blocks with half the width and half the height of the first coding block, and the division depth of the second coding block can be increased to k+1.
[0153] The second coding block 310 with a division depth of k+1 may be divided into a plurality of third coding blocks with a division depth of k+2. The division of the second coding block 310 may be performed by selectively using one of a quadtree and a binary tree according to a division method. Here, the division method may be determined based on at least one of information indicating a quadtree-based division or information indicating a binary tree-based division.
[0154] When the second coding block 310 is divided based on a quadtree, the second coding block 310 may be divided into four third coding blocks 310a whose width and height are halved relative to the second coding block, and the division depth of the third coding block 310a may be increased to k+2. In contrast, when the second coding block 310 is divided based on a binary tree, the second coding block 310 may be divided into two third coding blocks. Here, each of the two third coding blocks may be a non-square block whose width and height are halved relative to the second coding block, and the division depth may be increased to k+2. The second coding block may be determined as a non-square block in the horizontal direction or the vertical direction according to the division direction, and the division direction may be determined based on information about whether the binary tree-based division is performed in the vertical direction or in the horizontal direction.
[0155] Meanwhile, the second coding block 310 may be determined as a leaf coding block that is no longer divided based on a quadtree or a binary tree. In this case, the leaf coding block may be used as a prediction block or a transform block.
[0156] Similar to the division of the second coding block 310, the third coding block 310a may be determined as a leaf coding block, or may be further divided based on a quadtree or a binary tree.
[0157] At the same time, the third coding block 310b based on the binary tree division can be further divided into a coding block 310b-2 in the vertical direction or a coding block 310b-3 in the horizontal direction based on the binary tree, and the division depth of the relevant coding block can be increased to k+3. Alternatively, the third coding block 310b can be determined as a leaf coding block 310b-1 that is no longer divided based on the binary tree. In this case, the coding block 310b-1 can be used as a prediction block or a transform block. However, the above division processing can be performed restrictively based on at least one of the following items: information about the size / depth of the coding block that allows quadtree-based division; information about the size / depth of the coding block that allows binary tree-based division; or information about the size / depth of the coding block that does not allow binary tree-based division.
[0158] The number of candidate sizes may be limited to a predetermined number, or the size of a coding block in a predetermined unit may have a fixed value. As an example, the size of a coding block in a sequence or in a picture may be forced to have one of 256×256, 128×128, or 32×32. Information indicating the size of a coding block in a sequence or in a picture may be signaled through a sequence header or a picture header.
[0159] As a result of quadtree and binary tree based partitioning, the coding unit may be represented in a square or rectangular shape of an arbitrary size.
[0160] Due to the partitioning based on quadtree, binary tree or ternary tree, the coding block that is no longer divided can be used as a prediction block or a transform block. That is, the coding block that is no longer divided can be used as a coding block, a prediction block or a transform block generated due to quadtree partitioning or binary tree partitioning. For example, a predicted image can be generated on the basis of each coding block, and a residual signal as the difference between the original image and the predicted image can be transformed on the basis of each coding block. In order to generate a predicted image on the basis of each coding block, motion information can be determined on the basis of the coding block, or an intra-frame prediction mode can be determined on the basis of the coding block. Therefore, the coding block can be encoded using at least one of skip mode, intra-frame prediction or inter-frame prediction.
[0161] Alternatively, the plurality of coding blocks generated by dividing the coding block may be configured to share at least one of motion information, merge candidates, reference samples, reference sample lines, or intra-frame prediction modes. As an example, when the coding block is divided by ternary tree-based division, the divisions generated by dividing the coding block may share at least one of motion information, merge candidates, reference samples, reference sample lines, or intra-frame prediction modes according to the size and shape of the coding block. Alternatively, only some of the plurality of coding blocks may be set to share the information, and other coding blocks may be set not to share the information.
[0162] As another example, by partitioning the coding block, a prediction block or a transform block having a size smaller than that of the coding block may be used.
[0163] Hereinafter, a method of performing transformation on a coding block or a transformation block generated by dividing the coding block will be described in detail. In an embodiment described later, the current block may refer to a transformation block or a coding block as a basic unit for performing transformation and quantization.
[0164] Fig. 9 : is a flowchart showing a residual sample encoding process as an embodiment to which the present invention is applied.
[0165] The residual samples can be obtained by subtracting the predicted samples from the original samples.
[0166] When the residual sample is obtained, it can be determined whether to skip the transformation for the obtained residual sample (S910). The encoder can encode information indicating whether to skip the transformation for the current block. For example, "transform_skip_flag" indicates whether the transformation for the current block is skipped. The value of transform_skip_flag is 1, indicating that the transformation for the current block is skipped, and the value of transform_skip_flag is 0, indicating that the transformation for the current block is not skipped.
[0167] The encoder can encode information for determining the size of a block that allows skipping of transforms. For example, "log2_transform_skip_max_size_minus2" indicates the maximum size of a block that allows skipping of transforms. The encoder can encode transform_skip_flag only when the current block is less than or equal to the maximum size that allows skipping of transforms.
[0168] Alternatively, it may be determined whether to skip the transformation for each of the horizontal direction and the vertical direction. The encoder may encode at least one of information about whether to skip the transformation for the horizontal direction or information indicating whether to skip the transformation for the vertical direction. For example, at least one of "hor_transform_skip_flag" indicating whether to skip the transformation for the horizontal direction or "ver_transform_skip_flag" indicating whether to skip the transformation for the vertical direction may be encoded.
[0169] When it is determined that the transform skip is not applied to the current block, a primary transform may be performed on the current block (S920). The primary transform may be performed based on a change core such as DCT or DST. The change core may be determined based on the transform type of the current block. The encoder may encode information for determining the transform type of the current block. Alternatively, the transform type of the current block may be implicitly derived in the encoder and the decoder according to a predetermined rule. For example, the transform type of the current block may be determined based on at least one of the size of the current block, the shape of the current block, the encoding mode of the current block, or the intra-prediction mode of the current block.
[0170] A secondary transform may be performed on the transform coefficients of the primary transform (S930). The secondary transform may be selectively performed only when the primary transform is performed on the current block. Whether to perform the secondary transform may be determined based on at least one of the size, shape, transform type, coding mode, or intra prediction mode of the current block. Alternatively, the encoder may encode information indicating whether to perform the secondary transform.
[0171] The secondary transform may be performed on a partial area of the current block. The position and / or size of the area where the secondary transform is performed may be predefined in the encoder and the decoder. Alternatively, information indicating the area where the secondary transform is performed may be encoded in the bitstream. Alternatively, the area where the secondary transform is performed may be determined based on at least one of the size, shape, coding mode, or intra-prediction mode of the current block.
[0172] When it is determined to skip the transformation for the current block, the residual samples of the current block may be scaled (S940). The scaling may be performed using a scaling parameter. The scaling parameter may be determined based on the width, height, or size of the current block.
[0173] The transform coefficient generated by transforming the residual coefficient or the scaled residual coefficient generated by scaling the residual coefficient may be quantized (S950). The residual coefficient generated due to the quantization may be arranged in one dimension, and the residual coefficient may be encoded (S960). The arrangement order of the residual coefficient may follow at least one of diagonal scanning, zigzag scanning, upward scanning, vertical scanning, or horizontal scanning.
[0174] Fig.10 is a flowchart illustrating a process of decoding residual samples according to an embodiment to which the present invention is applied.
[0175] First, the residual coefficients of the current block can be obtained (S1010). The decoder can obtain the residual coefficients by coefficient scanning. The coefficient scanning can be performed based on at least one of diagonal scanning, zigzag scanning, upward scanning, vertical scanning or horizontal scanning. The scanning order of the current block can be determined based on at least one of the size, shape, coding mode or intra-frame prediction mode of the current block. As a result of the coefficient scanning, the residual coefficients in the form of a two-dimensional block can be obtained.
[0176] Inverse quantization may be performed on the residual coefficient of the current block (S1020).
[0177] It may be determined whether to skip the inverse transform in the inverse quantized residual coefficients of the current block (S1030). Information indicating whether to skip the inverse transform for the current block may be signaled through the bitstream. For example, "transform_skip_flag" indicates whether to skip the inverse transform for the current block. A value of transform_skip_flag of 1 indicates that the inverse transform for the current block is skipped, while a value of transform_skip_flag of 0 indicates that the inverse transform for the current block is not skipped.
[0178] Information for determining the size of a block for which inverse transform is allowed to be skipped may be signaled via a bitstream. For example, "log2_transform_skip_max_size_minus2" indicates that the maximum size of a block for which inverse transform is allowed to be skipped is allowed. transform_skip_flag may be encoded only when the current block is less than or equal to the maximum size for which inverse transform is allowed to be skipped. When encoding of transform_skip_flag is omitted, the value of transform_skip_flag may be derived as a predefined value. The predefined value may be 0. Therefore, when the size of the current block is greater than the maximum size for which inverse transform is allowed to be skipped, inverse transform may be performed on the current block.
[0179] Alternatively, it may be determined whether to skip inverse transform for each of the horizontal direction and the vertical direction. At least one of information indicating whether to skip inverse transform for the horizontal direction or information indicating whether to skip inverse transform for the vertical direction may be signaled through a bitstream. For example, at least one of “hor_transform_skip_flag” indicating whether to skip inverse transform for the horizontal direction or “ver_transform_skip_flag” indicating whether to skip inverse transform for the vertical direction may be signaled through a bitstream.
[0180] hor_transform_skip_flag and ver_transform_skip_flag may be encoded only if the current block is smaller than the maximum size that allows skipping inverse transforms. As an example, hor_transform_skip_flag may be encoded only if the width of the current block is smaller than the maximum size that allows skipping inverse transforms. Additionally, ver_transform_skip_flag may be encoded only if the height of the current block is smaller than the maximum size that allows skipping inverse transforms.
[0181] Alternatively, whether to encode hor_transform_skip_flag and ver_transform_skip_flag may be determined based on the shape of the current block. As an example, hor_transform_skip_flag may be encoded only when the current block is a non-square block with a width greater than a height. In addition, ver_transform_skip_flag may be encoded only when the current block is a non-square block with a height greater than a weight.
[0182] When the encoding of the information is omitted, the value of the information may be derived as a predetermined value. The predetermined value may be 0.
[0183] When it is determined to perform an inverse transform on the current block, the inverse quantized residual coefficients of the current block may be inversely transformed (S1040, S1050). The inverse transform may be performed in the order of a secondary inverse transform and a primary inverse transform. The secondary inverse transform may be performed on a partial area of the current block. The transform matrix of the secondary inverse transform may be determined based on the size of the area to which the secondary inverse transform is applied.
[0184] The secondary inverse transform may be selectively performed. In other words, the primary inverse transform may be performed and the secondary inverse transform may be omitted. Whether to perform the secondary inverse transform may be determined based on at least one of the size, shape, transform type, coding mode, or intra-prediction mode of the current block. Alternatively, information indicating whether to perform the secondary inverse transform may be signaled via a bitstream.
[0185] The primary inverse transform may be performed based on a variation kernel such as DCT or DST. The encoder may encode information for determining the transform type of the current block. Alternatively, the transform type of the current block may be determined according to a rule predetermined by the encoder and the decoder. For example, the transform type of the current block may be determined based on at least one of the size of the current block, the shape of the current block, the coding mode of the current block, or the intra-prediction mode of the current block.
[0186] As a result of performing the inverse transform, residual samples of the current block may be obtained.
[0187] When it is determined that the inverse transform is skipped in the current block, the inverse quantized residual coefficient of the current block may be scaled (S1060). The scaling may be performed using a scaling parameter. The scaling parameter may be determined based on the width, height, or size of the current block. The residual coefficient of the current block may be obtained by scaling the inverse quantized residual coefficient based on the scaling parameter.
[0188] Transformation in at least one of the vertical direction or the horizontal direction may be skipped. When transform skipping is applied only in the horizontal direction, transformation may not be performed in the horizontal direction, but transformation may be performed in the vertical direction. When transform skipping is applied only in the vertical direction, transformation may not be performed in the vertical direction, but transformation may be performed in the horizontal direction.
[0189] It may be determined whether a transform skip technique for the current block can be used according to the partition type of the parent node block. For example, when the current block is generated by binary tree-based partitioning, the transform skip technique may not be allowed for the current block. That is, when the current block is generated by binary tree-based partitioning, the residual sample of the current block may be obtained by transforming the inverse quantized residual coefficients. When the current block is generated by binary tree-based partitioning, the encoding / decoding of information indicating whether to skip the transform (e.g., at least one of transform_skip_flag, hor_transform_skip_flag, or ver_transform_skip_flag) may be omitted.
[0190] Alternatively, when the current block is generated by binary tree-based partitioning, the transform skipping technique may be allowed only in the horizontal direction or the vertical direction. The direction in which the transform skipping technique is not allowed may be determined based on at least one of the information signaled by the bitstream, the size of the current block, the shape of the current block, or the intra-frame prediction mode of the current block. As an example, when the current block has a non-square shape with a width greater than the height, the transform skipping technique may be allowed only in the vertical direction. That is, when the current block is in the form of 2N×N, the transform may be performed in the horizontal direction, and the transform may be selectively performed in the vertical direction. Alternatively, when the current block is a non-square shape with a height greater than the width, the transform skipping technique may be allowed only in the horizontal direction. That is, when the current block is in the form of N×2N, the transform may be performed in the vertical direction, and the transform may be selectively performed in the horizontal direction. In contrast to the above example, when the current block is a non-square block with a width greater than the height, the transform skipping technique is allowed only in the horizontal direction, and when the current block is a non-square block with a height greater than the width, the transform skipping technique is allowed only in the vertical direction.
[0191] Alternatively, it may be determined whether to skip the transformation of the current block according to the partition type of the parent node block. As an example, when the current block is generated by binary tree-based partitioning, the transformation in the horizontal direction or the vertical direction may be omitted. That is, if the current block is generated by binary tree-based partitioning, the transformation in at least one of the horizontal direction or the vertical direction of the current block may be skipped without encoding / decoding information indicating whether to skip the transformation of the current block (e.g., transform_skip_flag, hor_transform_skip_flag, ver_transform_skip_flag).
[0192] Hereinafter, the primary transform / primary inverse transform and the secondary transform / secondary inverse transform will be described in detail. Hereinafter, the contents described with emphasis on the primary transform and the secondary transform may be extensively applied to the primary inverse transform and the secondary inverse transform.
[0193] The primary transform may be performed based on a transform kernel such as DCT or DST. As an example, Formula 1 represents a transform matrix A4 based on DST-VII. An inverse transform based on DST-VII may be performed based on A4T.
[0194] [Formula 1]
[0195]
[0196] Formula 2 represents the transformation matrix T8 based on DCT-II. T Performs an inverse DCT-II based transform.
[0197] [Formula 2]
[0198]
[0199] The change core of the current block may be determined based on the transform type of the current block. The transform type of the current block may be explicitly determined based on information signaled through a bitstream. The information may be index information indicating a transform type for a horizontal direction and a transform type for a vertical direction. Table 1 shows an example in which a transform type for a horizontal direction and a transform type for a vertical direction are determined based on index information.
[0200]
Table 1
[0201]
[0202]
[0203] In Table 1, HorTrType indicates a transform type in a horizontal direction, and VerTrType indicates a transform type in a vertical direction.
[0204] Information indicating whether index information for determining the transform type in the horizontal direction and the vertical direction is transmitted by signal may be encoded. The information may be transmitted by signal through a picture parameter set or a sequence parameter set. When the index information is not transmitted by signal, the transform type in the horizontal direction and the vertical direction may be implicitly determined based on at least one of the size, shape, or intra prediction mode of the current block.
[0205] For example, when the current block is encoded by inter-frame prediction, the transform type in the horizontal direction and the vertical direction can be determined based on the size or shape of the current block. Specifically, when the current block is a non-square shape with a width greater than a height and the width of the current block is included in a predefined range, the transform type in the horizontal direction of the current block can be determined as DCT-II. If the above conditions are not met, the transform type in the horizontal direction of the current block can be determined as DST-VII. In addition, when the current block is a non-square shape with a height greater than a width and the height of the current block is included in a predefined range, the transform type in the vertical direction of the current block can be determined as DCT-II. If the above conditions are not met, the transform type in the vertical direction of the current block can be determined as DST-VII. The predefined range can be defined as N or greater and / or M or less. N and M are different natural numbers and can be 4, 8, 16 or 32.
[0206] When the current block is encoded by intra prediction, the transform type in the horizontal direction and the transform type in the vertical direction may be determined based on at least one of the size, shape, or intra prediction mode of the current block. For example, when the intra prediction mode of the current block is the DC mode, the transform type in the horizontal direction and the transform type in the vertical direction of the current block may be determined as DST-VII.
[0207] When the intra prediction mode of the current block is the planar mode, the transform type in the horizontal direction and the transform type in the vertical direction of the current block can be determined based on whether the width and height of the current block are respectively included in the predefined range. For example, when the width of the current block is included in the predefined range, the transform type in the horizontal direction of the current block can be determined as DCT-II. On the other hand, when the width of the current block is not included in the predefined range, the transform type in the horizontal direction of the current block can be determined as DST-VII. When the height of the current block is included in the predefined range, the transform type in the vertical direction of the current block can be determined as DCT-II. On the other hand, when the height of the current block is not included in the predefined range, the transform type in the vertical direction of the current block can be determined as DST-VII.
[0208] When the intra prediction mode of the current block is a directional mode of the first direction, the transform type of the horizontal direction of the current block can be determined as DST-VII. When the intra prediction mode of the current block is the first directional mode, the transform type of the vertical direction of the current block can be determined as DST-VII or DCT-II according to whether the height of the current block is included in the predefined range.
[0209] When the intra prediction mode of the current block is a directional mode of the second direction, the transform type of the horizontal direction of the current block may be determined as DST-VII or DCT-II according to whether the width of the current block is included in the predefined range. When the intra prediction mode of the current block is the second directional mode, the transform type of the vertical direction of the current block may be determined as DST-VII. The first direction and the second direction may be different from each other.
[0210] Alternatively, the transform type of the current block may be determined based on at least one of the size, shape, encoding mode, or intra prediction mode of the current block, but the transform type in the vertical direction and the transform type in the horizontal direction may be set to be the same.
[0211] As an example, when the current block is a 4×4 block encoded in intra mode, a transform matrix based on DST (specifically, DST-VII) can be used. If the current block does not meet the above conditions, a transform matrix based on DCT (specifically, DCT-II) can be used.
[0212] Alternatively, the transform type of the current block may be determined based on at least one of an intra prediction mode of the current block or the number of samples included in the current block. The number of samples used to determine the transform type may have a value predefined by the encoder and the decoder. Alternatively, information for determining the number of samples may be signaled via a bitstream. The information may be signaled via a block, a slice header, or a picture parameter set.
[0213] When the number of samples indicates 16, if the current block is encoded by the intra prediction mode and the number of samples included in the current block is 16 or less, the DST-based transform matrix may be applied to the current block. For example, the DST-based transform matrix may be applied to a block of size 4x4, 2x8, or 8x2 encoded by intra prediction. On the other hand, when the current block does not satisfy the above conditions, a DCT-based transform matrix may be used.
[0214] The condition for determining the transform type may be set differently for each sequence, slice, or block. As an example, in slice 0, when the transform block encoded in intra mode has a size of 4×4, DST may be selected. On the other hand, in slice 1, when the size of the transform block encoded in intra mode is 8×8 or less, DST may be selected.
[0215] As another example, based on the transform set, the transform type of the current block can be determined. Index information specifying the transform type of the current block can be sent by a bitstream signal. Alternatively, the transform type of the current block can be determined based on at least one of the size, shape, coding mode, or intra-frame prediction mode of the current block. The transform set can be determined for a coding block or a transform block. Therefore, the transform sets between the coding block and the transform block can be different from each other. Alternatively, multiple transform blocks can share a transform set. Multiple transform blocks can be included in a coding block having a predetermined size or a predetermined shape.
[0216] Table 2 shows a number of transformation sets.
[0217]
Table 2
[0218] Transformation Collection Index Transform Candidate 0 Transformation candidate 1 0 DST-VII DCT-II 1 DST-VII DST-I 2 DST-VII DCT-VIII
[0219] The transform set may include at least one transform type candidate. At least one of the number of transform type candidates or the types of transform type candidates included in each of the multiple transform sets may be different. That is, at least one of the number of transform type candidates or the types of transform type candidates included in one transform set may be different from the number of transform type candidates or the types of transform type candidates included in another transform set.
[0220] Table 1 shows that each transform set includes two transform type candidates. A transform set including one, three, four or more transform type candidates may be defined. Alternatively, transform type candidate 0 and transform type candidate 1 may define the same transform set.
[0221] Alternatively, the maximum number of transform type candidates included in the transform set may be signaled through the bitstream. The number of transform type candidates included in each transform set may be less than or equal to the maximum number signaled through the bitstream. This information may be signaled through a slice header or a sequence header.
[0222] Any one of the transform type candidates included in the transform set may be determined as the transform type of the current block. The transform type of the current block may be determined based on at least one of the size, shape, coding mode, or intra prediction mode of the current block. The intra prediction mode represents an intra prediction mode of a prediction block or coding block corresponding to the transform block.
[0223] As an example, when the current block is a 4x4 block coded in intra mode, transform type candidate 0 in the transform set may be determined as the transform type of the current block. On the other hand, when the current block does not meet the above conditions, transform type candidate 1 in the transform set may be determined as the transform type of the current block.
[0224] As a specific example, when the transform set index 0 is determined as the transform set of the current block, if the current block is a block of size 4x4 encoded in intra-frame mode, the transform can be performed based on DST-VII. On the other hand, when the current block does not meet the above conditions, the transform can be performed based on DCT-II. When the transform set index 2 is determined as the transform set of the current block, if the current block is a block of size 4×4 or 8×8 encoded in intra-frame mode, the transform can be performed based on DST-VII. If the current block does not meet the above conditions, the transform can be performed based on DCT-VIII.
[0225] The selection condition of the transform type candidate may be determined based on at least one of the size, shape, coding mode, or intra prediction mode of the coding block. For example, when the size of the coding block is less than or equal to 32×32, transform type candidate 0 may be applied only when the transform block is encoded in intra mode and the size of the transform block is 4×4. On the other hand, when the size of the coding block is greater than 32×32, transform type candidate 0 may be applied only when the transform block is encoded in intra mode and the size of the transform block is 8×8 or less.
[0226] Alternatively, information for designating any one of a plurality of transform type candidates may be signaled through a bitstream.
[0227] A transform skip may be set as a transform type candidate. Whether the transform skip can be used as a transform type candidate may be determined based on whether the transform skip is allowed in the current block. Depending on whether the transform skip can be used as a transform skip candidate, at least one of the types of the included transform type candidates or the number of transform type candidates in the transform set may be set differently.
[0228] Table 3 shows a plurality of transform sets in which transform skip is set as a transform skip candidate.
[0229]
Table 3
[0230] Transformation Collection Index Transform Candidate 0 Transformation candidate 1 Transformation Candidate 2 0 DST-VII DCT-II Transform Skip 1 DST-VII DST-I Transform Skip 2 DST-VII DCT-VIII Transform Skip
[0231] As shown in Table 3, when transform_skip_enabled_flag indicating whether transform skipping is allowed in a predetermined picture is 1, a transform set including transform skipping as a transform type candidate may be used. On the other hand, as shown in Table 2, when transform_skip_enabled_flag is 0, a transform set not including transform skipping as a transform type candidate may be used.
[0232] The transform type in the horizontal direction and the transform type in the vertical direction of the current block may be set equally. For example, when determining the transform type of the current block, the determined transform type may be applied to both the determined horizontal transform and the determined vertical transform. Alternatively, the transform type in the horizontal direction and the transform type in the vertical direction of the current block may be set differently. For example, when determining the transform set of the current block, transform type candidate 0 may be determined as the transform type for the horizontal direction, and transform type candidate 1 may be determined as the transform type for the vertical direction.
[0233] As another example, a transform set for the horizontal direction and a transform set for the vertical direction may be determined separately. To this end, a first transform set index for specifying a transform set in the horizontal direction and a second transform set index for specifying a transform set in the vertical direction may be signaled through a bitstream. Any one of the transform type candidates included in the first transform set specified by the first transform set index may be determined as a transform type in the horizontal direction. Any one of the transform type candidates included in the second transform set specified by the second transform set index may be determined as a transform type in the vertical direction.
[0234] As an example, it is assumed that the transform set corresponding to the transform set index 0 is determined as the transform set in the horizontal direction, and the transform set corresponding to the transform set index 1 is determined as the transform set in the vertical direction. When the current block is not encoded by intra prediction or the size of the current block is not 4×4, the transform type candidate 1 may be determined as the transform type of the current block. That is, for the transform type of the vertical transform, the transform type candidate 1 (i.e., DCT-II) included in the transform set index 0 is determined as the transform type in the horizontal direction, and the transform type candidate 1 (i.e., DST-I) included in the transform set index 1 may be determined as the transform type in the vertical direction.
[0235] The first transform set index and the second transform set index may represent the same transform set. Alternatively, the second transform set index may specify one of the residual transform sets other than the transform set specified by the first transform set index.
[0236] Whether the transform set for the horizontal direction and the transform set for the vertical direction are the same can be determined based on at least one of the size, shape, coding mode or intra-prediction mode of the current block. For ease of explanation, the transform set for horizontal transform is defined as the horizontal transform set, and the transform set for vertical transform is defined as the vertical transform set. For example, when the intra-prediction mode of the current block is an intra-prediction mode similar to the horizontal direction or an intra-prediction mode similar to the vertical direction, the horizontal transform set and the vertical transform set may be different from each other. Here, the intra-prediction mode similar to the horizontal direction may mean an intra-prediction mode in which the difference in mode value of the horizontal direction or the horizontal direction intra-prediction mode is within a predefined value. In addition, the intra-prediction mode similar to the vertical direction may mean an intra-prediction mode in which the difference in mode value of the vertical direction or the vertical direction intra-prediction mode is within a predefined value.
[0237] On the other hand, when the intra prediction mode of the current block is a non-directional mode or a directional mode that does not satisfy the above conditions, the horizontal transform set and the vertical transform set may be the same. When the horizontal transform set and the vertical transform set are set to be the same, at least one of the first index information for specifying the horizontal transform set or the second index information for specifying the vertical transform set may be omitted.
[0238] Alternatively, when the intra prediction mode of the current block is a non-directional mode, the horizontal direction transform set and the vertical direction transform set of the current block may be set differently from each other.
[0239] Fig.11 : is a diagram showing whether a transform set in the vertical direction and a transform set in the horizontal direction are the same for 33 intra prediction modes.
[0240] exist Fig.11 In the example shown in , when the intra prediction mode of the current block is included in the range of 7 to 13 or 23 to 29, the horizontal direction transform set and the vertical direction transform set are shown to be different from each other. On the other hand, when the intra prediction mode of the current block is a directional mode not included in the above range, it is shown that the horizontal direction transform set and the vertical direction transform set are the same.
[0241] When there is a block having the same intra prediction mode as the current block in the predetermined block, the transform set of the current block can be set equal to the transform set of the block having the same intra prediction mode as the current block. Here, the predetermined block can be a coding block, a coding tree block or a block with a predefined size.
[0242] As an example, assuming that the direction of the intra prediction mode corresponding to the first transform block in the coding block is vertical (e.g., mode number 26), the horizontal direction transform set is index 2, and the vertical direction transform set is index 0. Here, the order of the transform blocks can be determined according to the scanning order of the coding block.
[0243] When there is still a transform block with an intra prediction mode in the vertical direction in the coding block, the transform set index value may not be signaled to the newly scanned transform block. Instead, the transform set of the transform block with the intra prediction mode in the vertical direction may be applied to the transform set of the newly scanned transform block. That is, the horizontal transform set and the vertical transform set of the newly scanned transform block may be determined as index 2 and index 0, respectively.
[0244] As another example, when there is a block having an intra prediction mode similar to the current block in the predetermined blocks, the transform set of the current block may be set equal to the transform set of the block having an intra prediction mode similar to the current block. Here, the intra prediction mode similar to the current block may refer to a standard intra prediction mode or an intra prediction mode whose mode value difference with the standard intra prediction mode is less than or equal to a predetermined value. The standard intra prediction mode may include an intra prediction mode in the horizontal direction or an intra prediction mode in the vertical direction.
[0245] As an example, assuming that the direction of the intra prediction mode corresponding to the first transform block in the coding block is vertical (eg, mode number 26), the horizontal direction transform set is index 2, and the vertical direction transform set is index 0.
[0246] When there is a transform block with an intra prediction mode (e.g., mode number 27) similar to that in the vertical direction in the coding block, the transform set index value may not be signaled to the newly scanned transform block. Instead, the transform set of the transform block with an intra prediction mode similar to that of the current block may be applied to the transform set of the newly scanned transform block. That is, the horizontal direction transform set of the newly scanned transform block may be determined as index 2, and the vertical direction transform set may be determined as index 0.
[0247] Based on the intra prediction mode of the current block, a transform set for the horizontal direction and / or a transform set for the vertical direction may be determined. Table 4 shows a horizontal transform set and a vertical transform set according to the intra prediction mode of the current block.
[0248]
Table 4
[0249]
[0250] When the current block is encoded by inter prediction, the current block may use a predefined transform set. The predefined transform set may be a transform set with an index of 0.
[0251] Alternatively, when a coding block is encoded by inter prediction, a transform set is selected for the coding block, and a transform block within the coding block may use a transform type candidate included in the transform set of the coding block. In this case, the transform type of each transform block may be determined by the size and / or shape of the transform block, or information identifying the transform type selected for each transform block may be signaled through a bitstream.
[0252] Selecting a transform type for a current block from a plurality of transform type candidates may be defined as adaptive multiple transform (AMT). Whether to apply adaptive multiple transform to a current block may be determined based on at least one of the size, shape, coding mode, or intra prediction mode of the current block. Information for determining at least one of the size or shape of a block allowing adaptive multiple transform may be signaled through a bitstream. The size information may include at least one of a maximum size or a minimum size allowing multiple transforms. The size of a block allowing multiple transforms and / or information for determining the size of a block may be signaled through at least one of a block, a slice header, or a sequence header.
[0253] Next, the secondary transformation will be described.
[0254] After performing a primary transform on the current block, the encoder may perform a secondary transform. After performing a secondary inverse transform on the current block, the decoder may perform a primary inverse transform.
[0255] Whether to apply the secondary transform to the current block may be determined based on at least one of the size, shape, transform type, or quantization parameter size of the current block. For example, when at least one of the width or height of the current block is less than a predefined value, the secondary transform may not be performed on the current block.
[0256] The secondary transform may be performed on the entire region of the current block or a partial region of the current block. Fig.12 is a diagram showing an execution area of a secondary transformation.
[0257] In the case where the primary transform is performed on the entire region of the current block, the secondary transform may be performed on a partial region of the current block.
[0258] The position and / or size of the area where the secondary transform is performed may be predefined in the encoder and decoder. For example, the area may have a size of M×N, and the position of the upper left corner of the area may coincide with the upper left corner of the current block. M and N may be 2, 4, 8, or 16. M and N may have the same value or different values. For example, the secondary transform may be applied to a 4×4 block.
[0259] Alternatively, the position and / or size of the area where the secondary transform is performed may be determined based on at least one of the size, shape, transform type, size of the quantization parameter, or whether residual flipping is performed on the current block. For example, when at least one of the height or width of the current block is equal to or less than a threshold, a secondary transform may be performed on a 4×4 block. When the height and width of the current block are greater than a threshold, a secondary transform may be performed on an 8×8 block. Here, the threshold may be a natural number, such as 4, 8, or 16.
[0260] Alternatively, information for determining the location and / or size of the region where the secondary transform is performed may be signaled through a bitstream. The information may be signaled through a slice, a picture, or a sequence.
[0261] The secondary transform may be a separable transform (separable transform) or an inseparable transform (non-separable transform). The separable transform and the non-separable transform may be classified according to whether a transform for a horizontal direction and a transform for a vertical direction are performed separately when performing the secondary transform.
[0262] Specifically, separable transform means that horizontal transform and vertical transform are performed separately for a block having a size of M×N. Non-separable transform means that horizontal transform and vertical transform are not separated for a block having a size of M×N.
[0263] The inseparable transformation may be performed based on a one-dimensional matrix in which the transformation matrix T and the transformation coefficient included in the block having the size of M×N are arranged in one dimension. Specifically, a (M×N)×1 matrix may be created by arranging the transformation coefficients included in the block having the size of M×N in one dimension, and a transformation coefficient having a size of K×1 may be generated by multiplying the transformation matrix T having a size of K×(M×N) by the one-dimensional matrix. The decoder may perform a secondary inverse transformation using the transformation matrix TT.
[0264] The transformation matrix T may be determined based on the type of the secondary transformation. Information for determining the type of the secondary transformation may be signaled through a bitstream. Alternatively, the type of the secondary transformation may be determined based on at least one of the size, width, coding mode, or intra prediction mode of the current block.
[0265] Formula 3 shows transform coefficients included in a 4×4 block, and Formula 4 shows an example of arranging the transform coefficients included in a 4×4 block in one dimension.
[0266] [Formula 3]
[0267]
[0268] [Formula 4]
[0269]
[0270] By arranging the transform coefficients included in the block having a size of 4×4 in one row, a matrix having a size of 16×1 can be obtained. The order of arrangement of the transform coefficients may follow the scanning order of the current block. That is, the order of arrangement of the transform coefficients may follow at least one of vertical scanning, horizontal scanning, raster scanning, zigzag scanning or diagonal scanning.
[0271] Formula 5 shows the one-dimensional matrix An example of performing a secondary transformation with the transformation matrix T.
[0272] [Formula 5]
[0273]
[0274] When the size of the transformation matrix T is 16×16, a matrix having a size of 16×1 may be generated due to the secondary transformation.
[0275] Performing a non-separable transform only on some regions of a transform block may be referred to as a sub-non-separable transform. Formula 6 shows a sub-non-separable transform.
[0276] [Formula 6]
[0277] F i ′=T′·X i '
[0278] T' represents the transformation matrix, and X i ' represents a matrix in which transform coefficients included in a sub-block are arranged in one dimension. i ' represents an inseparable transform coefficient generated by performing inseparable transform. The inseparable transform coefficient of a region where inseparable transform is not performed may be set to a predefined value. The predefined value may be an integer including 0.
[0279] The size of each matrix may be determined based on the size of the sub-block on which the inseparable transform is performed. As an example, assume that the inseparable transform is performed only in an area corresponding to a 2×2 size sub-block of a 4×4 size transform block. i ' denotes a 4x1 matrix in which transform coefficients included in a 2x2 subblock are arranged in one dimension, and T' denotes a 16x4 transform matrix. Fi' denotes a 16x1 transform coefficient matrix generated as a result of performing an inseparable transform.
[0280] For example, it is assumed that the inseparable transform is performed only in the region corresponding to the 8×4 size sub-block of the 8×8 size transform block. i ' represents a 32×1 matrix in which transform coefficients included in an 8×4 sub-block are arranged in one dimension, and T' represents a 64×32 transform matrix. i ' represents a transform coefficient matrix having a size of 64×1 generated as a result of performing the non-separable transform.
[0281] As an example, it is assumed that the inseparable transform is performed only in the region corresponding to the 4x4 size sub-block of the 8x8 size transform block. i ' denotes a 16x1 matrix in which transform coefficients included in a 4x4 subblock are arranged in one dimension, and T' denotes a 64x16 transform matrix. Fi' denotes a transform coefficient matrix having a size of 64x1 generated as a result of performing an inseparable transform.
[0282] The area where the sub-transform is performed does not necessarily need to be square. For example, a non-square area of 2×8 or 8×2 or a polygonal area in which three blocks of 4×4 are combined can be determined as the area where the sub-transform is performed.
[0283] The size or area of the area where the secondary transformation is performed may be determined according to the size, shape, coding mode, or intra-prediction mode of the current block. For example, when the current block is a square, the area where the secondary transformation is performed may be set to a square. When the current block is a non-square, the area where the secondary transformation is performed may be set to a non-square.
[0284] An inseparable transform can be composed of multiple sub-inseparable transforms. Formula 7 shows multiple inseparable transforms.
[0285] [Formula 7]
[0286] F0′=T′·X0′, F1′=T′·X1′, F2′=T′·X2′,…F n-1 ′=T′·X n-1 '
[0287] In Formula 7, X n ' represents the following matrix, in which the transform coefficients contained in the Nth sub-block are arranged in one dimension. The non-separable transform coefficients F of the Nth sub-block can be generated by the non-separable transform n '.
[0288] The set of non-separable transform coefficients of each sub-block may be set as the non-separable transform coefficients of the current block. Formula 8 shows the non-separable transform coefficient F of the current block.
[0289] [Formula 8]
[0290]
[0291] The size of the transform matrix T may be determined based on the size of the region to which the secondary transform is applied. For example, when the secondary transform is applied to a 4×4 block, a 16×16 transform matrix may be used, and when the secondary transform is applied to an 8×8 block, a 64×64 transform matrix may be used.
[0292] Alternatively, a reduced transform matrix may be used in which the number of rows or columns is less than the number of samples included in the region to which the secondary transform is applied. For example, for a 4×4 block, a 4×16 reduced transform matrix may be used instead of a 16×16 transform matrix to perform the secondary transform. When the reduced transform matrix is used, four transform coefficients may be output due to the secondary transform. The four samples may be mapped to a block having a size of 2×2 at the upper left of the block, and the transform coefficient of the residual region may be set to 0.
[0293] Alternatively, for an 8x8 block, a 16x64 reduction transform matrix may be used instead of a 64x64 transform matrix to perform a secondary transform. When a reduction transform matrix is used, 16 transform coefficients may be output as a result of performing the secondary transform. The 16 samples may be mapped to a block having a size of 4x4 at the upper left of the block, and the transform coefficient of the residual region may be set to 0.
[0294] Information for determining the size of the reduced transform matrix may be signaled via the bitstream. The information may indicate a reduction parameter. The number of columns or rows of the reduced transform matrix may be determined by scaling the number of samples included in the execution region of the secondary transform using the reduction parameter.
[0295] Whether to apply the reduced transform matrix may be determined based on at least one of the size, shape, coding mode, intra prediction mode, or transform mode of the current block. For example, when at least one condition in which the width and / or height of the current block is greater than or equal to a threshold or the current block is encoded by intra prediction is satisfied, a transform matrix based on a reduced secondary transform may be allowed.
[0296] Alternatively, information indicating whether the downscaling transform matrix is allowed may be signaled through a bitstream.
[0297] In the above-described embodiment, it has been described that the residual coefficients are encoded in the order of primary transform, secondary transform, and quantization. Unlike the described example, the residual coefficients may be encoded in the order of primary transform, quantization, and secondary transform.
[0298] It is within the scope of the present invention to apply the embodiments described with emphasis on the decoding process or the encoding process to the encoding process or the decoding process. It is also within the scope of the present invention to change the embodiments described in a predetermined order to a different order.
[0299] Although the above-mentioned embodiments have been described based on a series of steps or flow charts, these embodiments do not limit the time series order of the present invention, and can be executed simultaneously or in different orders as required. In addition, each of the components (e.g., units, modules, etc.) constituting the block diagram in the above-mentioned embodiments can be implemented by hardware devices or software and multiple components. Or, multiple components can be combined and implemented by a single hardware device or software. The above-mentioned embodiments can be implemented in the form of program instructions, which can be executed by various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include one or a combination of program commands, data files, data structures, etc. The example of a computer-readable medium includes: magnetic media such as hard disks, floppy disks and tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floppy disks, media, and hardware devices such as ROMs, RAMs, flash memories, etc. that are specifically configured to store and execute program instructions. The hardware device can be configured to operate as one or more software modules for performing processing according to the present invention, and vice versa.
[0300] Industrial Applicability
[0301] The present invention can be applied to electronic devices capable of encoding / decoding videos.
[0302] The solution of the present invention also includes:
[0303] (1) A method for decoding an image, the method comprising:
[0304] Get the residual coefficient of the current block;
[0305] Inverse quantizing the residual coefficients;
[0306] performing a secondary inverse transform on the inverse quantized residual coefficients; and
[0307] performing a primary inverse transform on a result of performing said secondary inverse transform,
[0308] Wherein, the secondary inverse transform is performed on a partial area of the current block.
[0309] (2) The method according to (1), wherein the secondary inverse transform is performed using a transform matrix and a one-dimensional matrix generated by arranging the inverse quantized residual coefficients included in the partial region in one dimension.
[0310] (3) The method according to (2), wherein the number of rows or columns of the transformation matrix is smaller than the number of samples included in the partial region.
[0311] (4) The method according to (1), wherein the size or shape of the partial area is determined based on the size or shape of the current block.
[0312] (5) The method according to (1), wherein the transform type of the primary inverse transform is determined based on index information signaled via a bitstream.
[0313] (6) The method according to (5), wherein the index information specifies any one of a plurality of transformation sets,
[0314] The first transform type candidate included in the transform set is determined as the horizontal transform type of the current block, and
[0315] The second transform type candidate included in the transform set is determined as the vertical transform type of the current block.
[0316] (7) The method according to (1), wherein the transform type of the primary inverse transform is determined based on a comparison result of the width of the current block and the height of the current block.
[0317] (8) A method for encoding an image, the method comprising:
[0318] Performing a primary transform on the residual samples;
[0319] performing a secondary transform on primary transform coefficients generated as a result of the primary transform;
[0320] quantizing secondary transform coefficients resulting from the secondary transform; and
[0321] Encode the quantized transform coefficients,
[0322] The secondary transformation is performed on a partial area of the current block.
[0323] (9) The method according to (8), wherein the secondary transform is performed using a transform matrix and a one-dimensional matrix, and the one-dimensional matrix is generated by arranging the primary transform coefficients included in the partial region in one dimension.
[0324] (10) The method according to (9), wherein the number of rows or columns of the transformation matrix is smaller than the number of samples included in the partial region.
[0325] (11). The method according to (8), wherein the size or shape of the partial area is determined based on the size or shape of the current block.
[0326] (12) The method according to (8), wherein index information indicating the transform type of the primary transform is encoded in a bitstream.
[0327] (13) The method according to (12), wherein the index information specifies any one of a plurality of transformation sets,
[0328] The first transform type candidate included in the transform set is determined as the horizontal transform type of the current block, and
[0329] The second transform type candidate included in the transform set is determined as the vertical transform type of the current block.
[0330] (14) The method according to (8), wherein the transform type of the primary transform is determined based on a comparison result of the width of the current block and the height of the current block.
[0331] (15) A device for decoding an image, the device comprising:
[0332] A decoding unit, which is used to decode the residual coefficients of the current block;
[0333] an inverse quantization unit, configured to inverse quantize the residual coefficient; and
[0334] an inverse transform unit, which is used to perform a secondary inverse transform on the inverse quantized residual coefficients, and to perform a primary inverse transform on the result of the secondary inverse transform,
[0335] Wherein, the secondary inverse transform is performed on a partial area of the current block.
Claims
1. A method for decoding an image, the method comprising: Divide the coding block into a plurality of child node blocks, the current block being one of the child node blocks; Obtaining residual coefficients of the current block; performing inverse quantization on the residual coefficients; performing a secondary inverse transform on the current block; as well as performing a primary inverse transform on a result of performing said secondary inverse transform, wherein the secondary inverse transform is performed on the inverse quantized residual coefficients obtained from the inverse quantization and included in the partial area of the current block, wherein the secondary inverse transform is performed by using an inverse transform matrix and an input matrix generated by arranging the inverse quantized residual coefficients in the two-dimensional form in the partial region into a one-dimensional form, and The number of rows or columns of the inverse transformation matrix is greater than the number of the inverse quantized residual coefficients included in the partial region.
2. The method according to claim 1, wherein: The size of the partial area is adaptively determined based on the size of the current block.
3. The method according to claim 1, wherein: The transform type of the primary inverse transform is determined based on index information signaled via the bitstream.
4. The method according to claim 3, wherein: The index information specifies any one of a plurality of transformation sets. The first transform type candidate included in the transform set is determined as the horizontal transform type of the current block, and The second transform type candidate included in the transform set is determined as the vertical transform type of the current block.
5. A method for encoding an image, the method comprising: Divide the coding block into a plurality of child node blocks, the current block being one of the child node blocks; performing a primary transform on residual samples of the current block; performing a secondary transform on the current block; quantizing transform coefficients resulting from the secondary transform; as well as Encoding the quantized transform coefficients, wherein the secondary transform is performed on the intermediate transform coefficients obtained from the primary transform and included in the partial area of the current block, wherein the secondary transform is performed by using a transform matrix and an input matrix generated by arranging the intermediate transform coefficients in a two-dimensional form into a one-dimensional form, and The number of rows or columns of the transformation matrix is smaller than the number of the intermediate transformation coefficients included in the partial region.
6. A device for transmitting compressed video data, the device comprising: a processor configured to obtain the compressed video data; as well as a sending unit, configured to send the compressed video data, Wherein, obtaining the compressed video data comprises: performing a primary transform on residual samples of the current block; performing a secondary transform on the current block; quantizing transform coefficients resulting from the secondary transform; and Encoding the quantized transform coefficients, wherein the secondary transform is performed on the intermediate transform coefficients obtained from the primary transform and included in the partial area of the current block, wherein the secondary transform is performed by using a transform matrix and an input matrix generated by arranging the intermediate transform coefficients in a two-dimensional form into a one-dimensional form, and The number of rows or columns of the transformation matrix is smaller than the number of the intermediate transformation coefficients included in the partial region.
