Method for encoding / decoding an image and method for storing a bit stream

Abstract

The present invention relates to a method for encoding / decoding an image and a method for storing a bitstream. The method for encoding an image includes: generating geometric modification information of a current block; determining a geometric modification flag; determining geometric modification type information of the current block; determining a reference geometric modification index indicating reference geometric modification information; determining predicted geometric modification information of the current block based on the reference geometric modification information of the current block; determining residual geometric modification information; generating a predicted block of the current block by performing inter-frame prediction based on a reference picture of the current block and the geometric modification information of the current block; generating a residual block based on residual data of the current block; and encoding the current block based on the predicted block and the residual block of the current block; and encoding a bitstream including a geometric modification flag, a reference geometric modification index, residual geometric modification information, and geometric modification type information, wherein the geometric modification information is used to perform affine modification on a reference area referenced by the current block.

Description

[0001] This application is a divisional application of the invention patent application with the application date of November 18, 2016, application number 201680066455.0, and invention name "Method and device for encoding / decoding images using geometrically modified pictures". Technical Field

[0002] The present invention generally relates to a method and apparatus for encoding / decoding an image by using a geometrically modified picture, wherein the geometrically modified picture is generated by geometrically modifying a reference picture. Background Art

[0003] As high-definition (HD) broadcasting expands across the country and around the world, many users have become accustomed to images with high resolution and high picture quality. As a result, many organizations are driving the development of the next generation of image devices. In addition, as interest in ultra-high definition (UHD), which has a resolution several times higher than HDTV, continues to grow, technologies that can compress and process images with higher resolution and higher image quality are needed.

[0004] As image compression techniques, there are various techniques such as inter-frame prediction techniques in which pixel values ​​included in a current picture are predicted based on pictures before or after the current picture; intra-frame prediction techniques in which pixel values ​​included in a current picture are predicted using pixel information in the current picture; transform and quantization techniques for compressing energy of a residual signal; and entropy coding techniques in which short codes are assigned to values ​​with high frequency of occurrence and long codes are assigned to values ​​with low frequency of occurrence. Using these image compression techniques, image data can be transmitted and stored in a state in which the image data is effectively compressed.

[0005] When global motion is included in the reference picture referred to during inter-frame prediction, the similarity between the reference picture and the current picture is reduced. The reduced similarity between the reference picture and the current picture may lead to a reduction in prediction efficiency. Moreover, when performing intra-frame prediction, since the prediction direction of the intra-frame prediction of the current block is limited, the efficiency of the intra-frame prediction may deteriorate. Therefore, improvements are needed to solve the above problems. Summary of the invention

[0006] Technical issues

[0007] The present invention aims to provide a method and device for effectively encoding / decoding images.

[0008] Furthermore, the present invention provides a method and apparatus for efficiently performing intra prediction and / or inter prediction.

[0009] In addition, the present invention provides a method and apparatus for generating a geometrically modified picture by geometrically modifying a reference picture.

[0010] In addition, the present invention provides a method and apparatus for efficiently signaling information related to a geometry-modified picture.

[0011] In addition, the present invention provides a method and apparatus for performing intra prediction and / or inter prediction by referring to a geometrically modified picture.

[0012] Technical Solution

[0013] According to one aspect of the present invention, a method for encoding an image is provided. The method may include: generating a geometrically modified picture by geometrically modifying a reference picture; and generating a first prediction block of a current block by performing inter-frame prediction with reference to the geometrically modified picture.

[0014] According to the encoding method of the present invention, the method may further include: generating a second prediction block of the current block by performing inter-frame prediction with reference to the reference picture; and selecting a final prediction block for motion compensation of the current block from the first prediction block and the second prediction block.

[0015] According to the encoding method of the present invention, the final prediction block can be selected by selecting one having less error with the current block among the first prediction block and the second prediction block.

[0016] According to the encoding method of the present invention, the method may further include generating geometry modification information for generating a geometry modification picture, and the generating of the geometry modification picture may be performed based on the geometry modification information and the reference picture.

[0017] According to the encoding method of the present invention, the method may also include encoding geometric modification information.

[0018] According to the encoding method of the present invention, encoding the geometry modification information may include reconfiguring the geometry modification information, and reconfiguring the geometry modification information may include simplifying or predicting the geometry modification information.

[0019] According to the encoding method of the present invention, the geometry modification information may be generated based on the change in pixel information between a current picture including a current block and a reference picture.

[0020] According to the encoding method of the present invention, the geometry modification information may be generated based on the matching information between the feature points extracted from each of the current picture and the reference picture.

[0021] According to the encoding method of the present invention, generating a geometrically modified picture may include: identifying a point in a reference picture, which corresponds to a point in the geometrically modified picture; and setting pixel information of the corresponding point in the reference picture as pixel information of the point in the geometrically modified picture.

[0022] According to the encoding method of the present invention, when a point corresponding to a point in the geometrically modified picture does not exist in the reference picture, a point in the reference picture that is closest to the corresponding point can be identified as the corresponding point.

[0023] According to another aspect of the present invention, a method for decoding an image is provided. The method may include generating a geometrically modified picture by geometrically modifying a reference picture; and generating a prediction block of a current block by performing inter-frame prediction of the reference geometrically modified picture.

[0024] According to the decoding method of the present invention, the method may also include: receiving motion compensation related information; and determining whether a geometric modification picture is used for motion compensation of a current block based on the motion compensation related information, and when determining, when it is determined that a reference geometric modification picture is used for motion compensation of the current block, the generating of the geometric modification picture and the generating of the prediction block may be performed.

[0025] According to the decoding method of the present invention, the determination may be performed based on whether the motion compensation related information includes geometry modification information or based on information on whether to reference a geometry modification picture, the information being included in the motion compensation related information.

[0026] According to the decoding method of the present invention, when determining, when determining to reference a geometric modification picture in motion compensation of a current block, the method may further include reconstructing the geometric modification information based on information about the geometric modification information included in the motion compensation related information, and the generating of the geometric modification picture may be performed based on the reference picture and the reconstructed geometric modification information.

[0027] According to the decoding method of the present invention, the information about the geometric modification information may include residual geometric modification information or a scaling coefficient, and the reconstruction of the geometric modification information may be performed based on the residual geometric modification information and the scaling coefficient and at least one of the previously stored geometric modification information.

[0028] According to the decoding method of the present invention, the motion compensation related information may include information about previously stored geometric modification information, and the information about the previously stored geometric modification information may be information identifying one geometric modification information among one or more previously stored geometric modification information.

[0029] According to the decoding method of the present invention, generating a geometrically modified picture may include: identifying a point in a reference picture, which corresponds to a point in the geometrically modified picture; and setting pixel information of the corresponding point in the reference picture as pixel information of the point in the geometrically modified picture.

[0030] According to the decoding method of the present invention, when a point corresponding to the point in the geometrically modified picture does not exist in the reference picture, a point in the reference picture that is closest to the corresponding point can be identified as the corresponding point.

[0031] According to the decoding method of the present invention, when a corresponding point within the reference picture has real coordinates, one or more points within the reference picture that have integer coordinates and are adjacent to the corresponding point can be identified, and pixel information of the corresponding point within the reference picture can be derived based on the pixel information of each one or more points with integer coordinates.

[0032] According to the decoding method of the present invention, the generating of the geometrically modified picture is performed in at least one unit of a picture, a slice, a tile, a coding unit and a prediction unit.

[0033] Beneficial Effects

[0034] According to the present invention, images can be efficiently encoded / decoded.

[0035] In addition, according to the present invention, inter prediction and / or intra prediction can be efficiently performed.

[0036] Furthermore, according to the present invention, a geometrically modified picture may be generated by geometrically modifying a reference picture.

[0037] In addition, according to the present invention, information related to a geometry modified picture can be efficiently signaled.

[0038] Furthermore, according to the present invention, intra prediction and / or inter prediction may be performed by referring to a geometry modified picture. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a block diagram showing the configuration of an image encoding device to which an embodiment of the present invention is applied.

[0040] Figure 2 is a block diagram showing the configuration of an image decoding device to which an embodiment of the present invention is applied.

[0041] Figure 3 The diagram schematically shows a partition structure of an image when encoding the image.

[0042] Figure 4is a diagram showing a form of a prediction unit (PU) that can be included in a coding unit (CU).

[0043] Figure 5 is a diagram illustrating a form of a transform unit (TU) that may be included in a coding unit (CU).

[0044] Figure 6 is a diagram illustrating an example of intra prediction processing.

[0045] Figure 7 is a diagram illustrating an example of inter-frame prediction processing.

[0046] Figure 8 is a diagram showing a transfer modification of an embodiment of a geometric modification of an image according to the present invention.

[0047] Fig. 9 is a diagram showing size modification of an embodiment of geometric modification of an image according to the present invention.

[0048] Fig.10 is a diagram showing rotation modification of an embodiment of geometric modification of an image according to the present invention.

[0049] Fig.11 is a diagram showing an affine modification of an embodiment of a geometric modification of an image according to the present invention.

[0050] Fig.12 is a diagram showing a projection modification of an embodiment of a geometric modification of an image according to the present invention.

[0051] Fig.13 is a diagram showing an example of a method of implementing homography according to the present invention.

[0052] Fig.14 is an example method of deriving a relational expression between two points corresponding between two images according to the present invention.

[0053] Fig.15 is a diagram illustrating a method of generating a geometry modification image based on a geometry modification matrix and an original image according to the present invention.

[0054] Fig.16 is a diagram illustrating a method of generating a geometry-modified image by using inverse mapping according to the present invention.

[0055] Fig.17 is a diagram illustrating a method of generating a geometry modification image based on a geometry modification matrix and an original image according to the present invention, wherein the geometry modification matrix may correspond to geometry modification information.

[0056] Fig.18 is a reference diagram showing an embodiment of the present invention. Fig.17A graph of bilinear interpolation among the various interpolation methods shown.

[0057] Fig.19 is a diagram illustrating a method of performing motion prediction according to the present invention, in which a video encoder generates geometry modification information and a geometry modification picture, and performs motion prediction by using a reference picture and the geometry modification picture.

[0058] Fig. 20 is a block diagram showing the configuration of an image encoding device to which another embodiment of the present invention is applied.

[0059] Fig.21 It shows that according to Fig. 20 A diagram showing the configuration and operation of a geometrically modified picture generation unit 2010 of an example encoding device is shown.

[0060] Fig. 22 It shows that according to Fig. 20 A diagram showing the configuration and operation of a geometrically modified picture predictor 2015 of an example encoding device is shown.

[0061] Fig.23 is a flowchart illustrating motion prediction according to an embodiment of the present invention.

[0062] Fig.24 is a flowchart illustrating a method of generating a geometry-modified picture according to an embodiment of the present invention.

[0063] Fig.25 is a flowchart illustrating an inter-frame prediction method according to an embodiment of the present invention.

[0064] Fig.26 is a flowchart illustrating a method for encoding geometry modification information according to an embodiment of the present invention.

[0065] Fig. 27 is a diagram showing various examples for modifying geometry modification information to reduce the amount of bits used to encode the geometry modification information.

[0066] Fig.28 is a diagram illustrating a motion compensation method in which a decoder generates a geometry modified picture from a reference picture by using geometry modification information and performs motion compensation by using the reference picture and the geometry modified picture.

[0067] Fig.29 is a block diagram showing the configuration of a decoding device to which another embodiment of the present invention is applied.

[0068] Fig.30 It is shown Fig.29 A diagram illustrating the operation and configuration of a geometrically modified picture generator 2970 of a decoding device is shown.

[0069] Fig.31 It is shown Fig.29 A diagram illustrating the operation and configuration of a geometrically modified picture predictor 2960 of a decoding device is shown.

[0070] Fig.32 is a flowchart illustrating motion compensation of a decoder according to an embodiment of the present invention.

[0071] Fig.33 is a conceptual diagram illustrating extended intra prediction according to an embodiment of the present invention.

[0072] Fig.34 is a diagram illustrating the operation and configuration of an extended intra prediction unit according to an embodiment of the present invention.

[0073] Fig.35 is a diagram illustrating extended intra prediction according to an embodiment of the present invention.

[0074] Fig.36 is a diagram showing an intra prediction direction according to the present invention.

[0075] Fig.37 is a diagram showing an embodiment of performing extended intra prediction according to the present invention.

[0076] Specific implementation mode

[0077] Since various modifications can be made to the present invention, and there are various embodiments of the present invention, examples will now be provided with reference to the accompanying drawings and will be described in detail. However, the present invention is not limited thereto, and the exemplary embodiments may be interpreted as including all modifications, equivalents or substitutes within the technical concept and technical scope of the present invention. In various aspects, similar reference numerals refer to the same or similar functions. In the accompanying drawings, for the sake of clarity, the shapes and sizes of the elements may be exaggerated, and the same reference numerals are used throughout the text to designate the same or similar elements. In the following detailed description of the present invention, reference is made to the accompanying drawings showing specific embodiments in which the present invention can be practiced in an illustrative manner. These embodiments are described in sufficient detail to enable those skilled in the art to implement the present disclosure. It should be understood that the various embodiments of the present disclosure, although different, are not necessarily mutually exclusive. For example, the specific features, structures and characteristics described herein in conjunction with one embodiment may be implemented in other embodiments without departing from the spirit and scope of the present disclosure. In addition, it should be understood that the position or arrangement of each element within each disclosed embodiment may be modified without departing from the spirit and scope of the present disclosure. Therefore, the following detailed description should not be regarded as having a limiting meaning, and the scope of the present disclosure and the full scope equivalent to the scope claimed by the claims are limited and properly interpreted only by the attached claims.

[0078] The terms "first", "second", etc. used in the specification may be used to describe various components, but these components should not be construed as being limited to these terms. These terms are only used to distinguish one component from other components. For example, a "first" component may be referred to as a "second" component without departing from the scope of the present invention, and the "second" component may also be similarly referred to as a "first" component. The term "and / or" includes a combination of multiple items or any one of the multiple items.

[0079] When it is mentioned that an element is "coupled" or "connected" to another element, it may mean that it is directly coupled or connected to another element, but it should be understood that there may be another element between them. On the other hand, when it is mentioned that an element is "directly coupled" or "directly connected" to another element, it should be understood that there are no other elements between them.

[0080] In addition, the components shown in the embodiments of the present invention are shown independently to represent the characteristic functions that are different from each other. Therefore, this does not mean that each component is composed of a separate hardware or software component. In other words, for convenience, each component includes each of the enumerated components. Therefore, at least two components of each component can be combined to form a component, or a component can be divided into multiple components to perform each function. Without departing from the essence of the present invention, the embodiment of combining each component and the embodiment in which a component is divided are also included in the scope of the present invention.

[0081] The terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention. Expressions used in the singular include expressions in the plural form unless they have clear different meanings in the context. In this specification, it should be understood that terms such as "including", "having" etc. are intended to indicate the existence of features, numbers, steps, actions, elements, parts or combinations thereof disclosed in the specification, and are not intended to exclude the possibility that one or more other features, numbers, steps, actions, elements, parts or combinations thereof may exist or may be added. In other words, when a particular element is referred to as "comprising", elements other than the corresponding element are not excluded, but additional elements may be included in an embodiment of the present invention or in the scope of the present invention.

[0082] In addition, some components may not be indispensable components for performing the basic functions of the present invention, but are selective components that only improve its performance. In addition to components for improving performance, the present invention can be implemented by only including basic indispensable components for implementing the present invention. In addition to selective components that are only used to improve performance, structures that only include indispensable components are also included in the scope of the present invention.

[0083] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. When describing the exemplary embodiments of the present invention, well-known functions or configurations will not be described in detail because they may unnecessarily obscure the understanding of the present invention. The same component elements in the accompanying drawings are represented by the same reference numerals, and repeated descriptions of the same elements will be omitted.

[0084] In addition, hereinafter, an image may refer to a picture configuring a video, or may refer to a video. For example, "encoding and / or decoding an image" may refer to "encoding and / or decoding a video", or may refer to "encoding and / or decoding a single image among images configuring a video". Here, a picture may refer to an image.

[0085] Encoder: can refer to an encoding device.

[0086] Decoder: can refer to a decoding device.

[0087] Parsing: may refer to determining syntax element values ​​by performing entropy decoding, or may refer to an entropy decoder.

[0088] Block: can refer to an MxN matrix of samples, where M and N are positive integers. Block can refer to a two-dimensional matrix of samples.

[0089] Unit: may refer to a unit for encoding or decoding an image. When encoding and decoding an image, a unit may be a region generated by partitioning the image. Alternatively, when an image is subdivided and encoded or decoded, a unit may refer to a divided unit of an image. While encoding and decoding, a predetermined process may be performed for each unit. A single unit may be divided into smaller sub-units. The unit may also refer to a block, a macroblock (MB), a coding unit (CU), a prediction unit (PU), a transform unit (TU), a coding block (CB), a prediction block (PB), or a transform block (TB) according to its function. A unit may refer to an object including a luminance component block for each block, its corresponding chrominance component block, and a syntax element for indicating a distinction from a block. The unit may have different sizes and shapes. Specifically, the shape of the unit may include a two-dimensional form such as a rectangle, a cube, a trapezoid, a triangle, a pentagon, etc. In addition, the shape of the unit may include a geometric figure. In addition, the unit information may include at least one of a unit type (such as a coding unit, a prediction unit, a transform unit, etc.), a unit size, a unit depth, and a sequence of unit encoding and decoding.

[0090] Reconstruction neighboring unit: may refer to a reconstructed unit that has been encoded or decoded in space / time and is adjacent to the encoding / decoding target unit.

[0091] Depth: Indicates the degree of partitioning of a cell. In a tree structure, the highest node may refer to the root node and the lowest node may refer to the leaf node.

[0092] Symbol: It can refer to syntax elements and coding parameters of the encoding / decoding target unit, the value of the transform coefficient, etc.

[0093] Parameter set: may correspond to header information in a structure within a bitstream. At least one of a video parameter set, a sequence parameter set, a picture parameter set, and an adaptation parameter set may be included in the parameter set. In addition, the parameter set may include information of a slice header and a tile header.

[0094] Bitstream: May refer to a string of bits that includes encoded image information.

[0095] Coding parameters: may include not only information encoded by the encoder and then sent to the decoder together with syntax elements, but also information that can be derived during the encoding or decoding process, or may refer to parameters necessary for encoding and decoding. For example, coding parameters may include at least one value and / or statistics as follows: intra prediction mode, inter prediction mode, intra prediction direction, motion information, motion vector, reference image index, inter prediction direction, inter prediction indicator, reference image list, motion vector predictor, motion merge candidate, transform type, transform size, information on whether to use additional transform, filter information within the loop, information on whether there is a residual signal, quantization parameter, context model, transform coefficient, transform coefficient level, encoded block pattern, encoded block flag, image display / output order, slice information, tile information, picture type, information on whether to use motion merge mode, information on whether to use skip mode, block size, block depth, block partition information, unit size, unit partition information, etc.

[0096] Prediction unit: may refer to a basic unit when performing inter-frame prediction or intra-frame prediction and when compensating for prediction. A prediction unit may be divided into a plurality of partitions. Each of the partitions may also be a basic unit when performing inter-frame prediction or intra-frame prediction and when compensating for prediction. A partitioned prediction unit may also refer to a prediction unit. In addition, a single prediction unit may be divided into smaller sub-units. A prediction unit may have various sizes and shapes. Specifically, the shape of the unit may include two-dimensional forms such as a rectangle, a cube, a trapezoid, a triangle, a pentagon, etc. In addition, the shape of the unit may include a geometric figure.

[0097] Prediction unit partition: may refer to the partitioning form of the prediction unit.

[0098] Reference picture list: may refer to a list including at least one reference picture used for inter prediction or motion compensation. The types of reference lists may include combined list (LC), L0 (list 0), L1 (list 1), L2 (list 2), L3 (list 3), etc. At least one reference picture list may be used for inter prediction.

[0099] Inter prediction indicator: may refer to the inter prediction direction (unidirectional prediction, bidirectional prediction) of the encoding / decoding target block. Alternatively, the indicator may refer to the number of reference pictures used to generate the prediction block of the encoding / decoding target block, or may refer to the number of prediction blocks used when the encoding / decoding target block performs motion compensation.

[0100] Reference picture index: may refer to the index of a specific picture within a reference picture list.

[0101] Reference picture: may refer to a reference picture referred to by a specific unit for inter prediction or motion compensation. Alternatively, reference image may refer to a reference picture.

[0102] Motion vector: refers to a two-dimensional matrix used for inter-frame prediction or motion compensation, or may be an offset between an encoding / decoding target image and a reference image. For example, (mvX, mvY) may indicate a motion vector, mvX may be a horizontal component, and mvY may be a vertical component.

[0103] Motion vector candidate: may refer to a unit that becomes a prediction candidate when predicting a motion vector, or may refer to a motion vector of a unit.

[0104] Motion vector candidate list: may refer to a list configured with motion vector candidates.

[0105] Motion vector candidate index: may refer to an indicator indicating a motion vector candidate within a motion vector candidate list, or may refer to an index of a motion vector predictor.

[0106] Motion information: may refer to information including at least one of a motion vector, a reference picture index, an inter prediction indicator, reference picture list information, a reference picture, a motion vector candidate, a motion vector candidate index, and the like.

[0107] Transform unit: When performing transformation of transform coefficients, inverse transformation, quantization, inverse quantization, and encoding / decoding of residual signals, a basic unit may be referred to. A single unit may be divided into smaller sub-units. The unit may have different sizes and shapes. Specifically, the shape of the unit may include two-dimensional forms such as rectangles, cubes, trapezoids, triangles, pentagons, etc. In addition, the shape of the unit may also include geometric figures.

[0108] Scaling: may refer to the process of multiplying factors by transform coefficient levels and the result may generate transform coefficients. Scaling may also refer to inverse quantization.

[0109] Quantization parameter: may refer to a value used to scale the level of transform coefficients in quantization and inverse quantization. Here, the quantization parameter may be a value mapped to a quantization step size.

[0110] Differential quantization parameter: may refer to a residual value between a predicted quantization parameter and a quantization parameter of an encoding / decoding target unit.

[0111] Scan: can refer to the method of reordering the order of coefficients within a block or matrix. For example, reordering a two-dimensional matrix into a one-dimensional matrix can be referred to as scanning or inverse scanning.

[0112] Transform coefficient: may be a coefficient value generated after performing transformation. In the present invention, a transform coefficient level quantized by applying quantization to the transform coefficient may be included in the transform coefficient.

[0113] Non-zero transform coefficient: may refer to a transform coefficient whose value or magnitude is not zero.

[0114] Quantization matrix: can refer to the matrix used for quantization and inverse quantization to improve the quality of the image. Quantization matrix can also refer to scaling list.

[0115] Quantization matrix coefficients: can refer to each element of the quantization matrix. Quantization matrix coefficients can also refer to matrix coefficients.

[0116] Default matrix: may refer to a predetermined quantization matrix defined in advance in an encoder and a decoder.

[0117] Non-default matrix: can refer to the quantization matrix sent / received by the user and is not defined in advance in the encoder and decoder.

[0118] Figure 1 is a block diagram showing the configuration of an image encoding device to which an embodiment of the present invention is applied.

[0119] The encoding device 100 may be a video encoding device or an image encoding device. A video may include at least one image. The encoding device 100 may encode at least one image of a video in time sequence.

[0120] refer to Figure 1 , the encoding device 100 may include a motion prediction unit 111, a motion compensation unit 112, an intra-frame prediction unit 120, a switch 115, a subtractor 125, a transform unit 130, a quantization unit 140, an entropy encoding unit 150, an inverse quantization unit 160, an inverse transform unit 170, an adder 175, a filter unit 180 and a reference picture buffer 190.

[0121] The encoding device 100 may encode the input image in intra mode or inter mode or both. In addition, the encoding device 100 may generate a bit stream by encoding the input image, and may output the generated bit stream. When the intra mode is used as the prediction mode, the switch 115 may be switched to intra. When the inter mode is used as the prediction mode, the switch 115 may be switched to inter. Here, the intra mode may be referred to as the intra prediction mode, and the inter mode may be referred to as the inter prediction mode. The encoding device 100 may generate a prediction signal for an input block of the input image. The prediction signal as a block unit may be referred to as a prediction block. In addition, after generating the prediction block, the encoding device 100 may encode the residual value between the input block and the prediction block. The input image may be referred to as the current image as the target of the current encoding. The input block may be referred to as the current block, or the encoding target block as the target of the current encoding.

[0122] When the prediction mode is intra mode, the intra prediction unit 120 may use the pixel value of the previously encoded block adjacent to the current block as a reference pixel. The intra prediction unit 120 may perform spatial prediction by using the reference pixel for spatial prediction, and may generate a prediction sample of the input block by using the spatial prediction. Here, intra prediction may mean intra-frame prediction.

[0123] When the prediction mode is the inter mode, the motion prediction unit 111 may search for an area that best matches an input block of a reference image during motion prediction and may derive a motion vector by using the searched area. The reference image may be stored in the reference picture buffer 190.

[0124] The motion compensation unit 112 may generate a prediction block by performing motion compensation using a motion vector. Here, the motion vector may be a two-dimensional vector used in inter-frame prediction. In addition, the motion vector may indicate an offset between a current image and a reference image. Here, inter-frame prediction may refer to inter-frame prediction.

[0125] When the value of the motion vector is not an integer, the motion prediction unit 111 and the motion compensation unit 112 may generate a prediction block by applying an interpolation filter to a partial area in the reference image. In order to perform inter-frame prediction or motion compensation, based on the coding unit, a motion prediction method of a prediction unit included in the coding unit and a compensation method of motion prediction may be determined among the skip mode, the merge mode, and the AMVP mode. In addition, inter-frame prediction or motion compensation may be performed according to the mode.

[0126] The subtractor 125 may generate a residual block by using a residual between the input block and the prediction block. The residual block may be referred to as a residual signal.

[0127] The transform unit 130 may generate a transform coefficient by transforming the residual block, and may output the transform coefficient. Here, the transform coefficient may be a coefficient value generated by transforming the residual block. In the transform skip mode, the transform unit 130 may skip the transform of the residual block.

[0128] The quantized transform coefficient levels may be generated by applying quantization to the transform coefficients. Hereinafter, in the embodiments of the present invention, the quantized transform coefficient levels may be referred to as transform coefficients.

[0129] The quantization unit 140 may generate quantized transform coefficient levels by quantizing the transform coefficients according to the quantization parameters, and may output the quantized transform coefficient levels. Here, the quantization unit 140 may quantize the transform coefficients by using a quantization matrix.

[0130] According to the probability distribution, the entropy coding unit 150 may generate a bit stream by performing entropy coding on the value calculated by the quantization unit 140 or the encoding parameter value calculated in the encoding process, and the like, and may output the bit stream. The entropy coding unit 150 may entropy code information for decoding an image and information of pixels of the image. For example, the information for decoding an image may include a syntax element, and the like.

[0131] When entropy coding is applied, a symbol is represented by allocating a small number of bits to a symbol with a high probability of occurrence and a large number of bits to a symbol with a low probability of occurrence, thereby reducing the size of the bit stream of the encoded target symbol. Therefore, the compression performance of image coding can be increased by entropy coding. For entropy coding, the entropy coding unit 150 can use a coding method such as exponential Golomb, context adaptive variable length coding (CAVLC) and context adaptive binary arithmetic coding (CABAC). For example, the entropy coding unit 150 can perform entropy coding by using a variable length coding / code (VLC) table. In addition, the entropy coding unit 150 can derive a binarization method of the target symbol and a probability model of the target symbol / bin, and can perform arithmetic coding using the derived binarization method or the derived probability model afterwards.

[0132] In order to encode the transformation coefficient level, the entropy coding unit 150 can change the two-dimensional block form coefficient into a one-dimensional vector form by using a transformation coefficient scanning method. For example, by scanning the coefficients of the block using an upper right scan, the two-dimensional block form coefficient can be changed into a one-dimensional vector form. According to the size of the transformation unit and the intra-frame prediction mode, a vertical scan that scans the two-dimensional block form coefficient in the column direction and a horizontal scan that scans the two-dimensional block form coefficient in the row direction can be used instead of using an upper right scan. In other words, the scanning method can be determined among an upper right scan, a vertical direction scan, and a horizontal direction scan according to the size of the transformation unit and the intra-frame prediction mode.

[0133] The coding parameters may include not only information encoded by the encoder and then transmitted to the decoder together with the syntax elements, but also information that can be derived during the encoding or decoding process, or may refer to parameters necessary for encoding and decoding. For example, the coding parameters may include at least one value or statistic as follows: intra prediction mode, inter prediction mode, intra prediction direction, motion information, motion vector, reference image index, inter prediction direction, inter prediction indicator, reference image list, motion vector predictor, motion merge candidate, transform type, transform size, information on whether to use additional transform, filter information within the loop, information on whether there is a residual signal, quantization parameter, context model, transform coefficient, transform coefficient level, encoded block pattern, encoded block flag, image display / output order, slice information, tile information, picture type, information on whether to use motion merge mode, information on whether to use skip mode, block size, block depth, block partition information, unit size, unit partition information, etc.

[0134] The residual signal may mean the difference between the original signal and the prediction signal. Alternatively, the residual signal may be a signal generated by transforming the difference between the original signal and the prediction signal. Alternatively, the residual signal may be a signal generated by transforming and quantizing the difference between the original signal and the prediction signal. The residual block may be the residual signal, which is a block unit.

[0135] When the encoding device 100 performs encoding by using inter-frame prediction, the encoded current image can be used as a reference image for other images to be processed later. Therefore, the encoding device 100 can decode the encoded current image and store the decoded image as a reference image. In order to perform decoding, inverse quantization and inverse transformation can be performed on the encoded current image.

[0136] The quantized coefficients may be dequantized by the inverse quantization unit 160, and may be inversely transformed by the inverse transform unit 170. The dequantized and inversely transformed coefficients may be added to the prediction block by the adder 175, thereby generating a reconstructed block.

[0137] The reconstructed block may pass through the filter unit 180. The filter unit 180 may apply at least one of a deblocking filter, a sample adaptive offset (SAO), and an adaptive loop filter (ALF) to the reconstructed block or the reconstructed image. The filter unit 180 may be referred to as an in-loop filter.

[0138] The deblocking filter can remove block distortion that occurs at the boundary between blocks. In order to determine whether the deblocking filter is operated, it can be determined whether the deblocking filter is applied to the current block based on the pixels in several rows or columns included in the block. When the deblocking filter is applied to the block, a strong filter or a weak filter can be applied according to the required deblocking filter strength. In addition, when applying the deblocking filter, when performing vertical filtering and horizontal filtering, horizontal direction filtering and vertical direction filtering can be processed in parallel.

[0139] Sample adaptive offset can add the optimal offset value to the pixel value to compensate for the coding error. Sample adaptive offset can use pixels to correct the offset between the deblocking filtered image and the original image. In order to perform offset correction on a specific picture, a method of applying offset correction considering the edge information of each pixel, or a method of partitioning the pixels of the image into a predetermined number of regions, determining the region to perform offset correction, and applying offset correction to the determined region can be used.

[0140] The adaptive loop filter can filter based on the value obtained by comparing the reconstructed image and the original image. The pixels of the image can be partitioned into predetermined groups, a single filter applied to each group is determined, and different filtering can be performed in each group. Information about whether to apply the adaptive loop filter can be sent to each coding unit (CU). The shape and filter coefficients of the adaptive loop filter applicable to each block can vary. In addition, an adaptive loop filter with the same form (fixed form) can be applied regardless of the characteristics of the target block.

[0141] The reconstructed block that has passed through the filter unit 180 may be stored in the reference picture buffer 190 .

[0142] Figure 2 is a block diagram showing the configuration of an image decoding device to which an embodiment of the present invention is applied.

[0143] The decoding device 200 may be a video decoding device or an image decoding device.

[0144] refer to Figure 2 , the decoding device 200 may include an entropy decoding unit 210, an inverse quantization unit 220, an inverse transform unit 230, an intra-frame prediction unit 240, a motion compensation unit 250, an adder 255, a filter unit 260 and a reference picture buffer 270.

[0145] The decoding device 200 may receive the bitstream output from the encoding device 100. The decoding device 200 may decode the bitstream in an intra mode or an inter mode. In addition, the decoding device 200 may generate a reconstructed image through decoding, and may output the reconstructed image.

[0146] When the intra mode is used as the prediction mode used in decoding, the switch can be switched to intra. When the inter mode is used as the prediction mode used in decoding, the switch can be switched to inter.

[0147] The decoding apparatus 200 may obtain a reconstructed residual block from an input bit stream, and may generate a prediction block.

[0148] When the reconstructed residual block and the prediction block are obtained, the decoding apparatus 200 may generate a reconstructed block as a decoding target block by adding the reconstructed residual block and the prediction block. The decoding target block may be referred to as a current block.

[0149] The entropy decoding unit 210 may generate symbols by entropy decoding the bitstream according to the probability distribution. The generated symbols may include symbols in the form of quantized transform coefficient levels.

[0150] Here, the entropy decoding method may be similar to the entropy encoding method described above. For example, the entropy decoding method may be the inverse process of the entropy encoding method described above.

[0151] In order to decode the transform coefficient level, the entropy decoding unit 210 can change the one-dimensional block form coefficient into a two-dimensional vector form by using a transform coefficient scanning method. For example, by scanning the coefficient of the block using an upper right scan, the one-dimensional block form coefficient can be changed into a two-dimensional vector form. Depending on the size of the transform unit and the intra-frame prediction mode, vertical scanning and horizontal scanning can be used instead of using an upper right scan. In other words, the scanning method can be determined among upper right scanning, vertical direction scanning, and horizontal direction scanning according to the size of the transform unit and the intra-frame prediction mode.

[0152] The quantized transform coefficient levels may be dequantized by the inverse quantization unit 220 and may be inversely transformed by the inverse transform unit 230. The quantized transform coefficient levels are dequantized and inversely transformed to generate a reconstructed residual block. Here, the inverse quantization unit 220 may apply a quantization matrix to the quantized transform coefficient levels.

[0153] When the intra mode is used, the intra prediction unit 240 may generate a predicted block by performing spatial prediction using pixel values ​​of previously decoded blocks around a decoding target block.

[0154] When the inter-frame mode is used, the motion compensation unit 250 may generate a prediction block by performing motion compensation using both the motion vector and the reference image stored in the reference picture buffer 270. When the value of the motion vector is not an integer, the motion compensation unit 250 may generate a prediction block by applying an interpolation filter to a partial area in the reference image. In order to perform motion compensation, based on the coding unit, the motion prediction method of the prediction unit included in the coding unit and the compensation method of the motion prediction may be determined among the skip mode, the merge mode, and the AMVP mode. In addition, inter-frame prediction or motion compensation may be performed according to the mode. Here, the current picture reference mode may mean a prediction mode using a previously reconstructed area within the current picture with the decoding target block. The previously reconstructed area may not be adjacent to the decoding target block. In order to specify the previously reconstructed area, a fixed vector may be used for the current picture reference mode. In addition, a flag or index indicating whether the decoding target block is a block decoded in the current picture reference mode may be sent by a signal, and the flag or index may be derived by using the reference picture index of the decoding target block. The current picture for the current picture reference mode may exist at a fixed position (e.g., a position of refIdx=0 or the last position) within the reference picture list for the decoding target block. Additionally, it may be variably located within the reference picture list, and to this end, an additional reference picture index indicating the position of the current picture may be signaled.

[0155] The reconstructed residual block may be added to the prediction block by the adder 255. A block generated by adding the reconstructed residual block and the prediction block may pass through the filter unit 260. The filter unit 260 may apply at least one of a deblocking filter, a sample adaptive offset, and an adaptive loop filter to the reconstructed block or the reconstructed image. The filter unit 260 may output the reconstructed image. The reconstructed image may be stored in the reference picture buffer 270 and may be used in inter-frame prediction.

[0156] Figure 3 The diagram schematically shows the partition structure of an image when encoding and decoding the image. Figure 3 An example of partitioning a single unit into multiple units of a lower layer is schematically shown.

[0157] In order to efficiently partition an image, a coding unit (CU) may be used while encoding and decoding. A unit may refer to 1) a syntax element, and 2) a block including a sample image. For example, "a partition of a unit" may refer to "a partition of a block corresponding to the unit". Block partition information may include depth information of the unit. The depth information may indicate the number of partitions in the unit or / and the degree of the partition.

[0158] refer to Figure 3, the image 300 is partitioned in order of the largest coding unit (hereinafter referred to as LCU), and the partition structure is determined based on the LCU. Here, the LCU may be used as a coding tree unit (CTU). A single unit may include depth information based on a tree structure and may be partitioned hierarchically. Each of the partition units of the lower layer may include depth information. The depth information indicates the number of partitions in the unit or / and the degree of partitioning, and thus may include unit size information of the lower layer.

[0159] The partition structure may refer to the distribution of coding units (CUs) within the LCU 310. A CU may be a unit for efficiently encoding an image. The distribution may be determined based on whether a single CU will be partitioned into multiple (positive integers greater than 2, including 2, 4, 8, 16, etc.). The width size and height size of each partitioned CU may be half the width size and half the height size of a single CU. Alternatively, depending on the number of partition units, the width size and height size of each partitioned CU may be smaller than the width size and height size of a single CU. Similarly, a partitioned CU may be recursively partitioned into multiple CUs, each CU reducing the width size and height size by half from the partitioned CU.

[0160] Here, the partitioning of the CU may be performed recursively until a predetermined depth. The depth information may be information indicating the size of the CU. The depth information of each CU may be stored therein. For example, the depth of the LCU may be 0, and the depth of the minimum coding unit (SCU) may be a predetermined maximum depth. Here, the LCU may be a CU with a maximum CU size as described above, and the SCU may be a CU with a minimum CU size.

[0161] Whenever the LCU 310 is partitioned and its width and height sizes are reduced, the depth of the CU increases by 1. A CU on which partitioning has not been performed may have a 2N×2N size for each depth, and a CU on which partitioning has been performed may be partitioned from a CU having a 2N×2N size into a plurality of CUs, each of which has an N×N size. Whenever the depth increases by 1, the size of N is halved.

[0162] refer to Figure 3 , the size of an LCU with a minimum depth of 0 may be 64×64 pixels, and the size of an SCU with a maximum depth of 3 may be 8×8 pixels. Here, an LCU with 64×64 pixels may be represented by a depth of 0, a CU with 32×32 pixels may be represented by a depth of 1, a CU with 16×16 pixels may be represented by a depth of 2, and an SCU with 8×8 pixels may be represented by a depth of 3.

[0163] In addition, information about whether a particular CU will be partitioned can be represented by 1 bit of partition information for each CU. All CUs except SCU may contain partition information. For example, when the CU is not partitioned, the partition information may be 0. Alternatively, when the CU is partitioned, the partition information may be 1.

[0164] Figure 4 is a diagram showing a form of a prediction unit (PU) that can be included in a CU.

[0165] A CU that is no longer partitioned from the CU partitioned from the LCU may be partitioned into at least one PU. Such a process may also be referred to as partitioning.

[0166] A prediction unit (PU) may be a basic unit of prediction. A PU may be encoded and decoded in any of a skip mode, an inter prediction mode, and an intra prediction mode. A PU may be partitioned in various forms according to each mode.

[0167] like Figure 4 As shown, in skip mode, there may be no partitions within the CU. In addition, a 2N×2N mode 410 having the same size as the CU may be supported without partitions within the CU.

[0168] In the inter prediction mode, 8 partitioning types may be supported within the CU, for example, 2N×2N mode 410, 2N×2N mode 415, N×2N mode 420, N×N mode 425, 2N×nU mode 430, 2N×nD mode 435, nL×2N mode 440 and nR×2N mode 445.

[0169] Figure 5 is a diagram showing a form of a transform unit (TU) that may be included in a CU.

[0170] A transform unit (TU) may be a basic unit for transform, quantization, inverse transform, and inverse quantization processes within a CU. A TU may have a rectangular or square form. A TU may be independently determined by the size and / or form of a CU.

[0171] A CU that is no longer partitioned from a CU partitioned from an LCU may be partitioned into one or more TUs. Here, the partition structure of the TU may be a quadtree structure. For example, Figure 5As shown, depending on the quadtree structure, a single CU 510 may be partitioned once or multiple times so that the CU 510 is formed of TUs of various sizes. Alternatively, a single CU 510 may be partitioned into at least one TU based on the number of horizontal lines and / or vertical lines that partition the CU. The CU may be partitioned into TUs that are symmetrical to each other, or may be partitioned into TUs that are asymmetrical to each other. In order to partition into asymmetric TUs, information on the size and form of the TU may be signaled, or the information may be derived from information on the size and form of the CU.

[0172] While performing the transformation, the residual block may be transformed by using one of the predetermined methods. For example, the predetermined method may include a discrete cosine transform (DCT), a discrete sine transform (DST), or a Karhunen-Loève transform (KLT). In order to determine the method of transforming the residual block, the method may be determined by using at least one of the inter-frame prediction mode information of the prediction unit, the intra-frame prediction mode information of the prediction unit, or the size and form of the transform block. Alternatively, information indicating the method may be sent by a signal in some cases.

[0173] Figure 6 is a diagram showing an example of an intra prediction mode.

[0174] The number of intra prediction modes may vary according to the size of the prediction unit (PU), or may be fixed to N regardless of the size of the prediction unit (PU). Here, N may include 35 and 67, or may be a positive integer greater than 1. For example, the predetermined intra prediction modes of the encoder / decoder may include 2 non-directional modes and 65 directional modes, such as Figure 6 The two non-directional modes may include a DC mode and a planar mode.

[0175] The number of intra prediction modes may vary depending on the type of color component. For example, whether the color component is a luminance signal or a chrominance signal, the number of intra prediction modes may vary.

[0176] A PU may have a square size of NxN or 2Nx2N. NxN sizes may include 4x4, 8x8, 16x16, 32x32, 64x64, 128x128, etc. Alternatively, a PU may have an MxN size. Here, M and N may be positive integers greater than 2, and M and N may be different numbers. The unit of a PU may be the size of at least one of a CU, a PU, and a TU.

[0177] Intra-frame encoding and / or decoding may be performed by using sample values ​​or encoding parameters included in adjacent reconstruction units.

[0178] In intra prediction, a prediction block can be generated by applying a reference sampling filter to a reference pixel using at least one of the sizes of the encoding / decoding target block. The type of reference filter applied to the reference pixel may be different. For example, the reference filter may be different depending on the intra prediction mode of the encoding / decoding target block, the size / form of the encoding / decoding target block, or the position of the reference pixel. "The type of reference filter may be different" may refer to that the filter coefficients of the reference filter, the number of filter taps, the filter strength, or the number of filtering processes may be different.

[0179] In order to perform intra prediction, the intra prediction mode of the current prediction unit may be predicted by the intra prediction mode of the neighboring prediction unit adjacent to the current prediction unit. When the intra prediction mode of the current prediction unit is predicted by using the intra prediction mode information of the neighboring prediction unit, and the two modes are the same, the information that the two modes are the same may be sent by using a predetermined flag. Alternatively, when the modes are different, all prediction mode information within the encoding / decoding target block may be encoded by entropy coding.

[0180] Figure 7 is a diagram illustrating an example of inter-frame prediction processing.

[0181] Figure 7 The rectangle can refer to an image (or picture). In addition, Figure 7 The arrow may indicate the prediction direction. In other words, the image may be encoded and / or decoded according to the arrow direction. According to the encoding type, each image may be classified into an I picture (intra picture), a P picture (unidirectional prediction picture), and a B picture (bidirectional prediction picture), etc. Each picture may be encoded and decoded according to the encoding type of each picture.

[0182] When the encoding target image is an I picture, the target image itself can be intra-encoded while performing inter-frame prediction. When the encoding target image is a P picture, the target image can be encoded by using inter-frame prediction or motion compensation of a reference image in the forward direction. When the encoding target image is a B picture, the target image can be encoded by using inter-frame prediction or motion compensation of reference images in the forward and reverse directions. Alternatively, the target image can be encoded by using inter-frame prediction of reference images in the forward and reverse directions. Here, in the case of the inter-frame prediction mode, the encoder can perform inter-frame prediction or motion compensation, and the decoder can perform motion compensation in response to the encoder. Images of P pictures and B pictures encoded and / or decoded by using reference images are used for inter-frame prediction.

[0183] Hereinafter, inter prediction according to an embodiment is described in detail.

[0184] Inter prediction or motion compensation can be performed by using a reference image and motion information. In addition, inter prediction can use the skip mode described above.

[0185] The reference picture may be at least one of a previous picture of the current picture or a subsequent picture of the current picture. Here, in inter-frame prediction, a block of the current picture based on the reference picture may be predicted. Here, an area within the reference picture may be specified by using a reference picture index refIdx indicating the reference picture and a motion vector to be described later.

[0186] In inter prediction, a reference block corresponding to a current block within a reference picture may be selected. A prediction block of the current block may be generated by using the selected reference block. The current block may be a current encoding or decoding target block among blocks of the current picture.

[0187] The motion information may be derived from the inter-frame prediction process of the encoding device 100 and the decoding device 200. In addition, the derived motion information may be used for inter-frame prediction. Here, the encoding device 100 and the decoding device 200 may improve the efficiency of encoding and / or decoding by using the motion information of the reconstructed adjacent blocks and / or the motion information of the collocated blocks (col blocks). The collocated blocks may be blocks that spatially correspond to the encoding / decoding target blocks within the reconstructed collocated pictures (col pictures). The reconstructed adjacent blocks may be blocks within the current pictures, and reconstructed blocks through encoding and / or decoding. In addition, the reconstructed blocks may be blocks adjacent to the encoding / decoding target blocks, and / or blocks located at the outer corners of the encoding / decoding target blocks. Here, the blocks located at the outer corners of the encoding / decoding target blocks may be blocks adjacent in the vertical direction, and the blocks adjacent in the vertical direction are adjacent to the encoding / decoding target blocks in the horizontal direction. Alternatively, the blocks located at the outer corners of the encoding / decoding target blocks may be blocks adjacent in the horizontal direction, and the blocks adjacent in the horizontal direction are adjacent to the encoding / decoding target blocks in the vertical direction.

[0188] Each of the encoding device 100 and the decoding device 200 may determine a predetermined relative position based on a block existing at a position spatially corresponding to the current block within the collocated picture. The predetermined relative position may be located inside and / or outside the block existing at a position spatially corresponding to the current block. In addition, the encoding device 100 and the decoding device 200 may derive the collocated block based on the determined relative position. Here, the collocated picture may be at least one picture of the reference pictures included in the reference picture list.

[0189] The method of deriving motion information may vary according to the prediction mode of the encoding / decoding target block. For example, the prediction mode applied to inter-frame prediction may include an advanced motion vector predictor (AMVP) mode, a merge mode, etc. Here, the merge mode may refer to a motion merge mode.

[0190] For example, in the case of applying the Advanced Motion Vector Predictor (AMVP) mode, the encoding device 100 and the decoding device 200 may generate a prediction motion vector candidate list by using the motion vector of the restored neighboring block and / or the motion vector of the collocated block. In other words, the motion vector of the restored neighboring block and / or the motion vector of the collocated block may be used as a prediction motion vector candidate. Here, the motion vector of the collocated block may refer to a temporal motion vector candidate, and the motion vector of the restored neighboring block may refer to a spatial motion vector candidate.

[0191] The encoding device 100 may generate a bitstream, and the bitstream may include a motion vector candidate index. In other words, the encoding device 100 may entropy encode the motion vector candidate index to generate a bitstream. The motion vector candidate index may indicate an optimal prediction motion vector selected from among the prediction motion vector candidates included in the motion vector candidate list. The motion vector candidate index may be sent from the encoding device 100 to the decoding device 200 via a bitstream.

[0192] The decoding device 200 may entropy-decode the motion vector candidate index through the bitstream, and select the motion vector candidate of the decoding target block among the motion vector candidates included in the motion vector candidate list by using the entropy-decoded motion vector candidate index.

[0193] The encoding device 100 may calculate a motion vector difference (MVD) between a motion vector of a coding target block and a motion vector candidate, and may entropy encode the motion vector difference (MVD). The bitstream may include the entropy-encoded MVD. The MVD is sent to the decoding device 200 through the bitstream. Here, the decoding device 200 may entropy decode the MVD from the bitstream. The decoding device 200 may derive the motion vector of the decoding target block by summing the decoded MVD and the motion vector candidate.

[0194] The bitstream may include a reference picture index indicating a reference picture. The reference picture index may be entropy encoded and transmitted from the encoding device 100 to the decoding device 200 through the bitstream. The decoding device 200 may predict the motion vector of the current block by using the motion information of the adjacent blocks, and may derive the motion vector of the decoding target block by using the predicted motion vector and the residual of the predicted motion vector. The decoding device 200 may generate a prediction block of the decoding target block based on the derived motion vector and the reference picture index information.

[0195] As another method of deriving motion information, a merge mode may be used. The merge mode may refer to motion merging of multiple blocks. The merge mode may refer to applying the motion information of a single block to another block. When the merge mode is applied, the encoding device 100 and the decoding device 200 may generate a merge candidate list by using the motion information of the restored adjacent blocks and / or the motion information of the collocated blocks. Here, the motion information may include at least one of 1) a motion vector, 2) a reference picture index, and 3) an inter-frame prediction indicator. The prediction indicator may indicate unidirectional (LO prediction, L1 prediction) or bidirectional.

[0196] Here, the merge mode can be applied in the unit of the coding unit or the prediction unit (PU). In the case where the merge mode is performed by the CU unit or the PU unit, the encoding device 100 can generate a bit stream by entropy encoding the predetermined information and send the bit stream to the decoding device 200. The bit stream may include predetermined information. The predetermined information may include 1) a merge flag indicating whether the merge mode is used for each block partition, and 2) a merge index including information on which block among the adjacent blocks adjacent to the encoding target block is merged. For example, the adjacent blocks adjacent to the encoding target block may include the left adjacent block of the current block, the upper adjacent block of the encoding target block, the temporal adjacent blocks of the encoding target block, etc.

[0197] The merge candidate list may represent a list in which motion information is stored. The merge candidate list may be generated before the merge mode is executed. The motion information stored in the merge candidate list may be at least one of the following information: motion information of a neighboring block adjacent to the encoding / decoding target block, motion information of a collocated block corresponding to the encoding / decoding target block in a reference picture, motion information newly generated by combining motion information that existed in the merge motion candidate list in advance, and a zero merge candidate. Here, the motion information of a neighboring block adjacent to the encoding / decoding target block may refer to a spatial merge candidate, and the motion information of a collocated block corresponding to the encoding / decoding target block in a reference picture may refer to a temporal merge candidate.

[0198] In the case of skip mode, the skip mode applies motion information of neighboring blocks to the encoding / decoding target block. The skip mode may be one of other modes for inter-frame prediction. When the skip mode is used, the encoding device 100 may generate a bit stream by entropy encoding information of neighboring blocks that may be used for the encoding target block, and send the bit stream to the decoding device 200. The encoding device 100 may not send other information such as syntax information to the decoding device 200. The syntax information may include at least one of residual information of a motion vector, a coding block flag, and a transform coefficient level.

[0199] Figures 8 to 18 is a diagram illustrating a method of generating a geometrically modified image by geometrically modifying the image.

[0200] The geometric modification of an image may refer to geometrically modifying the optical information of the image. The optical information may refer to the brightness, color, or chromaticity of each point of the image. Alternatively, the optical information may refer to the pixel values ​​in a digital image. The geometric modification may refer to the parallel movement of each point within the image, the rotation of the image, the change in the size of the image, etc.

[0201] Figures 8 to 12 1 and 2 are diagrams respectively showing the geometric modification of an image according to the present invention. (x, y) of each diagram refers to a point of the original image before modification. (x', y') refers to a point corresponding to point (x, y) after modification. Here, a corresponding point refers to a point where the light information of (x, y) is moved by geometric modification.

[0202] Figure 8 is a diagram showing a transfer modification of an embodiment of a geometric modification of an image according to the present invention.

[0203] exist Figure 8 In , tx refers to the displacement of each point that has been shifted on the x-axis, and ty refers to the displacement of each point that has been shifted on the y-axis. Thus, a point (x', y') within the image is derived by adding tx and ty to the point (x, y) that was a point within the image before the modification. The shift modification can be done with Figure 8 The matrix shown is used to represent it.

[0204] Fig. 9 is a diagram showing size modification of an embodiment of geometric modification of an image according to the present invention.

[0205] exist Fig. 9 In , sx refers to the size modification factor in the x-axis direction, and sy refers to the size modification factor in the y-axis direction. The size modification factor may refer to the size ratio of the image before modification to the image after modification. When the size modification factor is equal to 1, it indicates that the size of the image before modification is equal to the size of the image after modification. When the size modification factor is greater than 1, it indicates that the size of the image after modification is enlarged. When the size modification factor is less than 1, it indicates that the size of the image after modification is reduced. The size modification factor always has a value greater than 0. Therefore, the point (x', y') in the modified image with the modified size is derived by multiplying sx and sy by the point (x, y) in the image before modification. The size modification can be used Fig. 9 The matrix shown represents .

[0206] Fig.10 is a diagram showing rotation modification of an embodiment of geometric modification of an image according to the present invention.

[0207] exist Fig.10 In , θ refers to the rotation angle of the image. Fig.10In the embodiment of the present invention, the rotation is performed with the point (0,0) of the image before modification as the center. The point (x', y') in the modified image after rotation modification can be derived by using θ and trigonometric functions. The rotation modification can be used Fig.10 The matrix representation shown in .

[0208] Fig.11 is a diagram showing an affine modification of an embodiment of a geometric modification of an image according to the present invention.

[0209] Affine modification refers to the case where transfer modification, size modification and rotation modification are performed in a composite manner. The geometric modification of the affine modification can vary depending on the order in which the transfer modification, size modification and / or rotation modification are applied to the image. Depending on the order of application between the multiple modifications that make up the affine modification and the composite of each modification, the image can be modified in the form of tilt as well as transfer modification, size modification and rotation modification.

[0210] exist Fig.11 In, M i It can be a 3×3 matrix for a translation modification, a size modification, or a rotation modification. Depending on the order of the modifications that make up the affine modification, the 3x3 matrix can be obtained by matrix multiplying each matrix used for the modification with each other. Fig.11 In the example, matrix A can be mapped to matrix M1 through matrix M n The 3x3 matrix obtained by matrix product of . The matrix A can be composed of elements a1 to a6. The matrix p is the point in the image before modification, where the modification is represented by a matrix. The matrix p' is the point in the image after modification and corresponds to the point p in the image before modification. Therefore, the affine modification can be represented by the matrix equation p'=Ap.

[0211] Fig.12 is a diagram showing a projection modification of an embodiment of a geometric modification of an image according to the present invention.

[0212] Projection modification can be an extended affine modification, where a perspective modification is added to the affine modification. When an object in three-dimensional space is projected into a two-dimensional plane, a perspective modification may occur depending on the viewing angle of the camera or observer. In perspective modification, distant objects are represented as small and nearby objects are represented as large.

[0213] exist Fig.12 In the example, the matrix H can be used for projection modification. The elements h1 to h6 constituting the matrix H can correspond to the elements constituting Fig.11 Elements a1 to a6 of the matrix A of the affine modification. Therefore, the projection modification may include the affine modification. Elements h7 and h8 constituting the matrix H may be elements related to the perspective modification.

[0214] The geometric modification of an image is a method of geometrically modifying an image into a specific form. Points in the geometrically modified image corresponding to points in the image before the geometric modification can be calculated by the geometric modification defined in a matrix. In contrast, homography refers to a method of inversely deriving a mutual geometric modification matrix from two images that have points that correspond to each other.

[0215] Fig.13 is a diagram showing an example of a method of implementing homography according to the present invention.

[0216] Homography can derive a geometrically modified relationship between two images based on identifying two points that are located in the two images and correspond to each other. For this purpose, feature point matching can be used. Feature points of an image refer to points within the image that have descriptive characteristics.

[0217] In steps S1301 and S1302, the homography implementation method can extract feature points from the original image and the geometrically modified image. The feature points of the image can be extracted differently according to the extraction method or according to the purpose of use. Points where the brightness value in the image changes significantly, the center points of an area with a specific shape, or the outer corner points of an object in the image can be used as feature points. Feature points can be extracted by using algorithms such as scale-invariant feature transform (SIFT), speeded up robust features (SURF), blob detection, etc.

[0218] In step S1303, the homography implementation method can match feature points based on feature points extracted from the original image and the geometrically modified image. In detail, each extracted feature point is descriptive, and feature points between the two images can be matched by finding points with similar descriptive information. The matched feature points can be used as points where the original image and the geometrically modified image correspond to each other.

[0219] However, feature point matching may not match points that actually correspond to each other. Therefore, in step S1304, valid feature points among the derived feature points can be selected. The method of selecting valid feature points can vary according to the calculation algorithm. For example, the following methods can be used, such as a method of excluding feature points that do not meet the baseline based on description information, a method of excluding feature points with very low similarity through the distribution of matching results, or a method using a random sample consensus (RANSAC) algorithm. The homography implementation method can selectively perform step S1304 based on the matching results of the feature points. In other words, step S1304 may not be performed depending on the situation. Alternatively, steps S1303 and S1304 can be merged. Alternatively, the homography implementation method can perform a matching process for valid feature points without performing steps S1303 and S1304.

[0220] In step S1305, the homography implementation method may derive a relationship between the original image and the geometrically modified image by using the selected valid points. In step S1306, the homography implementation method may derive a geometric matrix by using the derived formula. Alternatively, the homography implementation method may not perform step S1306 and output information of the derived formula obtained in step S1305 in addition to the geometrically modified matrix in a different form.

[0221] Fig.14 is an exemplary method of deriving a relationship between two corresponding points in two images according to the present invention.

[0222] The geometric modification of the image can be performed using a 3×3 matrix H. Therefore, the simultaneous equations including the elements h1 to h9 of the matrix H as unknown quantities can be derived from the matrix formula p'=Hp. Here, p means a point within the original image, and p' represents a point within the geometrically modified image corresponding to the point p. By dividing all elements of the matrix H by h9, the equations can be simply calculated by fixing H9 to 1. In addition, the number of unknown quantities can be reduced from 9 to 8.

[0223] Fig.14 Elements k1 to k8 of correspond to the values ​​of h1 to h8 divided by h9. The same geometry modification can be performed in a geometry matrix where h9 is changed to 1 and h1 to h8 are changed to k1 to k8, respectively. Thus, eight unknown values ​​may need to be calculated. Fig.14 In , the final formula for a pair of points that match each other in the form of a single point can be expressed in two forms of x' and y'. At least 4 pairs of points that match each other may be required because there are 8 unknown values. However, as mentioned above, a pair of points may not match each other. Or, a pair of points may be incorrectly matched. This error can occur even if valid feature points are selected. This error can be reduced by using many pairs of points that match each other while calculating the geometry modification matrix. Therefore, taking these characteristics into account, the number of pairs of points to be used can be determined.

[0224] Fig.15 is a diagram illustrating a method of generating a geometry modification image based on a geometry modification matrix and an original image according to the present invention.

[0225] like Fig.15 As shown, by using the light information of points within the original image, the generation of the geometrically modified image can correspond to the generation of light information of corresponding points within the geometrically modified image. Fig.15The (x0, y0), (x1, y1) and (x2, y2) in the geometric modification image refer to different points in the original image. In addition, (x'0, y'0), (x'1, y'1) and (x'2, y'2) are points corresponding to (x0, y0), (x1, y1) and (x2, y2) in the geometric modification image, respectively. Function f calculates the corresponding x' coordinate of the x-axis in the geometric modification image by using the point (x, y) in the original image and the additional information α for geometric modification. Function g calculates the corresponding y' coordinate of the y-axis in the geometric modification image by using the point (x, y) in the original image and the additional information β for geometric modification. When (x, y), (x', y'), function f and function g are expressed in a matrix formula, the matrix H can refer to the geometric modification method. Therefore, points corresponding to each other in the original image and the geometric modification image can be found by using the matrix H.

[0226] Fig.15 The geometric modification method may be problematic in discretely sampled image signals because light information is included only in points with integer coordinates of the discrete image signal. Therefore, when a point within the geometrically corrected image and corresponding to a point within the original image has a real coordinate, light information of the closest integer coordinate is assigned to the point within the geometrically corrected image. Therefore, light information may be overlapped to a portion of a point with real coordinates within the geometrically corrected image, or light information may not be assigned. In this case, inverse mapping may be used.

[0227] Fig.16 is a diagram illustrating a method of generating a geometry-modified image by using inverse mapping according to the present invention.

[0228] Fig.16 The dotted rectangular area refers to the actually observed area. The points within the original image corresponding to each point within the dotted rectangular area can be derived. Therefore, the light information of the original image can be assigned to all points within the geometrically modified image. However, the point (x3, y3) corresponding to (x'3, y'3) may be located outside the original image. In this case, the light information of the original image may not be assigned to the point (x'3, y'3). Among the points to which the light information of the original image is not assigned, the adjacent light information of the original image can be assigned. In other words, the light information of the closest point in the original image (for example, (x4, y4)) can be assigned.

[0229] Fig.17 is a diagram illustrating a method of generating a geometry modification image based on a geometry modification matrix and an original image according to the present invention, wherein the geometry modification matrix may correspond to geometry modification information.

[0230] In step S1701, the generation method may receive an input original image, a geometry modification matrix and / or information about a current point of the geometry modified image. The generation method may calculate a point of the original image corresponding to the current point of the geometry modified image by using the original image and the geometry modification matrix. The calculated corresponding point of the original image may be a real corresponding point with real coordinates.

[0231] In step S1702, the generation method may determine whether the calculated corresponding point is located inside the original image.

[0232] In step S1702, when the calculated corresponding point is not located inside the original image, in step S1703, the generation method may change the point closest to the calculated corresponding point in the original image with the corresponding point.

[0233] In step S1702, when the calculated corresponding point is located inside the original image, the generation method may execute step S1704. When the calculated corresponding point is changed in step 1703, the generation method may execute step S1704.

[0234] In step S1704, when the corresponding point has real coordinates, the generation method may identify the closest point with integer coordinates. When the corresponding point has integer coordinates, the generation method may skip steps S1704 and S1705 and perform step S1706.

[0235] In step S1705, the generation method can generate light information of a point having real coordinates by interpolating light information (e.g., pixel values) of an identification point having integer coordinates. As an interpolation method, Lanczos interpolation, S-Spline interpolation, or bicubic interpolation can be used.

[0236] In step S1706, the generation method may check whether all points within the geometrically modified image have completed their geometrical modification.Then, the generation method may finally output the generated geometrically modified image.

[0237] When it is determined in step S1706 that the geometry modification is not completed, in step S1707, the generation method may change the current point of the geometry modified image to another point, and may repeat steps S1701 to S1706.

[0238] Fig.18 is a reference diagram showing an embodiment according to the present invention Fig.17 A diagram explaining bilinear interpolation among various interpolation methods.

[0239] exist Fig.18 In the example, the real coordinates (x, y) can correspond to Fig.17The four points (i, j), (i, j+1), (i+1, j) and (i+1, j+1) adjacent to the coordinates (x, y) can correspond to Fig.17 The closest point with integer coordinates mentioned in step S1704 of . I(x, y) may refer to light information of point (x, y), such as brightness. a refers to the x-axis distance between i and x, and b refers to the y-axis distance between j and y. 1-a refers to the x-axis distance between i+1 and x, and 1-b refers to the y-axis distance between j+1 and y. The light information of point (x, y) may be calculated from the light information of points (i, j), (i, j+1), (i+1, j), and (i+1, j+1) by using the ratio of a to 1-a in the x-axis and the ratio of b to 1-b in the y-axis.

[0240] When the inter-frame prediction unit of the video encoder performs motion prediction, the inter-frame prediction unit can predict the encoding target area (current area or current block) within the encoding target picture (current picture) by referring to a reference picture. Here, when the time interval between the reference picture and the encoding target picture is large, or when a global motion such as rotation, enlargement, reduction, or change of the viewing angle of the target has occurred between the two images, the pixel similarity between the two images is reduced. Therefore, the prediction accuracy may be reduced and the coding efficiency may be reduced. In this case, the encoder can calculate the change in motion between the encoding target picture and the reference picture, and geometrically modify the reference picture so that the reference picture has a form similar to the encoding target picture. The reference picture can be geometrically modified in units of frames, slices and / or blocks. The picture generated by geometrically modifying the reference picture can be defined as a geometrically modified picture. Motion prediction accuracy is improved by referring to the geometrically modified picture instead of the reference picture.

[0241] According to the present invention, the reference picture and / or the geometrically modified picture is not limited to the frame size or picture size of one frame of the configured video. In the present invention, a partial area of ​​a reference picture having a picture (frame) size defined in a parameter set may refer to a reference picture. A geometrically modified picture generated by modifying a reference picture or a partial area of ​​a reference picture may have a size corresponding to the size of the image before being modified. In other words, when the image before being modified is a reference picture having a picture (frame) size, its geometrically modified picture may have a picture (frame) size. When the image before being modified is a partial area of ​​a reference picture having a picture (frame) size, the geometrically modified picture may have a size corresponding to the partial area of ​​the reference picture.

[0242] When the video encoder has performed encoding by referring to the geometry modification picture, the video decoder may receive information required for the geometry modification. The information required for the geometry modification may refer to the geometry modification information. The video decoder may generate a geometry modification picture from the reference picture based on the received geometry modification information. The video decoder may perform inter-frame prediction by referring to the generated geometry modification picture.

[0243] Fig.19 is a diagram illustrating motion prediction, in which a video encoder generates geometry modification information and performs motion prediction by using a reference picture and a geometry modification picture.

[0244] The geometric modification picture generator of the video encoder may generate geometric modification information from a reference picture. The geometric modification picture generator of the video encoder may generate a geometric modification picture by using the geometric modification information. The video encoder may store the generated geometric modification picture so that the geometric modification picture predictor uses the geometric modification picture as a reference signal. The geometric modification picture may be stored in at least one of a reconstructed picture buffer DPB, a reference picture list, and a geometric modification picture buffer of the video encoder. The geometric modification picture predictor of the video encoder may perform inter-frame prediction using the generated geometric modification picture and / or the reference picture. An encoder including a geometric modification picture generator, a geometric modification picture buffer, and / or a geometric modification picture predictor will be described later.

[0245] like Fig.19 As shown, geometric modification information can be generated based on the encoding target picture and the reference picture. The reference picture can be selected from a reference picture list configured with at least one decoded reconstructed picture. The reference picture can be a reference picture with global motion. The geometric modification information can be generated by reflecting the global motion between the encoding target picture and the reference picture with global motion. The geometric modification information can be configured or generated in units of the entire image or a part of the image (such as a frame, a slice, a block, etc.). When generating the geometric modification information, the whole or a part of the global motion can be reflected. The global motion can be a motion related to the geometric modification of the above-mentioned image, such as transfer, enlargement, reduction, rotation, etc. A geometric modification picture can be generated based on the reference picture and the generated geometric modification information. The geometric modification picture can be a geometric modification picture reflecting the global motion.

[0246] As a method of configuring geometric modification information related to the transfer, enlargement, reduction, or rotation of pixels between the encoding target picture and the reference picture, a reference Fig.13 Homography can provide information about geometric modifications that reflect rotation, scaling, and translation between two images to explain the 2D geometric modifications of planar objects.

[0247] By using geometric modification information to rotate, enlarge, reduce or translate the image, Fig.19 The image within the reference picture can be geometrically modified to Fig.19 The geometry of the image within the picture is modified. Fig.19 Reference pictures and Fig.19 Geometric Modification Images can be geometrically modified to have rectangles of different shapes. Fig.19 Any coordinate (a, b) in the reference image is obtained by Fig.14 The relation derived in becomes (a', b'), and (a, b) can correspond to Fig.19 The geometry of the modified image is at coordinates (a', b').

[0248] Therefore, the video encoder can generate an image similar to the encoding target picture from the reference picture by calculating the geometric modification information between the encoding target picture and the reference picture. Fig.19 The geometric modification of the image area A and Fig.19 The similarity between region A and region B within the encoding target picture is very high. In other words, the similarity of pixel values ​​between region A and region B is very high, so the prediction accuracy of the encoder can be improved by referring to the geometrically modified picture while performing motion prediction.

[0249] As described above, when inter-frame prediction is performed in a video encoder, the video encoder may refer to a reference picture configured with an already decoded picture. However, since the reference picture is not the same as the encoding target picture, there may be a change in pixel values ​​during the time interval between the reference picture and the encoding target picture. Therefore, the video encoder preferably refers to a geometric modification picture generated based on geometric modification information reflecting the change in pixel values ​​between the reference picture and the encoding target picture. The change in pixel value may include, for example, something caused by global motion. While performing motion prediction, an optimal prediction signal may be generated by referring to the reference picture and the geometric modification picture.

[0250] In detail, since the geometrically modified picture is an image that reflects the change in pixel values ​​between the reference picture and the encoding target picture, the pixel distribution similarity between the encoding target picture and the geometrically modified picture is very high. However, when new pixels that did not exist in the previous image appear in the image, such as a new object, or the error of the geometric modification information is large, errors or noise may appear in the geometrically modified picture. Here, the motion prediction with reference to the geometrically modified picture may be degraded. Therefore, when performing motion prediction, the first motion prediction is performed by referring to both the reference picture and the geometrically modified picture. Then, the information with high prediction accuracy can be updated to the best prediction information. Therefore, the reduction in encoding performance caused by noise or errors in the geometrically modified picture can be prevented. The video encoder can encode the motion prediction information with the best encoding efficiency and send the encoded information through the bitstream. Here, the video encoder may not encode the geometric modification information of the area that is not referenced by the geometrically modified picture. In other words, the encoding efficiency can be improved by encoding only the necessary geometric modification information.

[0251] Fig. 20 is a block diagram showing the configuration of an image encoding device to which another embodiment of the present invention is applied.

[0252] Fig. 20 The encoding device shown may include a geometry modification picture generation unit 2010, a geometry modification picture predictor 2015, an extended intra-frame prediction unit 2020, a subtractor 2025, a transform unit 2030, a quantization unit 2040, an entropy encoding unit 2050, an inverse quantization unit 2060, an inverse transform unit 2070, an adder 2075, a deblocking filter unit 2080 and a sampling adaptive offset unit 2090.

[0253] The geometry modification picture generation unit 2010 may generate a geometry modification picture 2012 by calculating geometry modification information reflecting a change in pixel values ​​between the encoding target picture 2011 and a reference picture of a reference picture list stored in the reconstructed picture buffer 2013. The generated geometry modification picture 2012 may be stored in a geometry modification picture buffer 2016.

[0254] The geometry modified picture predictor 2015 may include a geometry modified picture buffer 2016 and an inter prediction unit 2017. The geometry modified picture buffer 2016 may store the geometry modified picture generated in the geometry modified picture generation unit 2010. The inter prediction unit 2017 may perform motion prediction by using a reference picture of a reference picture list stored in the reconstructed picture buffer 2013 as a reference signal. When the geometry modified picture is referenced while performing motion prediction, geometry modification information for generating the geometry modified picture may be transmitted to the entropy encoding unit 2050 and encoded in the entropy encoding unit 2050.

[0255] The geometry modification picture generation unit 2010 may perform reconfiguration of geometry modification information which will be described later. The reconfiguration of geometry modification information may be performed when generating a geometry modification picture.

[0256] The extended intra prediction unit 2020 may perform extended intra prediction by referring to already encoded / decoded signals of the current picture and the geometry modification picture. The extended intra prediction unit 2020 will be described later.

[0257] Fig.21 It shows that according to Fig. 20 A diagram showing the configuration and operation of a geometrically modified picture generation unit 2010 of an example encoding device is shown.

[0258] The geometry modification picture generation unit 2010 may generate a geometry modification picture 2108, which may be used as a reference signal in the geometry modification picture predictor 21015. The geometry modification information configurator 2105 may configure the geometry modification information 2106 by receiving the reference picture 2103 of the reference picture list 2102 configured by the reconstructed picture buffer 2101, and the encoding target picture 2104. The picture geometry modification calculator 2107 may generate the geometry modification picture 2108 by modifying the reference picture 2103 using the geometry modification information 2106.

[0259] The geometry modification information configurator 2105 may calculate the change in pixel values ​​between the reference picture 2103 and the encoding target picture 2104, and specifically, calculate global motion information. The geometry modification information configurator 2105 may find matching feature points between the two images, and calculate motion information by calculating the shift, rotation, and / or size change between the matching feature points. The geometry modification information configurator 2105 may configure and output geometry modification information capable of generating a geometry modification picture based on the calculated global motion information.

[0260] The picture geometry modification calculator 2107 may receive the reference picture 2103 and the geometry modification information 2106 to generate a geometry modified picture 2108. The picture geometry modification calculator 2107 may generate the geometry modified picture 2108 from the reference picture 2103 by using the geometry modification information 2106. The generated geometry modified picture 2108 may be stored in the geometry modified picture buffer 2016 and may be referenced by the inter prediction unit 2017 while performing inter prediction.

[0261] Fig. 22 It shows that according to Fig. 20 A diagram showing the configuration and operation of a geometrically modified picture predictor 2015 of an example encoding device is shown.

[0262] The reference picture 2202 of the reference picture list 2201 and the geometry modified picture 2204 of the geometry modified picture buffer 2203 may be input to the inter predictor 2205 and used as reference signals for motion prediction.

[0263] While performing motion prediction, the inter-frame predictor 2205 may derive optimal motion prediction information 2206 by referring to both the reference picture 2202 and the geometry modified picture 2204. The optimal motion prediction information 2206 may refer to motion prediction information having high prediction accuracy and optimal encoding efficiency. When the optimal motion prediction information 2206 is derived by referring to the geometry modified picture 2204, the geometry modification information may be encoded and transmitted through a bitstream.

[0264] refer to Figures 20 to 22 The configuration of the encoding device shown is merely one of various embodiments of the present invention and is not limited thereto. Figures 20 to 22 Some configurations of the encoding device shown may be combined with other configurations or omitted. Alternatively, other configurations may be added. In addition, a portion of the multiple configurations included in the geometry modification picture generation unit 2010 and the geometry modification picture predictor 2015 may be configured independently from the geometry modification picture generation unit 2010 and the geometry modification picture predictor 2015. Alternatively, it may be included in a sub-configuration of another configuration, or combined with another configuration.

[0265] Fig.23 is a flowchart illustrating motion prediction according to an embodiment of the present invention.

[0266] In steps S2301 and S2302, a coding target picture and a reference picture may be specified. The reference picture may be selected from a reference picture list.

[0267] In step S2303, the geometry modification information may be generated based on the encoding target picture and the reference picture. The geometry modification information may be generated by using the above-mentioned method.

[0268] In step S2304, a geometry modification picture may be generated based on the generated geometry modification information. The generated geometry modification picture may be stored in a geometry modification picture buffer.

[0269] In step S2305, motion prediction may be performed by referring to the reference picture and / or the geometry modified picture.

[0270] In step S2306, the best prediction information may be stored and updated based on the reference signal having the best coding efficiency. The rate distortion cost (RD Cost) may be used as an index for determining the best coding efficiency.

[0271] In step S2307, when not all reference pictures are applied to steps S2301 to S2306, steps S2302 to S2306 are repeated. In step S2308, when all reference pictures are applied, the best motion prediction information and / or geometric modification information finally determined may be encoded. When the geometric modification picture is used while performing motion prediction, only the geometric modification information may be encoded and transmitted.

[0272] Fig.24 is a flowchart illustrating a method for generating a geometry-modified picture according to an embodiment of the present invention.

[0273] In step S2401 , the generating method may receive input of an encoding target picture and a reference picture selected from a reference picture list.

[0274] In step S2402, the generation method may extract feature points from the two images and match the extracted feature points. The extracted feature points may be used to calculate the change in pixel values ​​between the two images, specifically to calculate global motion information. As described above, a feature point may refer to a pixel that can be distinguished from adjacent pixels. In step S2403, geometric modification information reflecting the rotation, enlargement, reduction, and / or transfer of feature points within the two images may be calculated by matching the extracted feature points. The geometric modification information may be calculated using various algorithms, for example, using Fig.13 The homography of .

[0275] In step S2404, the generating method may generate a geometry modified picture. By applying the geometry modification information to each pixel of the reference picture, the geometry modified picture may be generated by deriving each pixel in the geometry modified picture corresponding to each pixel of the reference picture.

[0276] In step S2405, the generating method may store the generated geometrically modified picture. For example, the generated geometrically modified picture may be stored in a geometrically modified picture buffer. However, it is not limited thereto, and the geometrically modified picture may be stored in a reconstructed picture buffer or in a reference picture list.

[0277] Fig.25 is a flowchart illustrating an inter-frame prediction method according to an embodiment of the present invention.

[0278] In steps S2501 and S2502, the inter-frame prediction method performs motion prediction by referring to a reference picture and motion prediction by referring to a geometrically modified picture. The order of motion prediction is not limited to this. In other words, motion prediction by referring to a geometrically modified picture may be performed first. Alternatively, the two motion predictions may be performed simultaneously. The reference picture may be selected from a reference picture list.

[0279] In step S2503, the inter prediction method may determine which of the motion prediction with reference to the reference picture and the motion prediction with reference to the geometry modified picture is more efficient.

[0280] When the motion prediction of the reference picture is determined in step S2503, in step S2504, the inter prediction method may store or update the reference picture information as the optimal prediction information.

[0281] When the motion prediction of the reference geometry modified picture is determined in step S2503, in step S2505, the inter prediction method may store or update the geometry modified picture information as the optimal prediction information.

[0282] In step S2506, when the geometric modification picture information is finally encoded as the best prediction information, the inter-frame prediction method can encode the geometric modification information.

[0283] refer to Fig.26 and Fig. 27 , a method for efficiently encoding geometry modification information is described.

[0284] Geometry modification information can be configured with multiple factors, such as Fig.14. In addition, these factors may be real numbers. When a large number of bits are used to encode a matrix configured with real factors, the coding efficiency may be reduced. Various methods can be applied to reduce the amount of bits while sending geometry modification information. For example, real factors can be approximated to integer factors. Alternatively, the geometry modification information can be simplified by reducing a portion of the generated geometry modification information. Alternatively, the geometry modification information to be encoded can be predicted from pre-used geometry modification information, and only the residual values ​​between the geometry modification information can be transmitted. In addition to the described method, various methods can be used to reduce the amount of bits. Alternatively, one or more of the above methods can be applied together.

[0285] Fig.26 is a flowchart illustrating a method for encoding geometry modification information according to an embodiment of the present invention.

[0286] In step S2601, the encoding method may receive input of encoding target geometry modification information.

[0287] In step S2602, the encoding method may simplify the geometry modification information. In step S2603, the encoding method may predict the geometry modification information. Steps S2602 and S2603 may be performed in reverse. Alternatively, one of steps S2602 and S2603 may be selectively performed.

[0288] In step S2602, the geometry modification information may be simplified by approximating the real number factors constituting the geometry modification information as integer factors. The number of bits expressing these factors may be reduced by approximation. To reduce the number of bits, various operations may be used, such as rounding, raising, lowering, discarding, cutting based on a predetermined number of bits. As an example of approximation, a real number may be converted to an integer.

[0289] In step S2603, the encoding method may predict the geometric modification information based on the previously used geometric modification information. The encoding method may send a residual value between the predicted and previously used information. The previously used geometric modification information may refer to the geometric modification information used just before, or the geometric modification information of the previously encoded picture. When multiple geometric modification information can be used to predict the geometric modification information, the encoding method may additionally encode information indicating the referenced geometric modification information. Here, the decoder may determine the referenced geometric modification information based on the additionally sent information, and decode the geometric modification information based on the information. When only specific geometric modification information (for example, the geometric modification information used just before) is used as the geometric modification information, the encoding method may not additionally send information indicating the referenced geometric modification information.

[0290] In step S2604, the encoding method may encode information required to reconstruct the geometry modification information.

[0291] Fig. 27 is a diagram showing various examples for modifying geometry modification information to reduce the amount of bits used to encode the geometry modification information.

[0292] Fig. 27 (1) is an example of converting real number factors into integer factors.

[0293] Fig. 27 (2) is an example of simplifying the geometry modification information by removing a part of it. Among the removed factors, 0.1 and 0.7 as the left factors are used for reference Fig.12 Described factors for projection modification. Factors used for projection modification statistically appear at small values ​​close to zero. Factors close to zero have a small effect when generating geometrically modified pictures. Therefore, even if omitted, it will not adversely affect the accuracy of the geometrically modified pictures. In addition, among the three removed factors, the rightmost factor 1 may not be sent to the decoder because it is always fixed to 1 while generating geometric modification information.

[0294] Fig. 27 (3) is an example of configuring geometry modification information B by multiplying coefficient 2 by geometry modification information A. Geometry modification information A and coefficient 2 may be transmitted to the decoder instead of transmitting geometry modification information A and B. Modification information B may be reconstructed by transmitting geometry modification information A and coefficient 2.

[0295] Fig. 27 (4) is an example of configuring geometry modification information B by using geometry modification information A as reference geometry modification information. Geometry modification information A and residual geometry modification information B d is encoded. d It may correspond to the difference between the geometry modification information A and B. The decoder may add the geometry modification information A and B. d To reconstruct the geometric modification information B.

[0296] Fig.28 is a diagram illustrating a motion compensation method in which a decoder generates a geometry modified picture from a reference picture by using geometry modification information and performs motion compensation by using the reference picture and the geometry modified picture.

[0297] The video decoder according to the present invention can receive the geometry modification information generated by the video encoder through a bitstream.

[0298] The video decoder may generate a geometry modified picture by applying the geometry modification information to a reference picture selected from a reference picture list. The geometry modified picture may be generated by a geometry modified picture generator of the decoder to be described later. The unit of the geometry modified picture generated by the geometry modified picture generator may vary according to a unit determined when encoding the geometry modified picture, and the unit may be a frame, a slice, a block, etc.

[0299] The method for generating a geometrically modified picture by applying the geometric modification information to the reference picture may correspond to the method for generating a geometrically modified picture in the encoder as described above. However, unlike the video encoder, the decoder receives the geometric modification information through the bitstream. Therefore, the configuration complexity of the video decoder is not significantly increased. The generated geometrically modified picture can be stored in a geometrically modified picture buffer to be described later.

[0300] The video decoder may refer to both the reference picture and the geometrically modified picture for inter-frame prediction of the decoding target picture. Inter-frame prediction with reference to the reference picture and / or the geometrically modified picture may be performed by an inter-frame prediction unit of the decoder to be described later. The inter-frame prediction unit of the decoder may perform inter-frame prediction by using the reference picture of the reconstructed picture buffer DPB and / or the geometrically modified picture of the geometrically modified picture buffer as a reference signal.

[0301] Information on whether to use a reference picture or a geometrically modified picture for inter prediction of a decoding target picture or decoding target region may be signaled through a bitstream. Such information may be signaled through an additional syntax element. Alternatively, such information may be signaled through the presence of geometric modification information.

[0302] Based on the information indicating that the decoding target picture or the decoding target area is predicted by referring to the geometry modification picture, the video decoder may receive the geometry modification information, generate the geometry modification picture, and / or perform inter prediction by referring to the geometry modification picture.

[0303] Based on the information indicating that the decoding target picture or the decoding target area is predicted by referring to the reference picture, the video decoder may not receive the geometry modification information, not generate the geometry modification picture, and / or not perform inter prediction by referring to the geometry modification picture.

[0304] Fig.29 is a block diagram showing the configuration of a decoding device to which another embodiment of the present invention is applied.

[0305] Fig.29The decoder shown may include an entropy decoding unit 2910, an inverse quantization unit 2920, an inverse transform unit 2930, a subtractor 2935, a filter unit 2940, an extended intra prediction unit 2950, ​​a geometry modified picture predictor 2960, and a geometry modified picture generator 2970. The decoder may output a decoded picture 2980 by receiving the bitstream 2900.

[0306] The geometry modification picture generator 2970 may generate a geometry modification picture 2972 ​​by using the geometry modification information extracted from the bitstream 2900 and the reference pictures of the reference picture list stored in the reconstructed picture buffer 2971 and entropy decoded.

[0307] The geometry modified picture predictor 2960 may be configured with a geometry modified picture buffer 2961 for storing the geometry modified picture 2972 ​​and an inter prediction unit 2962 .

[0308] The geometry modified picture 2972 ​​generated in the geometry modified picture generator 2970 may be stored in the geometry modified picture buffer 2961. The geometry modified picture 2972 ​​stored in the geometry modified picture buffer 2961 may be used as a reference signal in the inter prediction unit 2962.

[0309] The inter prediction unit 2962 may reconstruct a decoding target picture by using a reference picture and / or a geometrically modified picture as a reference signal for motion prediction based on the information transmitted from the encoder.

[0310] Fig.30 It is shown Fig.29 A diagram illustrating the operation and configuration of a geometrically modified picture generator 2970 of a decoding device is shown.

[0311] The geometry modification calculator 3005 of the geometry modification picture generator 2970 may generate a geometry modification picture 3006 by receiving the geometry modification information 3004 and the reference picture 3003. The reference picture 3003 may be selected from the reference picture list 3002 configured from the reconstructed picture buffer 3001. Information for selecting the reference picture 3003 may be included in the bitstream and transmitted through the bitstream. Information for selecting the reference picture 3003 may be transmitted as an additional syntax element, or may be explicitly or implicitly included in the geometry modification information 3004.

[0312] As described above, the generated geometry modified picture 3006 may be output to the geometry modified picture buffer 2961 .

[0313] Fig.31 It is shown Fig.29 A diagram illustrating the operation and configuration of a geometrically modified picture predictor 2960 of a decoding device is shown.

[0314] The inter-frame predictor 3105 of the geometric modification picture predictor 2960 can perform inter-frame prediction by referring to the reference picture 3102 and / or the geometric modification picture 3104, and output a prediction signal 3106. Information on whether to use the reference picture 3102 or the geometric modification picture 3104 or both for inter-frame prediction of the decoding target picture or the decoding target area can be signaled through the bitstream. As described above, such information signaling can be performed through additional syntax elements. Alternatively, such information signaling can be performed through the presence of geometric modification information.

[0315] The reference picture 3102 may be selected from the reference picture list 3101. The geometry modified picture 3104 may be selected from the geometry modified picture buffer 3103. The geometry modified picture 3104 may be configured with a portion required for inter prediction.

[0316] refer to Figures 29 to 31 The configuration of the decoding device shown is merely one of various embodiments of the present invention, but is not limited thereto. Figures 29 to 31 Some configurations of the encoding device shown may be combined with other configurations or omitted. Alternatively, other configurations may be added. In addition, a portion of the multiple configurations included in the geometry modified picture generator 2970 and the geometry modified picture predictor 2960 may be configured independently from the geometry modified picture generator 2970 and the geometry modified picture predictor 2960. Alternatively, it may be included in a sub-configuration of another configuration, or combined with another configuration.

[0317] Fig.32 is a flowchart illustrating motion compensation of a decoder according to an embodiment of the present invention.

[0318] In step S3201, the decoder may obtain motion compensation related information by parsing the bitstream. The motion compensation related information may include at least one of reference picture information and geometric modification information. The reference picture information may be information for specifying a reference picture within a reference picture included in a reference picture list. Alternatively, the reference picture information may be information indicating whether a reference image or a geometrically modified picture is used while performing motion compensation. Alternatively, the reference picture information may be information indicating whether both a reference image and a geometrically modified picture are used while performing motion compensation. Alternatively, the reference picture information may be information including at least two of the above information. The reference picture information may be signaled by a single syntax element or by multiple single syntax elements. The reference picture information may be signaled by a method explicitly or implicitly defined by an encoder and a decoder. Whether a reference picture or a geometrically modified picture is referenced during inter-frame prediction may be signaled by the presence of geometric modification information.

[0319] In step S3202, based on the motion compensation related information, the decoder may determine whether to refer to the geometrically modified picture when performing motion compensation. As described above, the decoder may make a decision based on the presence of geometric modification information and / or reference picture information. When the decoder uses the presence of geometric modification information, for example, when the geometric modification information is received, the decoder may determine to perform motion prediction by referring to the geometrically modified picture.

[0320] If the result in step S3202 is "yes", then in step S3203, the decoder may generate a geometry modification picture. The geometry modification picture may be generated based on a reference picture selected from a reference picture list and geometry modification information parsed from a bitstream. Here, the decoder may generate a portion of the geometry modification picture required for motion compensation.

[0321] In step S3204, the decoder may perform motion compensation by referring to the generated geometry modified picture.

[0322] Alternatively, if the result in step S3202 is "No", in step S3205, the decoder may perform motion compensation by referring to a reference picture selected from the reference picture list.

[0323] In step S3203, refer to Fig.32 The explained motion compensation generates a geometry modified picture based on whether the geometry modified picture is used (S3202). However, it is not limited thereto, and when the geometry modification information is received, the decoder can generate a geometry modified picture regardless of its reference.

[0324] As described above, a geometry modification picture can be generated from a reference picture by using geometry modification information, thereby improving inter-frame prediction accuracy by referring to the reference picture and / or the geometry modification picture while performing inter-frame prediction.

[0325] Prediction using geometrically modified pictures is not limited to inter-frame prediction. For example, intra-frame prediction can be performed by using geometrically modified pictures. Figure 33 to Figure 37 Extended intra prediction using geometrically modified pictures according to an embodiment of the present invention is described.

[0326] Fig.33 is a conceptual diagram showing extended intra prediction according to an embodiment of the present invention. Fig.33 In , the current block may refer to a decoding target block.

[0327] like Fig.33As shown in (1), an area that is a decoded area of ​​the current block can be referenced for intra-frame prediction of the current block. In other words, the left area, the upper area, the upper left area and / or the upper right area of ​​the current block can be referenced for intra-frame prediction. However, since the right area, the lower area, the lower left area and / or the lower right area of ​​the current block have not been decoded, these areas cannot be referenced for intra-frame prediction.

[0328] like Fig.33 As shown in (2), for the right area, lower area, lower left area and / or lower right area of ​​the current block that cannot be referenced, a signal of an already decoded picture corresponding to the current picture, or a signal corresponding to a geometrically modified picture generated by geometrically modifying the already decoded picture, can be referenced as a reference signal for intra prediction. The geometrically modified picture can be a geometrically modified picture generated by reconfiguring a reference picture of the current picture using geometric modification information, so that the reconfigured reference picture can be similar to the current picture.

[0329] like Fig.33 (3) By using Fig.33 (1) and all reference signals of 33 (2) can obtain all reference signals of the area surrounding the current block. Therefore, intra prediction of the current block can be performed in all directions (left, right, upper left, upper right, right, bottom, lower left and / or lower right area of ​​the current block). Therefore, the efficiency of intra prediction can be improved.

[0330] Fig.34 is a diagram illustrating the operation and configuration of an extended intra prediction unit according to an embodiment of the present invention.

[0331] Fig.34 The extended intra prediction unit may correspond to Fig. 20 The extended intra prediction unit of the encoder shown in .

[0332] The extended intra prediction unit may include a reference signal selector 3404 and a prediction performer 3405. The reference signal selector 3404 may receive a current block 3403 as a decoding target. In addition, the reference signal selector 3404 may receive a current picture 3401 and / or a geometrically modified picture 3402 used as a reference signal for intra prediction. The received current picture 3401 may be a signal of an area that is encoded / decoded earlier than the current block 3403. The received geometrically modified picture 3402 may be a reference image of the current picture 3401. Alternatively, the received geometrically modified picture 3402 may be a geometrically modified picture generated by geometrically modifying a reference picture of the current picture 3401 using geometric modification information. The information about the current block 3403 received by the reference signal selector 3404 may be information about the position and / or size of the current block 3403 within the current picture 3401. The reference signal selector 3404 may select a reference signal from the current picture 3401 and the geometrically modified picture 3402. Alternatively, the reference signal selector 3404 may select the current picture 3401 or the geometrically modified picture 3402. For example, when performing intra prediction from the right area of ​​the current block 3403, the reference signal selector 3404 may select the geometrically modified picture 3402 as a reference picture.

[0333] When the reference picture is selected, the prediction executor 3405 may perform intra prediction and generate a prediction block (prediction signal 3406) of the current block 3403. In addition, the decoder may generate intra prediction information 3407 required for intra prediction. The intra prediction information 3407 may include an intra prediction direction or a reference signal, etc.

[0334] Intra prediction can be performed for all available reference signals, and the reference signal with the best efficiency can be selected.

[0335] The extended intra prediction unit of the decoder may selectively reference a decoded area of ​​the current picture or a geometrically modified picture (or reference picture) based on the intra prediction information sent from the encoder. For example, when the intra prediction direction indicated by the intra prediction information is the right area of ​​the current block, the extended intra prediction unit may perform intra prediction by using the geometrically modified picture. When referencing the geometrically modified picture, the decoder may only generate the geometrically modified picture. As described above, the information about whether the geometrically modified picture is used may be determined by the intra prediction direction. Alternatively, the information about whether the geometrically modified picture is used may be determined by the presence of geometric modification information. Alternatively, the information about whether the geometrically modified picture is used may be signaled by additional one or more syntax elements, or may be signaled using a method explicitly or implicitly defined by the encoder and decoder.

[0336] In addition, the method for generating the referenced geometry modification picture during intra prediction according to the present invention may be the same as the method for generating the referenced geometry modification picture during inter prediction according to the present invention. In addition, the method for configuring and using geometry modification information may be the same as the method for inter prediction.

[0337] Fig.35 is a diagram illustrating extended intra prediction according to an embodiment of the present invention.

[0338] In step S3501, the extended intra prediction selects a reference area to be used for intra prediction. The reference area can be selected from all directions of the current block. The intra prediction direction can be expressed as a direction subdivided at a predetermined interval, and each direction can be expressed as an index. In addition, all other possible expressions can be used. The encoder can perform intra prediction in all directions sequentially or in parallel. In the decoder, a single intra prediction direction can be specified based on the intra prediction information.

[0339] In step S3502, the extended intra prediction can determine whether the reference area indicated from the intra prediction direction in the current picture is valid. In other words, if the selected reference area is located in the area encoded / decoded before the current block, the reference area is valid, otherwise, the reference area is invalid. The result of step S3502 can be determined by selecting the reference area in step S3501. For example, it can be determined whether the reference area is valid for the current picture based on the angle, index, or direction used to indicate the selected reference area.

[0340] In step S3504, when the reference region is valid, the extended intra prediction can perform intra prediction of the current block by referring to the decoded signal. In step S3503, when the reference region is invalid, the extended intra prediction can perform intra prediction of the current block by referring to the geometrically modified picture.

[0341] Fig.36 is a diagram showing an intra prediction direction according to the present invention.

[0342] exist Fig.36 In , prediction modes 0 and 1 are non-angular modes. Prediction mode 0 refers to intra planar mode, and prediction mode 1 refers to DC mode (intra DC). Fig.36 In (1), prediction modes 2 to 34 refer to angle modes. Fig.36 In (2), prediction modes 2 to 65 refer to angle modes. Fig.36 In FIG. 1 , the arrow may indicate a reference signal for intra prediction of the current block.

[0343] Fig.36 (1) is a diagram showing the intra prediction direction when only the decoded signal of the current picture can be referenced. Fig.36As shown in (1), intra prediction of prediction modes 2 to 34 can refer to only the left, top, left-top and / or right-top signals of the current block.

[0344] Fig.36 (2) is a diagram showing the intra prediction direction when a decoded signal of a current image, a decoded picture, or a geometrically corrected image of a decoded picture can be referenced. Fig.36 As shown in (2), intra-frame prediction of prediction modes 2 to 65 can refer not only to the left, top, upper-left and / or upper-right signals of the current block, but also to the right, bottom, lower-left and / or lower-right signals of the current block.

[0345] Fig.37 is a diagram showing an embodiment of performing extended intra prediction according to the present invention.

[0346] exist Fig.37 In the above description, the neighboring pixels within the current picture refer to the pixels within the current picture that are referenced when performing intra prediction. The neighboring pixels within the geometrically modified picture refer to the pixels within the geometrically modified picture (or reference picture) corresponding to the current picture. The area without any information refers to an area that does not have pixel information that can be referenced when performing intra prediction among the neighboring pixels. Fig.37 The arrows with lines in (a) and 37(b) indicate the intra prediction direction when intra prediction is performed by referring to neighboring pixels within the current picture. Here, the intra prediction direction may correspond to Fig.36 Prediction modes 2 to 34. Fig.37 The dotted arrow of (d) indicates the intra prediction direction when intra prediction is performed by referring to neighboring pixels in a geometrically modified picture (or reference picture) corresponding to the current picture. Here, the intra prediction direction may correspond to Fig.36 The prediction mode is 35 to 65. Fig.37 The dashed arrows in (c) indicate directions in which intra prediction is not possible due to the absence of a reference signal.

[0347] exist Fig.37 In (a) and 37(c), intra prediction is performed by referring to neighboring pixels in the current picture. Since there is a reference signal, it can be performed Fig.37 Intra prediction in the intra prediction direction shown in (a). However, since the reference signal does not exist, it may not be performed. Fig.37 Intra prediction in the intra prediction direction shown in (c).

[0348] exist Fig.37 (b) and Fig.37 In (d), intra prediction is performed by referring not only to neighboring pixels within the current picture but also to neighboring pixels within a geometrically modified picture (or reference picture) corresponding to the current picture. Fig.37Intra prediction in the intra prediction direction shown in (b) may be performed by referring to neighboring pixels within the current picture. Fig.37 Intra prediction in the intra prediction direction shown in (d) can be performed by referring to neighboring pixels within the geometrically modified picture. Fig.37 The intra prediction of (b) and 37(d) can improve the prediction accuracy by performing intra prediction with reference to neighboring pixels in the current picture and neighboring pixels in the geometrically modified picture, thereby improving the coding efficiency.

[0349] According to the present invention, while generating a geometric modification picture, a geometric modification picture can be configured in a coding unit (CU), a prediction unit (PD), a slice unit, or an image frame unit. When a change in pixel value occurs in an image frame unit of a slice unit, it is preferred to generate a geometric modification picture in a large unit such as a slice unit or an image frame unit rather than a small unit such as a CU or a PU. Therefore, repeated geometric modification information generated when generating a geometric modification picture in a small unit can be avoided, and it can be more efficient in terms of complexity. In addition, when a geometric modification picture is not referenced in an area of ​​a frame unit or a slice unit, information on whether the geometric modification picture is referenced is included in the frame unit or the slice unit. However, the encoder / decoder may not configure unnecessary modification information. For example, the semantics of a picture parameter set (PPS), a syntax configuration, and a slice header may be applied.

[0350] In Tables 1 and 2, "modification_image_generation_flag" may be information about whether a geometry modification picture is used as a reference picture when predicting motion information. "Modification_image_generation_flag" may be included in a sequence unit, a frame unit, a slice unit, etc., and includes information about whether a geometry modification picture is used for each unit. When a geometry modification picture is used as a reference picture for each unit, "modification_image_generation_flag" may be set to "1". When a geometry modification picture is not used as a reference picture, "modification_image_generation_flag" may be set to "0". Alternatively, "modification_image_generation_flag" may be configured to have an opposite value in each case. "Modification_image_generation_flag" may be used as information about whether a geometry modification picture is used in each unit, or as information about whether geometry modification information is included in a bitstream.

[0351] Tables 1 and 2 are examples of syntax configuration when each of "MODIFICATION_PICTURE_GENERATION_FLAG" is transmitted through the PPS and the slice header.

[0352] [Table 1]

[0353]

[0354]

[0355] [Table 2]

[0356]

[0357]

[0358] "Image_modification_info" may be a syntax element related to geometry modification information and configured with a matrix or nonlinear data. "Image_modification_info" may be a syntax element present in a bitstream when a geometry modification picture is used in each unit such as a frame unit, a slice unit, etc. (when "modification_image_generation_flag" is 1). Tables 1 and 2 show examples when "modification_image_generation_flag" has geometry modification information configured in a matrix form. As described above, the geometry modification information matrix may include 9 coefficients. Since a fixed value may be used for one of the 9 coefficients, 8 coefficients may be transmitted. By approximation or prediction of the coefficients, the matrix coefficients to be transmitted may be reduced to 8 or less. In Tables 1 and 2, the matrix coefficients may be transmitted in the form of "image_modification_info[x][y][z]". Here, x may refer to an index of a reference picture list, y may refer to an index of a reference picture within the reference picture list, and z may refer to an index of the matrix coefficient. Therefore, all reference pictures may have different matrices, respectively.

[0359] When the encoder determines that the geometric modification picture being configured in the coding unit and / or prediction unit is the best case, as shown in Tables 3 and 4, "Modification_Image_Generation_Flag" and "Image_Modification_Information" may be included as syntax elements of CO and / or PU. In Tables 3 and 4, a CU may include reference picture information referenced by itself, and therefore may require a single geometric modification information for geometrically modifying the corresponding reference picture. Therefore, unlike Tables 1 and 2, the matrix coefficients are sent in the form of "Image_Modification_Information[z]". Here, z refers to the index of the matrix coefficient.

[0360] Table 3 and Table 4 are examples of syntax configurations when each of “modification_image_generation_flag” and “image_modification_information” is signaled by the CU and the PU.

[0361] [Table 3]

[0362] coding_unit(x0,y0,log2CbSize){ describe .. modification_image_generation_flag ae(v) if(modification_image_generation_flag){ for(i=0;i<=8;i++){ image_modification_info[i] ae(v) ..

[0363] [Table 4]

[0364] prediction_unit(x0,y0,nPbW,nPbH){ describe .. modification_image_generation_flag ae(v) if(modification_image_generation_flag){ for(i=0;i<=8;i++){ image_modification_info[i] ae(v) ..

[0365] When the PPS or slice header includes information about the geometry modification information as shown in Tables 1 and 2, the geometry modification information may not be transmitted in units of CU or PU. Therefore, the CU or PU may be configured as shown in Tables 5 and 6. In Tables 5 and 6, "modification_image_reference_flag" of 1 may indicate that a geometry modification picture is used as a reference signal, or "modification_image_reference_flag" of 0 may indicate that a geometry modification picture is not used as a reference signal. Alternatively, the "modification_image_reference_flag" may be configured to have an opposite value in each case. When a geometry modification picture is used as reference information in a CU or PU (or when the "modification_image_generation_flag" is "1"), the CU or PU may include "reference_modification_info" of information required for reference. The information required for reference may include a reference index, a residual signal, and the like.

[0366] Table 5 and Table 6 are examples of syntax configuration when each “reference_modification_information” is signaled by the CU and the PU.

[0367] [Table 5]

[0368]

[0369]

[0370] [Table 6]

[0371] prediction_unit(x0,y0,nPbW,nPbH){ describe .. modification_image_reference_flag ae(v) if(modification_image_reference_flag){ reference_modification_info ae(v) ..

[0372] Unidirectional prediction or bidirectional prediction can be used in motion prediction. When a geometrically modified picture is used in motion prediction, the motion prediction type can be signaled in CU and PU through the syntax configuration of Table 7 and Table 8.

[0373] In detail, a geometry modification picture may be generated by using geometry modification information signaled in an upper layer such as a slice header or PPS. In addition, a prediction type such as bidirectional prediction using a geometry modification picture, unidirectional prediction using a geometry modification picture, or prediction not using a geometry modification picture may be signaled in a CU or PU. In addition, information on whether a geometry modification picture is used in each direction may be signaled through flag information.

[0374] In Tables 7 and 8, "modification_image_reference_type" may be a flag indicating one of bidirectional prediction using a geometry modification picture, unidirectional prediction using a geometry modification picture, or prediction not using a geometry modification picture. "ref_0_modification_flag" may be a flag indicating whether the current decoded picture refers to a geometry modification picture of a reference picture in a first reference picture list of two reference picture lists. "ref_1_modification_flag" may be a flag indicating whether the current decoded picture refers to a geometry modification picture of a reference picture in a second reference picture list of two reference picture lists. When "ref_X_modification_flag" is set to true (1) (here, X may be 0 or 1), prediction may be performed by referring to a geometry modification picture of a corresponding reference picture. When "ref_X_modification_flag" is set to false (0) (here, X may be 0 or 1), prediction may be performed without referring to a geometry modification picture of a corresponding reference picture.

[0375] When "MODIFICATION_IMAGE_MODIFICATION_TYPE" in CU or PU is "NONE_USE", prediction in both directions can be performed without referring to the geometric modification picture of the reference picture by setting "REFERENCE_0_MODIFICATION_FLAG" and "REFERENCE_1_MODIFICATION_FLAG" to false.

[0376] When MODIFICATION_IMAGE_MODIFICATION_TYPE is REF_0_USE, prediction in the list 0 direction may be performed by referring to the geometry modification picture by setting REF_0_MODIFICATION_FLAG to true and REF_1_MODIFICATION_FLAG to false.

[0377] When MODIFICATION_IMAGE_MODIFICATION_TYPE is REF_1_USE, prediction in list 1 direction may be performed by referencing the geometry modification picture by setting REF_0_MODIFICATION_FLAG to FALSE and REF_1_MODIFICATION_FLAG to TRUE.

[0378] When "modification_image_modification_type" is "BOTH_USE", prediction in two directions can be performed by referring to the geometric modification picture of the reference picture by setting "reference_0_modification_flag" and "reference_1_modification_flag" to true.

[0379] Table 7 and Table 8 are examples of syntax configuration when each of “MODIFIED_PICTURE_REFERENCE_TYPE” is signaled by the CU and the PU.

[0380] [Table 7]

[0381]

[0382]

[0383] [Table 8]

[0384]

[0385] As an application example of the present invention, a geometric modification picture that is geometrically modified in an upper layer (such as a slice header, PPS, etc.) and geometric modification information sent from the upper layer may be corrected and used in the CU layer and / or the PU layer. As shown in Tables 9 and 10, the CU layer and / or the PU layer may include "modification_image_using_revision_flag" and "modification_revision_info".

[0386] The “modify_image_use_correction_flag” being 1 may indicate that the CU layer or the PU layer includes information to be corrected. Here, “modify_correction_information” may be additionally signaled.

[0387] A "modified_image_use_correction_flag" of 0 may indicate that the CU layer or the PU layer does not include information to be corrected. Since there is no information to be corrected, the "modified_correction_information" may not be signaled.

[0388] "Modify_picture_use_correction_flag" and "Modify_correction_info" may be directly signaled as syntax elements, as shown in the embodiments of Tables 9 and 10. Alternatively, "Modify_picture_use_correction_flag" and "Modify_correction_info" may be explicitly and / or implicitly signaled according to a predetermined rule defined in the encoder and the decoder.

[0389] Tables 9 and 10 are examples of syntax configurations when each of “modification_image_use_correction_flag” and “modification_correction_information” is signaled by the CU and the PU.

[0390] [Table 9]

[0391] coding_unit(x0,y0,log2CbSize){ describe .. modification_image_using_revision_flag ae(v) if (modification_image_using_revision_flag) { modification_revision_info ae(v) ..

[0392] [Table 10]

[0393] prediction_unit(x0, y0, nPbW, nPbH) { description .. modification_image_using_revision_flag ae(v) if (modification_image_using_revision_flag) { modification_revision_info ae(v) ..

[0394] In the above embodiments, the method is described based on a flow chart with a series of steps or units, but the present invention is not limited to the order of these steps, but some steps can be performed simultaneously with other steps or in a different order. In addition, it should be understood by those skilled in the art that the steps in the flow chart do not exclude each other, and other steps can be added to the flow chart, or some steps can be deleted from the flow chart without affecting the scope of the present invention.

[0395] The above description includes examples of various aspects. Of course, it is not possible to describe every possible combination of components or methods for the purpose of describing various aspects, but those of ordinary skill in the art will recognize that many further combinations and arrangements are possible. Therefore, this specification is intended to cover all such changes, modifications and variations that fall within the spirit and scope of the appended claims.

[0396] The computer-readable storage medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded in the computer-readable storage medium may be any program instructions specially designed and constructed for the present invention or known to a person skilled in the art of computer software. Examples of computer-readable storage media include magnetic recording media (such as hard disks, floppy disks, and magnetic tapes); optical data storage media (such as CD-ROMs or DVD-ROMs); magneto-optical media (such as optical magnetic disks); hardware devices specially constructed to store and implement program instructions (such as read-only memory (ROM), random access memory (RAM), and flash memory). Examples of program instructions include not only machine language codes formatted by a compiler, but also high-level language codes that can be implemented by a computer using an interpreter. Hardware devices may be configured to operate through one or more software modules (or vice versa) to perform processing according to the present invention.

[0397] Although the present invention has been described according to specific items such as detailed elements and limited embodiments and drawings, they are only provided to help a more general understanding of the present invention, and the present invention is not limited to the above embodiments. It will be appreciated by those skilled in the art that various modifications and changes can be made from the above description.

[0398] Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the full scope of the appended claims and their equivalents should fall within the scope and spirit of the present invention.

[0399] Industrial Applicability

[0400] The present invention can be used for encoding / decoding images.

Claims

1. A method for encoding an image, the method comprising: Generate geometry modification information of the current block; determining a geometry modification flag indicating whether geometry modification is used for motion compensation of the current block; determining geometry modification type information of the current block, the geometry modification type information indicating a number of affine references used for the motion compensation of the current block; Determining predicted geometry modification information of the current block based on reference geometry modification information of the current block; determining a reference geometry modification index indicating the reference geometry modification information among a plurality of reference geometry modification information, the reference geometry modification information comprising a plurality of the affine references as many as the number of the affine references; determining residual geometry modification information indicating a difference between the predicted geometry modification information of the current block and the geometry modification information of the current block; generating a prediction block of the current block by performing inter-frame prediction based on a reference picture of the current block and the geometry modification information of the current block; Generate a residual block of the current block based on the residual data of the current block; encoding the current block based on the prediction block of the current block and the residual block of the current block; as well as encoding a bit stream including the geometry modification flag, the reference geometry modification index, the residual geometry modification information, and the geometry modification type information, The geometric modification information enables the prediction block of the current block to be derived from an affine-modified shape region in the reference picture referenced by the current block. 2 . The method of claim 1 , wherein encoding the geometry modification information comprises reconfiguring the geometry modification information, and the reconfiguring the geometry modification information comprises simplifying or predicting the geometry modification information. 3 . The method of claim 1 , wherein the geometry modification information is generated based on a change in pixel information between a current picture including the current block and the reference picture. 4 . The method of claim 3 , wherein the geometry modification information is generated based on matching information between feature points extracted from each of the current picture and the reference picture.

5. The method of claim 1, wherein the method further comprises: identifying a point within the reference picture, the point corresponding to a point within the geometrically modified picture; as well as The pixel information of the corresponding point in the reference picture is set as the pixel information of the point in the geometry modified picture. 6 . The method of claim 5 , wherein when a point corresponding to the point within the geometry-modified picture does not exist within the reference picture, a point within the reference picture that is closest to the corresponding point is identified as the corresponding point.

7. A method for decoding an image, the method comprising: Obtaining a geometry modification flag of a current block from a bitstream, the geometry modification flag indicating whether geometry modification is used for motion compensation of the current block; Obtaining geometry modification type information of the current block from the bitstream, the geometry modification type information indicating the number of affine references used for the motion compensation of the current block; Obtaining, from the bitstream, a reference geometry modification index indicating one reference geometry modification information among a plurality of reference geometry modification information; Retrieving residual geometry modification information from the bitstream, the residual geometry modification information indicating a difference between predicted geometry modification information of the current block and geometry modification information of the current block; When the geometry modification flag indicates that geometry modification is used for motion compensation of the current block, determining reference geometry modification information of the current block from previously used geometry modification information, the reference geometry modification information including a plurality of the affine references indicated by the geometry modification type information; Determining the predicted geometry modification information of the current block based on the reference geometry modification information specified by the reference geometry modification index; Determining the geometry modification information of the current block based on the predicted geometry modification information of the current block and the residual geometry modification information of the current block; generating a prediction block of the current block by performing inter-frame prediction based on a reference picture of the current block and the geometry modification information of the current block; Generate a residual block of the current block based on the residual data of the current block; as well as reconstructing the current block based on the prediction block of the current block and the residual block of the current block, The geometric modification information enables the prediction block to be derived from an affine modified area in the reference picture referenced by the current block.

8. The method of claim 7, wherein the method further comprises: identifying a point within the reference picture, the point corresponding to a point within the geometrically modified picture; as well as The pixel information of the corresponding point in the reference picture is set as the pixel information of the point in the geometry modified picture.

9. The method of claim 8, wherein when a point corresponding to the point within the geometry-modified picture does not exist within the reference picture, a point within the reference picture that is closest to the corresponding point is identified as the corresponding point.

10. The method of claim 8, wherein when the corresponding point within the reference picture has real coordinates, one or more points within the reference picture each having integer coordinates and adjacent to the corresponding point are identified, and pixel information of the corresponding point within the reference picture is derived based on the pixel information of each of the one or more points having integer coordinates.

11. The method of claim 8, wherein the generating of the geometry modified picture is performed in at least one of a picture, a slice, a tile, a coding unit, and a prediction unit.

12. A method for storing a bit stream, comprising: Generate a bitstream including residual data of the current block, a geometry modification flag, geometry modification type information, a reference geometry modification index, and residual geometry modification information; as well as storing the bitstream, in, The geometry modification flag indicates whether geometry modification is used for motion compensation of the current block; The geometry modification type information indicates the number of affine references used for the motion compensation of the current block; The reference geometry modification index indicates a piece of reference geometry modification information among a plurality of reference geometry modification information; The reference geometry modification information specified by the reference geometry modification index is used to determine the predicted geometry modification information of the current block and includes a plurality of the affine references indicated by the geometry modification type information; The residual geometry modification information indicates a difference between the predicted geometry modification information of the current block and the geometry modification information of the current block; The prediction block of the current block is generated by performing inter-frame prediction based on a reference picture of the current block and the geometry modification information of the current block; The geometric modification information enables the prediction block to be derived from an affine-modified region in the reference picture referred to by the current block; A residual block of the current block is generated based on the residual data of the current block; and The current block is reconstructed based on the prediction block of the current block and the residual block of the current block.

13. A method for decoding an image, the method comprising: Receiving a bitstream including a plurality of flag syntax elements corresponding to a plurality of reference pictures, respectively, wherein each of the plurality of flag syntax elements indicates whether the bitstream includes a global geometry modification information syntax element for the corresponding reference picture; If a flag syntax element indicates that the bitstream includes global geometry modification information for a reference picture corresponding to the flag syntax element, reading from the bitstream a global geometry modification information syntax element associated with the reference picture corresponding to the flag syntax element; reconstructing geometry modification information of the reference picture corresponding to the flag syntax element based on the geometry modification information syntax element of the reference picture corresponding to the flag syntax element; geometrically modifying the reference picture corresponding to the flag syntax element based on the reconstructed geometry modification information of the reference picture corresponding to the flag syntax element to generate a prediction block; Generate a residual block by performing dequantization and inverse transformation on the residual data; as well as A current block is reconstructed based on the residual block and the prediction block.

14. The method of claim 13, wherein reconstructing the geometry modification information comprises: The geometry modification information is further reconstructed based on previously stored geometry modification information.

15. The method of claim 13, wherein geometrically modifying the reference picture corresponding to the flag syntax element comprises: identifying a point within the reference picture that corresponds to the flag syntax element, wherein the point within the reference picture that corresponds to the flag syntax element corresponds to a point within the prediction block; as well as Pixel information of the prediction block is determined based on the one point in the reference picture corresponding to the flag syntax element.

16. The method of claim 15, wherein the geometry modification information represents a relationship between the one point within the prediction block and the one point within the reference picture corresponding to the flag syntax element.

17. A method for encoding an image, the method comprising: determining a plurality of flag syntax elements corresponding to a plurality of reference pictures, respectively, wherein each of the plurality of flag syntax elements indicates whether a bitstream includes a global geometry modification information syntax element for the corresponding reference picture; If the flag syntax element is determined to indicate that the bitstream includes global geometry modification information for a reference picture corresponding to the flag syntax element, determining a global geometry modification information syntax element associated with the reference picture corresponding to the flag syntax element; determining, based on the geometry modification information syntax element of the reference picture corresponding to the flag syntax element, geometry modification information for the reference picture corresponding to the flag syntax element; geometrically modifying the reference picture corresponding to the flag syntax element based on the geometry modification information of the reference picture corresponding to the flag syntax element to generate a prediction block; Generate a residual block by performing dequantization and inverse transformation on the residual data; as well as A current block is reconstructed based on the residual block and the prediction block.

Images