Image Encoding / Decoding Method, Apparatus, and Recording Medium for Storing a Bitstream Based on Illumination Compensation

Abstract

An image encoding / decoding method and apparatus are provided. The image decoding method according to the present disclosure includes: obtaining information regarding luminance compensation for a current block; deriving luminance compensation parameters for each of two or more luminance compensation candidate modes for the luminance compensation; applying luminance compensation to a predetermined first adjacent region adjacent to a reference block of the current block based on the luminance compensation parameters to obtain a luminance-compensated first adjacent region; obtaining a final error value for each of the luminance compensation candidate modes based on an error between sample values in the luminance-compensated first adjacent region and sample values in a predetermined second adjacent region adjacent to the current block; rearranging the luminance compensation candidate modes based on the final error values; and determining a luminance compensation candidate mode for the current block based on the rearranged luminance compensation candidate modes and the information regarding the luminance compensation, wherein the first adjacent region and the second adjacent region may be corresponding positions.

Description

Technical Field

[0001] The present disclosure relates to an image encoding / decoding method, apparatus, and a recording medium storing a bitstream, and more particularly, to an image encoding / decoding method, apparatus, and a recording medium storing a bitstream generated by the image encoding method / apparatus of the present disclosure based on illumination compensation.

Background Art

[0002] Recently, demands for high-resolution, high-quality images, such as HD (High Definition) images and UHD (Ultra High Definition) images, have been increasing in various fields. As the image data becomes higher in resolution and quality, the amount of information or bits to be transmitted relatively increases compared to conventional image data. The increase in the amount of information or bits to be transmitted brings about an increase in transmission costs and storage costs.

[0003] Accordingly, there is a need for a highly efficient image compression technique for effectively transmitting, storing, and reproducing information of high-resolution, high-quality images.

Summary of the Invention

Problems to be Solved by the Invention

[0004] An object of the present disclosure is to provide an image encoding / decoding method and apparatus with improved encoding / decoding efficiency.

[0005] Another object of the present disclosure is to provide an image encoding / decoding method and apparatus that perform illumination compensation.

[0006] Another object of the present disclosure is to provide an image encoding / decoding method and apparatus that rearrange illumination compensation candidates based on a template cost when performing illumination compensation.

[0007] Also, an object of the present disclosure is to provide an image encoding / decoding method and apparatus that perform luminance compensation when a prediction mode is a merge mode.

[0008] Also, an object of the present disclosure is to provide a non-transitory computer-readable recording medium that stores a bitstream generated by an image encoding method or apparatus according to the present disclosure.

[0009] Also, an object of the present disclosure is to provide a non-transitory computer-readable recording medium that stores a bitstream received by an image decoding apparatus according to the present disclosure, decoded, and used for restoring an image.

[0010] Also, an object of the present disclosure is to provide a method for transmitting a bitstream generated by an image encoding method or apparatus according to the present disclosure.

[0011] The technical problems to be solved in the present disclosure are not limited to the above-described technical problems, and other technical problems not described above will be clearly understood by those of ordinary skill in the technical field to which the present disclosure pertains from the following description.

Means for Solving the Problems

[0012] According to an embodiment of the present disclosure, an image decoding method performed by an image decoding apparatus, the method including: obtaining information regarding luminance compensation for a current block; deriving luminance compensation parameters for each of two or more luminance compensation candidate modes for the luminance compensation; applying luminance compensation to a predetermined first adjacent region adjacent to a reference block of the current block based on the luminance compensation parameters to obtain a luminance-compensated first adjacent region; obtaining a final error value for each of the luminance compensation candidate modes based on an error between sample values in the luminance-compensated first adjacent region and sample values in a predetermined second adjacent region adjacent to the current block; rearranging the luminance compensation candidate modes based on the final error values; and determining a luminance compensation candidate mode for the current block based on the rearranged luminance compensation candidate modes and the information regarding the luminance compensation, wherein the first adjacent region and the second adjacent region may be corresponding positions.

[0013] According to an embodiment of the present disclosure, the information regarding the luminance compensation may include at least one of information indicating whether luminance compensation is executable for the current block, information indicating whether the luminance compensation is applied to the current block, or an index indicating one of the luminance compensation candidate modes.

[0014] According to an embodiment of the present disclosure, the luminance compensation candidate modes may include at least one of a first mode that uses both a left adjacent region and an upper adjacent region of the current block and the reference block, a second mode that uses only the left adjacent region of the current block and the reference block, or a third mode that uses only the upper adjacent region of the current block and the reference block.

[0015] According to an embodiment of the present disclosure, the rearranging may mean arranging the final error values in ascending order.

[0016] According to an embodiment of the present disclosure, based on the prediction mode of the current block being the merge mode, the information regarding the luminance compensation can be obtained as the information regarding the luminance compensation of the selected merge candidate of the current block.

[0017] According to an embodiment of the present disclosure, the information regarding the luminance compensation of the selected merge candidate of the current block can be obtained based on at least one of the size, prediction type, split type, or transform type of the current block.

[0018] According to an embodiment of the present disclosure, the information indicating whether the luminance compensation is applied to the current block can be obtained based on a first error value for the luminance compensation candidate mode calculated based on the first adjacent region and the second adjacent region before the luminance compensation, and the final error value.

[0019] According to an embodiment of the present disclosure, the acquisition of the information indicating whether the luminance compensation is applied to the current block can be further determined based on a threshold value.

[0020] According to an embodiment of the present disclosure, the acquisition of the information indicating whether the luminance compensation is applied to the current block may be performed by comparing a difference value between the final error value and the first error value with the threshold value.

[0021] According to an embodiment of the present disclosure, an image encoding method performed by an image encoding apparatus, the method comprising: encoding information related to luminance compensation for a current block; deriving luminance compensation parameters for each of two or more luminance compensation candidate modes for the luminance compensation; applying luminance compensation to a predetermined first adjacent region adjacent to a reference block of the current block based on the luminance compensation parameters for each of the luminance compensation candidate modes to obtain a luminance-compensated first adjacent region; obtaining a final error value for each of the luminance compensation candidate modes based on an error between sample values in the luminance-compensated first adjacent region and sample values in a predetermined second adjacent region adjacent to the current block; rearranging the luminance compensation candidate modes based on the final error values for each of the luminance compensation candidate modes; and determining a luminance compensation candidate mode for the current block based on the rearranged luminance compensation candidate modes and the information related to the luminance compensation, wherein the first adjacent region and the second adjacent region may be collocated regions.

[0022] According to an embodiment of the present disclosure, a method for transmitting a bitstream generated by an image encoding method, the image encoding method comprising: encoding information related to luminance compensation for a current block; deriving luminance compensation parameters for each of two or more luminance compensation candidate modes for the luminance compensation; applying luminance compensation to a predetermined first adjacent region adjacent to a reference block of the current block based on the luminance compensation parameters for each of the luminance compensation candidate modes to obtain a luminance-compensated first adjacent region; obtaining a final error value for each of the luminance compensation candidate modes based on an error between sample values in the luminance-compensated first adjacent region and sample values in a predetermined second adjacent region adjacent to the current block; rearranging the luminance compensation candidate modes based on the final error values for each of the luminance compensation candidate modes; and determining a luminance compensation candidate mode for the current block based on the rearranged luminance compensation candidate modes and the information related to the luminance compensation, wherein the first adjacent region and the second adjacent region may be collocated regions.

Advantages of the Invention

[0023] According to the present disclosure, it is possible to provide an image encoding / decoding method and apparatus with improved encoding / decoding efficiency.

[0024] In addition, the present disclosure can provide an image encoding / decoding method and apparatus that perform illumination compensation.

[0025] In addition, when performing luminance compensation, the present disclosure can provide an image encoding / decoding method and apparatus that rearrange luminance compensation candidates based on template cost.

[0026] In addition, when the prediction mode is a merge mode, the present disclosure can provide an image encoding / decoding method and apparatus that perform luminance compensation.

[0027] In addition, according to the present disclosure, a non-transitory computer-readable recording medium for storing a bitstream generated by the image encoding method or apparatus according to the present disclosure can be provided.

[0028] In addition, according to the present disclosure, a non-transitory computer-readable recording medium for storing a bitstream received by the image decoding apparatus according to the present disclosure, decoded, and used for restoring an image can be provided.

[0029] According to the present disclosure, a method for transmitting a bitstream generated by the image encoding method or apparatus according to the present disclosure can be provided.

[0030] The effects obtained in the present disclosure are not limited to the above-described effects, and other effects not described above will be clearly understood by those of ordinary skill in the technical field to which the present disclosure pertains from the following description.

Brief Description of the Drawings

[0031]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Embodiments for Carrying Out the Invention

[0032] Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those having ordinary knowledge in the technical field to which the present disclosure belongs can easily implement them. However, the present disclosure can be realized in various different forms and is not limited to the embodiments described here.

[0033] In describing embodiments of the present disclosure, when it is determined that a detailed description of known configurations or functions may obscure the gist of the present disclosure, the detailed description thereof will be omitted. In the drawings, parts not related to the description of the present disclosure are omitted, and the same reference numerals are given to the same parts.

[0034] In the present disclosure, when a component is “connected,” “coupled,” or “joined” to another component, this can include not only a direct connection relationship but also an indirect connection relationship in which another component exists between them. Also, when a component “includes” or “has” (comprises; constitutes; constructs; sets; encloses; includes; contains) another component, this means that, unless otherwise stated to the contrary, it does not exclude other components but can further include other components.

[0035] In the present disclosure, terms such as “first,” “second,” etc. are used only for the purpose of distinguishing one component from another and do not limit the order or importance between components, etc., unless otherwise specifically mentioned. Therefore, within the scope of the present disclosure, the first component of one embodiment may be referred to as the second component in another embodiment, and similarly, the second component of one embodiment may be referred to as the first component in another embodiment.

[0036] In the present disclosure, components that are distinguished from each other are for clearly explaining their respective features and do not necessarily mean that the components are separated. That is, a plurality of components may be integrated and configured as one hardware or software unit, or one component may be distributed and configured as a plurality of hardware or software units. Therefore, without separate mention, such integrated or distributed embodiments are also included in the scope of the present disclosure.

[0037] In the present disclosure, the components described in various embodiments do not necessarily mean essential components, and some may be optional components. Therefore, embodiments constituted by a subset of the components described in one embodiment are also included in the scope of the present disclosure. Further, embodiments that include other components in addition to the components described in various embodiments are also included in the scope of the present disclosure.

[0038] The present disclosure relates to the encoding and decoding of images, and the terms used in the present disclosure can have the ordinary meanings in the technical field to which the present disclosure belongs, unless newly defined in the present disclosure.

[0039] In the present disclosure, "video" can mean a set of a series of images over time.

[0040] In the present disclosure, "picture" generally means a unit indicating any one image in a specific time period, and a slice / tile is an encoding unit constituting a part of a picture, and one picture can be composed of one or more slices / tiles. Further, a slice / tile can include one or more CTUs (coding tree units).

[0041] In the present disclosure, "pixel" or "pel" can mean the smallest unit constituting one picture (or image). Further, the term "sample" can be used as a term corresponding to a pixel. A sample can generally indicate a pixel or a pixel value, and can also indicate only the pixel / pixel value of the luma component, or can also indicate only the pixel / pixel value of the chroma component.

[0042] In the present disclosure, "unit" can indicate the basic unit of image processing. A unit can include at least one of a specific region of a picture and information related to the region. A unit can, in some cases, be used interchangeably with terms such as "sample array", "block", or "area". Generally, an M×N block can include a set (or array) of samples (or sample arrays) or transform coefficients consisting of M columns and N rows.

[0043] In the present disclosure, "current block" can mean any one of "current coding block", "current coding unit", "block to be coded", "block to be decoded", or "block to be processed". When prediction is performed, "current block" can mean "current prediction block" or "block to be predicted". When transformation (inverse transformation) / quantization (inverse quantization) is performed, "current block" can mean "current transformation block" or "block to be transformed". When filtering is performed, "current block" can mean "block to be filtered".

[0044] Also, in the present disclosure, unless explicitly stated as a chroma block, "current block" can mean a block that includes both a luma component block and a chroma component block or the "luma block of the current block". The luma component block of the current block can be explicitly expressed including an explicit description of the luma component block such as "luma block" or "current luma block". Also, the chroma component block of the current block can be explicitly expressed including an explicit description of the chroma component block such as "chroma block" or "current chroma block".

[0045] In the present disclosure, " / " and "," can be interpreted as "and / or". For example, "A / B" and "A, B" can be interpreted as "A and / or B". Also, "A / B / C" and "A, B, C" can mean "at least one of A, B, and / or C".

[0046] In the present disclosure, "or" can be interpreted as "and / or". For example, "A or B" can mean 1) only "A", 2) only "B", or 3) "A and B". Alternatively, in the present disclosure, "or" can mean "additionally or alternatively".

[0047] In the present disclosure, "at least one of A, B, and C" can mean "only A", "only B", "only C", or "any all combinations of A, B, and C". Also, "at least one of A, B or C" or "at least one of A, B and / or C" can mean "at least one of A, B, and C".

[0048] The parentheses used in the present disclosure can mean "for example". For example, when displayed as "prediction (intra prediction)", "intra prediction" can be proposed as an example of "prediction". In other words, "prediction" in the present disclosure is not limited to "intra prediction", and "intra prediction" can be proposed as an example of "prediction". Also, when displayed as "prediction (i.e., intra prediction)", "intra prediction" can be proposed as an example of "prediction".

[0049] Overview of Video Coding System

[0050] FIG. 1 is a diagram schematically showing a video coding system to which an embodiment according to the present disclosure can be applied.

[0051] A video coding system according to an embodiment can include an encoding device 10 and a decoding device 20. The encoding device 10 can transmit encoded video and / or image information or data to the decoding device 20 in a file or streaming format via a digital storage medium or a network.

[0052] The encoding device 10 according to an embodiment can include a video source generation unit 11, an encoding unit 12, and a transmission unit 13. The decoding device 20 according to an embodiment can include a reception unit 21, a decoding unit 22, and a rendering unit 23. The encoding unit 12 can be referred to as a video / image encoding unit, and the decoding unit 22 can be referred to as a video / image decoding unit. The transmission unit 13 can be included in the encoding unit 12. The reception unit 21 can be included in the decoding unit 22. The rendering unit 23 can also include a display unit, and the display unit can be configured as a separate device or an external component.

[0053] The video source generation unit 11 can obtain video / images through processes such as capture, synthesis, or generation of video / images. The video source generation unit 11 can include a video / image capture device and / or a video / image generation device. The video / image capture device can include, for example, one or more cameras, a video / image archive including previously captured video / images, etc. The video / image generation device can include, for example, a computer, a tablet, and a smartphone, etc., and can (electronically) generate video / images. For example, virtual video / images can be generated via a computer or the like, and in this case, the video / image capture process can be replaced by a process in which related data is generated.

[0054] The symbolization unit 12 can encode the input video / image. The symbolization unit 12 can perform a series of procedures such as prediction, transformation, quantization, etc. for compression and encoding efficiency. The symbolization unit 12 can output the encoded data (encoded video / image information) in the form of a bitstream.

[0055] The transmission unit 13 can acquire the encoded video / image information or data output in the form of a bitstream, and transmit this to the receiving unit 21 of the decoding device 20 or other external objects via a digital storage medium or network in file or streaming format. The digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray (registered trademark), HDD, SSD, etc. The transmission unit 13 can include elements for generating a media file via a predetermined file format, and can include elements for transmission via a broadcast / communication network. The transmission unit 13 can be provided as a transmission device separate from the symbolization unit 120. In this case, the transmission device can include at least one processor that acquires the encoded video / image information or data output in the form of a bitstream and a transmission unit that transmits this in file or streaming format. This receiving unit 21 can extract / receive the bitstream from the storage medium or network and transmit it to the decoding unit 22.

[0056] The decoding unit 22 can decode the video / image by performing a series of procedures such as inverse quantization, inverse transformation, prediction, etc. corresponding to the operations of the symbolization unit 12.

[0057] The rendering unit 23 can render the decoded video / image. The rendered video / image can be displayed via the display unit.

[0058] Overview of Image Encoding Device

[0059] FIG. 2 is a diagram schematically showing an image encoding apparatus to which an embodiment according to the present disclosure can be applied.

[0060] As shown in FIG. 2, the image encoding apparatus 100 can include an image division unit 110, a subtraction unit 115, a conversion unit 120, a quantization unit 130, an inverse quantization unit 140, an inverse conversion unit 150, an addition unit 155, a filtering unit 160, a memory 170, an inter prediction unit 180, an intra prediction unit 185, and an entropy encoding unit 190. The inter prediction unit 180 and the intra prediction unit 185 can be collectively referred to as a "prediction unit". The conversion unit 120, the quantization unit 130, the inverse quantization unit 140, and the inverse conversion unit 150 can be included in a residual processing unit. The residual processing unit can further include the subtraction unit 115.

[0061] All or at least a part of a plurality of components constituting the image encoding apparatus 100 can be realized by one hardware component (for example, an encoder or a processor) according to an embodiment. Further, the memory 170 can include a DPB (decoded picture buffer) and can be realized by a digital storage medium.

[0062] The image segmentation unit 110 can divide an input image (or picture, frame) input to the image encoding device 100 into one or more processing units. As an example, the processing unit can be called a coding unit (CU). The coding unit can be obtained by recursively dividing a coding tree unit (CTU) or a largest coding unit (LCU) in a QT / BT / TT (Quad-tree / Binary-tree / Ternary-tree) structure. For example, one coding unit can be divided into a plurality of coding units with a deeper depth based on a quadtree structure, a binary tree structure, and / or a ternary tree structure. For the division of the coding unit, the quadtree structure can be applied first, and the binary tree structure and / or the ternary tree structure can be applied later. Based on the final coding unit that cannot be divided further, the coding procedure according to the present disclosure can be performed. The largest coding unit can be used as the final coding unit, and the coding units with a lower depth obtained by dividing the largest coding unit can also be used as the final coding unit. Here, the coding procedure can include procedures such as prediction, transformation, and / or restoration described later. As another example, the processing unit of the coding procedure can be a prediction unit (PU: Prediction Unit) or a transformation unit (TU: Transform Unit). The prediction unit and the transformation unit can be divided or partitioned from the final coding unit respectively. The prediction unit can be a unit of sample prediction, and the transformation unit can be a unit for deriving transformation coefficients and / or a unit for deriving a residual signal from the transformation coefficients.

[0063] The prediction unit (inter prediction unit 180 or intra prediction unit 185) can perform prediction on a block to be processed (current block) and generate a predicted block including prediction samples for the current block. The prediction unit can determine whether intra prediction is applied in units of the current block or CU, or whether inter prediction is applied. The prediction unit can generate various information regarding the prediction of the current block and transmit it to the entropy encoding unit 190. The information regarding the prediction can be encoded by the entropy encoding unit 190 and output in the form of a bitstream.

[0064] The intra prediction unit 185 can predict the current block by referring to samples within the current picture. The samples to be referred to can be located in the neighborhood of the current block or at a distance according to the intra prediction mode and / or intra prediction technique. The intra prediction mode can include a plurality of non-directional modes and a plurality of directional modes. The non-directional modes can include, for example, the DC mode and the Planar mode. The directional modes can include, for example, 33 directional prediction modes or 65 directional prediction modes according to the degree of fineness of the prediction direction. However, this is only an example, and more or fewer directional prediction modes can be used based on the settings. The intra prediction unit 185 can also determine the prediction mode to be applied to the current block using the prediction mode applied to the neighboring blocks.

[0065] The inter prediction unit 180 can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, the motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter prediction, the neighboring blocks can include spatial neighboring blocks existing within the current picture and temporal neighboring blocks existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same or different from each other. The temporal neighboring block can be referred to by names such as a collocated reference block, a collocated CU (colCU), etc. The reference picture including the temporal neighboring block can be referred to as a collocated picture (colPic). For example, the inter prediction unit 180 can construct a motion information candidate list based on neighboring blocks and generate information indicating which candidate is used to derive the motion vector and / or reference picture index of the current block. Inter prediction can be performed based on various prediction modes. For example, in the case of the skip mode and the merge mode, the inter prediction unit 180 can use the motion information of neighboring blocks as the motion information of the current block. In the case of the skip mode, different from the merge mode, the residual signal cannot be transmitted.In the case of the motion information prediction (MVP) mode, the motion vectors of neighboring blocks are used as motion vector predictors, and the motion vector difference and the indicator for the motion vector predictor are encoded to signal the motion vector of the current block. The motion vector difference can mean the difference between the motion vector of the current block and the motion vector predictor.

[0066] The prediction unit can generate a prediction signal based on various prediction methods and / or prediction techniques described below. For example, the prediction unit can not only apply intra prediction or inter prediction for the prediction of the current block, but also apply intra prediction and inter prediction simultaneously. A prediction method that applies intra prediction and inter prediction simultaneously for the prediction of the current block can be called CIIP (combined inter and intra prediction). In addition, the prediction unit can also perform intra block copy (IBC) for the prediction of the current block. Intra block copy can be used for content image / video coding such as games, for example, like SCC (screen content coding). IBC is a method of predicting the current block using a restored reference block within the current picture at a position separated from the current block by a predetermined distance. When IBC is applied, the position of the reference block within the current picture can be encoded as a vector (block vector) corresponding to the predetermined distance. IBC basically performs prediction within the current picture, but can be performed in the same manner as inter prediction in terms of deriving a reference block within the current picture. That is, IBC can use at least one of the inter prediction techniques described in the present disclosure.

[0067] The prediction signal generated by the prediction unit can be used to generate a restored signal or can be used to generate a residual signal. The subtraction unit 115 can subtract the prediction signal (predicted block, predicted sample array) output from the prediction unit from the input image signal (original block, original sample array) to generate a residual signal (residual signal, residual block, residual sample array). The generated residual signal can be transmitted to the conversion unit 120.

[0068] The conversion unit 120 can apply a conversion technique to the residual signal to generate transform coefficients. For example, the conversion technique can include at least one of DCT (Discrete Cosine Transform), DST (Discrete Sine Transform), KLT (Karhunen-Loeve Transform), GBT (Graph-Based Transform), or CNT (Conditionally Non-linear Transform). Here, GBT means the transform obtained from this graph when representing the relationship information between pixels with a graph. CNT means the transform obtained based on generating a prediction signal using all previously reconstructed pixels. The conversion process can also be applied to a pixel block having the same size of a square or can be applied to a non-square, variable-size block.

[0069] The quantization unit 130 can quantize the transform coefficients and transmit them to the entropy encoding unit 190. The entropy encoding unit 190 can encode the quantized signal (information regarding the quantized transform coefficients) and output it in the form of a bitstream. The information regarding the quantized transform coefficients can be called residual information. The quantization unit 130 can reorder the block-form quantized transform coefficients into a one-dimensional vector form based on the coefficient scan order, and can also generate the information regarding the quantized transform coefficients based on the quantized transform coefficients in the one-dimensional vector form.

[0070] The entropy encoding unit 190 can perform various encoding methods such as, for example, exponential Golomb, CAVLC (context-adaptive variable length coding), CABAC (context-adaptive binary arithmetic coding). In addition to the quantized transform coefficients, the entropy encoding unit 190 can also encode, together or separately, information necessary for video / image restoration (for example, values of syntax elements). The encoded information (for example, encoded video / image information) can be transmitted or stored in the form of a bitstream in units of NAL (network abstraction layer) units. The video / image information can further include information regarding various parameter sets such as an adaptive parameter set (APS), a picture parameter set (PPS), a sequence parameter set (SPS), or a video parameter set (VPS). Also, the video / image information can further include general constraint information. The signaling information, the transmitted information, and / or the syntax elements referred to in the present disclosure can be encoded through the above-described encoding procedure and included in the bitstream.

[0071] The bitstream can be transmitted via a network or stored in a digital storage medium. Here, the network can include a broadcast network and / or a communication network, etc., and the digital storage medium can include various storage media such as USB, SD, CD, DVD, Blu-ray (registered trademark), HDD, SSD, etc. A transmission unit (not shown) for transmitting and / or a storage unit (not shown) for storing the signal output from the entropy encoding unit 190 can be provided as internal / external elements of the image encoding apparatus 100, or the transmission unit can also be provided as a component of the entropy encoding unit 190.

[0072] The quantized transform coefficients output from the quantization unit 130 can be used to generate a residual signal. For example, by applying inverse quantization and inverse transformation to the quantized transform coefficients via the inverse quantization unit 140 and the inverse transformation unit 150, a residual signal (residual block or residual sample) can be restored.

[0073] The addition unit 155 can generate a reconstructed signal (reconstructed picture, reconstructed block, reconstructed sample array) by adding the restored residual signal to the prediction signal output from the inter prediction unit 180 or the intra prediction unit 185. When there is no residual for the processing target block as in the case where the skip mode is applied, the predicted block can be used as the reconstructed block. The addition unit 155 can be called a restoration unit or a reconstructed block generation unit. The generated reconstructed signal can be used for intra prediction of the next processing target block in the current picture and can also be used for inter prediction of the next picture after passing through filtering as described later.

[0074] On the other hand, LMCS (luma mapping with chroma scaling) can also be applied in the picture encoding and / or restoration process.

[0075] The filtering unit 160 can apply filtering to the restored signal to improve the subjective / objective image quality. For example, the filtering unit 160 can apply various filtering methods to the restored picture to generate a modified restored picture, and the modified restored picture can be stored in the memory 170, specifically, in the DPB of the memory 170. The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, and the like. The filtering unit 160 can generate various information related to filtering as described later in the description of each filtering method and transmit it to the entropy encoding unit 190. The information related to filtering can be encoded by the entropy encoding unit 190 and output in the form of a bitstream.

[0076] The modified restored picture transmitted to the memory 170 can be used as a reference picture by the inter prediction unit 180. When inter prediction is applied through the image encoding device 100, the image encoding device 100 can avoid prediction mismatches between the image encoding device 100 and the image decoding device, and can also improve the encoding efficiency.

[0077] The DPB in the memory 170 can store the modified restored picture for use as a reference picture by the inter prediction unit 180. The memory 170 can store the motion information of the blocks in which the motion information in the current picture has been derived (or encoded) and / or the motion information of the blocks in the already restored picture. The stored motion information can be transmitted to the inter prediction unit 180 for utilization as the motion information of the spatial neighboring blocks or the motion information of the temporal neighboring blocks. The memory 170 can store the restored samples of the restored blocks in the current picture and transmit them to the intra prediction unit 185.

[0078] Overview of Image Decoding Device

[0079] FIG. 3 is a diagram schematically showing an image decoding apparatus to which an embodiment according to the present disclosure can be applied.

[0080] As shown in FIG. 3, the image decoding apparatus 200 can be configured to include an entropy decoding unit 210, an inverse quantization unit 220, an inverse transformation unit 230, an addition unit 235, a filtering unit 240, a memory 250, an inter prediction unit 260, and an intra prediction unit 265. The inter prediction unit 260 and the intra prediction unit 265 can be collectively referred to as a "prediction unit". The inverse quantization unit 220 and the inverse transformation unit 230 can be included in a residual processing unit.

[0081] All or at least a part of a plurality of components constituting the image decoding apparatus 200 can be realized by one hardware component (e.g., a decoder or a processor) according to an embodiment. Further, the memory 170 can include a DPB and can be realized by a digital storage medium.

[0082] The image decoding apparatus 200 that has received a bitstream including video / image information can execute a process corresponding to the process performed by the image encoding apparatus 100 of FIG. 2 to restore an image. For example, the image decoding apparatus 200 can perform decoding using the processing unit applied in the image encoding apparatus. Therefore, the decoding processing unit can be, for example, a coding unit. The coding unit can be obtained by dividing a coding tree unit or a maximum coding unit. Then, the restored image signal decoded and output via the image decoding apparatus 200 can be reproduced via a reproducing apparatus (not shown).

[0083] The image decoding device 200 can receive the signal output from the image encoding device of FIG. 2 in the form of a bitstream. The received signal can be decoded via the entropy decoding unit 210. For example, the entropy decoding unit 210 can parse the bitstream to derive information (e.g., video / image information) necessary for image restoration (or picture restoration). The video / image information can further include information regarding various parameter sets such as an Adaptive Parameter Set (APS), a Picture Parameter Set (PPS), a Sequence Parameter Set (SPS), or a Video Parameter Set (VPS). Also, the video / image information can further include general constraint information. The image decoding device can further use the information regarding the parameter set and / or the general constraint information to decode the image. The signaling information, the received information, and / or the syntax elements referred to in the present disclosure can be obtained from the bitstream by being decoded via the decoding procedure. For example, the entropy decoding unit 210 can decode the information in the bitstream based on a coding method such as exponential Golomb coding, CAVLC, or CABAC, and output the values of the syntax elements necessary for image restoration and the quantized values of the transform coefficients regarding the residual. More specifically, the CABAC entropy decoding method receives the bin corresponding to each syntax element from the bitstream, determines a context model using the syntax element information to be decoded, the decoding information of the surrounding blocks and the block to be decoded, or the information of the symbols / bins decoded in the previous step, and performs arithmetic decoding of the bin by predicting the occurrence probability of the bin based on the determined context model, thereby generating a symbol corresponding to the value of each syntax element. At this time, the CABAC entropy decoding method can update the context model using the information of the decoded symbols / bins for the context model of the next symbol / bin after determining the context model.Among the information decoded by the entropy decoding unit 210, the information related to prediction is provided to the prediction units (inter prediction unit 260 and intra prediction unit 265), and the residual values that have undergone entropy decoding in the entropy decoding unit 210, that is, the quantized transform coefficients and related parameter information, can be input to the inverse quantization unit 220. Also, among the information decoded by the entropy decoding unit 210, the information related to filtering can be provided to the filtering unit 240. On the other hand, a receiving unit (not shown) that receives the signal output from the image encoding device can be further provided as an internal / external element of the image decoding device 200, or the receiving unit can be provided as a component of the entropy decoding unit 210.

[0084] On the other hand, the image decoding device according to the present disclosure can be referred to as a video / image / picture decoding device. The image decoding device can also include an information decoder (video / image / picture information decoder) and / or a sample decoder (video / image / picture sample decoder). The information decoder can include the entropy decoding unit 210, and the sample decoder can include at least one of the inverse quantization unit 220, the inverse transform unit 230, the addition unit 235, the filtering unit 240, the memory 250, the inter prediction unit 260, and the intra prediction unit 265.

[0085] In the inverse quantization unit 220, the quantized transform coefficients can be inverse quantized to output the transform coefficients. The inverse quantization unit 220 can reorder the quantized transform coefficients in a two-dimensional block format. In this case, the reordering can be performed based on the coefficient scan order performed in the image encoding device. The inverse quantization unit 220 can perform inverse quantization on the quantized transform coefficients using a quantization parameter (for example, quantization step size information) to obtain the transform coefficients.

[0086] In the inverse conversion unit 230, the conversion coefficients can be inversely converted to obtain a residual signal (residual block, residual sample array).

[0087] The prediction unit can perform prediction on the current block and generate a predicted block including predicted samples for the current block. The prediction unit can determine whether intra prediction or inter prediction is applied to the current block based on the information regarding the prediction output from the entropy decoding unit 210, and can determine a specific intra / inter prediction mode (prediction technique).

[0088] The reason that the prediction unit can generate a prediction signal based on various prediction methods (techniques) described later is the same as that described in the explanation of the prediction unit of the image encoding apparatus 100.

[0089] The intra prediction unit 265 can predict the current block by referring to samples within the current picture. The explanation for the intra prediction unit 185 can also be similarly applied to the intra prediction unit 265.

[0090] The inter prediction unit 260 can derive a predicted block for the current block based on a reference block (reference sample array) specified by a motion vector on a reference picture. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, motion information can be predicted in units of blocks, sub-blocks, or samples based on the correlation of motion information between neighboring blocks and the current block. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction direction (L0 prediction, L1 prediction, Bi prediction, etc.) information. In the case of inter prediction, neighboring blocks can include spatial neighboring blocks existing within the current picture and temporal neighboring blocks existing in the reference picture. For example, the inter prediction unit 260 can construct a motion information candidate list based on neighboring blocks and derive the motion vector and / or reference picture index of the current block based on the received candidate selection information. Inter prediction can be performed based on various prediction modes (techniques), and the information regarding the prediction can include information indicating the mode (technique) of inter prediction for the current block.

[0091] The adder 235 can generate a restored signal (restored picture, restored block, restored sample array) by adding the obtained residual signal to a predicted signal (predicted block, predicted sample array) output from a prediction unit (including the inter prediction unit 260 and / or the intra prediction unit 265). When there is no residual for the processing target block as in the case where the skip mode is applied, the predicted block can be used as the restored block. The description of the adder 155 can be similarly applied to the adder 235. The adder 235 may also be referred to as a restoration unit or a restored block generation unit. The generated restored signal can be used for intra prediction of the next processing target block within the current picture and can also be used for inter prediction of the next picture through filtering as described later.

[0092] The filtering unit 240 can apply filtering to the restored signal to improve the subjective / objective image quality. For example, the filtering unit 240 can apply various filtering methods to the restored picture to generate a modified restored picture, and the modified restored picture can be stored in the memory 250, specifically in the DPB of the memory 250. The various filtering methods can include, for example, deblocking filtering, sample adaptive offset, adaptive loop filter, bilateral filter, and the like.

[0093] The (modified) restored picture stored in the DPB of the memory 250 can be used as a reference picture in the inter prediction unit 260. The memory 250 can store the motion information of the block in which the motion information in the current picture has been derived (or decoded) and / or the motion information of the block in the already restored picture. The stored motion information can be transmitted to the inter prediction unit 260 for utilization as the motion information of the spatial neighboring blocks or the motion information of the temporal neighboring blocks. The memory 250 can store the restored samples of the restored blocks in the current picture and transmit them to the intra prediction unit 265.

[0094] In this specification, the embodiments described in the filtering unit 160, the inter prediction unit 180, and the intra prediction unit 185 of the image encoding device 100 can also be applied to the filtering unit 240, the inter prediction unit 260, and the intra prediction unit 265 of the image decoding device 200 in the same or corresponding manner.

[0095] Inter Prediction

[0096] The prediction units of the image encoding device 100 and the image decoding device 200 can derive prediction samples by performing inter prediction in block units. The inter prediction can be a prediction derived in a method that depends on data elements (such as sample values or motion information) of pictures other than the current picture. When inter prediction is applied to the current block, a predicted block (prediction sample array) for the current block can be derived based on a reference block (reference sample array) specified by a motion vector on a reference picture pointed to by a reference picture index. At this time, in order to reduce the amount of motion information transmitted in the inter prediction mode, based on the correlation of the motion information between the peripheral block and the current block, the motion information of the current block can be predicted in block, sub-block, or sample units. The motion information can include a motion vector and a reference picture index. The motion information can further include inter prediction type (L0 prediction, L1 prediction, Bi prediction, etc.) information. When inter prediction is applied, the peripheral block can include a spatial neighboring block existing in the current picture and a temporal neighboring block existing in the reference picture. The reference picture including the reference block and the reference picture including the temporal neighboring block may be the same as or different from each other. The temporal neighboring block can be called by names such as a collocated reference block or a collocated CU (colCU), and the reference picture including the temporal neighboring block can also be called a collocated picture (colPic). For example, a motion information candidate list can be configured based on the peripheral blocks of the current block, and flag or index information indicating which candidate is selected (used) to derive the motion vector and / or reference picture index of the current block can be signaled.Inter-prediction can be performed based on various prediction modes. For example, in the case of skip mode and merge mode, the motion information of the current block may be the same as that of the selected neighboring block. In the case of skip mode, different from merge mode, the residual signal may not be transmitted. In the case of motion vector prediction (MVP) mode, the motion vector of the selected neighboring block can be used as a motion vector predictor, and the motion vector difference can be signaled. In this case, the motion vector of the current block can be derived using the sum of the motion vector predictor and the motion vector difference.

[0097] The motion information can include L0 motion information and / or L1 motion information according to the inter prediction type (such as L0 prediction, L1 prediction, Bi prediction, etc.). The motion vector in the L0 direction can be called the L0 motion vector or MVL0, and the motion vector in the L1 direction can be called the L1 motion vector or MVL1. The prediction based on the L0 motion vector can be called L0 prediction, the prediction based on the L1 motion vector can be called L1 prediction, and the prediction based on both the L0 motion vector and the L1 motion vector can be called dual (Bi) prediction. Here, the L0 motion vector can represent a motion vector related to the reference picture list L0 (L0), and the L1 motion vector can represent a motion vector related to the reference picture list L1 (L1). The reference picture list L0 can include previous pictures as reference pictures in the output order earlier than the current picture, and the reference picture list L1 can include later pictures in the output order than the current picture. The previous picture can be called a forward (reference) picture, and the later picture can be called a backward (reference) picture. The reference picture list L0 can further include later pictures as reference pictures in the output order later than the current picture. In this case, the previous picture can be indexed first within the reference picture list L0, and the later picture can be indexed next. The reference picture list L1 can further include previous pictures as reference pictures in the output order earlier than the current picture. In this case, the later picture can be indexed first within the reference picture list 1, and the previous picture can be indexed next. Here, the output order can correspond to the POC (picture order count) order.

[0098] FIG. 4 is a diagram schematically showing the inter prediction unit (180) of the image decoding apparatus 100, and FIG. 5 is a flowchart showing a method of encoding an image based on inter prediction.

[0099] The image encoding device 100 can perform inter prediction on the current block (S510). The image encoding device 100 can derive an inter prediction mode and motion information for the current block and generate a prediction sample for the current block. Here, the procedures for determining the inter prediction mode, deriving the motion information, and generating the prediction sample may be performed simultaneously, or any one of the procedures may be performed prior to the other procedures. For example, the inter prediction unit 180 of the image encoding device 100 can include a prediction mode determination unit 181, a motion information derivation unit 182, and a prediction sample derivation unit 183. The prediction mode determination unit 181 can determine a prediction mode for the current block, the motion information derivation unit 182 can derive the motion information for the current block, and the prediction sample derivation unit 183 can derive the prediction sample for the current block. For example, the inter prediction unit 180 of the image encoding device 100 can search for a block similar to the current block within a certain region (search region) of a reference picture via motion estimation and derive a reference block whose difference from the current block is the minimum or below a certain criterion. Based on this, a reference picture index indicating the reference picture in which the reference block is located can be derived, and a motion vector can be derived based on the positional difference between the reference block and the current block. The image encoding device 100 can determine a mode to be applied to the current block among various prediction modes. The image encoding device 100 can compare the RD costs for the various prediction modes and determine an optimal prediction mode for the current block.

[0100] For example, when the skip mode or the merge mode is applied to the current block, the image encoding device 100 constructs a merge candidate list described later, and among the reference blocks pointed to by the merge candidates included in the merge candidate list, a reference block whose difference from the current block is the smallest or equal to a certain criterion can be derived. In this case, a merge candidate related to the derived reference block can be selected, and merge index information indicating the selected merge candidate can be generated and signaled to the decoding device. The motion information of the current block can be derived using the motion information of the selected merge candidate.

[0101] As another example, when the (A)MVP mode is applied to the current block, the image encoding device 100 constructs an (A)MVP candidate list described later, and among the mvp (motion vector predictor) candidates included in the (A)MVP candidate list, the motion vector of the selected mvp candidate can be used as the mvp of the current block. In this case, for example, the motion vector pointing to the reference block derived by the above-described motion estimation can be used as the motion vector of the current block, and among the mvp candidates, the mvp candidate having the motion vector with the smallest difference from the motion vector of the current block can be the selected mvp candidate. An MVD (motion vector difference), which is the difference obtained by subtracting the mvp from the motion vector of the current block, can be derived. In this case, information regarding the MVD can be signaled to the image decoding device 200. Also, when the (A)MVP mode is applied, the value of the reference picture index can be configured with reference picture index information and signaled separately to the image decoding device 200.

[0102] The image encoding device 100 can derive a residual sample based on the prediction sample (S520). The image encoding device 100 can derive the residual sample by comparing the original sample of the current block with the prediction sample.

[0103] The image encoding device 100 can encode image information including prediction information and residual information (S530). The image encoding device 100 can output the encoded image information in the form of a bitstream. The prediction information can include, as information related to the prediction procedure, prediction mode information (e.g., skip flag, merge flag, or mode index, etc.) and information related to motion information. The information related to the motion information can include candidate selection information (e.g., merge index, mvp flag, or mvp index) which is information for deriving a motion vector. Also, the information related to the motion information can include information related to the above-mentioned MVD and / or reference picture index information. Also, the information related to the motion information can include information indicating whether L0 prediction, L1 prediction, or bi-prediction is applied. The residual information is information related to the residual samples. The residual information can include information related to the quantized transform coefficients for the residual samples.

[0104] The output bitstream may be stored in a (digital) storage medium and transmitted to the image decoding device 200, or may be transmitted to the image decoding device 200 via a network.

[0105] On the other hand, as described above, the image encoding device 100 can generate a reconstructed picture (including reconstructed samples and reconstructed blocks) based on the reference samples and the residual samples. This is because the same prediction result as that performed by the image decoding device 200 is derived by the image encoding device 100, and thus the coding efficiency can be improved. Therefore, the image encoding device 100 can store the reconstructed picture (or reconstructed samples, reconstructed blocks) in the memory and utilize it as a reference picture for inter prediction. As described above, loop filter procedures and the like can be further applied to the reconstructed picture.

[0106] FIG. 6 is a diagram schematically showing an inter prediction unit 260 of the image decoding apparatus 200, and FIG. 7 is a flowchart showing a method of decoding an image based on inter prediction.

[0107] The image decoding apparatus 200 can perform operations corresponding to the operations performed by the image encoding apparatus 100. The image decoding apparatus 200 can perform prediction on the current block based on the received prediction information and derive a prediction sample.

[0108] Specifically, the image decoding apparatus 200 can determine a prediction mode for the current block based on the received prediction information (S710). The image decoding apparatus 200 can determine which inter prediction mode is applied to the current block based on the prediction mode information in the prediction information.

[0109] For example, based on the merge flag, it can be determined whether the merge mode is applied to the current block or whether the (A)MVP mode is determined. Alternatively, one of various inter prediction mode candidates can be selected based on the mode index. The inter prediction mode candidates can include a skip mode, a merge mode, and / or the (A)MVP mode, or can include various inter prediction modes described later.

[0110] The image decoding device 200 can derive the motion information of the current block based on the determined inter prediction mode (S720). For example, when the skip mode or the merge mode is applied to the current block, the image decoding device 200 can construct a merge candidate list described later and select one merge candidate from among the merge candidates included in the merge candidate list. The selection can be made based on the selection information (merge index) described above. The motion information of the current block can be derived using the motion information of the selected merge candidate. The motion information of the selected merge candidate can be used as the motion information of the current block.

[0111] As another example, when the (A)MVP mode is applied to the current block, the image decoding device 200 can construct an (A)MVP candidate list described later, and among the mvp (motion vector predictor) candidates included in the (A)MVP candidate list, the motion vector of the selected mvp candidate can be used as the mvp of the current block. The selection can be made based on the selection information (mvp flag or mvp index) described above. In this case, the MVD of the current block can be derived based on the information regarding the MVD, and the motion vector of the current block can be derived based on the mvp and the MVD of the current block. Also, the reference picture index of the current block can be derived based on the reference picture index information. The picture pointed to by the reference picture index within the reference picture list regarding the current block can be derived as the reference picture to be referred to for the inter prediction of the current block.

[0112] On the other hand, as will be described later, the motion information of the current block can be derived without constructing a candidate list. In this case, the motion information of the current block can be derived according to the procedure disclosed in the prediction mode described later. In this case, the construction of the candidate list as described above can be omitted.

[0113] The image decoding device 200 can generate a prediction sample for the current block based on the motion information of the current block (S730). In this case, the reference picture is derived based on the reference picture index of the current block, and the prediction sample of the current block can be derived using the samples of the reference block pointed to by the motion vector of the current block on the reference picture. In this case, as will be described later, in some cases, a prediction sample filtering procedure for all or part of the prediction sample of the current block can be further performed.

[0114] For example, the inter prediction unit 260 of the image decoding device 200 can include a prediction mode determination unit 261, a motion information derivation unit 262, and a prediction sample derivation unit 263. The prediction mode determination unit determines the prediction mode for the current block based on the prediction mode information received by the prediction mode determination unit 181, and the motion information derivation unit derives the motion information (such as a motion vector and / or a reference picture index) of the current block based on the information related to the motion information received by the motion information derivation unit 182, and the prediction sample derivation unit 183 can derive the prediction sample of the current block.

[0115] The image decoding device 200 can generate a residual sample for the current block based on the received residual information (S740). The image decoding device 200 can generate a restored sample for the current block based on the prediction sample and the residual sample, and generate a restored picture based on this (S750). As described above, an in-loop filtering procedure and the like can be further applied to the restored picture hereafter.

[0116] Referring to FIG. 8, as described above, the inter prediction procedure may include an inter prediction mode determination step (S810), a motion information derivation step (S820) according to the determined prediction mode, and a prediction execution (prediction sample generation) step (S830) based on the derived motion information. The inter prediction procedure can be performed by the image encoding device 100 and the image decoding device (200) as described above.

[0117] Determination of Inter Prediction Mode

[0118] Various inter prediction modes can be used to predict the current block within a picture. For example, various modes such as the merge mode, skip mode, MVP (motion vector prediction) mode, Affine mode, sub-block merge mode, MVVD (merge with MVD) mode, etc. can be used. DMVR (Decoder side motion vector refinement) mode, AMVR (adaptive motion vector resolution) mode, Bi-prediction with CU-level weight (BCW), Bi-directional optical flow (BDOF), etc. can be used additionally or alternatively as associated modes. The Affine mode may also be called the affine motion prediction mode. The MVP mode may also be called the AMVP (advanced motion vector prediction) mode. In this document, some modes and / or motion information candidates derived by some modes may also be included as one of the motion information related candidates of other modes. For example, the HMVP candidate may be added as a merge candidate in the merge / skip mode, or may be added as an mvp candidate in the MVP mode.

[0119] Prediction mode information indicating the inter prediction mode of the current block can be signaled from the image encoding device 100 to the image decoding device 200. The prediction mode information can be included in the bitstream and received by the image decoding device 200. The prediction mode information can include index information indicating one of a number of candidate modes. Alternatively, the inter prediction mode can also be indicated via hierarchical signaling of flag information. In this case, the prediction mode information can include one or more flags. For example, a skip flag is signaled to indicate whether the skip mode is applied. If the skip mode is not applied, a merge flag is signaled to indicate whether the merge mode is applied. If the merge mode is not applied, it can be indicated that the MVP mode is applied, or flags for further classification can also be signaled. The affine mode may be signaled as an independent mode, or may be signaled as a dependent mode such as the merge mode or the MVP mode. For example, the affine mode can include an affine merge mode and an affine MVP mode.

[0120] Derivation of Motion Information

[0121] Inter-prediction can now be performed using the motion information of the current block. The image encoding device 100 can derive the optimal motion information for the current block through a motion estimation procedure. For example, the image encoding device 100 can search for a highly correlated similar reference block within a predetermined search range in the reference picture in units of fractional pixels using the original block in the original picture for the current block, and thereby derive the motion information. The similarity of the blocks can be derived based on the difference in phase-based sample values. For example, the similarity of the blocks can be calculated based on the SAD between the current block (or the template of the current block) and the reference block (or the template of the reference block). In this case, the motion information can be derived based on the reference block with the smallest SAD in the search area. The derived motion information can be signaled to the image decoding device 200 according to various methods based on the inter-prediction mode.

[0122] Generation of Prediction Sample

[0123] Based on the motion information derived according to the prediction mode, a predicted block for the current block can be derived. The predicted block can include prediction samples (prediction sample arrays) of the current block. When the motion vector of the current block indicates a fractional sample unit, an interpolation procedure can be performed, whereby the prediction samples of the current block can be derived based on reference samples in fractional sample units within the reference picture. When Affine inter-prediction is applied to the current block, prediction samples can be generated based on sample / sub-block unit MVs. When dual prediction is applied, the prediction samples derived through the weighted sum or weighted average (by phase) of the prediction samples derived based on L0 prediction (i.e., prediction using the reference picture within reference picture list L0 and MVL0) and the prediction samples derived based on L1 prediction (i.e., prediction using the reference picture within reference picture list L1 and MVL1) can be used as the prediction samples of the current block. When dual prediction is applied and the reference picture used for L0 prediction and the reference picture used for L1 prediction are located in different temporal directions with respect to the current picture (i.e., when it corresponds to bidirectional prediction while being dual prediction), this can be referred to as true dual prediction.

[0124] Based on the derived prediction samples, restored samples and a restored picture can be generated, and then procedures such as in-loop filtering can be performed, as described above.

[0125] Template matching(TM)

[0126] FIG. 9 is a diagram for explaining an encoding / decoding method of a template matching base according to the present disclosure.

[0127] Template Matching (TM) is a method for deriving motion vectors at the decoder end. By finding the template in the reference picture that is most similar to the template (hereinafter referred to as the "current template") adjacent to the current block (e.g., current coding unit, current CU), the motion information of the current block can be refined. The current template may be the upper adjacent block and / or the left adjacent block of the current block, or a part of these adjacent blocks. Also, the reference template can be determined to be the same size as the current template.

[0128] As shown in FIG. 9, once the initial motion vector of the current block is derived, a search for a better motion vector can be performed in the surrounding area of the initial motion vector. For example, the range of the surrounding area where the search is performed can be within the [-8, +8]-pel search area centered on the initial motion vector. Also, the size of the search step for performing the search can be determined based on the AMVR mode of the current block. Also, template matching may be performed continuously with the bilateral matching process in the merge mode.

[0129] When the prediction mode of the current block is the AMVP mode, the motion vector predictor candidate (MVP candidate) can be determined based on the template matching error. For example, the motion vector predictor candidate (MVP candidate) that minimizes the error between the current template and the reference template can be selected. Then, template matching for improving the motion vector can be performed on the selected motion vector predictor candidate. At this time, template matching for improving the motion vector may not be performed on the motion vector predictor candidates that are not selected.

[0130] More specifically, the improvement for the selected motion vector predictor candidate can start from full-pel (integer-pel) accuracy within the [-8, +8]-pel search region using iterative diamond search. Or, in the case of the 4-pel AMVR mode, it can start from 4-pel accuracy. Thereafter, depending on the AMVR mode, half-pel and / or quarter-pel accuracy search can continue. According to the search process, the motion vector predictor candidate can maintain the same motion vector accuracy as that indicated by the AMVR mode even after the template matching process. In the iterative search process, if the difference between the previous minimum cost and the current minimum cost is smaller than any threshold value, the search process ends. The threshold value can be the same as the area of the block, i.e., the number of samples within the block. Table 1 is an illustration of search patterns for the AMVR mode and the merge mode with AMVR.

[0131]

Table 1

[0132] When the prediction mode of the current block is the merge mode, a similar search method can be applied to the merge candidates indicated by the merge index. As shown in Table 1 above, template matching can be performed up to 1 / 8-pel accuracy, or skipping can be performed for accuracies below half-pel, which can be determined dependently based on whether an Alternative Interpolation Filter is used based on the merge motion information. At this time, the alternative interpolation filter may be a filter used when the AMVR is in the half-pel mode. Also, when template matching is available, depending on whether bilateral matching (BM) is available, the template matching can operate as an independent process, or can operate as an additional motion vector improvement process between block-based bilateral matching and sub-block-based bilateral matching. The availability of the template matching and / or the availability of the bilateral matching can be determined by checking the availability conditions. The accuracy of the motion vector in the above can mean the accuracy of the motion vector difference (MVD).

[0133] Hereinafter, an image encoding / decoding method according to various embodiments of the present disclosure will be described in detail.

[0134] The present disclosure relates to illumination compensation, and more particularly, to an illumination compensation method and an illumination compensation candidate rearrangement method.

[0135] Example 1

[0136] The mathematical formula for compensating the luminance difference between the reference picture and the current picture according to the present disclosure is as shown in Equation 1 below.

[0137] [Equation]

[0138] In Equation 1, a and b can be parameters for luminance compensation (hereinafter referred to as "luminance compensation parameters"). x represents the signal of the reference picture, and y can represent the signal of the current picture or the signal of the original image. In Equation 1, the difference between both sides can be regarded as an error, and the image encoding device 100 or the image decoding device 200 can obtain luminance compensation parameters a and / or b that can minimize this error.

[0139] The luminance compensation parameters can be generated based on various units such as pictures, tiles, slices, CTUs, CUs, sub-CUs, etc. Hereinafter, for convenience of explanation, it will be described based on CUs.

[0140] There can be the following three methods for obtaining the luminance compensation parameters.

[0141] - [Method 1] The luminance compensation parameters can be calculated from the original image pixels of the current CU to which luminance compensation is to be applied and the pixel information of the CU in the reference picture.

[0142] - [Method 2] The luminance compensation parameters can be calculated from the surrounding pixel information of the current CU and the reference picture CU to which luminance compensation is to be applied.

[0143] - [Method 3] The luminance compensation parameters can be calculated by selectively combining the above Method 1 and Method 2.

[0144] In the case of the above Method 1, according to Equation 1, when y is the original image pixel corresponding to the current CU, a CU block close to the original image can be generated from the combination of the pixel information of the reference picture CU and the luminance compensation parameters a and b. Therefore, when using Method 1, it may be useful for reducing the residual signal. However, when viewed from the side of the image decoding device 200, since there is no original image, the image encoding device 100 may have to encode and transmit the compensation parameters a and b even after they are generated.

[0145] In the case of the method 2, according to Equation 1, y can be the peripheral pixels of the current CU, and x can be the peripheral pixels of the reference picture CU. Different from Method 1, since the relationship of the peripheral pixels is used to obtain the luminance compensation parameters a and b, peripheral pixels can exist even when viewed from the side of the image decoding apparatus 200. Therefore, luminance compensation information can be generated without directly encoding the luminance compensation parameters a and b.

[0146] The derived luminance compensation parameters a and b can always be applied to the CU to which luminance compensation is applied. Alternatively, one or more of the derived luminance compensation parameters a and b may not be used or may be replaced with fixed values. Or, whether to use the luminance compensation parameters a and b can be determined according to a predefined condition. When whether to use the luminance compensation parameters is determined according to a predefined condition, the calculation and implementation complexity can be reduced even if a part of the improvement in compression efficiency is abandoned. That is, when the degree of improvement in compression efficiency obtained through luminance compensation applying the luminance compensation parameters is small, or when the calculation and implementation complexity are unnecessarily increased, it can be determined not to use the luminance compensation parameters, and the calculation and implementation complexity can be reduced.

[0147] FIG. 10 is a diagram showing the positions of samples in a reference region for inducing luminance compensation parameters according to an embodiment of the present disclosure. When using the peripheral samples of the current CU and the reference CU for generating the luminance compensation parameters, the peripheral samples of the reference CU can be selected by the following various methods.

[0148] (a) All peripheral samples of the reference CU are used, but more peripheral samples than the size of the reference CU can be used (1010).

[0149] (b) All peripheral samples of the reference CU are used, but only the peripheral samples within the same range as the size of the reference CU can be used (1020).

[0150] (c) Only one or more parts of the peripheral samples of the reference CU can be used (1030).

[0151] (d) Only one or more parts of the peripheral samples of the reference CU are used, but one or more parts of the peripheral samples of the reference CU can be used in a subsampled form (1040).

[0152] (e) Only one or more parts of the peripheral samples of the reference CU are used, but one or more parts of the peripheral samples of the reference CU can be used in an irregular form (1050).

[0153] (f) Only a specific sample representing the peripheral samples of the reference CU can be used (1060). For example, only two peripheral samples are used, but this value may be the maximum and minimum values among the available pixels.

[0154] (g) The methods of (a) to (f) above can be selectively combined and used.

[0155] As a method for selecting the peripheral samples of the reference CU, the width and / or height of the block can be used as a reference. For example, the size of the largest available block is arbitrarily determined, and all peripheral samples are used up to a specific width or height. If the width or height is larger, only some samples can be used. As another example, a subsampling ratio is determined and some samples can be adaptively selected according to the size of the width or height of the block. If the subsampling ratio is 2 and the width or height of the block is 4, only two samples can be used. Or, if the subsampling ratio is 2 and the width or height of the block is 8, only four samples can be used.

[0156] As another example, any threshold can be used as a method for selecting surrounding samples of the reference CU. That is, the values of the surrounding samples of the reference CU are compared with any threshold, and only the samples corresponding to the case where the sample value is greater than or less than the threshold can be used. The method for selecting the surrounding samples of the reference CU mentioned in the present disclosure can be selectively combined.

[0157] The above-mentioned subsampling ratio or any threshold can be defined in advance commonly in the image encoding device 100 and / or the image decoding device 200. Alternatively, the subsampling ratio or any threshold can be transmitted in units of specific headers (such as SPS, PPS, Picture Header, Slice Header, etc.).

[0158] FIG. 11 is a diagram showing the positions of sample lines in a reference region for deriving a luminance compensation parameter according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the position or number of sample lines in the surrounding region of the reference CU for deriving the luminance compensation parameter can be various.

[0159] (a) The surrounding samples of the reference CU for deriving the luminance compensation parameter can be selected from a region composed of 1-line samples adjacent to the reference CU (1110).

[0160] (b) The surrounding samples of the reference CU for deriving the luminance compensation parameter can be selected from a region composed of 1-line samples not adjacent to the reference CU (1120).

[0161] (c) The surrounding samples of the reference CU for deriving the luminance compensation parameter can be selected from a region composed of any multi-line samples adjacent to the reference CU (1130).

[0162] (d) The methods (a) to (c) can be selectively used.

[0163] According to an embodiment of the present disclosure, as a method for selecting a region to which a peripheral sample belongs, the width and / or height of a block can be used as a reference. For example, the minimum number of samples required to obtain a luminance compensation parameter can be arbitrarily determined. In this case, for a CU having a specific width or height, the number of samples can be ensured only in a 1-line sample region. At this time, 1-line can mean adjacent or non-adjacent peripheral sample lines. For a CU smaller than a specific width or height, peripheral samples included in a multi-line sample region can be used.

[0164] As another example, an arbitrary threshold can be used as a method for selecting a peripheral sample. That is, by comparing the value of a peripheral sample with an arbitrary threshold, only the peripheral sample can be used when the peripheral sample value is greater than or less than the threshold. In this case, when the required number of samples is satisfied in an adjacent or non-adjacent 1-line sample region, samples are selected only in the 1-line sample region, and otherwise, luminance compensation parameters can be obtained using samples included in a multi-line sample region. The method for selecting a sample region for selecting a peripheral sample of a reference CU referred to in the present disclosure can be selectively combined.

[0165] Information for selecting a sample region can be defined in advance in common by the image encoding device 100 and / or the image decoding device 200. Alternatively, information for selecting a sample region can be transmitted in units of a specific header (such as SPS, PPS, Picture Header, Slice Header, etc.).

[0166] Example 2

[0167] According to an embodiment of the present disclosure, a plurality of modes can be used to compensate for the luminance change of a reference picture. Also, the reference samples used to derive the plurality of modes can be selected based on position. The luminance compensation parameter can be generated based on various units such as a picture, a tile, a slice, a CTU, a CU, a sub-CU, etc. However, for the sake of convenience of explanation, the present disclosure will be described based on a CU.

[0168] The degree of luminance compensation can depend on the luminance compensation parameters a and b. That is, when one parameter set (a, b) is derived through samples around the current CU and its corresponding reference CU, the luminance compensation can be composed of only one form. If a plurality of luminance compensation parameter sets are derived from the surrounding samples, the luminance compensation may be performed in various forms.

[0169] FIG. 12 is a diagram showing luminance compensation candidate modes according to an embodiment of the present disclosure. According to an embodiment of the present disclosure, the present disclosure can divide the surrounding samples into n sets for deriving luminance compensation parameters for generating a plurality of luminance compensation candidate modes. FIG. 12 shows a method of generating a plurality of luminance compensation candidate modes based on the positions of the surrounding samples. The luminance compensation candidate mode (a) 1210 can use all of the surrounding samples of the current CU and the reference CU as a sample region. The luminance compensation candidate mode (b) 1220 can use only the left surrounding samples as a sample region. The luminance compensation candidate mode (c) 1230 can use only the upper surrounding samples as a sample region. FIG. 12 is an example for generating a number of luminance compensation candidate modes, and the luminance compensation parameters can be variously derived using the method described in Example 1.

[0170] Example 3

[0171] When using multiple luminance compensation candidate modes, the present disclosure can define the number of luminance compensation candidate modes to be used. The multiple luminance compensation candidate modes are not limited to three as in Embodiment 2, and the number of luminance compensation candidate modes can be determined to be the same N in the image encoding device 100 and / or the image decoding device 200. Here, N can be a natural number. Alternatively, the number of the maximum possible luminance compensation candidate modes at a higher-level syntax can be transmitted.

[0172] According to an embodiment of the present disclosure, a flag indicating whether luminance compensation is available can be transmitted in the SPS, and a syntax indicating the number of the maximum possible luminance compensation candidate modes can be transmitted. Table 2 is a syntax showing a method of transmitting a flag indicating whether luminance compensation is available and a syntax indicating the number of the maximum possible luminance compensation candidate modes.

[0173]

Table 2

[0174] In the syntax of Table 2, sps_illumination_compensation_enabled_flag can indicate whether luminance compensation can be performed on a CU for which inter prediction has been performed. When the value of sps_illumination_compensation_enabled_flag is 1, it can indicate that luminance compensation can be performed on a CU inter-predicted for CLVS. When the value of sps_illumination_compensation_enabled_flag is 0, it can indicate that luminance compensation cannot be performed on a CU inter-predicted for CLVS.

[0175] In the syntax of Table 1, sps_max_num_ic_cand can indicate the maximum number of luminance compensation candidates supported by the SPS. Table 2 shows the method of transmission in the SPS, and the number of luminance compensation candidate modes can be transmitted at various header (such as VPS, PPS, Picture Header, Slice Header, etc.) positions other than the SPS.

[0176] According to another embodiment of the present disclosure, the number of luminance compensation candidate modes can be determined according to additional criteria in the same reference unit. As additional criteria, various characteristics such as the size of the block (such as the width / height of the block), prediction type, partition type, transform type, etc. can be used. For example, when using the block size as an additional criterion, the number of luminance compensation candidate modes can be defined according to the same criterion determined in advance by the image encoding device 100 and / or the image decoding device 200. Or, the number of luminance compensation candidate modes can be defined according to whether it is a square block in the form of N×N or a non-square block in the form of M×N. Or, the number of luminance compensation candidate modes can be defined according to the number of samples included in the block. The number of luminance compensation candidate modes according to additional criteria is not limited to the above-exemplified cases, and can be variously defined by other combinations.

[0177] Example 4

[0178] This embodiment relates to an index transmission method when using a large number of luminance compensation candidate modes. According to an embodiment of the present disclosure, a flag indicating whether luminance compensation is used can be transmitted in CU units. The following Table 3 shows the method of transmitting an index when luminance compensation is used.

[0179]

Table 3

[0180] The cu_ic_flag is a flag indicating whether luminance compensation is currently applied to the CU. When the value of cu_ic_flag is 1, it can indicate that luminance compensation is currently applied to the CU. The cu_ic_idx may be an index indicating a luminance compensation candidate mode in a luminance compensation list. The cu_ic_idx can be defined as shown in Table 4 below.

[0181]

Table 4

[0182] When cu_ic_idx is 0, the luminance compensation parameter can be derived using both the upper sample and the left sample. When cu_ic_idx is 1, the luminance compensation parameter can be derived using only the upper sample. When cu_ic_idx is 2, the luminance compensation parameter can be derived using only the left sample.

[0183] According to another embodiment of the present disclosure, information on whether luminance compensation is applied to the CU and index information can also be integrated and transmitted in one syntax. In this case, one of the index entries indicates that no luminance compensation is applied, and the remaining index entries can indicate a specific mode among multiple luminance compensation candidate modes. At this time, the entry indicating that no luminance compensation is applied may be the first (0th) entry. Table 5 below shows a method of integrating and transmitting information on whether luminance compensation is applied and index information in one syntax.

[0184]

Table 5

[0185] The meaning of cu_ic_idx in Table 5 can be defined as shown in Table 6 below.

[0186]

Table 6

[0187] The above example for the index transmission of the luminance compensation candidate mode may also be an example for the case of using three luminance compensation candidate modes as in Embodiment 2. When the number of maximum luminance compensation candidate modes is defined by Embodiment 3, the index can be transmitted according to the number. That is, the cu_ic_idx value may be the same as or smaller than sps_max_num_ic_cand. The index transmission of the luminance compensation candidate mode according to the present disclosure is not limited to the above-described examples and can be transmitted in other forms.

[0188] Example 5

[0189] This embodiment relates to a method for constructing an index candidate list when using a large number of luminance compensation candidate modes. Embodiment 4 relates to the case where the positions of the luminance compensation candidate modes for deriving luminance compensation parameters according to the index positions are predefined and fixedly used. However, in the present disclosure, the luminance compensation candidate modes can be rearranged by a template cost.

[0190] FIG. 13 is a flowchart showing a method for rearranging luminance compensation candidate modes according to an embodiment of the present disclosure. Hereinafter, the image encoding apparatus 100 will be mainly described, but it is obvious that the image decoding apparatus 200 can also be the main body.

[0191] The image encoding apparatus 100 can derive parameters of luminance compensation candidates (S1310). That is, the image encoding apparatus 100 can derive luminance compensation parameters for each luminance compensation candidate mode. When using {IC_TOP_LEFT, IC_TOP, IC_LEFT} as the luminance compensation candidate modes, the luminance compensation parameters {a, b} can be derived using the samples at the positions.

[0192] The image encoding device 100 can apply luminance compensation to the template area (S1320). That is, the luminance compensation parameter can be applied to the template area of the reference block. Here, the template area can be a predetermined adjacent area adjacent to the reference block. Thereafter, the image encoding device 100 can calculate the template cost (S1330). Specifically, the image encoding device 100 can calculate the luminance difference (template cost or template error) between the template area of the current block and the template area of the reference block to which luminance compensation has been applied. At this time, SAD (Sum of Absolute Difference), SATD (Sum of Absolute Transformed Difference), or SSD (Sum of Squared Difference) can be used to obtain the template cost.

[0193] FIG. 14 is a diagram showing a template position for calculating a template cost according to an embodiment of the present disclosure. The template 1410 of the current CU is the left and upper samples adjacent to the current CU, and the template 1420 of the reference block can be the left and upper samples adjacent to the reference block.

[0194] The image encoding device 100 can rearrange the luminance compensation candidates (S1340). Specifically, the image encoding device 100 can rearrange the luminance compensation candidate modes based on the template cost. Here, rearrangement can mean arranging in ascending order from the smallest template cost to the largest. The luminance compensation candidate list can be configured in the order of luminance compensation candidate modes with a high possibility of having luminance compensation applied through rearrangement based on the template cost. After performing the luminance compensation candidate rearrangement, the image encoding device 100 can transmit an index indicating the luminance compensation candidate mode as in Embodiment 4. At this time, the index indicating the luminance compensation candidate mode can be transmitted for as many of the entire luminance compensation candidate modes as possible, or can also be transmitted only for some candidates to reduce the flag transmission bits.

[0195] The method for transmitting the syntax indicating the luminance compensation index according to the present disclosure is as shown in Table 7 below.

[0196]

Table 7

[0197] In Table 7, when cu_ic_flag is 1, it indicates that luminance compensation can be applied to the current CU. When cu_ic_idx_flag is 0, it can indicate that the first luminance compensation candidate mode in the rearranged luminance compensation candidate list is used. When cu_ic_idx_flag is 1, it can indicate that the second luminance compensation candidate mode in the rearranged luminance compensation candidate list is used. The meaning of cu_ic_idx_flag can be defined as shown in Table 8 below.

[0198]

Table 8

[0199] When cu_ic_idx_flag is 0, the first candidate in the rearranged luminance compensation candidate list can be selected. When cu_ic_idx_flag is 1, the second candidate in the rearranged luminance compensation candidate list can be selected. The index transmission of the luminance compensation candidate mode according to the present disclosure is not limited to the above-mentioned examples only, and the range of the transmitted index value can be one of the same or fewer numbers than the maximum luminance compensation candidate mode.

[0200] Example 6

[0201] This embodiment relates to a method for inducing a luminance compensation candidate mode when the prediction mode of the current block is the merge mode. When the current CU is in the merge mode, the luminance compensation candidate mode information of the selected merge candidate can be used as it is. At this time, the luminance compensation candidate mode information can indicate whether the luminance compensation candidate mode is applied and the luminance compensation parameter derivation position information. Here, the position information can indicate the positions of peripheral samples necessary when calculating the luminance compensation parameters. The luminance compensation parameters of the current CU can be directly calculated from the peripheral samples of the current block. Or, the luminance compensation parameters of the current CU can also use the luminance compensation parameters of adjacent blocks.

[0202] According to another embodiment of the present disclosure, when the current CU is in the merge mode, only whether the luminance compensation candidate mode is applied can be induced from the selected merge candidate. When using luminance compensation, the image encoding device 100 and / or the image decoding device 200 can induce a plurality of luminance compensation candidate modes as in Embodiment 5 and rearrange the induced luminance compensation candidate list. By rearranging the luminance compensation candidate list, luminance compensation can be performed using the candidate having the minimum template cost.

[0203] According to another embodiment of the present disclosure, when the current CU is in the merge mode, depending on the luminance compensation candidate mode information of the selected merge candidate and additional criteria of the block, it can be adaptively determined whether to use the luminance compensation candidate mode information. The additional criteria can include various characteristics such as the size of the block, such as the width and / or height of the block, the prediction type, the split type, the transform type, etc. Or, whether to apply luminance compensation can be determined according to the template cost before and after luminance compensation.

[0204] FIG. 15 is a flowchart showing a method for applying adaptive luminance compensation according to an embodiment of the present disclosure. Hereinafter, FIG. 15 will be described mainly with respect to the image encoding device 100, but it is obvious that the image decoding device 200 can also be the main subject. The image encoding device 100 can calculate the template cost C1 before luminance compensation (S1510). Specifically, the image encoding device 100 can calculate the template difference (template cost) C1 between the template composed of the surrounding samples of the current CU and the template composed of the surrounding samples of the reference block. Here, the template of the reference block may be in a state where luminance compensation is not applied.

[0205] The image encoding device 100 can apply luminance compensation to the template area (S1520). Specifically, the image encoding device 100 can obtain information regarding luminance compensation from the selected merge mode. Thereafter, the image encoding device 100 can obtain luminance compensation parameters using the samples at the positions indicated by the information regarding luminance compensation.

[0206] The image encoding device 100 can apply luminance compensation to the template area (S1530). That is, the luminance compensation parameters can be applied to the template of the reference block. Thereafter, the image encoding device 100 can calculate the template cost C2 after luminance compensation (S1540). Specifically, the image encoding device 100 can calculate the template cost C2 between the template composed of the surrounding samples of the current CU and the template composed of the surrounding samples of the reference block to which luminance compensation has been applied.

[0207] The image encoding device 100 can determine whether to apply luminance compensation (S1550). Specifically, the image encoding device 100 can compare the difference between C1 and C2 and determine whether to apply luminance compensation based on a specific threshold. For example, when the value of C2 / C1 is greater than 0.9, luminance compensation can be applied. The present disclosure is not limited to the above-described examples, and any real number value between 0 and 10 can be used as the threshold for determining whether to apply luminance compensation.

[0208] FIG. 16 is a flowchart showing a luminance compensation list rearrangement method in an image encoding apparatus according to an embodiment of the present invention. The image encoding apparatus 100 can encode information related to luminance compensation for a current block (S1610). At this time, the information related to luminance compensation can include at least one of information indicating whether luminance compensation is executable for the current block, information indicating whether luminance compensation is applied to the current block, or an index indicating one of luminance compensation candidate modes. For example, the information indicating whether luminance compensation is executable for the current block may be sps_illumotion_compensation_enable_flag. The information indicating whether the luminance compensation is applied to the current block may be cu_ic_flag. The index may be cu_ic_idx_flag.

[0209] The image encoding apparatus 100 can derive luminance compensation parameters for each of two or more luminance compensation candidate modes for luminance compensation (S1620). Here, the luminance compensation mode can include at least one of a first mode that uses both the left adjacent region and the upper adjacent region of the current block and the reference block, a second mode that uses only the left adjacent region of the current block and the reference block, or a third mode that uses only the upper adjacent region of the current block and the reference block.

[0210] The image encoding apparatus 100 can apply luminance compensation to a predetermined first adjacent region adjacent to the reference block of the current block to obtain a luminance-compensated first adjacent region (S1630). Specifically, the image encoding apparatus 100 can obtain a luminance-compensated first adjacent region by applying luminance compensation to the first adjacent region based on the luminance compensation parameters for each of the luminance compensation candidate modes. Here, the first adjacent region may be a template region of the reference block.

[0211] The image encoding device 100 can calculate the final error value for each of the luminance compensation candidate modes (S1640). Specifically, the image encoding device 100 can obtain the final error value for each of the luminance compensation candidate modes based on the error between the sample value in the luminance-compensated first adjacent region and the sample value in a predetermined second adjacent region adjacent to the current block. Here, the second adjacent region can be the template region of the current block. Also, the first adjacent region and the second adjacent region can be corresponding positions. The final error value can be the template cost.

[0212] The image encoding device 100 can rearrange the luminance compensation candidate modes (S1650). Specifically, the luminance compensation candidate modes can be rearranged based on the final error value for each of the luminance compensation candidate modes. For example, the rearrangement of the luminance compensation candidate modes can be a process of arranging the final error values in ascending order.

[0213] The image encoding device 100 can determine the luminance compensation candidate mode for the current block (S1660). Specifically, the image encoding device 100 can determine the luminance compensation candidate mode for the current block based on the rearranged luminance compensation candidate modes and the information related to luminance compensation.

[0214] According to another embodiment of the present disclosure, when the prediction mode of the current block is the merge mode, the information related to luminance compensation can be obtained from the information related to the luminance compensation of the selected merge candidate of the current block. Here, the information related to the luminance compensation of the selected merge candidate of the current block can be obtained based on at least one of the size, prediction type, split type, or transform type of the current block.

[0215] According to another embodiment of the present disclosure, information indicating whether luminance compensation is applied to the current block can be obtained based on a first error value and a final error value for a luminance compensation candidate mode calculated based on a first adjacent region and a second adjacent region before luminance compensation. At this time, obtaining information indicating whether luminance compensation is applied to the current block can be further determined based on a threshold value. For example, obtaining information indicating whether luminance compensation is applied to the current block can be performed by comparing a difference value between the final error value and the first error value with the threshold value. When the difference value between the final error value and the first error value is greater than the threshold value, information indicating whether luminance compensation is applied to the current block may be obtained. When the difference value between the final error value and the first error value is the same as or smaller than the threshold value, information indicating whether luminance compensation is applied to the current block may not be obtained. Or, the case opposite to the above example may occur. For example, when the difference value between the final error value and the first error value is greater than the threshold value, information indicating whether luminance compensation is applied to the current block may not be obtained. When the difference value between the final error value and the first error value is the same as or smaller than the threshold value, information indicating whether luminance compensation is applied to the current block may be obtained.

[0216] FIG. 17 is a flowchart showing a method for rearranging a luminance compensation list in an image decoding apparatus according to an embodiment of the present invention. The image decoding apparatus 200 can obtain information regarding luminance compensation for the current block (S1710). At this time, the information regarding luminance compensation can include at least one of information indicating whether luminance compensation is executable for the current block, information indicating whether luminance compensation is applied to the current block, or an index indicating one of the luminance compensation candidate modes. For example, the information indicating whether luminance compensation is executable for the current block can be the sps_illumotion_compensation_enable_flag. The information indicating whether luminance compensation is applied to the current block can be the cu_ic_flag. The index can be the cu_ic_idx_flag.

[0217] The image decoding device 200 can derive luminance compensation parameters for each of two or more luminance compensation candidate modes for luminance compensation (S1720). Here, the luminance compensation mode can include at least one of a first mode that uses both the left adjacent region and the upper adjacent region of the current block and the reference block, a second mode that uses only the left adjacent region of the current block and the reference block, or a third mode that uses only the upper adjacent region of the current block and the reference block.

[0218] The image decoding device 200 can apply luminance compensation to a predetermined first adjacent region adjacent to the reference block of the current block to obtain a luminance-compensated first adjacent region (S1730). Specifically, the image decoding device 200 can obtain a luminance-compensated first adjacent region by applying luminance compensation to the first adjacent region based on the luminance compensation parameters for each of the luminance compensation candidate modes. Here, the first adjacent region can be a template region of the reference block.

[0219] The image decoding device 200 can calculate a final error value for each of the luminance compensation candidate modes (S1740). Specifically, the image decoding device 200 can obtain a final error value for each of the luminance compensation candidate modes based on the error between the sample values in the luminance-compensated first adjacent region and the sample values in a predetermined second adjacent region adjacent to the current block. Here, the second adjacent region can be a template region of the current block. Also, the first adjacent region and the second adjacent region can be corresponding positions. The final error value can be a template cost.

[0220] The image decoding device 100 can rearrange the luminance compensation candidate modes (S1750). Specifically, the luminance compensation candidate modes can be rearranged based on the final error value for each of the luminance compensation candidate modes. For example, the rearrangement of the luminance compensation candidate modes can be a process of sorting the final error values in ascending order.

[0221] The image decoding device 200 can determine a luminance compensation candidate mode for the current block (S1760). Specifically, the image decoding device 200 can determine the luminance compensation candidate mode for the current block based on the rearranged luminance compensation candidate mode and the information related to luminance compensation.

[0222] According to another embodiment of the present disclosure, when the prediction mode of the current block is the merge mode, the information related to luminance compensation can be obtained from the information related to the luminance compensation of the selected merge candidate of the current block. Here, the information related to the luminance compensation of the selected merge candidate of the current block can be obtained based on at least one of the size, prediction type, split type, or transform type of the current block.

[0223] According to another embodiment of the present disclosure, the information indicating whether luminance compensation is applied to the current block can be obtained based on a first error value and a final error value for the luminance compensation candidate mode calculated based on the first adjacent region and the second adjacent region before luminance compensation. At this time, the acquisition of the information indicating whether luminance compensation is applied to the current block can be further determined based on a threshold value. For example, the acquisition of the information indicating whether luminance compensation is applied to the current block can be performed by comparing the difference value between the final error value and the first error value with the threshold value. If the difference value between the final error value and the first error value is greater than the threshold value, the information indicating whether luminance compensation is applied to the current block may be obtained. If the difference value between the final error value and the first error value is the same as or smaller than the threshold value, the information indicating whether luminance compensation is applied to the current block may not be obtained. Or, the opposite case to the above example may occur. For example, if the difference value between the final error value and the first error value is greater than the threshold value, the information indicating whether luminance compensation is applied to the current block may not be obtained. If the difference value between the final error value and the first error value is the same as or smaller than the threshold value, the information indicating whether luminance compensation is applied to the current block may be obtained.

[0224] Exemplary methods of the present disclosure are presented in a series of operations for clarity of explanation, but this is not intended to limit the order in which the steps are performed. If necessary, each step can also be performed simultaneously or in a different order. To implement the methods according to the present disclosure, other steps can be further included in the exemplified steps, or the remaining steps can be included excluding some steps, or additional other steps can be included excluding some steps.

[0225] In the present disclosure, an image encoding device or an image decoding device that performs a predetermined operation (step) can perform an operation (step) of checking the execution conditions and situations of the operation (step). For example, when it is described that a predetermined operation is performed if a predetermined condition is satisfied, the image encoding device or the image decoding device can perform the predetermined operation after performing an operation of checking whether the predetermined condition is satisfied.

[0226] The various embodiments of the present disclosure do not list all possible combinations, but are for explaining representative aspects of the present disclosure. Matters described in the various embodiments may be applied independently or in combinations of two or more.

[0227] Also, the various embodiments of the present disclosure can be implemented by hardware, firmware, software, or a combination thereof. In the case of implementation by hardware, it can be implemented by one or more ASICs (Application Specific Integrated Circuits), DSPs (Digital Signal Processors), DSPDs (Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), general processors, controllers, microcontrollers, microprocessors, etc.

[0228] In addition, the image decoding device and the image encoding device to which the embodiments of the present disclosure are applied can be included in a multimedia broadcast transmission / reception device, a mobile communication terminal, a home cinema video device, a digital cinema video device, a surveillance camera, a video conversation device, a real-time communication device such as video communication, a mobile streaming device, a storage medium, a camcorder, an on-demand video (VoD) service providing device, an over-the-top video (OTT) device, an Internet streaming service providing device, a three-dimensional (3D) video device, an image phone video device, a medical video device, etc., and can be used to process video signals or data signals. For example, as an over-the-top video (OTT) device, it can include a game console, a Blu-ray player, an Internet-connected TV, a home theater system, a smartphone, a tablet PC, a digital video recorder (DVR), etc.

[0229] FIG. 18 is a diagram illustrating a content streaming system to which an embodiment of the present disclosure can be applied.

[0230] As shown in FIG. 18, the content streaming system to which the embodiments of the present disclosure are applied can generally include an encoding server, a streaming server, a web server, a media storage, a user device, and a multimedia input device.

[0231] The encoding server compresses the content input from a multimedia input device such as a smartphone, a camera, or a camcorder into digital data to generate a bitstream, and serves to transmit this to the streaming server. As another example, when a multimedia input device such as a smartphone, a camera, or a camcorder directly generates a bitstream, the server can be omitted.

[0232] The bitstream can be generated by an image encoding method and / or an image encoding apparatus to which the embodiments of the present disclosure are applied, and the streaming server can temporarily store the bitstream in the process of transmitting or receiving the bitstream.

[0233] The streaming server transmits multimedia data to a user device based on a user's request via a Web server, and the Web server can serve as a medium for informing the user of what services are available. When the user requests a desired service from the Web server, the Web server transmits this to the streaming server, and the streaming server can transmit multimedia data to the user. At this time, the content streaming system can include a separate control server. In this case, the control server can play a role in controlling commands / responses between each device in the content streaming system.

[0234] The streaming server can receive content from a media storage and / or an encoding server. For example, when receiving content from the encoding server, the content can be received in real time. In this case, in order to provide a smooth streaming service, the streaming server can store the bitstream for a certain period of time.

[0235] Examples of the user device may include a mobile phone, a smart phone, a laptop computer, a digital broadcast terminal, a PDA (personal digital assistants), a PMP (portable multimedia player), a navigation device, a slate PC, a tablet PC, an ultrabook, a wearable device, for example, a smartwatch, smart glass, an HMD (head mounted display), a digital TV, a desktop computer, a digital signage, and the like.

[0236] Each server in the content streaming system can be operated as a distributed server. In this case, the data received from each server can be processed distributively.

[0237] The scope of the present disclosure includes software or machine-executable commands (for example, an operating system, an application, firmware, a program, etc.) that enable operations according to the methods of various embodiments to be executed on a device or a computer, and a non-transitory computer-readable medium on which such software or commands are stored and can be executed on the device or the computer. 〔Industrial Applicability〕

[0238] Embodiments according to the present disclosure can be used to encode / decrypt images.

[0239] 〔Claims at the Time of International Application〕 〔Claim 1〕 An image decoding method performed by an image decoding apparatus, acquiring information regarding luminance compensation for a current block; inducing a luminance compensation parameter for each of two or more luminance compensation candidate modes for the luminance compensation; applying luminance compensation to a predetermined first adjacent region adjacent to a reference block of the current block based on the luminance compensation parameter to obtain a luminance-compensated first adjacent region; obtaining a final error value for each of the luminance compensation candidate modes based on an error between sample values in the luminance-compensated first adjacent region and sample values in a predetermined second adjacent region adjacent to the current block; rearranging the luminance compensation candidate modes based on the final error values; determining a luminance compensation candidate mode for the current block based on the rearranged luminance compensation candidate modes and information regarding the luminance compensation; and wherein the first adjacent region and the second adjacent region are corresponding positions, an image decoding method. [[Claim 2]] The information regarding the luminance compensation includes information indicating whether luminance compensation is executable for the current block, information indicating whether the luminance compensation is applied to the current block, or an index indicating one of the luminance compensation candidate modes, and includes at least one of them, the image decoding method according to claim 1. [[Claim 3]] The luminance compensation candidate mode includes a first mode that uses both a left adjacent region and an upper adjacent region of the current block and the reference block, a second mode that uses only a left adjacent region of the current block and the reference block, or a third mode that uses only an upper adjacent region of the current block and the reference block, and includes at least one of them, the image decoding method according to claim 1. [[Claim 4]] The rearrangement means arranging the final error values in ascending order, the image decoding method according to claim 1. [[Claim 5]] Based on the prediction mode of the current block being the merge mode, the information regarding the luminance compensation is obtained as information regarding the luminance compensation of the selected merge candidate of the current block. The image decoding method according to claim 2. 〔Claim 6〕 The information regarding the luminance compensation of the selected merge candidate of the current block is obtained based on at least one of the size, prediction type, split type, or transform type of the current block. The image decoding method according to claim 5. 〔Claim 7〕 The information indicating whether the luminance compensation is applied to the current block is obtained based on a first error value with respect to the luminance compensation candidate mode calculated based on the first adjacent region and the second adjacent region before the luminance compensation, and the final error value. The image decoding method according to claim 5. 〔Claim 8〕 The acquisition of the information indicating whether the luminance compensation is applied to the current block is further determined based on a threshold value. The image decoding apparatus according to claim 7. 〔Claim 9〕 The acquisition of the information indicating whether the luminance compensation is applied to the current block is performed by comparing a difference value between the final error value and the first error value with the threshold value. The image decoding apparatus according to claim 8. 〔Claim 10〕 An image encoding method performed by an image encoding apparatus, Encoding information regarding luminance compensation for a current block; Deriving luminance compensation parameters for each of two or more luminance compensation candidate modes for the luminance compensation; Applying luminance compensation to a predetermined first adjacent region adjacent to a reference block of the current block based on the luminance compensation parameters for each of the luminance compensation candidate modes to obtain a luminance-compensated first adjacent region; Obtaining a final error value for each of the luminance compensation candidate modes based on an error between a sample value in the luminance-compensated first adjacent region and a sample value in a predetermined second adjacent region adjacent to the current block; Rearranging the luminance compensation candidate modes based on the final error value for each of the luminance compensation candidate modes; Determining a luminance compensation candidate mode for the current block based on the rearranged luminance compensation candidate modes and information regarding the luminance compensation; and An image encoding method, wherein the first adjacent region and the second adjacent region are collocated regions. [[Claim 11]] A method for transmitting a bitstream generated by an image encoding method, comprising: wherein the image encoding method comprises: Encoding information regarding luminance compensation for a current block; Deriving luminance compensation parameters for each of two or more luminance compensation candidate modes for the luminance compensation; Applying luminance compensation to a predetermined first adjacent region adjacent to a reference block of the current block based on the luminance compensation parameters for each of the luminance compensation candidate modes to obtain a luminance-compensated first adjacent region; Obtaining a final error value for each of the luminance compensation candidate modes based on an error between a sample value in the luminance-compensated first adjacent region and a sample value in a predetermined second adjacent region adjacent to the current block; Rearranging the luminance compensation candidate modes based on the final error value for each of the luminance compensation candidate modes; Determining a luminance compensation candidate mode for the current block based on the rearranged luminance compensation candidate modes and information regarding the luminance compensation; and A bitstream transmission method, wherein the first adjacent region and the second adjacent region are collocated regions.

Claims

**Claim 1** An image decoding method performed by an image decoding apparatus, comprising: obtaining information regarding luminance compensation for a current block; deriving luminance compensation parameters for each of two or more luminance compensation candidate modes for the luminance compensation; applying luminance compensation to a predetermined first adjacent region adjacent to a reference block of the current block based on the luminance compensation parameters to obtain a luminance-compensated first adjacent region; obtaining a final error value for each of the luminance compensation candidate modes based on an error between sample values in the luminance-compensated first adjacent region and sample values in a predetermined second adjacent region adjacent to the current block; rearranging the luminance compensation candidate modes based on the final error values; determining a luminance compensation candidate mode for the current block based on the rearranged luminance compensation candidate modes and the information regarding the luminance compensation; and wherein the first adjacent region and the second adjacent region are corresponding positions. The image decoding method **Claim 2** The information regarding the luminance compensation includes information indicating whether luminance compensation is executable for the current block, information indicating whether the luminance compensation is applied to the current block, or an index indicating one of the luminance compensation candidate modes, at least one of which is included. The image decoding method according to claim 1 **Claim 3** The luminance compensation candidate modes include a first mode that uses both a left adjacent region and an upper adjacent region of the current block and the reference block, a second mode that uses only a left adjacent region of the current block and the reference block, or a third mode that uses only an upper adjacent region of the current block and the reference block, at least one of which is included. The image decoding method according to claim 1 **Claim 4** The rearrangement means arranging the final error values in ascending order. The image decoding method according to claim 1 **Claim 5** Based on the prediction mode of the current block being the merge mode, the information regarding the luminance compensation is obtained as information regarding the luminance compensation of a selected merge candidate of the current block. The image decoding method according to claim 2 **Claim 6** The information regarding the luminance compensation of the selected merge candidate of the current block is obtained based on at least one of the size, prediction type, split type, or transform type of the current block. The image decoding method according to claim 5.

7. The information indicating whether the luminance compensation is applied to the current block is obtained based on a first error value with respect to the luminance compensation candidate mode calculated based on the first adjacent region and the second adjacent region before the luminance compensation, and the final error value. The image decoding method according to claim 5.

8. The acquisition of the information indicating whether the luminance compensation is applied to the current block is further determined based on a threshold value. The image decoding apparatus according to claim 7.

9. The acquisition of the information indicating whether the luminance compensation is applied to the current block is performed by comparing a difference value between the final error value and the first error value with the threshold value. The image decoding apparatus according to claim 8.

10. An image encoding method performed by an image encoding apparatus, encoding information regarding luminance compensation for a current block; deriving luminance compensation parameters for each of two or more luminance compensation candidate modes for the luminance compensation; applying luminance compensation to a predetermined first adjacent region adjacent to a reference block of the current block based on the luminance compensation parameters for each of the luminance compensation candidate modes to obtain a luminance-compensated first adjacent region; obtaining a final error value for each of the luminance compensation candidate modes based on an error between sample values in the luminance-compensated first adjacent region and sample values in a predetermined second adjacent region adjacent to the current block; rearranging the luminance compensation candidate modes based on the final error value for each of the luminance compensation candidate modes; determining a luminance compensation candidate mode for the current block based on the rearranged luminance compensation candidate modes and the information regarding the luminance compensation; and The first adjacent region and the second adjacent region are collocate regions. An image encoding method.

11. A method of transmitting a bitstream generated by an image encoding method, wherein the image encoding method encodes information regarding luminance compensation for a current block; Steps of deriving luminance compensation parameters for each of two or more luminance compensation candidate modes for the luminance compensation; Steps of applying luminance compensation to a predetermined first adjacent region adjacent to the reference block of the current block based on the luminance compensation parameters for each of the luminance compensation candidate modes to obtain a luminance-compensated first adjacent region; Steps of obtaining a final error value for each of the luminance compensation candidate modes based on an error between sample values in the luminance-compensated first adjacent region and sample values in a predetermined second adjacent region adjacent to the current block; Steps of rearranging the luminance compensation candidate modes based on the final error value for each of the luminance compensation candidate modes; Steps of determining a luminance compensation candidate mode for the current block based on the rearranged luminance compensation candidate modes and information regarding the luminance compensation; comprising The first adjacent region and the second adjacent region are collocated regions, a bitstream transmission method.

Images