Selective use of local illumination compensation model, and device using same

Abstract

A selective use of a local illumination compensation (LIC) model, and a device using same are disclosed. A video encoder and a video decoder can derive initial prediction samples for the current block on the current picture on the basis of a reference block on a reference picture, determine an LIC model for the current block from among a plurality of available LIC models, and generate final prediction samples for the current block by applying local illumination compensation to the initial prediction samples for the current block on the basis of the LIC model for the current block.

Description

Optional use of local brightness compensation model and device using it

[0001] The present disclosure relates to encoding and decoding of video data, and more particularly, to a method for performing local brightness correction using a model that models local brightness changes between a current block and a reference block, and to an encoding device and a decoding device using the same.

[0002] The content described below merely provides background information related to the present invention and does not constitute prior art.

[0003] Since video data has a large amount of data compared to voice data or still image data, it requires a lot of hardware resources, including memory, to store or transmit it without processing for compression.

[0004] Therefore, when storing or transmitting video data, the encoder compresses the video data and stores or transmits it, and the decoder receives the compressed video data, decompresses it, and plays it back. These video compression technologies include H.264 / AVC, HEVC (High Efficiency Video Coding), and VVC (Versatile Video Coding), which improves encoding efficiency by about 30% compared to HEVC.

[0005] However, as the size, resolution, and frame rate of images are gradually increasing, and the amount of data that needs to be encoded is also increasing, a new compression technology that has better encoding efficiency and better image quality improvement than existing compression technologies is required.

[0006] The present disclosure presents a method for performing compensation for an inter-screen predictor based on multiple models to compensate for local brightness changes between a current block and a reference block.

[0007] The present disclosure presents various linear or nonlinear models that model local brightness changes between a current block and a reference block.

[0008] One aspect of the present disclosure provides a prediction method performed by a video decoding device. The method includes the steps of: deriving initial prediction samples for a current block in a current picture based on a reference block in a reference picture; determining a local illumination compensation (LIC) model for the current block from among a plurality of available LIC models; and applying local illumination compensation to the initial prediction samples for the current block based on the LIC model for the current block, thereby generating final prediction samples for the current block.

[0009] In some embodiments, determining a LIC model for the current block may include calculating, for each of the plurality of available LIC models, a cost for a template, which is a set of reconstructed samples neighboring the current block, and determining the LIC model for the current block based on the cost for the template for each of the plurality of available LIC models.

[0010] In some embodiments, the step of determining a LIC model for the current block may include the steps of decoding an index indicating a candidate in a merge candidate list for coding motion information of the current block from a bitstream; and determining a LIC model for the current block using LIC model information inherited from a block of the candidate indicated by the decoded index.

[0011] In some embodiments, the step of determining a LIC model for the current block may include the steps of constructing a LIC merge candidate list using LIC model information of other blocks already reconstructed on the current picture or the reference picture, decoding an index indicating a candidate in the LIC merge candidate list from a bitstream, and determining a LIC model for the current block using LIC model information of a candidate indicated by the decoded index.

[0012] In some embodiments, the step of generating final prediction samples may include the steps of determining a partitioning boundary dividing the current block into two subregions, identifying two subsets of the initial prediction samples corresponding to the two subregions, and determining a first set of parameters of the linear model to be applied to the first subset of the initial prediction samples corresponding to the first subregion of the two subregions, and a second set of parameters of the linear model to be applied to the second subset of the initial prediction samples corresponding to the second subregion of the two subregions.

[0013] In some embodiments, the step of generating the final prediction samples comprises the steps of identifying two subsets of the initial prediction samples based on comparing values ​​of the initial prediction samples to a threshold value, and determining a first set of parameters of the linear model to apply to the first subset of the initial prediction samples and a second set of parameters of the linear model to apply to the second subset of the initial prediction samples.

[0014] Another aspect of the present disclosure provides a prediction method performed by a video encoding device. The method includes the steps of: deriving initial prediction samples for a current block in a current picture based on a reference block in a reference picture; determining a local illumination compensation (LIC) model for the current block from among a plurality of available LIC models; and applying local illumination compensation to the initial prediction samples for the current block based on the LIC model for the current block, thereby generating final prediction samples for the current block.

[0015] Another aspect of the present disclosure provides a method for providing video data to a video decoding device. The method includes the steps of encoding the video data into a bitstream and transmitting the bitstream to the video decoding device, wherein the step of encoding the video data into the bitstream includes the steps of: deriving initial prediction samples for a current block in a current picture based on a reference block in a reference picture; determining a local illumination compensation (LIC) model for the current block from among a plurality of available LIC models; and applying local illumination compensation to the initial prediction samples for the current block based on the LIC model for the current block to generate final prediction samples for the current block.

[0016] According to embodiments of the present disclosure, a model suitable for a given block can be selected from among various linear models or nonlinear models, which enables better compensation for local brightness changes in the block compared to using a single linear model for all sample locations within the current block to compensate for local brightness changes, thereby improving the coding performance of the predictor for inter prediction.

[0017] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure.

[0018] Figure 2 is a drawing for explaining a method of dividing a block using the QTBTTT (QuadTree plus BinaryTree TernaryTree) structure.

[0019] FIGS. 3A and 3B are diagrams illustrating multiple intra prediction modes, including wide-angle intra prediction modes.

[0020] Figure 4 is an example diagram of the surrounding blocks of the current block.

[0021] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the techniques of the present disclosure.

[0022] Figure 6 is a conceptual diagram explaining local brightness compensation.

[0023] Figure 7 is a conceptual diagram illustrating an example where it is not appropriate to perform local brightness compensation for all samples of a block using a single linear model.

[0024] FIG. 8 is an example of a high-level syntax (HLS) for signaling a method for determining parameters of a model, according to one embodiment of the present disclosure.

[0025] FIG. 9A is a conceptual diagram illustrating spatially adjacent locations and spatially non-adjacent locations for a current block, according to some embodiments of the present disclosure.

[0026] FIG. 9b is a conceptual diagram illustrating the locations of temporal candidates according to some embodiments of the present disclosure.

[0027] FIG. 9c is a conceptual diagram illustrating the positions of shifted temporal candidates according to some embodiments of the present disclosure.

[0028] Figure 10 illustrates the spatial locations of the central sample and its neighboring samples.

[0029] Figure 11 is a conceptual diagram illustrating a segmentation boundary and the angle and offset that specify it.

[0030] Figure 12 is a conceptual diagram illustrating various segmentation boundaries that divide the current block to which local brightness compensation is applied into two sub-regions.

[0031] FIG. 13 is a conceptual diagram illustrating an example of using two different sets of parameters for a linear model based on one threshold value, according to some embodiments of the present disclosure.

[0032] FIG. 14 shows examples of subsampled samples used to compute or derive parameters of a LIC model, according to some embodiments of the present disclosure.

[0033] FIG. 15 is an example of a LIC-related syntax structure that may be included in a CU level according to one embodiment of the present disclosure.

[0034] FIG. 16 illustrates a CU level syntax including lic_mode_idx according to some embodiments of the present disclosure.

[0035] FIG. 17 is an example of binarization of lic_mode_idx for various LIC modes according to some embodiments of the present disclosure.

[0036] FIG. 18 is a flowchart illustrating a prediction method performed by each of a video encoder and a video decoder according to some embodiments of the present disclosure.

[0037] Hereinafter, embodiments of the present invention will be described in detail with reference to exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, in describing the present embodiments, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present embodiments.

[0038] FIG. 1 is an exemplary block diagram of an image encoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image encoding device and its subcomponents will be described with reference to the illustration in FIG. 1.

[0039] The video encoding device may be configured to include a picture segmentation unit (110), a prediction unit (120), a subtractor (130), a transformation unit (140), a quantization unit (145), a reordering unit (150), an entropy encoding unit (155), an inverse quantization unit (160), an inverse transformation unit (165), an adder (170), a loop filter unit (180), and a memory (190).

[0040] Each component of the video encoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0041] A single image (video) is composed of one or more sequences containing multiple pictures. Each picture is divided into multiple regions, and encoding is performed for each region. For example, a single picture is divided into one or more tiles and / or slices. Here, one or more tiles can be defined as a tile group. Each tile or slice is divided into one or more Coding Tree Units (CTUs). Each CTU is then divided into one or more Coding Units (CUs) by a tree structure. Information applied to each CU is encoded as the syntax of the CU, and information commonly applied to CUs included in a CTU is encoded as the syntax of the CTU. In addition, information commonly applied to all blocks within a single slice is encoded as the syntax of the slice header, and information applied to all blocks constituting one or more pictures is encoded in the Picture Parameter Set (PPS) or the picture header. Furthermore, information commonly referenced by multiple pictures is encoded in a Sequence Parameter Set (SPS). And, information commonly referenced by one or more SPS is encoded in a Video Parameter Set (VPS). In addition, information commonly applied to one tile or tile group may be encoded as syntax of a tile or tile group header. Syntaxes included in an SPS, PPS, slice header, tile or tile group header may be referred to as high level syntax.

[0042] The picture segmentation unit (110) determines the size of the CTU. Information about the size of the CTU (CTU size) is encoded as the syntax of SPS or PPS and transmitted to the image decoding device.

[0043] The picture segmentation unit (110) divides each picture constituting an image into a plurality of CTUs having a predetermined size, and then recursively divides the CTUs using a tree structure. A leaf node in the tree structure becomes a CU, which is a basic unit of encoding.

[0044] The tree structure may be a QuadTree (QT) in which an upper node (or parent node) is divided into four lower nodes (or child nodes) of the same size, a BinaryTree (BT) in which an upper node is divided into two lower nodes, or a TernaryTree (TT) in which an upper node is divided into three lower nodes in a 1:2:1 ratio, or a structure that mixes two or more of the QT structures, BT structures, and TT structures. For example, a QTBT (QuadTree plus BinaryTree) structure may be used, or a QTBTTT (QuadTree plus BinaryTree TernaryTree) structure may be used. Here, BTTT may be combined and referred to as a MTT (Multiple-Type Tree).

[0045] Figure 2 is a drawing for explaining a method of dividing a block using the QTBTTT structure.

[0046] As illustrated in FIG. 2, a CTU may first be split into a QT structure. The quadtree splitting may be repeated until the size of the splitting block reaches the minimum block size (MinQTSize) of the leaf node allowed in the QT. A first flag (QT_split_flag) indicating whether each node of the QT structure is split into four nodes of the lower layer is encoded by the entropy encoding unit (155) and signaled to the image decoding device. If the leaf node of the QT is not larger than the maximum block size (MaxBTSize) of the root node allowed in the BT, it may be further split into one or more of the BT structure or the TT structure. There may be multiple splitting directions in the BT structure and / or the TT structure. For example, there may be two directions in which the block of the corresponding node is split horizontally and two directions in which the block is split vertically. As illustrated in FIG. 2, when MTT splitting begins, a second flag (mtt_split_flag) indicating whether nodes have been split, and if splitting has occurred, a flag indicating the splitting direction (vertical or horizontal) and / or a flag indicating the splitting type (Binary or Ternary) are encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0047] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into four nodes of a lower layer, a CU split flag (split_cu_flag) indicating whether the node is split may be encoded. If the CU split flag (split_cu_flag) value indicates that the node is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a CU (coding unit), which is a basic unit of encoding. If the CU split flag (split_cu_flag) value indicates that the node is split, the video encoding device starts encoding from the first flag in the above-described manner.

[0048] As another example of a tree structure, when QTBT is used, there may be two types: a type that horizontally splits the block of the corresponding node into two blocks of the same size (i.e., symmetric horizontal splitting) and a type that vertically splits it (i.e., symmetric vertical splitting). A split flag (split_flag) indicating whether each node of the BT structure is split into blocks of a lower layer and split type information indicating the type of split are encoded by the entropy encoding unit (155) and transmitted to the image decoding device. Meanwhile, there may additionally be a type that splits the block of the corresponding node into two blocks of an asymmetrical shape. The asymmetric shape may include a shape that splits the block of the corresponding node into two rectangular blocks with a size ratio of 1:3, or a shape that splits the block of the corresponding node in a diagonal direction.

[0049] A CU can have various sizes depending on the QTBT or QTBTTT partitioning from the CTU. Hereinafter, the block corresponding to the CU to be encoded or decoded (i.e., the leaf node of the QTBTTT) is referred to as the "current block." Depending on the QTBTTT partitioning employed, the current block may be rectangular as well as square.

[0050] The prediction unit (120) predicts the current block and generates a prediction block. The prediction unit (120) includes an intra prediction unit (122) and an inter prediction unit (124).

[0051] In general, each current block within a picture can be predictively coded. Prediction of the current block can typically be performed using either intra-prediction (using data from the picture containing the current block) or inter-prediction (using data from a picture coded before the picture containing the current block). Inter-prediction encompasses both unidirectional and bidirectional prediction.

[0052] The intra prediction unit (122) predicts pixels within the current block using pixels (reference pixels) located around the current block within the current picture including the current block. There are multiple intra prediction modes depending on the prediction direction. For example, as shown in Fig. 3a, the multiple intra prediction modes may include two non-directional modes including the Planar mode and the DC mode, and 65 directional modes. The surrounding pixels to be used and the calculation formula are defined differently depending on each prediction mode.

[0053] For efficient directional prediction for a rectangular current block, directional modes (intra prediction modes 67 to 80 and -1 to -14) indicated by dotted arrows in Fig. 3b may be additionally used. These may be referred to as "wide-angle intra-prediction modes." In Fig. 3b, the arrows point to corresponding reference samples used for prediction, and do not indicate the prediction direction. The prediction direction is opposite to the direction indicated by the arrows. Wide-angle intra-prediction modes are modes that perform prediction in the opposite direction of a specific directional mode without additional bit transmission when the current block is rectangular. At this time, among the wide-angle intra-prediction modes, some wide-angle intra-prediction modes available for the current block may be determined based on the ratio of the width and height of the rectangular current block. For example, wide-angle intra prediction modes (intra prediction modes 67 to 80) having an angle less than 45 degrees are available when the current block is a rectangular shape whose height is smaller than its width, and wide-angle intra prediction modes (intra prediction modes -1 to -14) having an angle greater than -135 degrees are available when the current block is a rectangular shape whose width is larger than its height.

[0054] The intra prediction unit (122) can determine an intra prediction mode to be used to encode the current block. In some examples, the intra prediction unit (122) can encode the current block using multiple intra prediction modes and select an appropriate intra prediction mode to be used from the tested modes. For example, the intra prediction unit (122) can calculate bit-rate distortion values ​​using rate-distortion analysis for multiple tested intra prediction modes and select an intra prediction mode with the best bit-rate distortion characteristics among the tested modes.

[0055] The intra prediction unit (122) selects one intra prediction mode from among multiple intra prediction modes and predicts the current block using surrounding pixels (reference pixels) and an operation formula determined according to the selected intra prediction mode. Information about the selected intra prediction mode is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0056] The inter prediction unit (124) generates a prediction block for the current block using a motion compensation process. The inter prediction unit (124) searches for a block most similar to the current block within reference pictures that were encoded and decoded before the current picture, and generates a prediction block for the current block using the searched block. Then, a motion vector (MV) corresponding to the displacement between the current block within the current picture and the prediction block within the reference picture is generated. Generally, motion estimation is performed on the luma component, and the motion vector calculated based on the luma component is used for both the luma component and the chroma component. The motion information including information on the reference picture used to predict the current block and information on the motion vector is encoded by the entropy encoding unit (155) and transmitted to the image decoding device.

[0057] The inter prediction unit (124) may perform interpolation on a reference picture or a reference block to improve prediction accuracy. That is, subsamples between two consecutive integer samples are interpolated by applying filter coefficients to a plurality of consecutive integer samples including the two integer samples. When a process of searching for a block most similar to the current block is performed on the interpolated reference picture, the motion vector can be expressed up to a precision in decimal units rather than a precision in integer sample units. The precision or resolution of the motion vector can be set differently for each target region to be encoded, such as a slice, tile, CTU, CU, etc. When such adaptive motion vector resolution (AMVR) is applied, information on the motion vector resolution to be applied to each target region must be signaled for each target region. For example, when the target region is a CU, information on the motion vector resolution applied to each CU is signaled. Information on the motion vector resolution may be information indicating the precision of a differential motion vector, which will be described later.

[0058] Meanwhile, the inter prediction unit (124) can perform inter prediction using bi-prediction. In the case of bi-prediction, two reference pictures and two motion vectors indicating the block position most similar to the current block within each reference picture are used. The inter prediction unit (124) selects a first reference picture and a second reference picture from reference picture list 0 (RefPicList0) and reference picture list 1 (RefPicList1), respectively, and searches for a block similar to the current block within each reference picture to generate a first reference block and a second reference block. Then, the first reference block and the second reference block are averaged or weighted averaged to generate a prediction block for the current block. Then, motion information including information on two reference pictures used to predict the current block and information on two motion vectors is transmitted to the entropy encoding unit (155). Here, reference picture list 0 may be composed of pictures that are before the current picture in display order among the restored pictures, and reference picture list 1 may be composed of pictures that are after the current picture in display order among the restored pictures. However, this is not necessarily limited to this, and restored pictures that are after the current picture in display order may be additionally included in reference picture list 0, and conversely, restored pictures that are before the current picture may be additionally included in reference picture list 1.

[0059] Various methods can be used to minimize the number of bits required to encode motion information.

[0060] For example, if the reference picture and motion vector of the current block are identical to those of a neighboring block, the motion information of the current block can be transmitted to the image decoding device by encoding information that can identify the neighboring block. This method is called 'merge mode'.

[0061] In merge mode, the inter prediction unit (124) selects a predetermined number of merge candidate blocks (hereinafter referred to as 'merge candidates') from the surrounding blocks of the current block.

[0062] As the surrounding blocks for deriving merge candidates, all or part of the left block (A0), the lower left block (A1), the upper block (B0), the upper right block (B1), and the upper left block (B2) adjacent to the current block within the current picture may be used, as illustrated in FIG. 4. In addition, a block located within a reference picture (which may or may not be the same as the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as a merge candidate. For example, a block co-located with the current block within the reference picture or blocks adjacent to the block at the co-located block may be additionally used as a merge candidate. If the number of merge candidates selected by the method described above is less than a preset number, a 0 vector is added to the merge candidates.

[0063] The inter prediction unit (124) uses these surrounding blocks to construct a merge list containing a predetermined number of merge candidates. Among the merge candidates included in the merge list, the merge candidate to be used as motion information of the current block is selected and merge index information for identifying the selected candidate is generated. The generated merge index information is encoded by the entropy encoding unit (155) and transmitted to the video decoding device.

[0064] Merge Skip mode is a special case of merge mode. After quantization, when all transform coefficients for entropy encoding are close to zero, only neighboring block selection information is transmitted without transmitting residual signals. By utilizing merge skip mode, relatively high encoding efficiency can be achieved for low-motion images, still images, and screen content images.

[0065] Hereinafter, merge mode and merge skip mode are collectively referred to as merge / skip mode.

[0066] Another method for encoding motion information is Advanced Motion Vector Prediction (AMVP) mode.

[0067] In AMVP mode, the inter prediction unit (124) derives predicted motion vector candidates for the motion vector of the current block using neighboring blocks of the current block. As neighboring blocks used to derive predicted motion vector candidates, all or some of the left block (A0), the lower left block (A1), the upper block (B0), the upper right block (B1), and the upper left block (B2) adjacent to the current block in the current picture as shown in FIG. 4 may be used. In addition, a block located in a reference picture (which may or may not be the same as the reference picture used to predict the current block) other than the current picture in which the current block is located may be used as the neighboring block used to derive predicted motion vector candidates. For example, a block co-located with the current block in the reference picture or blocks adjacent to the block in the co-located block may be used. If the number of motion vector candidates is less than a preset number by the method described above, a 0 vector is added to the motion vector candidates.

[0068] The inter prediction unit (124) derives predicted motion vector candidates using the motion vectors of these surrounding blocks, and determines a predicted motion vector for the motion vector of the current block using the predicted motion vector candidates. Then, the predicted motion vector is subtracted from the motion vector of the current block to produce a differential motion vector.

[0069] The predicted motion vector can be obtained by applying a predefined function (e.g., median, mean, etc.) to the predicted motion vector candidates. In this case, the image decoding device also knows the predefined function. In addition, since the surrounding blocks used to derive the predicted motion vector candidates are blocks that have already been encoded and decoded, the image decoding device also already knows the motion vectors of the surrounding blocks. Therefore, the image encoding device does not need to encode information for identifying the predicted motion vector candidates. Therefore, in this case, information about the differential motion vector and information about the reference picture used to predict the current block are encoded.

[0070] Alternatively, the predicted motion vector can be determined by selecting one of the predicted motion vector candidates. In this case, information for identifying the selected predicted motion vector candidate is additionally encoded, along with information about the differential motion vector and the reference picture used to predict the current block.

[0071] The subtractor (130) subtracts the prediction block generated by the intra prediction unit (122) or inter prediction unit (124) from the current block to generate a residual block.

[0072] The transformation unit (140) transforms residual signals within a residual block having pixel values ​​in a spatial domain into transform coefficients in a frequency domain. The transformation unit (140) may transform the residual signals within the residual block using the entire size of the residual block as a transformation unit, or may divide the residual block into a plurality of sub-blocks and use the sub-blocks as transformation units to perform the transformation. Alternatively, the residual signals may be transformed using only the transformation domain sub-block as a transformation unit by dividing the sub-blocks into two sub-blocks, that is, a transformation domain and a non-transform domain. Here, the transformation domain sub-block may be one of two rectangular blocks having a size ratio of 1:1 with respect to the horizontal axis (or vertical axis). In this case, a flag (cu_sbt_flag) indicating that only a sub-block has been converted, directionality (vertical / horizontal) information (cu_sbt_horizontal_flag), and / or position information (cu_sbt_pos_flag) are encoded by the entropy encoding unit (155) and signaled to the image decoding device. In addition, the size of the conversion area sub-block may have a size ratio of 1:3 with respect to the horizontal axis (or vertical axis), and in this case, a flag (cu_sbt_quad_flag) distinguishing the corresponding division is additionally encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0073] Meanwhile, the transformation unit (140) can individually perform transformations on the residual block in the horizontal and vertical directions. For the transformation, various types of transformation functions or transformation matrices can be used. For example, a pair of transformation functions for horizontal transformation and vertical transformation can be defined as a Multiple Transform Set (MTS). The transformation unit (140) can select one transformation function pair with the best transformation efficiency among the MTS and transform the residual block in the horizontal and vertical directions, respectively. Information (mts_idx) on the transformation function pair selected among the MTS is encoded by the entropy encoding unit (155) and signaled to the image decoding device.

[0074] The quantization unit (145) quantizes the transform coefficients output from the transform unit (140) using quantization parameters and outputs the quantized transform coefficients to the entropy encoding unit (155). The quantization unit (145) may directly quantize a related residual block without transformation for a certain block or frame. The quantization unit (145) may also apply different quantization coefficients (scaling values) according to the positions of the transform coefficients within the transform block. The quantization matrix applied to the quantized transform coefficients arranged in two dimensions may be encoded and signaled to an image decoding device.

[0075] The rearrangement unit (150) can perform rearrangement of coefficient values ​​for quantized residual values.

[0076] The reordering unit (150) can change a two-dimensional coefficient array into a one-dimensional coefficient sequence by using coefficient scanning. For example, the reordering unit (150) can output a one-dimensional coefficient sequence by scanning from the DC coefficient to the coefficients of the high-frequency region by using a zig-zag scan or a diagonal scan. Depending on the size of the transformation unit and the intra prediction mode, a vertical scan that scans the two-dimensional coefficient array in the column direction or a horizontal scan that scans the two-dimensional block-shaped coefficients in the row direction may be used instead of the zig-zag scan. That is, depending on the size of the transformation unit and the intra prediction mode, the scanning method to be used may be determined among the zig-zag scan, the diagonal scan, the vertical scan, and the horizontal scan.

[0077] The entropy encoding unit (155) generates a bitstream by encoding a sequence of one-dimensional quantized transform coefficients output from the rearrangement unit (150) using various encoding methods such as CABAC (Context-based Adaptive Binary Arithmetic Code) and Exponential Golomb.

[0078] In addition, the entropy encoding unit (155) encodes information related to block division, such as CTU size, CU division flag, QT division flag, MTT division type, and MTT division direction, so that the image decoding device can divide the block in the same manner as the image encoding device. In addition, the entropy encoding unit (155) encodes information about a prediction type indicating whether the current block is encoded by intra prediction or inter prediction, and encodes intra prediction information (i.e., information about an intra prediction mode) or inter prediction information (information about an encoding mode of motion information (merge mode or AMVP mode), a merge index in the case of a merge mode, and a reference picture index and a differential motion vector in the case of an AMVP mode) according to the prediction type. In addition, the entropy encoding unit (155) encodes information related to quantization, that is, information about a quantization parameter and information about a quantization matrix.

[0079] The inverse quantization unit (160) inversely quantizes the quantized transform coefficients output from the quantization unit (145) to generate transform coefficients. The inverse transform unit (165) transforms the transform coefficients output from the inverse quantization unit (160) from the frequency domain to the spatial domain to restore the residual block.

[0080] An adder (170) adds the restored residual block and the predicted block generated by the prediction unit (120) to restore the current block. The pixels within the restored current block are used as reference pixels when intra-predicting the next block.

[0081] The loop filter unit (180) performs filtering on restored pixels to reduce blocking artifacts, ringing artifacts, blurring artifacts, etc. that occur due to block-based prediction and transformation / quantization. The loop filter unit (180) may include all or part of a deblocking filter (182), a sample adaptive offset (SAO) filter (184), and an adaptive loop filter (ALF, 186) as an in-loop filter.

[0082] The deblocking filter (182) filters the boundaries between restored blocks to remove blocking artifacts caused by block-based encoding / decoding, and the SAO filter (184) and the ALF (186) perform additional filtering on the deblocking-filtered image. The SAO filter (184) and the ALF (186) are filters used to compensate for the differences between restored pixels and original pixels caused by lossy coding. The SAO filter (184) improves not only subjective image quality but also encoding efficiency by applying an offset in units of CTUs. In contrast, the ALF (186) performs block-based filtering, and compensates for distortion by applying different filters by distinguishing the edges and degrees of variation of the corresponding block. Information on filter coefficients to be used in the ALF can be encoded and signaled to an image decoding device.

[0083] The restored blocks filtered through the deblocking filter (182), SAO filter (184), and ALF (186) are stored in the memory (190). When all blocks within a picture are restored, the restored picture can be used as a reference picture for inter-predicting blocks within a picture to be encoded later.

[0084] The video encoding device can store the bitstream of encoded video data on a non-transitory storage medium or transmit it to the video decoding device using a communication network.

[0085] FIG. 5 is an exemplary block diagram of an image decoding device capable of implementing the techniques of the present disclosure. Hereinafter, the image decoding device and its subcomponents will be described with reference to FIG. 5.

[0086] The video decoding device may be configured to include an entropy decoding unit (510), a rearrangement unit (515), an inverse quantization unit (520), an inverse transformation unit (530), a prediction unit (540), an adder (550), a loop filter unit (560), and a memory (570).

[0087] Similar to the video encoding device of FIG. 1, each component of the video decoding device may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0088] The entropy decoding unit (510) decodes the bitstream generated by the image encoding device to extract information related to block division, thereby determining the current block to be decoded, and extracts prediction information, information on residual signals, etc. required to restore the current block.

[0089] The entropy decoding unit (510) extracts information about the CTU size from the Sequence Parameter Set (SPS) or the Picture Parameter Set (PPS), determines the size of the CTU, and divides the picture into CTUs of the determined size. Then, the CTU is determined as the top layer of the tree structure, i.e., the root node, and the CTU is divided using the tree structure by extracting division information about the CTU.

[0090] For example, when splitting a CTU using the QTBTTT structure, first, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Then, for the nodes corresponding to the leaf nodes of QT, the second flag (mtt_split_flag) related to the splitting of MTT and the split direction (vertical / horizontal) and / or split type (binary / ternary) information are extracted, and the corresponding leaf nodes are split into the MTT structure. Accordingly, each node below the leaf nodes of QT are split recursively into the BT or TT structure.

[0091] As another example, when splitting a CTU using the QTBTTT structure, the CU split flag (split_cu_flag) indicating whether the CU is split is first extracted, and if the block is split, the first flag (QT_split_flag) may be extracted. During the splitting process, each node may undergo zero or more repeated QT splits followed by zero or more repeated MTT splits. For example, a CTU may undergo an MTT split right away, or conversely, may undergo only multiple QT splits.

[0092] As another example, when splitting a CTU using the QTBT structure, the first flag (QT_split_flag) related to the splitting of QT is extracted, and each node is split into four nodes of the lower layer. Furthermore, for nodes corresponding to leaf nodes of QT, a split flag (split_flag) indicating whether to further split into BTs and splitting direction information are extracted.

[0093] Meanwhile, when the entropy decoding unit (510) determines the current block to be decoded by using the division of the tree structure, it extracts information on the prediction type indicating whether the current block is intra-predicted or inter-predicted. If the prediction type information indicates intra-prediction, the entropy decoding unit (510) extracts syntax elements for intra-prediction information (intra-prediction mode) of the current block. If the prediction type information indicates inter-prediction, the entropy decoding unit (510) extracts syntax elements for inter-prediction information, i.e., information indicating a motion vector and a reference picture referenced by the motion vector.

[0094] Additionally, the entropy decoding unit (510) extracts information about the quantized transform coefficients of the current block as information related to quantization and information about residual signals.

[0095] The rearrangement unit (515) can change the sequence of one-dimensional quantized transform coefficients entropy-decoded in the entropy decoding unit (510) back into a two-dimensional coefficient array (i.e., block) in the reverse order of the coefficient scanning performed by the image encoding device.

[0096] The inverse quantization unit (520) inversely quantizes the quantized transform coefficients and inversely quantizes the quantized transform coefficients using the quantization parameters. The inverse quantization unit (520) may also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in two dimensions. The inverse quantization unit (520) may perform inverse quantization by applying a matrix of quantized coefficients (scaling values) from an image encoding device to a two-dimensional array of quantized transform coefficients.

[0097] The inverse transform unit (530) inversely transforms the inverse quantized transform coefficients from the frequency domain to the spatial domain to restore residual signals, thereby generating a residual block for the current block.

[0098] In addition, when the inverse transform unit (530) inversely transforms only a portion of a transform block (sub-block), it extracts a flag (cu_sbt_flag) indicating that only a sub-block of the transform block has been transformed, directionality (vertical / horizontal) information (cu_sbt_horizontal_flag) of the sub-block, and / or position information (cu_sbt_pos_flag) of the sub-block, and inversely transforms the transform coefficients of the corresponding sub-block from the frequency domain to the spatial domain to restore residual signals, and fills “0” values ​​with residual signals for areas that have not been inversely transformed, thereby generating a final residual block for the current block.

[0099] In addition, when MTS is applied, the inverse transform unit (530) determines a transform function or a transform matrix to be applied in the horizontal and vertical directions using MTS information (mts_idx) signaled from the image encoding device, and performs inverse transform on the transform coefficients within the transform block in the horizontal and vertical directions using the determined transform function.

[0100] The prediction unit (540) may include an intra prediction unit (542) and an inter prediction unit (544). The intra prediction unit (542) is activated when the prediction type of the current block is intra prediction, and the inter prediction unit (544) is activated when the prediction type of the current block is inter prediction.

[0101] The intra prediction unit (542) determines the intra prediction mode of the current block among a plurality of intra prediction modes from the syntax elements for the intra prediction mode extracted from the entropy decoding unit (510), and predicts the current block using reference pixels around the current block according to the intra prediction mode.

[0102] The inter prediction unit (544) uses the syntax elements for the inter prediction mode extracted from the entropy decoding unit (510) to determine the motion vector of the current block and the reference picture referenced by the motion vector, and predicts the current block using the motion vector and the reference picture.

[0103] An adder (550) adds the residual block output from the inverse transform unit (530) and the predicted block output from the inter prediction unit (544) or the intra prediction unit (542) to restore the current block. The pixels within the restored current block are used as reference pixels when intra-predicting a block to be decoded later.

[0104] The loop filter unit (560) may include a deblocking filter (562), an SAO filter (564), and an ALF (566) as in-loop filters. The deblocking filter (562) deblocks the boundaries between restored blocks to remove blocking artifacts caused by block-by-block decoding. The SAO filter (564) and the ALF (566) perform additional filtering on restored blocks after deblocking filtering to compensate for differences between restored pixels and original pixels caused by lossy coding. The filter coefficients of the ALF are determined using information about filter coefficients decoded from the non-stream.

[0105] The restored blocks filtered through the deblocking filter (562), SAO filter (564), and ALF (566) are stored in the memory (570). When all blocks within a picture are restored, the restored picture is used as a reference picture for inter-predicting blocks within a picture to be encoded later.

[0106] Hereinafter, improved coding techniques that can be performed by the aforementioned video encoder (e.g., the video encoding device illustrated in FIG. 1) or video decoder (e.g., the video decoding device illustrated in FIG. 5) are disclosed. In particular, the techniques of the present disclosure relate to local illumination compensation (LIC).

[0107] When performing inter-screen prediction, the region most similar to the current block is searched for in the reference picture, and the location of that region is expressed using motion information (motion vectors, reference picture IDs, etc.). When performing inter-screen prediction, there may be cases where the subject lighting between the current picture and the reference picture differs. A typical example is when a specific picture is illuminated. To compensate for this inter-screen brightness difference, the Local Light Compensation (LIC) technique was introduced.

[0108] Local Brightness Compensation (LIC) is an inter-prediction technique that models the local brightness change between a current block and a reference block as a function of the change between the current block template and the reference block template. The parameters of the function can be represented by the slope α and the offset β, which form the linear equation for compensating the brightness change as follows: P[x, y] = α*P[x, y] + β, where P[x, y] is the brightness-compensated prediction sample at position [x, y] in the current block, and P[x, y] is the reference sample pointed to by the motion vector at position [x, y] in the reference picture.

[0109] Figure 6 conceptually illustrates Local Brightness Compensation (LIC). As shown in Figure 6, when the current block is inter-predicted, the current block has a motion vector pointing to a reference block. The reconstructed neighboring samples of the current block (simply, the current template) and the reconstructed neighboring samples of the reference block (simply, the reference template) are used to compute the parameters of a linear model. The linear model is applied to the initial predictor P[x, y] of the inter-prediction to generate the final predictor P[x, y].

[0110] As such, LIC typically uses a single linear model for all samples in a block. However, a single linear model may not optimally model local brightness variations between the current block and the predicted block. For example, brightness variations may vary across samples even within a block. Furthermore, as illustrated in Figure 7, shadows may cause brightness variations in only some regions of a reference block within a reference picture. Therefore, using only a single linear model to compensate for local brightness variations across all samples in a block may not optimally compensate for local brightness variations within the block.

[0111] The present disclosure proposes several methods for further improving LIC coding efficiency. According to one aspect of the present disclosure, a method is presented for selectively using a model or models suitable for a given block among various models capable of compensating for local brightness variations. According to another aspect of the present disclosure, a method is presented for selecting a variety of valid samples for predicting the LIC parameters of each model.

[0112] In embodiments of the present disclosure, various LIC models, such as various linear models, nonlinear models, and / or machine learning models, may be configured to model local brightness changes between a current block and a predicted block. A video encoder and a video decoder may select one or more models from among a plurality of available LIC models to be used for local brightness compensation for the current block.

[0113] Here, selection of the LIC model can be performed based on predefined criteria, for example, selection of the LIC model for the current block can be performed based on coordinates of the sample, sample values, encoding information, LIC models of surrounding blocks, etc.

[0114] A video encoder can signal an indication in the bitstream that specifies one or more LIC models to be selected for local brightness compensation for the current block from among a plurality of available LIC models. Accordingly, a decoder can determine or select one or more LIC models to be used for local brightness compensation for the current block based on the indication obtained from the bitstream.

[0115] Alternatively, the video decoder may implicitly determine or select the LIC model or LIC models to use for local brightness compensation for the current block. For example, the video decoder may calculate the cost of the mean square error (MSE), sum of absolute difference (SAD), sum of absolute transformed difference (SATD), mean removed sum of absolute difference (MR-SAD), etc. for each of the available LIC models with respect to the template of the current block, and then select or use the model with the smallest cost for local brightness compensation for the current block. The cost for the template of the current block may be calculated based on the difference between the "reconstructed samples" of the template of the current block and the "compensated prediction samples" of the template of the current block obtained by applying the LIC model to the reconstructed samples of the template of the reference block.

[0116] Once a model or models to be used for the current block is selected from among multiple available models, the parameters of the selected model need to be determined.

[0117] Various methods can be employed to determine the parameters of the LIC model. For example, the parameters of a candidate selected from a merge candidate list can be used, the parameters can be estimated (or derived) at the decoder in the same manner as the encoder, or the parameters can be explicitly signaled by the encoder. The video encoder can signal to the encoder how to determine the model parameters using high-level syntax (HLS).

[0118] FIG. 8 is an example of a high-level syntax (HLS) for signaling a method for determining parameters of a model, according to an embodiment of the present disclosure. Some or all of the syntax elements illustrated in FIG. 8 may be signaled in a sequence parameter set (SPS), a picture parameter set (PPS), a picture header (PH), a slice header (SH), etc. In the high-level syntax (HLS) illustrated in FIG. 8, the signaling order of the syntax elements may be varied in various ways.

[0119] The syntax element cu_lic_flag is a flag indicating whether LIC use is allowed. If cu_lic_flag = 1, LIC mode is enabled, and if cu_lic_flag = 0, LIC mode is disabled.

[0120] The syntax element cu_lic_parameters_inherit is a syntax element that indicates whether to use the LIC merge mode. If cu_lic_parameters_inherit = 1, it means that the LIC merge mode is enabled, and if cu_lic_parameters_inherit = 0, it means that the LIC merge mode is disabled.

[0121] The syntax element cu_lic_parameters_implicit is a syntax element that indicates whether the LIC parameters are estimated (i.e., implicitly derived) in the decoder. If cu_lic_parameters_implicit = 1, it means that the decoder estimates the LIC parameters, and if cu_lic_parameters_implicit = 0, it means that the decoder does not estimate the LIC parameters.

[0122] The syntax element cu_lic_parameters_explicit is a syntax element that indicates whether the method in which LIC parameters are explicitly signaled is used. If cu_lic_parameters_explicit = 1, it means that LIC parameters are signaled, and if cu_lic_parameters_explicit = 0, it means that LIC parameters are not signaled.

[0123] Example 1 - Merge Candidate List

[0124] In this embodiment, a list of available LIC models for compensating for local brightness changes for a current block can be constructed. A video encoder can select a model to be used for the current block from the list of LIC models and signal an index identifying the selected model from the list of LIC models. A video decoder can select a model indicated by the index from the list of available LIC models and perform local brightness compensation for the current block using the selected model.

[0125] The list of available LIC models can be constructed using one of the following methods. The first method uses the merge candidate list constructed for motion information coding of the current block. The second method constructs an LIC merge candidate list using LIC model information of neighboring blocks of the current block, separate from the merge candidate list constructed for motion information coding.

[0126] When using the merge candidate list configured for motion information coding, which is the first method, the video decoder can decode an index indicating a candidate in the merge candidate list from the bitstream, and inherit LIC flag information and LIC model information from the block of the candidate indicated by the decoded index. The video decoder can use the inherited LIC model information to determine the LIC model (and its parameters) to be used for the current block.

[0127] The second method involves constructing a LIC merge candidate list using LIC model information used in other blocks. The LIC merge candidate list can be either predefined or adaptively sized. Furthermore, the LIC merge candidate list can be composed of various candidate types, as follows:

[0128] (1) Spatial adjacent candidates: LIC model information of spatially adjacent locations

[0129] (2) Spatial non-adjacent candidates: LIC model information of spatially non-adjacent locations

[0130] (3) Temporal candidate: LIC model information of the same temporal location

[0131] (4) History-based candidate: LIC model information from the history table that stores previously used LIC models.

[0132] (5) Shifted temporal candidate: LIC model information of the same temporal position shifted by the motion vector.

[0133] (6) Default candidate: Predefined LIC model information added to the remaining space in the LIC merge candidate list when the LIC merge candidate list is not fully filled.

[0134] Spatially adjacent positions and spatially non-adjacent positions are as shown in Fig. 9a, temporally identical positions are as shown in Fig. 9b, and temporally identical positions shifted by a motion vector are as shown in Fig. 9c.

[0135] In the example of FIG. 9a, positions 1 to 5 are spatially adjacent positions with respect to the current block, and positions 6 to 23 are spatially non-adjacent positions with respect to the current block. LIC information of one or more positions among positions 1 to 5 can be inserted into the LIC merge candidate list as one or more spatially adjacent merge candidates. LIC information of one or more positions among positions 6 to 23 can be inserted into the LIC merge candidate list as one or more spatially non-adjacent merge candidates. The distance between the spatially non-adjacent positions and the current block is based on the width and height of the current block.

[0136] As illustrated in Fig. 9b, temporal candidates can be selected from collocated pictures (colPic). C0 and C1 in Fig. 9b represent reference pictures corresponding to reference picture indices, respectively. The inclusion order of temporal candidates is (C01, C02, ..., C0 10 ) can be the same. If C0 i is outside the picture / slice boundary, and C1 i If C1 is inside the picture / slice boundary, i Go C0 i can be used instead of .

[0137] As illustrated in Fig. 9c, shifted temporal candidates can be selected from collocated pictures. The location of the collocated block and C0 i and C1 i The positions of can be shifted by the selected neighboring motion vectors. The inclusion order of the shifted temporal candidates can be the same as the inclusion order of the temporal candidates. The neighboring motion vectors can be selected from the motion vectors of the surrounding blocks (A0, A1, B0, B1, B2). The checking order is (L0 B1 , L1 B1 , L0 A1 , L1 A1 , L0 B0 , L1 B0 , L0 A0 , L1 A0 , L0 B2 , L1 B2 ) can be selected. The first motion vector that uses the same position picture as a reference picture can be selected.

[0138] To construct a list of LIC merge candidates, the search area for previously used LIC models can be limited to areas such as within the same CTU, within the same CTU row, or within the same picture. That is, the history table storing previously used LIC models can be initialized on a CTU basis, a CTU row basis, or a picture basis.

[0139] An example of a default candidate might be a LIC model generated by changing the parameter values ​​of a linear LIC model already included in the candidate list.

[0140] In addition to the types of candidates described here, other types of candidates may be used to construct the LIC merge candidate list. Furthermore, only some of the types of candidates described here may be used to construct the LIC merge candidate list.

[0141] Once the LIC merge candidate list is constructed, the candidates within the list can be reordered in descending order of cost by calculating the cost for the template of the current block. Loss functions such as SAD, SATD, and MR-SAD can be used to calculate the cost. The cost for the template of the current block can be calculated based on the difference between the reconstructed neighboring samples of the template of the current block and the compensated predictions of the template of the current block obtained by applying the candidate model to the corresponding neighboring samples of the reference block.

[0142] The decoder can decode an index pointing to a candidate in the LIC merge candidate list from the bitstream, and determine the LIC model (and its parameters) for the current block using the LIC model information of the candidate pointed to by the decoded index.

[0143] Example 2 - Compensation based on multiple sample inputs

[0144] In this embodiment, one or more multi-sample input-based compensation models may be used. The multi-sample input-based compensation model may output compensated sample values ​​using multiple samples of the initial predictor of inter prediction for a given prediction sample location. An autoregressive model (AR model) may be used as the multi-sample input-based compensation model. The mathematical expression below illustrates a method for compensating for local brightness changes using an autoregressive model. When modeling LIC using an autoregressive model, an offset term may be optional.

[0145]

[0146] P'[x, y] represents the sample value at the sample location (x, y) where compensation for local brightness change is performed, and P[x i , y i ] is an input value for performing compensation for local brightness changes, and is the location (x) within the reference picture. i , y i ) represents the sample value.

[0147] Figure 10 illustrates the spatial locations of a central sample (C) and its neighboring samples. The central sample (C) is a sample within a reference block corresponding to a sample position (x, y) within the current block. That is, the central sample (C) is a sample pointed to by a motion vector from position (x, y) within the reference picture.

[0148] When the autoregressive model does not use the central sample (C), but uses its northern (N), southern (S), eastern (E), and western (W) neighboring samples, the LIC model can be expressed as a convolutional filter as follows: c i are model parameters.

[0149]

[0150] When an autoregressive model uses a central sample (C) and its neighboring samples (N, S, E, W), the LIC model can be expressed as a convolutional filter as follows.

[0151]

[0152] When the autoregressive model additionally uses the non-linear term P(C) and the bias term B, the LIC model can be expressed as the following convolution filter. The non-linear term P(C) represents the normalized value of the square of the central sample (C) based on the bit depth, and the bias term B represents the median value according to the bit depth.

[0153]

[0154] When the autoregressive model uses the central sample (C), first-order neighbor samples (N, S, E, W) and second-order neighbor samples (NN, WW), the LIC model can be expressed as the following convolutional filter.

[0155]

[0156] Example 3: Single-Sample Input-Based Compensation

[0157] In this embodiment, a single-sample input-based compensation model is used for the current block, but a different set of model parameters is used for the samples of the current block. The single-sample input-based compensation model may be a linear model that generates a compensated prediction sample (i.e., a final prediction sample) using the initial prediction sample of the inter prediction for the current block.

[0158] The video encoder and video decoder can classify the initial prediction samples of the current block into several subsets and determine a set of parameters of a linear model to apply to each subset.

[0159] A video encoder and a video decoder can classify the neighboring samples of the current block and the neighboring samples of the reference block into multiple subsets, and use each subset as a training set to derive the parameters of a linear model for each subset. Instead of deriving the parameters of the linear model for each subset from the neighboring samples of the current block and the neighboring samples of the reference block, the video encoder can also signal an indication indicating the parameters of the linear model.

[0160] The criteria for classifying samples (such as initial prediction samples or neighboring samples) into subsets can be the coordinates of the sample (or the location of the sample) or the values ​​of the sample.

[0161] To classify samples into subsets based on their coordinates, a partitioning edge can be used that divides the current block into two sub-regions. The partitioning edge is a straight line determined based on an angle and an offset. Figure 11 is a conceptual diagram illustrating a partitioning edge and the angle and offset that specify it. Referring to Figure 11, the partitioning edge is specified by the angle of a perpendicular drawn from the center of the block to the partitioning edge and the offset, which is the distance from the center of the block to the partitioning edge. The angle of the perpendicular is measured counterclockwise from the horizontal axis. The angle and offset that specify the partitioning edge to be applied to the current block can each be signaled in the form of an index.

[0162] Figure 12 illustrates various segmentation boundaries that divide the current block into two sub-regions to which local brightness compensation is applied.

[0163] The video encoder and the video decoder can identify two subsets of initial prediction samples corresponding to two subregions, and individually determine parameters of an LIC model for the subsets of initial prediction samples. For example, the video encoder and the video decoder can identify a first subset of initial prediction samples corresponding to sample locations on a first subregion and a second subset of initial prediction samples corresponding to sample locations on a second subregion, and determine a first set of parameters of a linear model to be applied to the first subset of initial prediction samples and a second set of parameters of a linear model to be applied to the second subset of initial prediction samples. The parameters of the LIC model to be applied to the first subset of initial prediction samples of the current block may be different from the parameters of the LIC model to be applied to the second subset of initial prediction samples of the current block.

[0164] The first parameter set may be calculated based on reconstructed neighboring samples adjacent to the first sub-region and corresponding neighboring samples of the reference block, and the second parameter set may be calculated based on reconstructed neighboring samples adjacent to the second sub-region and corresponding neighboring samples of the reference block. That is, for a given subset of initial prediction samples, the parameter set may be derived from a portion of a template of a current block and a portion of a template of a reference block. The portion of the template of the current block may be referred to as a sub-template of the current block, and the portion of the template of the reference block may be referred to as a sub-template of the reference block.

[0165] In some embodiments, an index indicating a parameter set to be used for the LIC model applied to each subset of the subsets of initial prediction samples from a list of predefined parameter sets may be signaled. In some embodiments, syntax elements indicating the parameters to be used for the LIC model applied to each subset may be explicitly signaled.

[0166] Meanwhile, a threshold value compared to the sample value can be used to classify samples (such as initial prediction samples or neighboring samples) into subsets based on the sample value. The video encoder and video decoder can classify the initial prediction samples of the current block into two subsets based on the comparison between the sample values ​​of the initial prediction samples and the threshold value, and determine respective parameter sets of the linear model to be used to compensate for local brightness variations in each subset.

[0167] FIG. 13 is a conceptual diagram illustrating an example of using two different sets of parameters for a linear model based on a single threshold, according to some embodiments of the present disclosure. As shown in the mathematical equation below, a set of parameters of the linear model to be used for each of the initial prediction samples is determined based on a comparison of the initial prediction sample P[x, y] with the threshold.

[0168]

[0169] Since using different sets of parameters can be seen as using different models, we can also say that for each of the initial prediction samples, a linear model is selected from among two different linear models.

[0170] Depending on the implementation, more than one threshold value may be used. For example, two threshold values ​​may be used, such that for each initial prediction sample, one parameter set (or one linear model) may be selected from among three different parameter sets (or three linear models).

[0171] Thresholds can be derived implicitly or signaled explicitly.

[0172] As examples of methods for implicitly deriving threshold values, for example, a sample value at a predefined position (e.g., center, leftmost, topmost) of a template of a reference block corresponding to a template of a current block may be set as a threshold value, or an average value, median value, mode value, maximum value, etc. of sample values ​​at multiple predefined positions of the template of the reference block may be set as a threshold value.

[0173] As examples of how to explicitly signal a threshold, a syntax element indicating a threshold may be encoded on a block-by-block, sequence-by-sequence, or picture-by-picture basis. Furthermore, once a threshold is signaled for a block, the same threshold may be used for subsequent blocks until a new threshold is signaled.

[0174] The parameters of the LIC model for each subset can be derived from neighboring samples of the current block and neighboring samples of the reference block. For example, a video encoder and a video decoder can classify neighboring samples of a reference block into two subsets based on a comparison with a threshold. The video encoder and a video decoder can calculate or derive the parameters of the LIC model for each subset using the neighboring samples of each subset and the corresponding neighboring samples of the current block. Here, the same threshold can be used for classifying neighboring samples and classifying sample locations of the current block.

[0175] Instead of computing the parameters of the LIC model from neighboring samples of the current block and neighboring samples of the reference block, the video encoder may explicitly signal the parameters of the LIC model or explicitly signal an index pointing to a set of parameters.

[0176] Below, a mathematical method for deriving parameters of an m-th linear LIC model using neighboring samples of the m-th subset is described. The parameters of the m-th linear LIC model can be derived based on the Least Mean Square Error (LMSE) method using neighboring samples of the current block and neighboring samples of a reference block.

[0177] The mathematical expression of the mth linear LIC model is as follows.

[0178]

[0179] Here, p[x, y] is the brightness change compensated prediction sample at position [x, y] in the current block, and P[x, y] is the reference sample pointed to by the motion vector from position [x, y] in the reference picture. α m and β m represent the scaling parameter and offset parameter of the mth linear model, respectively, and α m and β m can be mathematically calculated as follows.

[0180]

[0181] Here, n is the number of neighboring samples used to derive the parameters ΣX, ΣY, ΣX 2 , the mathematical formulas for estimating ΣXY are as follows.

[0182]

[0183]

[0184]

[0185]

[0186] R above,m means the upper adjacent samples of the reference block used in the mth model, and R left,m denotes the left adjacent samples of the reference block used in the mth model, and Cabove,m means the upper adjacent samples of the current block used in the mth model, and C left,m refers to the left adjacent samples of the current block used in the mth model.

[0187] Below, we describe valid samples, which are neighboring samples of the current block used to calculate or derive the parameters of the LIC model.

[0188] Valid samples can be samples of the template of the current block (i.e., left column(s) and upper row(s)) or subsampled samples thereof. In addition, valid samples can be samples of the adjacent 0th reference line or nth reference line of the current block or subsampled samples thereof. Fig. 14 shows examples of subsampled samples used to compute or derive parameters of the LIC model from a template consisting of the left 1st column and the upper 1st row for an 8x8 CU and a 16x16 CU. The subsampling interval can depend on the block size as in the example of Fig. 14 or can be constant regardless of the block size.

[0189] In some embodiments, a method for selecting valid samples for each LIC model or each type of model may be predefined. For example, a video encoder and a video decoder may use samples of adjacent reference lines (i.e., left columns and upper rows) to derive parameters of a linear model, and use samples of adjacent templates (i.e., left columns and upper rows) to derive parameters of a nonlinear model. As another example, parameters of a first model may be derived based on samples of the left column(s), parameters of a second model may be derived based on samples of the upper row(s), and parameters of a third model may be derived based on samples of the left column(s) and upper row(s).

[0190] In some embodiments, a trained neural network or decision tree may be used to determine whether each neighboring sample of the current block is a valid sample. In some embodiments, the method for selecting valid samples on a picture-by-picture or sequence-by-sequence basis may be signaled.

[0191] The above-described embodiments or methods may be applied individually, but any combination of them may also be applied to a video encoder and a video decoder.

[0192] FIG. 15 is an example of a LIC-related syntax structure that may be included in a CU level according to one embodiment of the present disclosure.

[0193] When local brightness compensation is applied to the current CU, it can be signaled first whether the LIC merge mode corresponding to the method of the first embodiment is used.

[0194] lic_merge_flag, use_lic_cand_list, and merge_list_idx are syntax elements related to LIC merge mode, and their descriptions are as follows.

[0195] - lic_merge_flag is a syntax element that indicates whether LIC merge mode is used in the current CU. If it is 1, it means that LIC merge mode is used in the current CU, and if it is 0, it means that LIC merge mode is not used in the current CU.

[0196] - use_lic_cand_list is a syntax element that indicates whether a LIC merge candidate list is used in the LIC merge mode. If it is 1, the video decoder constructs a LIC merge candidate list and determines an LIC candidate to be used for the current CU based on the LIC merge candidate list. If it is 0, the video decoder determines a candidate to be used for the current CU based on a merge candidate list for motion information coding.

[0197] - merge_list_idx is a syntax element that indicates the index of the candidate to be used for the current CU within the candidate list.

[0198] In the example of FIG. 15, when the LIC merge mode is not used for the current CU, it is signaled whether the single pixel input based LIC mode corresponding to the method described in the third embodiment is used.

[0199] The syntax elements lic_single_sample_input, lic_single_sample_multi_model_idx, lic_angle_idx, lic_offset_idx, lic_explicit_parameter_flag, lic_model0_parameter, and lic_model1_parameter are related to the single-pixel input-based LIC mode.

[0200] - lic_single_sample_input is a syntax element that indicates whether to use the single-pixel input-based LIC mode. If it is 1, it means that the single-pixel input-based LIC mode is used, and if it is 0, it means that the single-pixel input-based LIC mode is not used.

[0201] - lic_single_sample_multi_model_idx is a syntax element that indicates whether to use the method of selecting a model to use among multiple LIC models based on coordinates for each sample when the single pixel input-based LIC mode is used. If it is 1, it means that the method of selecting a model to use among multiple LIC models based on coordinates for each sample is used, and if it is 0, it means that the method of selecting a model to use among multiple LIC models based on coordinates for each sample is not used.

[0202] - lic_angle_idx and lic_offset_idx are syntax elements that specify the partitioning edge that divides the current CU into two subsets when a method of selecting a model to use among multiple LIC models based on coordinates for each sample is used. lic_angle_idx means the angle of the perpendicular drawn from the center of the current CU to the partitioning edge, and lic_offset_idx means the distance from the center of the current CU to the partitioning edge. These syntax elements specify the geometric arrangement of the partitioning edge within the current CU.

[0203] - lic_explicit_parameter_flag is a syntax element that indicates whether the parameters of the LIC model applied to the current CU are explicitly signaled. If it is 1, the parameters of the LIC model applied to the current CU are signaled, and if it is 0, the parameters of the LIC model applied to the current CU are not signaled and are implicitly derived by the decoder.

[0204] - lic_model0_parameter and lic_model1_parameter are syntax elements representing parameters of the first LIC model and parameters of the second LIC model, respectively. If a single model is used instead of two models, the syntax element lic_model0_parameter syntax can be used to signal parameters of the single model.

[0205] The syntax elements lic_multi_sample_input, lic_regression_model0_idx and lic_regression_model1_idx relate to the multi-pixel input based LIC mode corresponding to the method described in the second embodiment.

[0206] - lic_multi_sample_input is a syntax element that indicates whether to use the multi-pixel input-based LIC mode. If it is 1, it means that the multi-pixel input-based LIC mode is used, and if it is 0, it means that the multi-pixel input-based LIC mode is not used.

[0207] - lic_regression_model0_idx and lic_regression_model1_idx are syntax elements representing the parameters of the first LIC model and the parameters of the second LIC model among the two LIC models used in the multi-pixel input-based LIC mode, respectively. If a single model is used instead of two models, the syntax element lic_regression_model0_idx can be used to signal the parameters of the single model.

[0208] FIG. 16 illustrates a CU level syntax including a syntax element lic_mode_idx indicating an index of a LIC mode applied to the current CU, according to some embodiments of the present disclosure. FIG. 17 illustrates an example of binarization of lic_mode_idx for various LIC modes, according to some embodiments of the present disclosure.

[0209] Instead of the syntax elements lic_merge_flag, lic_multi_sample_input, and lic_multi_sample_input illustrated in Fig. 16, the methods indicated by these syntax elements may each be defined as one available LIC mode. The video encoder may signal the index (lic_mode_idx) of the LIC mode selected from among the multiple available LIC modes.

[0210] The multiple available LIC modes may include a mode corresponding to the method using the LIC merge list of the first embodiment, a mode corresponding to the method using multiple autoregressive models of the second embodiment, a mode corresponding to the method using multiple linear models of the third embodiment, and a mode corresponding to the method using a single autoregressive model of the second embodiment. In addition, a conventional method of using a single linear LIC model for all samples within a block may be included as one available mode among the multiple available LIC modes.

[0211] FIG. 18 is a flowchart illustrating a prediction method performed by each of a video encoder and a video decoder according to some embodiments of the present disclosure.

[0212] The video encoder and video decoder can derive initial prediction samples for a current block in a current picture based on a reference block in a reference picture (S1810).

[0213] The video encoder and video decoder can determine a local brightness compensation (LIC) model for the current block from among a plurality of available LIC models (S1820).

[0214] The video encoder and video decoder can generate final prediction samples for the current block by applying local brightness compensation to initial prediction samples for the current block based on the LIC model for the current block (S1830).

[0215] In some embodiments, the video encoder and video decoder may, as part of step S1820, compute a cost for a template, which is a set of reconstructed samples neighboring the current block, for each of a plurality of available LIC models, and determine a LIC model for the current block based on the cost for the template for each of the plurality of available LIC models.

[0216] Here, the cost for a template can be calculated based on the difference between the reconstructed neighboring samples of the template and the compensated predictions of the template obtained by applying the candidate model to the corresponding neighboring samples of the reference block.

[0217] In some embodiments, the video encoder and the video decoder, as part of step S1820, may decode an index indicating a candidate in a merge candidate list for coding motion information of the current block from the bitstream, and determine a LIC model for the current block using LIC model information inherited from the block of the candidate indicated by the decoded index.

[0218] In some embodiments, the video encoder and the video decoder may, as part of step S1820, construct a LIC merge candidate list using LIC model information of other blocks already reconstructed on the current picture or reference picture. The video encoder and the video decoder may decode an index indicating a candidate in the LIC merge candidate list from the bitstream, and determine a LIC model for the current block using the LIC model information of the candidate indicated by the decoded index.

[0219] In some embodiments, at least one of the plurality of available LIC models may be a convolutional filter based on an autoregressive model. At least one of the plurality of available LIC models may be a linear model.

[0220] In some embodiments, the video encoder and the video decoder, as part of step S1830, may determine a partitioning boundary that divides the current block into two subregions, and identify two subsets of initial prediction samples corresponding to the two subregions. The video encoder and the video decoder may determine a first parameter set of a linear model to apply to a first subset of initial prediction samples corresponding to a first subregion of the two subregions, and a second parameter set of a linear model to apply to a second subset of initial prediction samples corresponding to a second subregion of the two subregions.

[0221] The first parameter set may be calculated based on reconstructed neighboring samples adjacent to the first sub-region and corresponding neighboring samples of the reference block, and the second parameter set may be calculated based on reconstructed neighboring samples adjacent to the second sub-region and corresponding neighboring samples of the reference block. The video encoder and the video decoder may apply a linear model having the first parameter set to the first subset of initial prediction samples and apply a linear model having the second parameter set to the second subset of initial prediction samples to generate final prediction samples for the current block.

[0222] In some embodiments, the video encoder and the video decoder may, as part of step S1830, identify two subsets of the initial prediction samples based on comparing the values ​​of the initial prediction samples to a threshold value, and determine a first set of parameters of the linear model to apply to the first subset of the initial prediction samples and a second set of parameters of the linear model to apply to the second subset of the initial prediction samples. Here, the threshold value may be calculated based on reconstructed samples neighboring a reference block corresponding to a template, which is a set of reconstructed samples neighboring the current block. For example, the threshold value may be an average value of some or all of the reconstructed samples neighboring the reference block.

[0223] The video encoder and the video decoder, as part of the step of determining the first parameter set and the second parameter set, can identify two subsets of neighboring samples of the reference block based on comparing neighboring samples of the reference block with the threshold value. The video encoder and the video decoder can determine the first parameter set based on the first subset of neighboring samples of the reference block and the first subset of neighboring samples of the corresponding current block, and can determine the second parameter set based on the second subset of neighboring samples of the reference block and the second subset of neighboring samples of the corresponding current block.

[0224] The video encoder and video decoder can generate final prediction samples for the current block by applying a linear model having a first set of parameters to a first subset of the initial prediction samples and by applying a linear model having a second set of parameters to a second subset of the initial prediction samples.

[0225] A video encoder can encode a current block based on the final prediction samples. For example, the video encoder can encode residual data representing the difference between the original samples of the current block and the final prediction samples into a bitstream. A video decoder can decode the current block based on the final prediction samples. For example, the video decoder can decode residual data representing the difference between the original samples of the current block and the final prediction samples from the bitstream, and reconstruct the current block by adding the final prediction samples to the residual data.

[0226] Although the flowchart / timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of ​​one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart / timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart / timing diagram is not limited to a chronological order.

[0227] It should be understood that the exemplary embodiments described above can be implemented in many different ways. The functions or methods described in one or more examples can be implemented in hardware, software, firmware, or any combination thereof. It should be understood that the functional components described herein are labeled as "units" to further emphasize their implementation independence.

[0228] Meanwhile, the various functions or methods described in this embodiment may also be implemented as instructions stored on a non-transitory storage medium that can be read and executed by one or more processors. Non-transitory storage media include, for example, all types of storage devices that store data in a form readable by a computer system. For example, non-transitory storage media include storage media such as erasable programmable read-only memory (EPROM), flash drives, optical drives, magnetic hard drives, and solid-state drives (SSDs).

[0229] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art to which the present embodiment pertains may make various modifications and variations without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The protection scope of the present embodiment should be interpreted by the following claims, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

[0230]

[0231]

[0232] CROSS-REFERENCE TO RELATED APPLICATION

[0233] This patent application claims priority to Korean patent application No. 10-2024-0050282, filed on April 15, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. In a prediction method performed by a video decoding device, A step of deriving initial prediction samples for a current block in a current picture based on a reference block in a reference picture; determining a local illumination compensation (LIC) model for the current block among multiple available LIC models; and A step of generating final prediction samples for the current block by applying local brightness compensation to initial prediction samples for the current block based on the LIC model for the current block. How to include.

2. In paragraph 1, The step of determining the LIC model for the current block above is: For each of the plurality of available LIC models, calculating a cost for a template, which is a set of reconstructed samples neighboring the current block; and determining a LIC model for the current block based on the cost for each of the templates of the plurality of available LIC models; A method characterized by including:

3. In paragraph 2, The cost for the above template is: A method characterized in that it is calculated based on the difference between the compensated predictions of the template obtained by applying the candidate model to the reconstructed neighboring samples of the template and the corresponding neighboring samples of the reference block.

4. In paragraph 1, The step of determining the LIC model for the current block above is: A step of decoding an index indicating a candidate in a merge candidate list for coding motion information of the current block from a bitstream; and A step of determining a LIC model for the current block using LIC model information inherited from a block of a candidate indicated by the decrypted index. A method characterized by including:

5. In paragraph 1, The step of determining the LIC model for the current block above is: A step of constructing a LIC merge candidate list using LIC model information of other blocks already reconstructed on the current picture or the reference picture; A step of decoding an index indicating a candidate in the LIC merge candidate list from the bitstream; and A step of determining a LIC model for the current block using the LIC model information of the candidate indicated by the decrypted index. A method characterized by including:

6. In paragraph 1, A method, characterized in that at least one of the plurality of available LIC models is an autoregressive model.

7. In paragraph 1, A method, characterized in that at least one of the plurality of available LIC models is a linear model.

8. In paragraph 7, If the LIC model for the current block is the linear model, the step of generating the final prediction samples is: A step of determining a division boundary that divides the current block into two sub-regions; identifying two subsets of the initial prediction samples corresponding to the two subregions; and A step of determining a first parameter set of the linear model to be applied to a first subset of the initial prediction samples corresponding to a first sub-region among the two sub-regions, and a second parameter set of the linear model to be applied to a second subset of the initial prediction samples corresponding to a second sub-region among the two sub-regions. A method characterized by including 9. In paragraph 8, A method characterized in that a first parameter set is calculated based on reconstructed neighboring samples adjacent to the first sub-region and corresponding neighboring samples of the reference block, and a second parameter set is calculated based on reconstructed neighboring samples adjacent to the second sub-region and corresponding neighboring samples of the reference block.

10. In paragraph 7, If the LIC model for the current block is the linear model, the step of generating the final prediction samples is: identifying two subsets of the initial prediction samples based on comparing the values ​​of the initial prediction samples with a threshold value; and A step of determining a first parameter set of the linear model to be applied to a first subset of the initial prediction samples and a second parameter set of the linear model to be applied to a second subset of the initial prediction samples. A method characterized by including:

11. In paragraph 10, The above threshold is, A method characterized in that the reconstructed samples neighboring the reference block corresponding to the template, which is a set of reconstructed samples neighboring the current block, are calculated based on the reconstructed samples neighboring the reference block.

12. In paragraph 10, The step of determining the first parameter set and the second parameter set comprises: A step of identifying two subsets of neighboring samples of the reference block based on comparing neighboring samples of the reference block with the threshold value; and A step of determining the first parameter set based on a first subset of neighboring samples of the reference block and a first subset of corresponding neighboring samples of the current block, and determining the second parameter set based on a second subset of neighboring samples of the reference block and a second subset of corresponding neighboring samples of the current block. A method characterized by including 13. In a prediction method performed by a video encoding device, A step of deriving initial prediction samples for a current block in a current picture based on a reference block in a reference picture; determining a local illumination compensation (LIC) model for the current block among multiple available LIC models; and A step of generating final prediction samples for the current block by applying local brightness compensation to initial prediction samples for the current block based on the LIC model for the current block. A method comprising:

14. A method for providing video data to a video decoding device. A step of encoding the above video data into a bitstream; and A step of transmitting the bitstream to the video decoding device and the step of encoding the video data into a bitstream includes: A step of deriving initial prediction samples for a current block in a current picture based on a reference block in a reference picture; determining a local illumination compensation (LIC) model for the current block among multiple available LIC models; and A step of generating final prediction samples for the current block by applying local brightness compensation to initial prediction samples for the current block based on the LIC model for the current block. A method characterized by including:

Images