Selective use of local brightness compensation models and devices using these models
By modeling the brightness change between the current block and the reference block using a local brightness compensation model, the problem of insufficient coding efficiency in existing video compression technologies is solved, and the coding performance of the inter-frame predictor is improved.
Patent Information
- Application Number
- CN202580027819.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-04-15
- Filing Date
- 2025-03-28
- Publication Date
- 2026-11-17
AI Technical Summary
Existing video compression technologies are insufficient in coding efficiency when faced with increases in video size, resolution, and frame rate, and cannot effectively compensate for local brightness changes between the current block and the reference block.
The Local Luminance Compensation (LIC) model is used to model the brightness variation between the current block and the reference block. By selecting a suitable linear or nonlinear model for compensation, the final prediction sample is generated.
It improves the coding performance of the inter-frame predictor and enhances the compensation effect for local brightness variations.
Smart Images

Figure CN123125872A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the encoding and decoding of video data, and more specifically, to a method for performing local brightness refinement by modeling the local brightness variations between the current block and a reference block, and to encoding and decoding devices using this method. Background Technology
[0002] The statements in this section are provided only as background information in connection with this disclosure and do not necessarily constitute prior art.
[0003] Compared to audio or still image data, unprocessed or uncompressed video data is much larger, and storing or transmitting it in its raw form requires significant hardware resources, including memory.
[0004] Therefore, when storing or transmitting video data, encoders are typically used to compress the video data for storage or transmission, while decoders receive the compressed video data and perform its decompression and reconstruction. These video compression techniques include H.264 / AVC, High Efficiency Video Coding (HEVC), and Multi-Functional Video Coding (VVC), which offer approximately 30% or more improvement in coding efficiency compared to HEVC.
[0005] However, as video size, resolution, and frame rate continue to increase, the amount of data that needs to be encoded also increases accordingly. Therefore, there is a need for new compression technologies that are more efficient in encoding and more effective in improving video quality than existing compression technologies. Summary of the Invention
[0006] Technical issues
[0007] This disclosure aims to provide a method for performing compensation on a predictor for inter-frame prediction based on multiple models to compensate for local brightness variations between the current block and a reference block.
[0008] This disclosure aims to provide various linear or nonlinear models for modeling local brightness variations between the current block and a reference block.
[0009] Technical solution
[0010] At least one aspect of this disclosure provides a prediction method performed by a video decoding device. The method includes: deriving an initial prediction sample of a current block in a current image based on a reference block in a reference image. The method further includes: determining a Local Luminance Compensation (LIC) model for the current block from a plurality of available LIC models. The method further includes: generating a final prediction sample of the current block by applying local luminance compensation to the initial prediction sample of the current block based on the LIC model of the current block.
[0011] In another aspect of this disclosure, determining the LIC model of the current block includes: for each of a plurality of available LIC models, calculating the cost of a template (i.e., a set of reconstructed samples adjacent to the current block), and determining the LIC model of the current block based on the template cost of each of the plurality of available LIC models.
[0012] In another aspect of this disclosure, determining the LIC model of the current block includes: decoding indices of candidates in a merge candidate list that indicate the encoding of motion information for the current block from the bitstream, and determining the LIC model of the current block using LIC model information inherited by the candidate blocks indicated by the decoded indices.
[0013] In another aspect of this disclosure, determining the LIC model of the current block includes: constructing a LIC merge candidate list using LIC model information of other blocks that have been reconstructed on the current picture or a reference picture, decoding indices indicating candidates in the LIC merge candidate list from the bitstream, and determining the LIC model of the current block using the LIC model information of the candidates indicated by the decoded indices.
[0014] In another aspect of this disclosure, generating the final prediction sample includes: determining the segmentation boundary that divides the current block into two sub-regions, identifying two subsets of the initial prediction samples corresponding to the two sub-regions, and determining a first set of parameters of a linear model to be applied to the first subset of the initial prediction samples corresponding to the first sub-region of the two sub-regions, and a second set of parameters of a linear model to be applied to the second subset of the initial prediction samples corresponding to the second sub-region of the two sub-regions.
[0015] In another aspect of this disclosure, generating the final predicted sample includes: identifying two subsets of the initial predicted sample based on comparing the values of the initial predicted sample with a threshold, and determining a first set of parameters of a linear model to be applied to the first subset of the initial predicted sample and a second set of parameters of a linear model to be applied to the second subset of the initial predicted sample.
[0016] Another aspect of this disclosure provides a prediction method performed by a video coding device. The method includes: deriving an initial prediction sample for a current block in a current image based on a reference block in a reference image. The method further includes: determining a local illumination compensation (LIC) model for the current block from multiple available LIC models. The method also includes: generating a final prediction sample for the current block by applying local illumination compensation to the initial prediction sample based on the LIC model for the current block.
[0017] Another aspect of this disclosure provides a method for providing video data to a video decoding device. The method includes: encoding the video data into a bitstream and transmitting the bitstream to the video decoding device. The encoding of the video data includes: deriving an initial prediction sample of a current block in a current image based on a reference block in a reference image. The encoding of the video data further includes: determining a Local Luminance Compensation (LIC) model for the current block from a plurality of available LIC models. The encoding of the video data further includes: generating a final prediction sample for the current block by applying local luminance compensation to the initial prediction sample of the current block based on the LIC model of the current block.
[0018] Beneficial effects
[0019] As described above, this disclosure allows selection of a suitable model from various linear or nonlinear models for a given block. Compared to using a single linear model to compensate for local brightness variations at all sample locations within the current block, this model provides improved compensation for local brightness variations within the block. Therefore, the coding performance of the predictor for inter-frame prediction can be improved. Attached Figure Description
[0020] Figure 1 This is a block diagram of a video encoding device that can implement the technology disclosed herein.
[0021] Figure 2 The diagram illustrates a method for segmenting blocks using a quadtree plus binary tree ternary tree (QTBTTT) structure.
[0022] Figure 3a and Figure 3b The illustration shows several intra-prediction modes, including the wide-angle intra-prediction mode.
[0023] Figure 4 The diagram shows the adjacent blocks of the current block.
[0024] Figure 5 This is a block diagram of a video decoding device that can implement the technology disclosed herein.
[0025] Figure 6 This is a conceptual diagram illustrating local brightness compensation.
[0026] Figure 7 This is a conceptual diagram illustrating an example of inappropriate local brightness compensation applied to all samples of a block using a single linear model.
[0027] Figure 8 This is an example of a high-level syntax (HLS) for a method of determining model parameters by signaling, according to one embodiment of this disclosure.
[0028] Figure 9a This is a conceptual diagram of spatially adjacent and spatially non-adjacent positions of the current block, based on some embodiments of this disclosure.
[0029] Figure 9b This is a conceptual diagram of the location of temporal candidates according to some embodiments of this disclosure.
[0030] Figure 9c This is a conceptual diagram of the position of the moved temporal candidate according to some embodiments of this disclosure.
[0031] Figure 10 The diagram illustrates the spatial location of the central sample and its adjacent samples.
[0032] Figure 11 It is a conceptual diagram illustrating the dividing boundary and the angle and offset of the specified dividing boundary.
[0033] Figure 12 This is a conceptual diagram illustrating various segmentation boundaries that divide the current block, which applies local brightness compensation, into two sub-regions.
[0034] Figure 13 This is a conceptual diagram illustrating examples of two different parameter sets using a linear model based on a threshold, according to some embodiments of this disclosure.
[0035] Figure 14 Examples of downsampled samples for calculating or deriving LIC model parameters according to some embodiments of this disclosure are shown.
[0036] Figure 15 This is an example of a LIC-related syntax structure that can be included at the CU level according to one embodiment of this disclosure.
[0037] Figure 16 The illustration shows CU-level syntax including lic_mode_idx according to some embodiments of this disclosure.
[0038] Figure 17 This is an example of binarization of various LIC modes of lic_mode_idx according to some implementations of this disclosure.
[0039] Figure 18 This is a flowchart of a prediction method performed by each of the video encoder and the video decoder according to some embodiments of this disclosure. Detailed Implementation
[0040] In the following description, some embodiments of the present disclosure are described in detail with reference to the accompanying drawings. In the following description, the same reference numerals denote the same elements, even if these elements are shown in different drawings. Furthermore, in the following description of some embodiments, detailed descriptions of relevant known components and functions may be omitted for clarity and brevity when it is thought that they might obscure the subject matter of the present disclosure.
[0041] Figure 1 This is a block diagram of a video encoding apparatus capable of implementing the technology disclosed herein. In the following, reference is made to... Figure 1 The diagram illustrates a video encoding device and its components.
[0042] The encoding device may include an image segmenter 110, a predictor 120, a subtractor 130, a transformer 140, a quantizer 145, a rearrangement unit 150, an entropy encoder 155, an inverse quantizer 160, an inverse transformer 165, an adder 170, a loop filter unit 180, and a memory 190.
[0043] Each component of the encoding device can be implemented as hardware or software, or a combination of hardware and software. Furthermore, the function of each component can be implemented as software, or a microprocessor can be implemented to execute the software function corresponding to each component.
[0044] A video consists of one or more sequences containing multiple images. Each image is segmented into multiple regions, and encoding is performed on each region. For example, an image is segmented into one or more tiles and / or stripes. Here, one or more tiles can be defined as a tile group. Each tile and / or strip is segmented into one or more coding tree units (CTUs). Furthermore, each CTU is segmented into one or more coding units (CUs) through a tree structure. Information applied to each coding unit (CU) is encoded as the syntax of that CU, and common information applied to the CUs included in a CTU is encoded as the syntax of that CTU. Furthermore, common information applied to all blocks in a strip is encoded as the syntax of the strip header, and information applied to all blocks constituting one or more images is encoded as a picture parameter set (PPS) or picture header. Furthermore, information commonly referenced by multiple images is encoded as a sequence parameter set (SPS). Furthermore, information commonly referenced by one or more SPSs is encoded as a video parameter set (VPS). Additionally, common information applied to a tile or tile group can also be encoded as the syntax of the tile or tile group header. Syntax included in SPS, PPS, strip headers, tile or tile group headers can be referred to as advanced syntax.
[0045] Image splitter 110 determines the size of the coding tree unit (CTU). Information about the size of the CTU (CTU size) is encoded into SPS or PPS syntax and passed to the video decoding device.
[0046] Image segmenter 110 segments each image constituting the video into multiple coding tree units (CTUs) of predetermined size, and then recursively segments the CTUs using a tree structure. The leaf nodes in the tree structure become coding units (CUs) that serve as the basic units of coding.
[0047] A tree structure can be a quadtree (QT), where a higher node (or parent node) is divided into four lower nodes (or child nodes) of the same size. A tree structure can also be a binary tree (BT), where a higher node is divided into two lower nodes. A tree structure can also be a ternary tree (TT), where a higher node is divided into three lower nodes in a 1:2:1 ratio. A tree structure can also be a mixture of two or more structures from QT, BT, and TT. For example, a quadtree plus binary tree (QTBT) structure can be used, as can a quadtree plus binary tree ternary tree (QTBTTT) structure. Here, a binary tree ternary tree (BTTT) is added to the tree structure, referred to as a multi-type tree (MTT).
[0048] Figure 2 This is a diagram used to describe the method of segmenting blocks using the QTBTTT structure.
[0049] like Figure 2 As shown, the CTU can first be segmented into a QT structure. Quadtree segmentation can be performed recursively until the size of the segmented block reaches the minimum block size (MinQTSize) allowed for a leaf node in the QT. A first flag (QT_split_flag) indicating whether each node in the QT structure is segmented into four lower-level nodes is encoded by the entropy encoder 155 and transmitted to the video decoding device. When a leaf node of the QT is not larger than the maximum block size (MaxBTSize) allowed for a root node in the BT, the leaf node can be further segmented into at least one of a BT structure or a TT structure. Multiple segmentation directions can exist in the BT structure and / or TT structure. For example, there can be two directions: the direction in which the block of the corresponding node is segmented horizontally and the direction in which it is segmented vertically. Figure 2 As shown, when MTT splitting begins, a second flag (mtt_split_flag) indicating whether a node is split, an additional flag indicating the splitting direction (vertical or horizontal), and / or a flag indicating the splitting type (binary or ternary) when a node is split are encoded by the entropy encoder 155 and transmitted to the video decoding device.
[0050] Alternatively, before encoding the first flag (QT_split_flag) indicating whether each node is split into the four lower-level nodes, a CU split flag (split_cu_flag) indicating whether the node is split can also be encoded. When the value of the CU split flag (split_cu_flag) indicates that each node is not split, the block of the corresponding node becomes a leaf node in the split tree structure and becomes a CU as the basic unit of encoding. When the value of the CU split flag (split_cu_flag) indicates that each node is split, the video encoding device first begins encoding the first flag according to the above scheme.
[0051] When using QTBT as another example of a tree structure, there can be two types: a type where the corresponding node's block is horizontally split into two blocks of the same size (i.e., symmetrical horizontal split) and a type where the corresponding node's block is vertically split into two blocks of the same size (i.e., symmetrical vertical split). A flag (split_flag) indicating whether each node in the BT structure is split into lower-level blocks and split type information indicating the split type are encoded by the entropy encoder 155 and transmitted to the video decoding device. Additionally, there can be a type where the corresponding node's block is split into two blocks that are asymmetrically divided. Asymmetrical forms can include the form where the corresponding node's block is split into two rectangular blocks with a 1:3 size ratio, or the form where the corresponding node's block is split along a diagonal direction.
[0052] The CU can be segmented from the CTU and has various sizes. In the following text, the block corresponding to the CU (i.e., the leaf node of the QTBTTT) to be encoded or decoded is referred to as the "current block". With the use of QTBTTT segmentation, the shape of the current block can be rectangular in addition to square.
[0053] Predictor 120 predicts the current block to generate a prediction block. Predictor 120 includes an intra-frame predictor 122 and an inter-frame predictor 124.
[0054] Typically, each current block in an image can be predictively encoded. This prediction can usually be performed using either intra-frame prediction (using data from the image that includes the current block) or inter-frame prediction (using data from an image encoded before the image that includes the current block). Inter-frame prediction includes both one-way and two-way prediction.
[0055] Intra-predictor 122 uses pixels located at neighboring positions (reference pixels) of the current block within the current image, including the current block, to predict pixels in the current block. Multiple intra-prediction modes exist depending on the prediction direction. For example, such as... Figure 3a As shown, multiple intra-frame prediction modes can include two non-directional modes, including a planar mode and a DC mode, and can include 65 directional modes. The neighboring pixels and arithmetic equations to be used are defined differently for each prediction mode.
[0056] To perform efficient directional prediction on the current block with a rectangular shape, additional methods can be used. Figure 3b The directional modes (intra-prediction modes #67 to #80, #-1 to #-14) are illustrated by dashed arrows. These directional modes can be referred to as "wide-angle intra-prediction modes." Figure 3bIn the diagram, the arrow indicates the corresponding reference sample used for prediction and does not indicate the prediction direction. The prediction direction is opposite to the direction indicated by the arrow. When the current block has a rectangular shape, the wide-angle intra-frame prediction mode is a mode that performs prediction in the opposite direction to the specific directional mode without requiring additional bit transmission. In this case, some wide-angle intra-frame prediction modes available for the current block can be determined by the ratio of the width to the height of the current block with a rectangular shape. For example, when the current block has a rectangular shape with a height less than its width, wide-angle intra-frame prediction modes with an angle less than 45 degrees can be used (intra-frame prediction modes #67 to #80). When the current block has a rectangular shape with a width greater than its height, wide-angle intra-frame prediction modes with an angle greater than -135 degrees can be used.
[0057] Intra-predictor 122 can determine the intra-prediction to be used for encoding the current block. In some examples, intra-predictor 122 can encode the current block using multiple intra-prediction modes, or it can select a suitable intra-prediction mode to use from the tested modes. For example, intra-predictor 122 can calculate rate-distortion values by using rate-distortion analysis on multiple tested intra-prediction modes, or it can select the intra-prediction mode with the best rate-distortion characteristics from the tested modes.
[0058] Intra-predictor 122 selects one intra-prediction mode from a plurality of intra-prediction modes and uses neighboring pixels (reference pixels) determined according to the selected intra-prediction mode and an arithmetic equation to predict the current block. Information about the selected intra-prediction mode is encoded by entropy encoder 155 and transmitted to video decoding device.
[0059] Inter-frame predictor 124 generates a predicted block for the current block using a motion compensation process. Inter-frame predictor 124 searches for the most similar block to the current block in a reference image that was encoded and decoded earlier than the current image, and uses the searched block to generate the predicted block for the current block. Furthermore, a motion vector (MV) is generated, corresponding to the displacement between the current block in the current image and the predicted block in the reference image. Typically, motion estimation is performed on the luma component, and the motion vector calculated based on the luma component is used for both the luma and chroma components. Motion information, including information about the reference image used to predict the current block and information about the motion vector, is encoded by entropy encoder 155 and transmitted to the video decoding device.
[0060] The inter-frame predictor 124 can also perform interpolation on a reference image or reference block to improve prediction accuracy. In other words, it interpolates subsamples between two consecutive integer samples by applying filter coefficients to multiple consecutive integer samples, including two integers. When performing the process of searching for the most similar block to the current block on the interpolated reference image, the motion vector can be represented with fractional precision instead of integer sample precision. The precision or resolution of the motion vector can be set differently for each target region to be encoded (e.g., stripe, tile, CTU, CU, etc.). When this adaptive motion vector resolution (AMVR) is applied, the information on the motion vector resolution to be applied to each target region should be signaled for each target region. For example, when the target region is a CU, the information on the motion vector resolution applied to each CU is signaled. The motion vector resolution information can be information representing the precision of the motion vector difference, which will be described later.
[0061] Simultaneously, the inter-frame predictor 124 can perform inter-frame prediction using bidirectional prediction. In the case of bidirectional prediction, two reference images and two motion vectors representing the location of the most similar block to the current block in each reference image are used. The inter-frame predictor 124 selects a first reference image and a second reference image from reference image list 0 (RefPicList0) and reference image list 1 (RefPicList1), respectively. The inter-frame predictor 124 also searches for the most similar block to the current block in each reference image to generate a first reference block and a second reference block. Furthermore, a predicted block for the current block is generated by averaging or weighted averaging the first reference block and the second reference block. In addition, motion information including information about the two reference images used to predict the current block and information about the two motion vectors is transmitted to the entropy encoder 155. Here, reference image list 0 may consist of images from the pre-reconstructed images that appear before the current image in display order, and reference image list 1 may consist of images from the pre-reconstructed images that appear after the current image in display order. However, although not particularly limited to this, pre-reconstructed images that appear after the current image in display order may be appended to reference image list 0. Conversely, pre-reconstructed images that precede the current image in the display order can also be included in the reference image list 1.
[0062] Various methods can be used to minimize the number of bits consumed in encoding motion information.
[0063] For example, when the reference image and motion vector of the current block are the same as those of the adjacent blocks, the encoder can identify the information of the adjacent blocks to transmit the motion information of the current block to the video decoding device. This method is called merging mode.
[0064] In merge mode, the inter-frame predictor 124 selects a predetermined number of merge candidate blocks (hereinafter referred to as "merge candidates") from the neighboring blocks of the current block.
[0065] As adjacent blocks used to derive merge candidates, they can be used as follows: Figure 4 The image shows all or part of the following blocks immediately adjacent to the current block: left block A0, lower left block A1, upper block B0, upper right block B1, and upper left block B2. Additionally, blocks located in a reference image outside the current image (which may be the same as or different from the reference image used to predict the current block) can also be used as merge candidates. For example, blocks in the reference image that are co-located with the current block or adjacent to a co-located block can be added as merge candidates. If the number of merge candidates selected using the above method is less than a preset number, a zero vector is added to the merge candidates.
[0066] The inter-frame predictor 124 uses a merge list containing a predetermined number of merge candidates by configuring adjacent blocks. It selects a merge candidate from the merge list to use as motion information for the current block and generates merge index information to identify the selected candidate. The generated merge index information is encoded by the entropy encoder 155 and transmitted to the video decoding device.
[0067] Merge skip mode is a special case of merging mode. After quantization, when all transform coefficients used for entropy coding are close to zero, only the adjacent block selection information is transmitted, and the residual signal is not transmitted. By using merge skip mode, relatively high coding efficiency can be achieved for images with slight motion, still images, and screen content images.
[0068] In the following text, merge mode and merge skip mode are collectively referred to as merge / skip mode.
[0069] Another method for encoding motion information is the Advanced Motion Vector Prediction (AMVP) pattern.
[0070] In AMVP mode, the inter-frame predictor 124 derives motion vector prediction candidates for the current block by using neighboring blocks of the current block. Neighboring blocks used to derive motion vector prediction candidates can be, for example... Figure 4The diagram shows all or part of the left block A0, lower left block A1, upper block B0, upper right block B1, and upper left block B2 that are immediately adjacent to the current block in the current image. Additionally, blocks located in a reference image outside the current image (which may be the same as or different from the reference image used to predict the current block) can also be used as neighboring blocks for deriving motion vector prediction candidates. For example, blocks in the reference image that are in the same position as the current block or adjacent to the block in the same position can be used. If the number of motion vector candidates selected by the above method is less than a preset number, a zero vector is added to the motion vector candidates.
[0071] Inter-frame predictor 124 derives motion vector prediction candidates using motion vectors from adjacent blocks, and determines a motion vector predictor for the motion vector of the current block using these candidates. Furthermore, the motion vector difference is calculated by subtracting the motion vector predictor from the motion vector of the current block.
[0072] Motion vector predictors can be obtained by applying predefined functions (such as center value and average value calculations) to motion vector prediction candidates. In this case, the video decoding device is also aware of the predefined function. Furthermore, since the neighboring blocks used to derive the motion vector prediction candidates are already encoded and decoded, the video decoding device may already know the motion vectors of those neighboring blocks. Therefore, the video encoding device does not need to encode information for identifying motion vector prediction candidates. Thus, in this case, information about the motion vector differences and information about the reference image used to predict the current block are encoded.
[0073] Alternatively, the motion vector predictor can be determined by selecting any one of the motion vector prediction candidates. In this case, the information used to identify the selected motion vector prediction candidate is encoded along with information about the motion vector difference and information about the reference image used to predict the current block.
[0074] Subtractor 130 generates a residual block by subtracting the prediction block generated by intra-predictor 122 or inter-predictor 124 from the current block.
[0075] Transformer 140 transforms the residual signal in the residual block, which has pixel values in the spatial domain, into transform coefficients in the frequency domain. Transformer 140 can use the total size of the residual block as a transform unit to transform the residual signal in the residual block, or it can divide the residual block into multiple sub-blocks and use these sub-blocks as transform units to perform the transformation. Alternatively, the residual block can be divided into two sub-blocks: a transform region and a non-transform region, so that only the transform region sub-block is used as a transform unit to transform the residual signal. Here, the transform region sub-block can be one of two rectangular blocks with a 1:1 size ratio based on the horizontal (or vertical) axis. In this case, a flag (cu_sbt_flag) indicates that only this sub-block is transformed, and the orientation (vertical / horizontal) information (cu_sbt_horizontal_flag) and / or position information (cu_sbt_pos_flag) are encoded by entropy encoder 155 and transmitted to the video decoding device. Furthermore, the size of the transform region sub-block can have a 1:3 size ratio based on the horizontal (or vertical) axis. In this case, the flag that defines the segment (cu_sbt_quad_flag) is additionally encoded by the entropy encoder 155 and transmitted to the video decoding device.
[0076] Simultaneously, the transformer 140 can perform transformations on the residual blocks separately in the horizontal and vertical directions. Various types of transformation functions or transformation matrices can be used for this transformation. For example, a pair of transformation functions for horizontal and vertical transformations can be defined as a Multiple Transform Set (MTS). The transformer 140 can select the most efficient pair of transformation functions from the MTS and transform the residual blocks in each of the horizontal and vertical directions. The information of the transformation function pair in the MTS (mts_idx) is encoded by the entropy encoder 155 and transmitted to the video decoding device.
[0077] Quantizer 145 uses quantization parameters to quantize the transform coefficients output from transformer 140 and outputs the quantized transform coefficients to entropy encoder 155. Quantizer 145 can also directly quantize relevant residual blocks without transforming any block or frame. Quantizer 145 can also apply different quantization coefficients (scaling values) based on the position of the transform coefficients within the transform block. The quantization matrix applied to the quantized transform coefficients arranged in two dimensions can be encoded and transmitted to a video decoding device.
[0078] The rearrangement unit 150 can rearrange the coefficient values of the quantized residuals.
[0079] The rearrangement unit 150 can transform a 2D coefficient array into a 1D coefficient sequence using coefficient scanning. For example, the rearrangement unit 150 can output a 1D coefficient sequence by scanning from DC coefficients to high-frequency domain coefficients using a zig-zag scan or a diagonal scan. Depending on the size of the transform unit and the intra-frame prediction mode, a vertical scan scanning the 2D coefficient array along the column direction and a horizontal scan scanning the 2D block-type coefficients along the row direction can also be used instead of a zig-zag scan. In other words, the scanning method to be used can be determined among zig-zag scan, diagonal scan, vertical scan, and horizontal scan, depending on the size of the transform unit and the intra-frame prediction mode.
[0080] The entropy encoder 155 generates a bitstream by encoding the 1D quantized transform coefficient sequence output from the rearrangement unit 150 using various encoding schemes, including context-based adaptive binary arithmetic code (CABAC), exponential Columbus, etc.
[0081] Furthermore, the entropy encoder 155 encodes block-segmentation related information (such as CTU size, CTU segmentation flag, QT segmentation flag, MTT segmentation type, MTT segmentation direction, etc.) to enable the video decoding device to segment blocks in the same way as the video encoding device. Additionally, the entropy encoder 155 encodes information indicating whether the current block is encoded using intra-frame prediction or inter-frame prediction. The entropy encoder 155 encodes intra-frame prediction information (i.e., information about the intra-frame prediction mode) or inter-frame prediction information (the merge index in merge mode, and information about the reference image index and motion vector difference in AMVP mode) based on the prediction type. Furthermore, the entropy encoder 155 encodes quantization-related information, namely information about quantization parameters and information about the quantization matrix.
[0082] Inverse quantizer 160 inverse quantizes the quantized transform coefficients output from quantizer 145 to generate transform coefficients. Inverse transformer 165 transforms the transform coefficients output from inverse quantizer 160 from the frequency domain to the spatial domain to reconstruct the residual block.
[0083] Adder 170 adds the reconstructed residual block to the prediction block generated by predictor 120 to reconstruct the current block. Pixels in the reconstructed current block can be used as reference pixels when performing intra-frame prediction in the next sequential block.
[0084] The loop filter unit 180 performs filtering on the reconstructed pixels to reduce block artifacts, ringing artifacts, blurring artifacts, etc., that occur due to block-based prediction and transform / quantization. The loop filter unit 180, as an in-loop filter, may include all or some of the following: a deblocking filter 182, a sample adaptive offset (SAO) filter 184, and an adaptive loop filter (ALF) 186.
[0085] Deblocking filter 182 filters the boundaries between reconstructed blocks to remove block artifacts caused by block unit encoding / decoding, while SAO filter 184 and ALF 186 perform additional filtering on the deblocked video. SAO filter 184 and ALF 186 are filters used to compensate for differences between reconstructed and original pixels caused by lossy coding. SAO filter 184 applies an offset in units of CTU to improve subjective image quality and coding efficiency. On the other hand, ALF 186 performs block unit filtering and compensates for distortion by applying different filters based on the boundaries and degree of variation of the corresponding blocks. Information about the filter coefficients used for ALF can be encoded and transmitted to the video decoding device.
[0086] The reconstructed blocks, filtered by deblocking filter 182, SAO filter 184, and ALF 186, are stored in memory 190. When all blocks in an image have been reconstructed, the reconstructed image can be used as a reference image for inter-frame prediction of blocks in subsequent images to be encoded.
[0087] Video encoding devices can store the bitstream of encoded video data in a non-transitory storage medium or transmit the bitstream to a video decoding device via a communication network.
[0088] Figure 5 This is a functional block diagram of a video decoding device that implements the technology disclosed herein. In the following, reference is made to... Figure 5 Describe video decoding equipment and its components.
[0089] The video decoding device may include an entropy decoder 510, a rearrangement unit 515, an inverse quantizer 520, an inverse transformer 530, a predictor 540, an adder 550, a loop filter unit 560, and a memory 570.
[0090] and Figure 1 Similar to video encoding devices, each component of a video decoding device can be implemented as hardware or software, or a combination of both. Furthermore, the function of each component can be implemented as software, or a microprocessor can be implemented to execute the software function corresponding to each component.
[0091] The entropy decoder 510 extracts information related to block segmentation by decoding the bitstream generated by the video encoding device to determine the current block to be decoded, and extracts the prediction information and information about the residual signal needed to reconstruct the current block.
[0092] The entropy decoder 510 determines the size of the CTU by extracting information about the CTU size from the sequence parameter set (SPS) or picture parameter set (PPS) and segments the image into CTUs with the determined size. Furthermore, the CTU is identified as the highest level of the tree structure, i.e., the root node, and segmentation information for the CTU can be extracted to segment the CTU using the tree structure.
[0093] For example, when using the QTBTTT structure to segment the CTU, the first flag (QT_split_flag) related to the QT segmentation is first extracted to split each node into four lower-level nodes. Furthermore, for the nodes corresponding to the leaf nodes of the QT, the second flag (mtt_split_flag), segmentation direction (vertical / horizontal), and / or segmentation type (binary / ternary) related to the MTT segmentation are extracted to segment the corresponding leaf nodes into an MTT structure. As a result, each node below the leaf node of the QT is recursively segmented into a BT or TT structure.
[0094] As another example, when using the QTBTTT structure to split a CTU, a CU split flag (split_cu_flag) indicating whether a CU has been split is extracted. A first flag (QT_split_flag) can also be extracted when the corresponding block is split. During the splitting process, for each node, zero or more recursive MTT splits may occur after zero or more recursive QT splits. For example, for a CTU, an MTT split may occur immediately, or conversely, only multiple QT splits may occur.
[0095] As another example, when using a QTBT structure to segment a CTU, the first flag (QT_split_flag) associated with the segmentation of the QT is extracted to split each node into four lower-level nodes. Additionally, flags (split_flag) indicating whether the node corresponding to a leaf node of the QT is further segmented into a BT and segmentation direction information are extracted.
[0096] Simultaneously, when the entropy decoder 510 determines the current block to be decoded using tree-structured segmentation, it extracts information indicating whether the current block is an intra-frame prediction or an inter-frame prediction. When the prediction type information indicates intra-frame prediction, the entropy decoder 510 extracts syntax elements for the intra-frame prediction information (intra-frame prediction mode) for the current block. When the prediction type information indicates inter-frame prediction, the entropy decoder 510 extracts information representing syntax elements for inter-frame prediction information, namely motion vectors and the reference images referenced by the motion vectors.
[0097] In addition, the entropy decoder 510 extracts quantization-related information and extracts information about the quantization transform coefficients of the current block as information about the residual signal.
[0098] The rearrangement unit 515 can transform the 1D quantization transform coefficient sequence entropy decoded by the entropy decoder 510 back into a 2D coefficient array (i.e., a block) in the reverse order of coefficient scanning performed by the video encoding device.
[0099] The inverse quantizer 520 performs inverse quantization on the quantized transform coefficients and uses quantization parameters to perform inverse quantization on the quantized transform coefficients. The inverse quantizer 520 can also apply different quantization coefficients (scaling values) to the quantized transform coefficients arranged in 2D. The inverse quantizer 520 can perform inverse quantization by applying a matrix of quantization coefficients (scaling values) from the video encoding device to a 2D array of quantized transform coefficients.
[0100] The inverse transformer 530 reconstructs the residual signal by inversely transforming the inversely quantized transform coefficients from the frequency domain to the spatial domain, thereby generating a residual block for the current block.
[0101] Furthermore, when the inverse transformer 530 performs an inverse transform on a portion (sub-block) of the transform block, it extracts the flag (cu_sbt_flag) of the sub-block that is transformed only, the orientation (vertical / horizontal) information (cu_sbt_horizontal_flag) of the sub-block, and / or the position information (cu_sbt_pos_flag) of the sub-block. The inverse transformer 530 also inversely transforms the transform coefficients of the corresponding sub-block from the frequency domain to the spatial domain to reconstruct the residual signal, and fills the untransformed regions with "0" values as the residual signal to generate the final residual block for the current block.
[0102] Furthermore, when applying MTS, the inverse transformer 530 uses the MTS information (mts_idx) transmitted from the video encoding device to determine the transform function or transform matrix to be applied in each of the horizontal and vertical directions. The inverse transformer 530 also performs an inverse transform on the transform coefficients in the transform block in the horizontal and vertical directions using the determined transform function.
[0103] Predictor 540 may include intra-predictor 542 and inter-predictor 544. Intra-predictor 542 is activated when the prediction type of the current block is intra-prediction, and inter-predictor 544 is activated when the prediction type of the current block is inter-prediction.
[0104] Intra-predictor 542 determines the intra-prediction mode for the current block from the syntax elements for intra-prediction modes extracted by self-entropy decoder 510 among multiple intra-prediction modes. Intra-predictor 542 also predicts the current block using neighboring reference pixels of the current block based on the intra-prediction mode.
[0105] The inter-frame predictor 544 uses the syntax elements for the inter-frame prediction mode extracted from the entropy decoder 510 to determine the motion vector of the current block and the reference picture referenced by the motion vector, and uses the motion vector and the reference picture to predict the current block.
[0106] Adder 550 reconstructs the current block by adding the residual block output by inverse transform 530 to the prediction block output by inter-predictor 544 or intra-predictor 542. Pixels in the reconstructed current block are used as reference pixels when performing intra-prediction on blocks to be decoded later.
[0107] The loop filter unit 560, serving as an in-loop filter, may include a deblocking filter 562, a SAO filter 564, and an ALF 566. The deblocking filter 562 performs deblocking filtering on the boundaries between reconstructed blocks to remove blocking artifacts caused by block unit decoding. The SAO filter 564 and ALF 566 perform additional filtering on the reconstructed blocks after deblocking to compensate for differences between reconstructed pixels and original pixels caused by lossy encoding. The filter coefficients of the ALF are determined using filter coefficient information decoded from the bitstream.
[0108] The reconstructed blocks, after being filtered by deblocking filter 562, SAO filter 564, and ALF 566, are stored in memory 570. When all blocks in an image have been reconstructed, the reconstructed image can be used as a reference image for inter-frame prediction of blocks in subsequent images to be encoded.
[0109] The following is disclosed that it can be done by Figure 1 The video encoder (i.e., video encoding device) shown Figure 5 The video decoder (i.e., video decoding device) shown implements an improved encoding technique. Specifically, the technique disclosed herein relates to Local Luminance Compensation (LIC).
[0110] During inter-frame prediction, the region most similar to the current block is searched in the reference image, and its position is represented using motion information (motion vector, reference image ID, etc.). However, during inter-frame prediction, there may be differences in the amount of light on objects between the current and reference images. For example, a particular image might be illuminated differently. To compensate for these brightness differences between images, Local Luminance Compensation (LIC) technology has been introduced.
[0111] Local Luminance Compensation (LIC) is an inter-frame prediction technique that models the local luminance variation between the current block and the reference block as a function of the variation between the current block template and the reference block template. The parameters of this function can be represented by the slope α and the offset β, forming the following linear equation: P[x,y] = α P[x,y] + β is used to compensate for brightness changes. Here, P'[x,y] is the brightness-compensated predicted sample at position [x,y] within the current block, and P[x,y] is the reference sample at position [x,y] within the reference image, indicated by the motion vector.
[0112] Figure 6 A conceptual illustration of Local Luminance Compensation (LIC) is provided. For example... Figure 6 As shown, when performing inter-frame prediction on the current block, the current block has a motion vector pointing to the reference block. The parameters of the linear model are computed using the reconstructed neighbor samples of the current block (referred to as the current template) and the reconstructed neighbor samples of the reference block (referred to as the reference template). The linear model is applied to the initial predictor P[x,y] of the inter-frame prediction to generate the final predictor P'[x,y].
[0113] This leads LIC to typically use a single linear model for all samples within a block. However, a single linear model may not optimally model the local brightness variations between the current block and the predicted block. For example, even within a block, the brightness variations of each sample may differ. Furthermore, as... Figure 7 As shown, due to shadows, brightness can only vary in certain regions of the reference block within the reference image. Therefore, using only a linear model to compensate for local brightness variations across all samples in the block may not be the optimal compensation for those local brightness variations.
[0114] This disclosure proposes various methods for further improving LIC encoding efficiency. According to one aspect of this disclosure, a method is provided for selectively using one or more models suitable for a given block among various models capable of performing compensation for local brightness variations. According to another aspect of this disclosure, a method is provided for selecting various effective samples for predicting LIC parameters for each model.
[0115] In embodiments of this disclosure, various LIC models (such as various linear models, nonlinear models, and / or machine learning models) can be configured to model local brightness compensation between the current block and the predicted block. The video encoder and video decoder can select one or more models from a plurality of available LIC models to be used for local brightness compensation of the current block.
[0116] Here, the selection of the LIC model can be performed based on predefined criteria. For example, the selection of the LIC model for the current block can be performed based on the coordinates of the sample, the sample value, the encoding information, the LIC models of adjacent blocks, etc.
[0117] The video encoder can specify one or more LIC models from a plurality of available LIC models for local brightness compensation of the current block using an indicator in the bitstream. Therefore, the decoder can determine or select one or more LIC models to be used for local brightness compensation of the current block based on the indicator obtained from the bitstream.
[0118] Alternatively, the video decoder can implicitly determine or select one or more LIC models to use for local brightness compensation of the current block. For example, the video decoder can calculate the cost of the template for the current block (such as mean squared error (MSE), sum of absolute differences (SAD), sum of absolute transform differences (SATD), or mean sum of absolute differences removed (MR-SAD)) for each available LIC model, and then select or use the model with the lowest cost for local brightness compensation of the current block. The cost of the template for the current block can be calculated based on the difference between the “reconstructed sample” of the template for the current block and the “compensated predicted sample” of the template for the current block obtained by applying the LIC model to the reconstructed sample of the template for a reference block.
[0119] When selecting one or more models from multiple available models for the current block, the parameters of the selected model need to be determined.
[0120] Various methods can be used to determine the parameters of the LIC model. For example, methods such as selecting parameters from a merge candidate list, estimating (or deriving) the parameters by the decoder in the same way as the encoder, and explicitly signaling the parameters to the encoder can be considered. The video encoder can signal the encoder to determine the model parameters using High-Level Syntax (HLS).
[0121] Figure 8 This is an example of a high-level syntax (HLS) for a method of determining model parameters by signaling, according to one embodiment of this disclosure. Figure 8 Some or all of the syntax elements illustrated in the diagram can be signaled in the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), Picture Header (PH), Strip Header (SH), etc. Figure 8 In the High-Level Syntax (HLS) shown, the order in which syntax elements are signaled can be changed in a variety of ways.
[0122] The syntax element cu_lic_flag is a flag indicating whether LIC is allowed; cu_lic_flag = 1 indicates that LIC mode is activated, and cu_lic_flag = 0 indicates that LIC mode is deactivated.
[0123] The syntax element cu_lic_parameters_inherit is a syntax element that indicates whether the LIC merge mode is used, where cu_lic_parameters_inherit = 1 indicates that the LIC merge mode is activated, and cu_lic_parameters_inherit = 0 indicates that the LIC merge mode is deactivated.
[0124] The syntax element cu_lic_parameters_implicit is a syntax element that indicates whether the LIC parameters are estimated in the decoder (i.e., implicitly derived). cu_lic_parameters_implicit = 1 indicates that the LIC parameters are estimated in the decoder, and cu_lic_parameters_implicit = 0 indicates that the LIC parameters are not estimated in the decoder.
[0125] The syntax element cu_lic_parameters_explicit is a syntax element that indicates whether to use the method of explicitly notifying LIC parameters with signals. cu_lic_parameters_explicit = 1 indicates that the LIC parameters are not notified with signals, and cu_lic_parameters_explicit = 0 indicates that the LIC parameters are not notified with signals.
[0126] First Implementation Method - Merging Candidate Lists
[0127] In this embodiment, a list of LIC models that can be configured to compensate for local brightness variations in the current block can be set. The video encoder can select a model from the LIC model list to use for the current block and can signal the index of the selected model in the LIC model list. The video decoder can select the model indicated by the index from the available LIC model list and perform local brightness compensation on the current block using the selected model.
[0128] The list of available LIC models can be configured using one of the following methods. The first method is to use a merge candidate list configured for the motion information encoding of the current block. The second method is to construct a LIC merge candidate list separately from the merge candidate list configured for motion information encoding, utilizing the LIC model information of the current block's neighboring blocks.
[0129] When using a merge candidate list configured for motion information encoding (i.e., the first method), the video decoder can decode the index of a candidate in the merge candidate list from the bitstream and inherit LIC flag information and LIC model information from the candidate block indicated by the decoded index. The video decoder can then determine the LIC model (and its parameters) to be used for the current block by using the inherited LIC model information.
[0130] The second approach utilizes LIC model information used for other blocks to construct the LIC merge candidate list. In this case, the size of the LIC merge candidate list can be a predetermined size or an adaptive size. Furthermore, the LIC merge candidate list can consist of various types of candidates.
[0131] (1) Spatial adjacency candidates: LIC model information of spatially adjacent locations
[0132] (2) Spatially non-adjacent candidates: LIC model information of spatially non-adjacent locations
[0133] (3) Temporal candidate: LIC model information that is in the same temporal position.
[0134] (4) Historical candidates: LIC model information from a history table storing previously used LIC models.
[0135] (5) Temporal candidate after movement: LIC model information that is in the same position in time as the motion vector.
[0136] (6) Default Candidate: Predefined LIC model information added to the remaining positions in the LIC merge candidate list when the LIC merge candidate list is not completely filled.
[0137] Spatial adjacent positions and spatial non-adjacent positions, such as Figure 9a As shown, positions that are in the same location in time are as follows: Figure 9b As shown, the positions that are in the same location in time due to the movement vector are as follows: Figure 9c As shown.
[0138] exist Figure 9a In the example, positions 1 to 5 are adjacent to the current block space, and positions 6 to 23 are non-adjacent to the current block space. The LIC information for one or more positions 1 to 5 can be inserted into the LIC merge candidate list as spatially adjacent merge candidates. The LIC information for one or more positions 6 to 23 can be inserted into the LIC merge candidate list as spatially non-adjacent merge candidates. The distance between the spatially non-adjacent positions and the current block is based on the width and height of the current block.
[0139] like Figure 9b As shown, temporal candidates can be selected from co-located images (colPic). Figure 9b In this context, C0 and C1 represent the reference images corresponding to the reference image indices. The order of inclusion of temporal candidates can be (C01, C02, ..., C0...). 10 If C0 i C1 is outside the image / strip boundary. iWithin the boundaries of the image / strip, C1 can be used. i Replace C0 i .
[0140] like Figure 9c As shown, the moved temporal candidates can be selected from the co-location images. The position of the co-location block and C0... i and C1 i The position can be moved by the selected adjacent motion vectors. The inclusion order of the moved temporal candidates can be the same as the inclusion order of the temporal candidates. Adjacent motion vectors can be selected from the motion vectors of adjacent blocks (A0, A1, B0, B1, B2). The checking order can be (L0... B1 L1 B1 L0 A1 L1 A1 L0 B0 L1 B0 L0 A0 L1 A0 L0 B2 L1 B2 You can choose to use a co-located image as the first motion vector of the reference image.
[0141] To construct the LIC merge candidate list, the search area for previously used LIC models can be limited to areas such as areas within the same CTU, areas within the same CTU row, or areas within the same image. In other words, the history table storing previously used LIC models can be initialized by CTU, by CTU row, or by image.
[0142] An example of a default candidate could be a LIC model generated by changing the parameter values of a linear LIC model already included in the candidate list.
[0143] In addition to the candidate types described in this paper, other candidate types can be used to construct the LIC merge candidate list. Furthermore, it is also possible to construct the LIC merge candidate list using only a subset of all the candidate types described in this paper.
[0144] Once the LIC candidate list is configured, the cost of the template for the current block can be calculated for each candidate in the list, and a rearrangement (reordering) can be performed in ascending order of cost. In this case, loss functions such as SAD, SATD, and MR-SAD can be used for cost calculation. The cost of the template for the current block can be calculated based on the difference between the reconstructed neighbor samples of the template for the current block and the compensated prediction of the template for the current block obtained by applying the candidate model to the corresponding neighbor samples of the reference block.
[0145] The decoder can decode the index of a candidate in the LIC merge candidate list from the bitstream and use the LIC model information of the candidate indicated by the decoded index to determine the LIC model (and its parameters) of the current block.
[0146] Second Implementation Method - Compensation Based on Multiple Sample Inputs
[0147] In this embodiment, one or more multi-sample input-based compensation models can be used. A multi-sample input-based compensation model can utilize the compensated sample values of multiple sample outputs from the initial predictor of the inter-frame prediction at a given predicted sample location. An autoregressive model (AR model) can be used as a multi-sample input-based compensation model. The following equation represents the mathematical expression of a method for performing local brightness variation compensation using an autoregressive model. When using an autoregressive model to model LIC, the offset term can be optional.
[0148]
Formula 1
[0149] P'[x,y] represents the sample value at position (x,y) where compensation has been performed for local brightness changes. i ,y i The x-axis represents the position in the reference image used as input for performing local brightness variation compensation. i ,y i The sample value at ().
[0150] Figure 10 The illustration shows the spatial locations of the center sample (C) and its neighboring samples. The center sample (C) is the sample in the reference block that corresponds to the sample position (x, y) within the current block. In other words, the center sample (C) is the sample in the reference image indicated by a motion vector originating from position (x, y).
[0151] When the autoregressive model does not use the center sample (C) but instead uses its north (N), south (S), east (E), and west (W) neighboring samples, the LIC model can be represented as the following convolutional filter. i These are model parameters.
[0152]
Formula 2
[0153] When the autoregressive model uses the center sample (C) and its neighboring samples (N, S, E, W), the LIC model can be represented as the following convolutional filter.
[0154]
Formula 3
[0155] When the autoregressive model additionally uses P(C) as a nonlinear term and B as a bias term, the LIC model can be represented as the following convolutional filter. The nonlinear term P(C) represents the value obtained by normalizing the square of the center sample (C) based on the bit depth, and the bias term B represents the value based on the median of the bit depth.
[0156]
Formula 4
[0157] When the autoregressive model uses center samples (C), first-order neighbor samples (N, S, E, W), and second-order neighbor samples (NN, WW), the LIC model can be represented as the following convolutional filter.
[0158]
Formula 5
[0159] Third implementation method: Compensation based on a single sample input
[0160] In this implementation, a compensation model based on single-sample input is used for the current block, but a different set of model parameters is used for the samples of the current block. The compensation model based on single-sample input can be a linear model that generates compensated prediction samples (i.e., final prediction samples) using the initial prediction samples from the inter-frame predictions of the current block.
[0161] The video encoder and video decoder can classify the initial predicted samples of the current block into multiple subsets and determine the set of parameters of the linear model to be applied to each subset.
[0162] Video encoders and decoders can classify neighboring samples of the current block and neighboring samples of the reference block into multiple subsets, and can use each subset as a training set to derive the parameters of a linear model for each subset. Instead of deriving the parameters of the linear model for each subset from neighboring samples of the current block and neighboring samples of the reference block, the video encoder can use signaling notions to indicate the linear model parameters.
[0163] The criteria for classifying samples (such as initial prediction samples or neighboring samples) into subsets can be the coordinates (or location) of the samples or the values of the samples.
[0164] To classify samples into subsets based on their coordinates, a segmentation boundary (segmentation edge) can be used to divide the current block into two sub-regions. The segmentation boundary is a straight line determined based on angle and offset. Figure 11 This is a conceptual diagram illustrating the dividing boundary and the angle and offset of the specified dividing boundary. (Reference) Figure 11 The split boundary is specified by the angle of the perpendicular line drawn from the center of the block to the split boundary and the offset as the distance from the center of the block to the split boundary. The angle of the perpendicular line is measured counterclockwise from the horizontal axis. The angle and offset to be applied to the split boundary of the current block can each be signaled as an index.
[0165] Figure 12 Various segmentation boundaries are shown for dividing the current block, to which local brightness compensation is applied, into two sub-regions.
[0166] The video encoder and decoder can identify two subsets of initial prediction samples corresponding to two sub-regions and can independently determine the parameters of the LIC model for these subsets of initial prediction samples. For example, the video encoder and decoder can identify a first subset of initial prediction samples corresponding to sample positions in a first sub-region and a second subset of initial prediction samples corresponding to sample positions in a second sub-region, and can determine a first set of parameters to be applied to the linear model of the first subset of initial prediction samples and a second set of parameters to be applied to the linear model of the second subset of initial prediction samples. The parameters of the LIC model applied to the first subset of initial prediction samples for the current block can be different from the parameters of the LIC model applied to the second subset of initial prediction samples for the current block.
[0167] The first parameter set can be calculated based on the reconstructed neighboring samples immediately adjacent to the first sub-region and the corresponding neighboring samples of the reference block. The second parameter set can be calculated based on the reconstructed neighboring samples immediately adjacent to the second sub-region and the corresponding neighboring samples of the reference block. In other words, for a given initial subset of predicted samples, it can be derived from a portion of the template of the current block and a portion of the template of the reference block. A portion of the template of the current block can be referred to as a sub-template of the current block, and a portion of the template of the reference block can be referred to as a sub-template of the reference block.
[0168] In some implementations, for each subset of the initial predicted sample subset in a predefined parameter set list, a signal may be used to indicate the index of the parameter set to be used for the LIC model applied to that subset. In some implementations, a signal may be used to explicitly indicate the syntax elements to be used for the parameters of the LIC model applied to each subset.
[0169] Simultaneously, a threshold can be used to compare with sample values in order to classify samples (such as initial predicted samples or neighboring samples) into subsets based on the sample values. The video encoder and video decoder can classify the initial predicted samples of the current block into two subsets based on the comparison between the sample values of the initial predicted samples and the threshold, and can determine the corresponding set of parameters for the linear model to be used to compensate for local brightness variations in each subset.
[0170] Figure 13This is a conceptual diagram illustrating examples of two different parameter sets of a linear model used based on a threshold, according to some embodiments of this disclosure. As shown in the equation below, the parameter set of the linear model for each initial prediction sample is determined based on a comparison of the initial prediction sample P[x,y] with the threshold.
[0171]
Formula 6
[0172] Since using different sets of parameters can be considered as using different models, it can also be said that the linear model to be used for each initial prediction sample is selected from two different linear models.
[0173] According to one implementation, two or more thresholds can be used. For example, two thresholds can be used such that for each initial prediction sample, a parameter set (or a linear model) can be selected from three different parameter sets (or three linear models).
[0174] Thresholds can be derived implicitly or explicitly signaled.
[0175] As an example of an implicitly derived threshold method, for instance, the sample value at a predefined position (e.g., center, leftmost, or topmost position) in the template of a reference block corresponding to the current block's template can be set as the threshold, or the average, median, mode, maximum, etc., of sample values at multiple predefined positions in the template of the reference block can be set as the threshold.
[0176] As an example of an explicit method of signaling a threshold, syntax elements indicating the threshold can be encoded by block, sequence, or image. Furthermore, once a threshold has been signaled for a block, subsequent blocks can use the same threshold until a new threshold is signaled.
[0177] The parameters of the LIC model for each subset can be derived from the neighboring samples of the current block and the neighboring samples of the reference block. For example, a video encoder and decoder can classify the neighboring samples of the reference block into two subsets based on a comparison with a threshold. The video encoder and decoder can compute or derive the parameters of the LIC model for each subset using the neighboring samples of each subset and the corresponding neighboring samples of the current block. Here, the same threshold can be used for classifying neighboring samples and classifying the sample positions of the current block.
[0178] Instead of calculating the parameters of the LIC model from neighboring samples of the current block and neighboring samples of the reference block, the video encoder can explicitly signal the parameters of the LIC model, or explicitly signal the index of the parameter set.
[0179] The following describes a mathematical method for deriving the parameters of the m-th linear LIC model using neighboring samples from the m-th subset. The parameters of the m-th linear LIC model can be derived based on the minimum mean square error (LMSE) method using neighboring samples from the current block and neighboring samples from the reference block.
[0180] The mathematical expression for the m-th linear LIC model is as follows.
[0181]
Formula 7
[0182] Here, p'[x,y] is the motion-compensated predicted sample at position [x,y] within the current block, and P[x,y] is the reference sample in the reference image indicated by the motion vector originating from position [x,y]. α m and β m Let α represent the scaling parameter and offset parameter of the m-th linear model, respectively. m and β m The following mathematical calculation can be performed.
[0183]
Formula 8
[0184] Here, n is the number of neighboring samples used to derive the parameters. The equations used to estimate ΣX, ΣY, ΣX², and ΣXY are as follows.
[0185]
Formula 9
[0186]
Formula 10
[0187]
Formula 11
[0188]
Formula 12
[0189] R above,m R represents the sample immediately above the reference block used for the m-th model. left,m C represents the left-hand neighboring sample of the reference block used for the m-th model. above,m C represents the immediate adjacent sample above the current block used in the m-th model. left,m This represents the left-hand neighboring sample of the current block used for the m-th model.
[0190] The following describes the valid samples among the neighboring samples of the current block used to calculate or derive LIC model parameters.
[0191] Valid samples can be samples from the template of the current block (i.e., the left column and the top row) or downsampled samples thereof. Additionally, valid samples can be samples from the current block immediately adjacent to the 0th or nth reference line, or downsampled samples thereof. Figure 14 This example illustrates the downsampling samples used to calculate or derive LIC model parameters from a template consisting of the left column and the top row, for 8×8 and 16×16 CUs. The downsampling interval can be as follows... Figure 14 As with the example, it depends on the size of the block, or it can be constant regardless of the size of the block.
[0192] In some implementations, a method for selecting valid samples for each LIC model or each model type can be predefined. For example, a video encoder and a video decoder can derive the parameters of a linear model using samples immediately adjacent to the reference line (i.e., the left column and the top row), and can derive the parameters of a nonlinear model using samples immediately adjacent to the template (i.e., the left column and the top row). As another example, the parameters of a first model can be derived based on samples from the left column, the parameters of a second model based on samples from the top row, and the parameters of a third model based on samples from both the left column and the top row.
[0193] In some implementations, a trained neural network or decision tree can be used to determine whether each neighboring sample in the current block is a valid sample. In some implementations, a method for selecting valid samples can be indicated by signals, either image-wise or sequence-wise.
[0194] The above-described implementation methods or approaches can be applied individually, or any combination thereof can be applied to video encoders and video decoders.
[0195] Figure 15 This is an example of a LIC-related syntax structure that can be included at the CU level according to one embodiment of this disclosure.
[0196] When applying local brightness compensation to the current CU, a signal can first be used to notify whether to use the LIC merging mode corresponding to the method of the first embodiment.
[0197] lic_merge_flag, use_lic_cand_list, and merge_list_idx are syntax elements related to LIC merge modes, and each is described below.
[0198] -lic_merge_flag is a syntax element that indicates whether the LIC merge mode is used in the current CU; if it is 1, it means that the LIC merge mode is used in the current CU, and if it is 0, it means that the LIC merge mode is not used in the current CU.
[0199] - use_lic_cand_list is a syntax element that indicates whether the LIC merge candidate list is used in LIC merge mode; if it is 1, the video decoder constructs the LIC merge candidate list and determines the LIC candidates to be used for the current CU based on the LIC merge candidate list; if it is 0, the video decoder determines the candidates to be used for the current CU based on the merge candidate list used for motion information encoding.
[0200] - merge_list_idx is a syntax element that represents the index of the candidate in the candidate list to be used for the current CU.
[0201] exist Figure 15 In the example, when the current CU does not use the LIC merging mode, a signal is used to indicate whether to use the LIC mode based on single-pixel input corresponding to the method described in the third embodiment.
[0202] The syntax elements lic_single_sample_input, lic_single_sample_multi_model_idx, lic_angle_idx, lic_offset_idx, lic_explicit_parameter_flag, lic_model0_parameter, and lic_model1_parameter are associated with LIC modes based on single-pixel input.
[0203] -lic_single_sample_input is a syntax element that indicates whether to use the LIC mode based on a single sample input; if it is 1, it means that the LIC mode based on a single sample input is used, and if it is 0, it means that the LIC mode based on a single sample input is not used.
[0204] -lic_single_sample_multi_model_idx is a syntax element that indicates whether to use a method based on the coordinates of each sample to select the model to use when using the LIC mode based on single-pixel input. If it is 1, it means that the method of selecting the model to use from multiple LIC models is used based on the coordinates of each sample. If it is 0, it means that the method of selecting the model to use from multiple LIC models is not used.
[0205] `-lic_angle_idx` and `lic_offset_idx` are syntax elements that specify the split boundaries (splitting edges) that divide the current CU into two subsets when using a method that selects the model to use from multiple LIC models based on the coordinates of each sample. `lic_angle_idx` refers to the angle perpendicular to the split boundary from the center of the current CU, and `lic_offset_idx` refers to the distance from the center of the current CU to the split boundary. These syntax elements specify the geometric arrangement of the split boundaries within the current CU.
[0206] -lic_explicit_parameter_flag is a syntax element that indicates whether to explicitly signal parameters applied to the LIC model of the current CU. If it is 1, the parameters applied to the LIC model of the current CU are signaled; if it is 0, the parameters applied to the LIC model of the current CU are not explicitly signaled, but are implicitly derived by the decoder.
[0207] `-lic_model0_parameter` and `lic_model1_parameter` are syntax elements representing the parameters of the first LIC model and the second LIC model, respectively. If a single model is used instead of two models, the `lic_model0_parameter` syntax element can be used to signal the parameters of that single model.
[0208] The syntax elements lic_multi_sample_input, lic_regression_model0_idx, and lic_regression_model1_idx are associated with the multi-pixel input-based LIC mode corresponding to the method described in the second embodiment.
[0209] -lic_multi_sample_input is a syntax element that indicates whether to use the LIC mode based on multi-pixel input; if it is 1, it means that the LIC mode based on multi-pixel input is used, and if it is 0, it means that the LIC mode based on multi-pixel input is not used.
[0210] `-lic_regression_model0_idx` and `lic_regression_model1_idx` are syntax elements that represent the parameters of the first LIC model and the second LIC model, respectively, used in LIC mode based on multi-pixel input. If a single model is used instead of two models, the parameters of that single model can be signaled using the syntax element `lic_regression_model0_idx`.
[0211] Figure 16 The illustration shows a CU-level syntax, including lic_mode_idx (which is a syntax element representing an index of the LIC mode applied to the current CU), according to some embodiments of this disclosure. Figure 17 This is an example of binarization of various LIC modes of lic_mode_idx according to some implementations of this disclosure.
[0212] to replace Figure 16 The syntax elements lic_merge_flag, lic_multi_sample_input, and lic_multi_sample_input shown represent methods that can each be defined as an available LIC mode. The video encoder can signal the index (lic_mode_idx) of the LIC mode selected from the multiple available LIC modes.
[0213] Multiple available LIC patterns may include patterns corresponding to the method of LIC merging lists using the first embodiment, patterns corresponding to the method of multiple autoregressive models using the second embodiment, patterns corresponding to the method of multiple linear models using the third embodiment, and patterns corresponding to the method of a single autoregressive model using the second embodiment. Furthermore, the conventional method of using a single linear LIC model for all samples within a block may be included as a single available pattern among the multiple available LIC patterns.
[0214] Figure 18 This is a flowchart illustrating a prediction method performed by each of the video encoder and video decoder according to some embodiments of the present disclosure.
[0215] The video encoder and video decoder can derive the initial prediction sample of the current block in the current image based on the reference block in the reference image (S1810).
[0216] The video encoder and video decoder can determine the LIC model of the current block from multiple available Local Luminance Compensation (LIC) models (S1820).
[0217] The video encoder and video decoder can apply local brightness compensation to the initial prediction sample of the current block based on the LIC model of the current block, thereby generating the final prediction sample of the current block (S1830).
[0218] In some implementations, the video encoder and video decoder may, as part of step S1820, calculate the cost of a template (a set of reconstructed samples of adjacent current blocks) for each of the multiple available LIC models, and may determine the LIC model of the current block based on the cost of the template for each of the multiple available LIC models.
[0219] Here, the cost of the template can be calculated based on the difference between the reconstructed neighboring samples of the template and the compensated prediction of the template obtained by applying the candidate model to the corresponding neighboring samples of the reference block.
[0220] In some implementations, the video encoder and video decoder may, as part of step S1820, decode indices from the bitstream in a merge candidate list used to encode motion information for the current block, and may use LIC model information inherited by the candidate blocks indicated by the decoded indices to determine the LIC model of the current block.
[0221] In some implementations, the video encoder and video decoder may construct a LIC merging candidate list as part of step S1820, based on LIC model information of other blocks reconstructed on the current or reference image. The video encoder and video decoder can decode indices indicating candidates in the LIC merging candidate list from the bitstream and can determine the LIC model of the current block using the LIC model information of the candidates indicated by the decoded indices.
[0222] In some implementations, at least one of the available LIC models can be a convolutional filter based on an autoregressive model. At least one of the available LIC models can be a linear model.
[0223] In some implementations, the video encoder and video decoder may, as part of step S1830, determine the segmentation boundary that divides the current block into two sub-regions, and may identify two subsets of the initial prediction samples corresponding to the two sub-regions. The video encoder and video decoder may determine a first set of parameters for a linear model to be applied to a first subset of the initial prediction samples corresponding to a first sub-region of the two sub-regions, and a second set of parameters for a linear model to be applied to a second subset of the initial prediction samples corresponding to a second sub-region of the two sub-regions.
[0224] The first parameter set can be calculated based on the reconstructed neighboring samples immediately adjacent to the first sub-region and the corresponding neighboring samples of the reference block. The second parameter set can be calculated based on the reconstructed neighboring samples immediately adjacent to the second sub-region and the corresponding neighboring samples of the reference block. The video encoder and video decoder can generate the final predicted sample for the current block by applying a linear model with the first parameter set to a first subset of the initial predicted samples and applying a linear model with the second parameter set to a second subset of the initial predicted samples.
[0225] In some implementations, the video encoder and video decoder may, as part of step S1830, identify two subsets of the initial prediction samples based on comparing the values of the initial prediction samples with a threshold, and may determine a first set of parameters for the linear model to be applied to the first subset of the initial prediction samples and a second set of parameters for the linear model to be applied to the second subset of the initial prediction samples. Here, the threshold may be calculated based on the reconstructed neighboring samples of a reference block corresponding to a template (which is a set of reconstructed samples adjacent to the current block). For example, the threshold may be the average of some or all of the reconstructed neighboring samples of the reference block.
[0226] The video encoder and video decoder can, as part of determining a first parameter set and a second parameter set, identify two subsets of neighboring samples of a reference block based on comparing neighboring samples of the reference block with a threshold. The video encoder and video decoder can determine the first parameter set based on a first subset of neighboring samples of the reference block and corresponding first subset neighboring samples of the current block, and can determine the second parameter set based on a second subset of neighboring samples of the reference block and corresponding second subset neighboring samples of the current block.
[0227] The video encoder and video decoder can generate the final predicted sample for the current block by applying a linear model with a first set of parameters to a first subset of the initial predicted samples and a linear model with a second set of parameters to a second subset of the initial predicted samples.
[0228] A video encoder can encode the current block based on the final predicted sample. For example, a video encoder can encode residual data, representing the difference between the original sample and the final predicted sample of the current block, into a bitstream. A video decoder can decode the current block based on the final predicted sample. For example, a video decoder can decode the residual data, representing the difference between the original sample and the final predicted sample of the current block, from the bitstream and reconstruct the current block by adding the final predicted sample to the residual data.
[0229] Although the steps in the various flowcharts are described as being executed sequentially, these steps are merely examples of the technical ideas of some embodiments of this disclosure. Therefore, those skilled in the art to which this disclosure pertains can perform these steps by changing the sequence described in the various figures, or by executing two or more steps in parallel. Thus, the steps in the various flowcharts are not limited to the illustrated temporal order.
[0230] It should be understood that the above description presents exemplary implementations that can be implemented in various other ways. The functionality described in some implementations can be implemented by hardware, software, firmware, and / or combinations thereof. It should also be understood that the functional components described in this disclosure are labeled "...unit" to strongly emphasize their potential for independent implementation.
[0231] Furthermore, the various methods or functions described in some embodiments can be implemented as instructions stored in one or more processor-readable and executable non-transitory recording media. Non-transitory recording media can, for example, include various types of recording devices that store data in a computer-system-readable form. For example, non-transitory recording media can include storage media such as erasable programmable read-only memory (EPROM), flash memory drives, optical drives, magnetic hard disk drives, and solid-state drives (SSDs), as well as other media.
[0232] Although embodiments of this disclosure have been described for illustrative purposes, those skilled in the art to which this disclosure pertains will understand that various modifications, additions, and substitutions can be made without departing from the spirit and scope of this disclosure. Therefore, embodiments of this disclosure have been described for the sake of brevity and clarity. The scope of the technical concept of the embodiments of this disclosure is not limited by the illustrations. Therefore, those skilled in the art to which this disclosure pertains should understand that the scope of this disclosure should not be limited by the embodiments explicitly described above, but rather by the claims and their equivalents.
[0233] Cross-references to related applications
[0234] This application claims priority and benefit to Korean Patent Application No. 10-2024-0050282, filed on April 15, 2024, the entire contents of which are incorporated herein by reference.
Claims
1. A prediction method performed by a video decoding device, the method comprising: Based on the reference block in the reference image, derive the initial prediction sample for the current block in the current image; Determine the LIC model for the current block from multiple available local illumination compensation (LIC) models; and Based on the LIC model used for the current block, local brightness compensation is applied to the initial prediction sample of the current block to generate the final prediction sample of the current block.
2. The method according to claim 1, wherein, Determining the LIC model for the current block includes: For each of the plurality of available LIC models, compute the cost of a template, which is a set of reconstructed samples adjacent to the current block; and The LIC model for the current block is determined based on the cost of the template for each of the plurality of available LIC models.
3. The method according to claim 2, wherein, The cost of the template is calculated based on the difference between the reconstructed neighboring samples of the template and the compensation prediction of the template, which is obtained by applying a candidate model to the corresponding neighboring samples of the reference block.
4. The method according to claim 1, wherein, Determining the LIC model for the current block includes: Decode from the bitstream the indexes used to indicate candidates in the merge candidate list, which is used to encode and decode the motion information of the current block; and The LIC model for the current block is determined using LIC model information inherited from the candidate block indicated by the decoded index.
5. The method according to claim 1, wherein, Determining the LIC model for the current block includes: A list of LIC merging candidates is constructed using LIC model information of other blocks that have been reconstructed on the current image or the reference image; Decode from the bitstream the index used to indicate the candidate in the LIC merge candidate list; and The LIC model for the current block is determined using the candidate LIC model information indicated by the decoded index.
6. The method according to claim 1, wherein, At least one of the available LIC models is an autoregressive model.
7. The method according to claim 1, wherein, At least one of the available LIC models is a linear model.
8. The method according to claim 7, wherein, When the LIC model used for the current block is the linear model, generating the final prediction sample includes: Determine the dividing boundary that divides the current block into two sub-regions; Identify two subsets of the initial prediction samples corresponding to the two sub-regions; and A first set of parameters is determined for the linear model to be applied to a first subset of the initial prediction samples corresponding to a first sub-region in the two sub-regions, and a second set of parameters is determined for the linear model to be applied to a second subset of the initial prediction samples corresponding to a second sub-region in the two sub-regions.
9. The method according to claim 8, wherein, The first parameter set is calculated based on the reconstructed neighboring samples immediately adjacent to the first sub-region and the corresponding neighboring samples of the reference block, and the second parameter set is calculated based on the reconstructed neighboring samples immediately adjacent to the second sub-region and the corresponding neighboring samples of the reference block.
10. The method according to claim 7, wherein, When the LIC model used for the current block is the linear model, generating the final prediction sample includes: Two subsets of the initial predicted samples are identified by comparing the values of the initial predicted samples with a threshold; and Determine a first set of parameters in the linear model to be applied to a first subset of the initial prediction samples, and determine a second set of parameters in the linear model to be applied to a second subset of the initial prediction samples.
11. The method according to claim 10, wherein, The threshold is calculated based on reconstructed samples that correspond to the template and are adjacent to the reference block, where the template is a set of reconstructed samples adjacent to the current block.
12. The method according to claim 10, wherein, Determining the first parameter set and the second parameter set includes: Two subsets of adjacent samples of the reference block are identified by comparing the adjacent samples of the reference block with the threshold; and The first parameter set is determined based on a first subset of the neighboring samples of the reference block and a first subset of the corresponding neighboring samples of the current block, and the second parameter set is determined based on a second subset of the neighboring samples of the reference block and a second subset of the corresponding neighboring samples of the current block.
13. A prediction method performed by a video encoding device, the method comprising: Based on the reference block in the reference image, derive the initial prediction sample for the current block in the current image; Determine the LIC model for the current block from multiple available local illumination compensation (LIC) models; and Based on the LIC model used for the current block, local brightness compensation is applied to the initial prediction samples of the current block to generate the final prediction samples of the current block.
14. A method for providing video data to a video decoding device, the method comprising: The video data is encoded into a bitstream; as well as The bitstream is transmitted to the video decoding device. Encoding the video data into the bitstream includes: Based on the reference block in the reference image, derive the initial prediction sample for the current block in the current image; Determine the LIC model for the current block from multiple available local illumination compensation (LIC) models; and Based on the LIC model used for the current block, local brightness compensation is applied to the initial prediction samples of the current block to generate the final prediction samples of the current block.
Citation Information
Patent Citations
Ultrasonic transducer operable at multiple resonant frequencies
KR1020240050282A
