Model adjustment for local illumination compensation in video coding

Abstract

The video decoder reconstructs a current frame of the video from the video bitstream based on the reconstructed reference frame. For a block of the current frame, the video decoder identifies a reference block in the reference frame based on a motion vector associated with the block. The decoder determines slope and offset parameters of a local illumination compensation model based on reconstructed pixels of the current frame and the reference frame. The video decoder decodes an adjustment to the slope from the video bitstream and updates the slope by applying the decoded adjustment. The decoder further determines an adjusted offset parameter of the local illumination compensation model. The decoder generates predicted pixels for the block by applying at least the local illumination compensation model with the updated parameters to the reference block.

Description

[Technical field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to U.S. provisional patent application Ser. No. 63 / 269,039, filed on March 8, 2022, entitled "Adjustment of Linear Model for Local Illumination Compensation for Video Coding," the entire contents of which are incorporated herein by reference.

[0002] This application relates to video processing, and in particular to model adjustment for local illumination compensation in video coding. [Background technology]

[0003] Thanks to ubiquitous imaging devices such as smartphones, tablets, personal computers, etc., taking videos and images has become easier than ever. However, even short videos can have a very large amount of data. Video coding techniques (including video encoding and decoding) can compress video data into smaller sizes, thereby enabling the storage and transmission of various videos. Video coding is widely used in fields such as digital television broadcasting, video transmission in the Internet and mobile networks, real-time applications (such as video chat, video conferencing, etc.), DVDs and Blu-ray discs, etc. It is desirable to improve the efficiency of video coding schemes to reduce the consumption of storage space for storing videos and / or network bandwidth for transmitting videos. Summary of the Invention

[0004] Some embodiments relate to model adjustment for local illumination compensation in video coding. In one example, a method for decoding inter-predicted frames of a video coded using local illumination compensation includes reconstructing one or more frames of the video from a video bitstream representing the video, and reconstructing a current frame based on a reference frame from the reconstructed one or more frames. Wherein, for a block of the current frame, reconstructing the current frame includes identifying a reference block in the reference frame based on a motion vector associated with the block, determining first and second parameters of a local illumination compensation model based on a first set of reconstructed pixels in the current frame and a second set of reconstructed pixels in the reference frame, decoding a first adjustment to the first parameters from the video bitstream, adjusting the first parameters by applying the first adjustment to the first parameters, determining the adjusted second parameters, and reconstructing pixels of the block by applying the local illumination compensation model having the adjusted first parameters and the adjusted second parameters to the reference block. The decoding method further includes displaying the reconstructed one or more frames and the reconstructed current frame.

[0005] In another example, a non-transitory computer-readable medium has program code stored thereon, the program code being executable by one or more processing devices to perform operations. The operations include reconstructing one or more frames of a video from a video bitstream representing the video, and reconstructing a current frame based on a reference frame from the reconstructed one or more frames, where for a block of the current frame, reconstructing the current frame includes identifying a reference block in the reference frame based on a motion vector associated with the block, determining first and second parameters of a local illumination compensation model based on a first set of reconstructed pixels in the current frame and a second set of reconstructed pixels in the reference frame, decoding a first adjustment to the first parameter from the video bitstream, adjusting the first parameter by applying the first adjustment to the first parameter, determining an adjusted second parameter, and reconstructing pixels of the block by applying the local illumination compensation model having the adjusted first and adjusted second parameters to the reference block. The operations further include displaying the reconstructed one or more frames and the reconstructed current frame.

[0006] In another example, a system includes a processing device and a non-transitory computer-readable medium communicatively coupled to the processing device. The processing device is configured to perform operations by executing program code stored on the non-transitory computer-readable medium. The operations include reconstructing one or more frames of a video from a video bitstream representing the video, and reconstructing a current frame based on a reference frame from the reconstructed one or more frames, where for a block of the current frame, reconstructing the current frame includes identifying a reference block in the reference frame based on a motion vector associated with the block, determining first and second parameters of a local illumination compensation model based on a first set of reconstructed pixels in the current frame and a second set of reconstructed pixels in the reference frame, decoding a first adjustment to the first parameters from the video bitstream, adjusting the first parameters by applying the first adjustment to the first parameters, determining an adjusted second parameter, and reconstructing pixels of the block by applying the local illumination compensation model having the adjusted first and adjusted second parameters to the reference block. The operations further include displaying the reconstructed one or more frames and the reconstructed current frame.

[0007] In yet another example, a method for performing inter prediction for a video frame using local illumination compensation includes accessing a plurality of frames of the video and performing inter prediction for a current frame within the plurality of frames, where performing inter prediction for a block of the current frame includes determining first and second parameters of a local illumination compensation model based on a first set of reconstructed pixels in the current frame and a second set of reconstructed pixels in a reference frame within the plurality of frames, and determining an adjustment to the first parameter based on a portion of the first parameter, the second parameter, original pixels in the block, and reconstructed pixels in a reference block in the reference frame, the reference block being associated with the block by a motion vector calculated for the block. The decoding method further includes encoding at least the adjustment to the first parameter into a bitstream representing the video.

[0008] In another example, a non-transitory computer-readable medium has stored thereon program code executable by one or more processing devices to perform operations. The operations include accessing a plurality of frames of a video and performing inter prediction for a current frame within the plurality of frames, where performing inter prediction for a block of the current frame includes determining first and second parameters of a local illumination compensation model based on a first set of reconstructed pixels in the current frame and a second set of reconstructed pixels in a reference frame within the plurality of frames, and determining an adjustment to the first parameter based on a portion of the first parameter, the second parameter, original pixels in the block, and reconstructed pixels in a reference block in the reference frame, the reference block being associated with the block by a motion vector calculated for the block. The operations further include encoding at least the adjustment to the first parameter into a bitstream representing the video.

[0009] In another example, a system includes a processing device and a non-transitory computer-readable medium communicatively coupled to the processing device. The processing device is configured to perform operations by executing program code stored on the non-transitory computer-readable medium. The operations include accessing a plurality of frames of a video and performing inter prediction for a current frame within the plurality of frames, where performing inter prediction for a block of the current frame includes determining first and second parameters of a local illumination compensation model based on a first set of reconstructed pixels in the current frame and a second set of reconstructed pixels in a reference frame within the plurality of frames, and determining an adjustment to the first parameter based on a portion of the first parameter, the second parameter, original pixels in the block, and reconstructed pixels in a reference block in the reference frame, the reference block being associated with the block by a motion vector calculated for the block. The operations further include encoding at least the adjustment to the first parameter into a bitstream representing the video.

[0010] These illustrative examples are not intended to limit or define the present disclosure, but are intended to facilitate understanding of the present disclosure by providing examples. Other examples are discussed and further explained in the specific embodiments. [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a block diagram illustrating an example of a video encoder for implementing embodiments presented herein. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a video decoder for implementing embodiments presented herein. [Diagram 3] FIG. 2 illustrates an example of a coding tree unit partition for a picture in a video, in accordance with some embodiments of the present disclosure. [Figure 4] FIG. 2 illustrates an example of a coding unit division of a coding tree unit in accordance with some embodiments of the present disclosure. [Diagram 5] FIG. 2 illustrates an example of pixels involved in local illumination compensation, according to some embodiments of the present disclosure. [Figure 6] FIG. 1 illustrates an example of a linear model for local illumination compensation and model adjustment, according to some embodiments of the present disclosure. [Figure 7] FIG. 2 illustrates another example process for performing inter prediction on video frames using local illumination compensation, according to some embodiments of the present disclosure. [Figure 8] FIG. 4 illustrates another example process for decoding inter-predicted frames of video encoded using local illumination compensation, according to some embodiments of the present disclosure. [Figure 9] FIG. 1 illustrates an example of a computing system that can be used to implement some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] The features, embodiments, and advantages of the present disclosure can be better understood by reading the following specific embodiments with reference to the drawings.

[0013] Each embodiment provides an adjustment of a local illumination compensation model for video coding. As described above, more and more video data is generated, stored, and transmitted. It is beneficial to improve the efficiency of video coding techniques. One method is by inter-prediction, i.e., using pixels or samples from other reconstructed frames when making predictions for video pixels or samples of a currently waiting to be decoded frame. However, there may be illumination changes between a current frame and a reference frame due to light changes (such as blinking). Therefore, a local illumination compensation (LIC) technique is used to compensate for illumination changes by modeling the local illumination variation between a current block and its predicted block as a linear function of the local illumination variation between a current block template (such as adjacent pixels of the current block) and a reference block template (such as adjacent pixels of the reference block). Since the local illumination compensation model is obtained by minimizing a predefined error measure using luminance samples from adjacent templates, without using luminance samples from the current block and the reference block, the obtained model may be inaccurate, which leads to a large prediction error and a decrease in coding efficiency.

[0014] Each embodiment described herein solves these problems by determining a local illumination compensation model for a block based on pixels of the block, thereby obtaining a more accurate local illumination compensation model. Information about the determined local illumination compensation model can be transmitted in a video bitstream, so that a decoder has sufficient information for decoding. For example, a video encoder can be configured to determine parameters of a basic local illumination compensation model for the block based on pixels of a reference frame and a current frame that can be obtained when the decoder decodes the current block. For example, the encoder can use reconstructed pixels adjacent to a reference block in a reference frame (referred to as a "reference block template") and pixels adjacent to the block in a current frame (referred to as a "current block template"). The encoder can further determine adjustments to parameters of the local illumination compensation model taking into account pixels of the current block, thereby determining an updated, more accurate local illumination compensation model, and transmitting one or more adjustments in the video bitstream for the decoder to use during decoding.

[0015] JPEG2025508549000002.jpg65170

[0016] To determine an adjustment to the gradient parameter, a set of candidate adjustments can be identified. For each candidate adjustment, a predicted pixel of the block can be determined by applying an updated local illumination compensation model with a corresponding adjustment. A distortion value between the predicted pixel and the original pixel in the current block can be determined. The candidate adjustment that results in the smallest distortion can be selected as the adjustment to the gradient parameter and encoded into the video bitstream.

[0017] In another embodiment, the updated local illumination compensation model is not limited, and both the adjustments of the slope and offset parameters can be transmitted in the video bitstream to achieve a more accurate local illumination compensation model. Therefore, any method available to the encoder can be used to determine the updated local illumination compensation model. For example, the decoder can use pixels in the current block and reconstructed pixels in the reference block to obtain the parameters of the updated local illumination compensation model. The difference between the obtained parameters and the parameters of the base model can be coded into the video bitstream as an adjustment to the parameters of the local illumination compensation model.

[0018] JPEG2025508549000003.jpg64170

[0019] As described herein, some embodiments improve the efficiency of video coding by adjusting a local illumination compensation model by considering pixels of a current block. Because the local illumination compensation model of a block is obtained based on pixels of the current block, not on neighboring pixels, the illumination compensation is more accurate and the prediction error of the reconstructed block is smaller. Thus, the video coding efficiency can be improved. In future video coding standards, this technique can be an effective coding tool.

[0020] Referring to the drawings, Fig. 1 is a block diagram illustrating an example of a video encoder 100 configured to realize embodiments proposed herein. In the example shown in Fig. 1, the video encoder 100 includes a partition module 112, a transform module 114, a quantization module 115, an inverse quantization module 118, an inverse transform module 119, a loop filter module 120, an intra prediction module 126, an inter prediction module 124, a motion estimation module 122, a decoded picture buffer 130, and an entropy coding module 116.

[0021] The input of the video encoder 100 is an input video 102 that includes a sequence of pictures (also called frames or images). In a block-based video encoder, for each picture, the video encoder 100 divides the picture into blocks 104 using a partition module 112, with each block including multiple pixels. These blocks may be macroblocks, coding tree units, coding units, prediction units, and / or prediction blocks. A picture may include blocks of different sizes, and block partitioning may be different for different pictures in a video. Each block may be coded using different predictions, such as intra prediction, inter prediction, or mixed intra and inter prediction.

[0022] Typically, the first picture of a video signal is an intra-coded picture, which is coded using only intra-prediction. Intra-prediction modes predict blocks of a picture using only previously coded data in the same picture. Intra-coded pictures can be decoded without using information from other pictures. To perform intra-prediction, the video encoder 100 shown in FIG. 1 can use an intra-prediction module 126. The intra-prediction module 126 is used to generate an intra-predicted block (prediction block 134) using reconstructed samples in a reconstructed block 136 of a neighboring block of the same picture. The intra-prediction is performed according to the intra-prediction mode selected for said block. Then, the video encoder 100 calculates the difference between the block 104 and the intra-predicted block 134. The difference is called the residual block 106.

[0023] To further remove redundancy within the block, the transform module 114 transforms the residual block 106 into a transform domain by applying a transform to the samples within the block. Examples of transforms may include, but are not limited to, a discrete cosine transform (DCT) or a discrete sine transform (DST). The transformed values ​​may be referred to as transform coefficients and represent the residual block in the transform domain. In some examples, the residual block may be directly quantized without transformation by the transform module 114. This mode is referred to as a transform skip mode.

[0024] The video encoder 100 may further quantize the transform coefficients using a quantization module 115 to obtain quantized coefficients. Quantization involves dividing a sample by a quantization step length and then rounding, whereas inverse quantization involves multiplying the quantized value by the quantization step length. This quantization process is called scalar quantization. Quantization is used to represent video samples with fewer bits by reducing the dynamic range of the video samples (transformed or untransformed).

[0025] Quantization of coefficients / samples within a block can be done independently, and some current video compression standards (such as H.264 and HEVC) adopt this quantization method. For one N*M block, a certain scan order can be used to convert the two-dimensional coefficients of the block into a one-dimensional array for coefficient quantization and encoding. Quantization of coefficients within a block can utilize scan order information. For example, quantization of a given coefficient within a block may depend on the state of one previous quantization value in the scan order. To further improve coding efficiency, multiple quantizers may be used. Which quantizer is used to quantize the current coefficient depends on the previous information of the current coefficient in the encoding / decoding scan order. This quantization method is called dependent quantization.

[0026] The degree of quantization may be adjusted by a quantization step length. For example, in the case of scalar quantization, different quantization step lengths may be adopted to achieve fine or coarse quantization. A small quantization step length corresponds to fine quantization, and a large quantization step length corresponds to coarse quantization. The quantization step length may be indicated by a quantization parameter (QP). The quantization parameter is provided in the encoded bitstream of the video, so that a video decoder can access and apply the quantization parameter for decoding.

[0027] The entropy coding module 116 then encodes the quantized samples to further reduce the size of the video signal. The entropy coding module 116 is configured to apply an entropy coding algorithm to the quantized samples. In some examples, the quantized samples are binarized with a binary container, and the coding algorithm further compresses the binary container into bits. Examples of binarization methods include, but are not limited to, truncated rice (TR), combined limited k-thorder Exp-Golomb (EGk), and k-th order Exp-Golomb binarization. Examples of entropy coding algorithms include, but are not limited to, variable length coding (VLC) schemes, context-adaptive VLC schemes (CAVLC), arithmetic coding schemes, binarization, context-adaptive binary arithmetic coding (CABAC), syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or other entropy coding techniques. The entropy coded data is added to the output coded video 132 bitstream.

[0028] As described above, the reconstructed block 136 from the neighboring blocks is used for intra prediction of the block of the picture. Generating the reconstructed block 136 of a block includes calculating a reconstructed residual of the block. The reconstructed residual can be determined by applying an inverse quantization and an inverse transform to the quantized residual of the block. The inverse quantization module 118 is configured to apply an inverse quantization to the quantized samples to obtain inverse quantized coefficients. The inverse quantization module 118 applies an inverse quantization scheme to the quantization module 115 by utilizing the same quantization step length as the quantization module 115. The inverse transform module 119 is configured to apply an inverse transform of the transform applied in the transform module 114 to the inverse quantized samples, such as an inverse DCT or an inverse DST. The output of the inverse transform module 119 is the reconstructed residual of the block in the pixel domain. The reconstructed residual may be added to the prediction block 134 of the block to obtain the reconstructed block 136 in the pixel domain. For blocks in which the transform is skipped, the inverse transform module 119 is not applied. The dequantized samples are the reconstructed residuals of the block.

[0029] Blocks in subsequent pictures after the first intra-predicted picture may be coded using inter prediction or intra prediction. In inter prediction, blocks in a picture are predicted from one or more previously coded video pictures. To perform inter prediction, video encoder 100 uses inter prediction module 124. Inter prediction module 124 is configured to perform motion compensation on the blocks based on motion estimates provided by motion estimation module 122.

[0030] The motion estimation module 122 performs motion estimation by comparing the current block 104 of the current picture with the decoded reference picture 108. The decoded reference picture 108 is stored in the decoded picture buffer 130. The motion estimation module 122 selects a reference block from the decoded reference picture 108 that best matches the current block. The motion estimation module 122 further identifies an offset between the location (e.g., x, y coordinates) of the reference block and the current block location. The offset, called a motion vector (MV), is provided to the inter prediction module 124 along with the selected reference block. In some cases, multiple reference blocks are identified for the current block in multiple decoded reference pictures 108. Thus, multiple motion vectors are generated and provided to the inter prediction module 124 along with the corresponding reference blocks.

[0031] The inter prediction module 124 performs motion compensation using the motion vector and other inter prediction parameters to generate a prediction of the current block, i.e., the inter prediction block 134. For example, based on the motion vector, the inter prediction module 124 can identify a prediction block to which the motion vector points in the corresponding reference picture. If there are higher prediction blocks, these prediction blocks are merged with a certain weight to generate the prediction block 134 of the current block.

[0032] For an inter-predicted block, video encoder 100 may subtract inter-predicted block 134 from block 104 to generate residual block 106. Residual block 106 may be transformed, quantized, and entropy coded in a manner similar to the residual of an intra-predicted block described above. Similarly, the residual may be inverse quantized and inverse transformed before being combined with the corresponding predicted block 134 to obtain a reconstructed block 136 for the inter-predicted block.

[0033] To obtain the decoded picture 108 for motion estimation, the reconstruction block 136 is processed by the loop filter module 120. The loop filter module 120 is configured to smooth pixel transitions, thereby improving video quality. The loop filter module 120 may be configured to implement one or more loop filters, such as an unlocked filter, a sampling adaptive offset (SAO) filter, an adaptive loop filter (ALF), etc.

[0034] 2 illustrates an example of a video decoder 200 configured to implement embodiments proposed herein. The video decoder 200 processes encoded video 202 in a bitstream and generates decoded pictures 208. In the example illustrated in FIG. 2, the video decoder 200 includes an entropy decoding module 216, an inverse quantization module 218, an inverse transform module 219, a loop filter module 220, an intra prediction module 226, an inter prediction module 224, and a decoded picture buffer 230.

[0035] The entropy decoding module 216 is configured to perform entropy decoding on the coded video 202. The entropy decoding module 216 decodes quantized coefficients, coding parameters (including intra-prediction parameters and inter-prediction parameters), and other information. In some examples, the entropy decoding module 216 decodes the bitstream of the coded video 202 into a binary representation and then converts the binary representation into quantization levels of the coefficients. The entropy decoded coefficient levels are then inverse quantized by the inverse quantization module 218 and inverse transformed to the pixel domain by the inverse transform module 219. The functions of the inverse quantization module 218 and the inverse transform module 219 are similar to the inverse quantization module 118 and the inverse transform module 119 described above with respect to FIG. 1, respectively. The inverse transformed residual block may be added to the corresponding prediction block 234 to generate the reconstruction block 236. The inverse transform module 219 is not applied to blocks for which the transform is skipped. The dequantized samples generated by the dequantization module 118 are used to generate a reconstruction block 236 .

[0036] A prediction block 234 for a particular block is generated based on the prediction mode of the block. If the coding parameters of the block indicate that the block is intra predicted, a reconstructed block 236 of a reference block in the same picture may be input to the intra prediction module 226 to generate the prediction block 234 for the block. If the coding parameters of the block indicate that the block is inter predicted, the prediction block 234 is generated by the inter prediction module 224. The intra prediction module 226 and the inter prediction module 224 function similarly to the intra prediction module 126 and the inter prediction module 124 of FIG. 1, respectively.

[0037] As described above with respect to Figure 1, inter prediction involves one or more reference pictures. The video decoder 200 generates a decoded picture 208 of the reference picture by applying a loop filter module 220 to the reconstructed blocks of the reference picture. The decoded picture 208 is stored in a decoded picture buffer 230 for use and output by the inter prediction module 224.

[0038] Referring to FIG. 3, FIG. 3 is a diagram illustrating an example of performing coding tree unit division on a picture in a video according to some embodiments of the present disclosure. As described above with respect to FIG. 1 and FIG. 2, in order to code a picture of a video, the picture is divided into blocks, such as CTUs (coding tree units) 302 of VVC shown in FIG. 3. For example, the CTUs 302 may be blocks of 128x128 pixels. The CTUs are processed according to an order (for example, the order shown in FIG. 3). In some examples, as shown in FIG. 4, each CTU 302 in a picture may be divided into one or more CUs (coding units) 402, and the CUs 402 may be further divided into prediction units or transform units (TUs) for prediction and transformation. Depending on the coding scheme, the CTUs 302 may be divided into CUs 402 in different manners. For example, in VVC, the CUs 402 may be rectangular or square, and may be coded if not further divided into prediction units or transform units. Each CU 402 may be the same size as the root CTU 302 or may be a smaller subdivision, such as a 4x4 block, of the root CTU 302. As shown in Figure 4, the division of CTUs 302 into CUs 402 in VVC may be a quad-tree division, a binary tree division, or a ternary tree division. In Figure 4, solid lines indicate a quad-tree division, and dashed lines indicate a binary tree or a ternary tree division.

[0039] About local lighting compensation Mixed video coding systems in VVC, HEVC, and many other practical video coding standards employ a generic tool that predicts video pixels or samples of a current frame waiting to be decoded using reconstructed pixels or samples from other frames. Coding tools following this generic architecture are usually called "inter prediction" tools, and the reconstructed frames can be called "reference frames". In still video scenes, inter prediction for pixels or samples of a current frame can be realized by using co-located pixels or samples from a reference frame. However, in video scenes involving motion, an inter prediction tool with motion compensation needs to be used. For example, a "current block" of a sample of a current frame can be predicted based on a "prediction block" or a "reference block" of a sample of a reference frame, whereas a "prediction block" or a "reference block" is first determined by decoding a "motion vector", which writes the position of the prediction block of the reference frame relative to the current block of the current frame into the bitstream. More complex inter prediction tools are used to develop video scenes with complex motion (such as occlusion and affine motion).

[0040] However, due to light changes (e.g., blinking), there may be illumination changes between the current picture and the reference picture. Conventional motion compensation, including affine-based motion compensation, cannot efficiently capture and process such light changes for encoding. Local illumination compensation (LIC) is an inter-prediction technique that models the local illumination variation between the current block and the reference block as a linear function of the local illumination variation between the current block template and the reference block template. FIG. 5 is a diagram illustrating an example of pixels involved in local illumination compensation, according to some embodiments of the present disclosure. For an M×N current block 502 in a current frame 504, a reference block 512 of the same size is determined from a reference frame 514 containing reconstructed pixels using transmitted or obtained side information (e.g., reference frame index, motion model mode selection, and motion vectors). A current block template 506 is determined based on the neighboring reconstructed pixels around the current block 502. Similarly, a reference block template 516 is determined based on the neighboring co-located pixels around the reference block 512. In FIG. 5, pixels in the current block 502 are shown as blank squares, and neighboring reconstructed / decoded pixels in the current block template 506 are shown as squares filled with vertical lines. Based on the known motion information, a reference block 512 for the current block 502 is determined and shown as a dotted square in the reference frame 514. In some examples, the reference block 512 is at an integer pixel location. If the motion vector of the current block points to a sub-pixel location, the closest inter pixel block in the reference frame is determined as the reference block. The reference block template 516 in the reference frame 514 contains the neighboring reconstructed / decoded pixels of the reference block 512, and the reference block template 516 has the same shape as the current block template 506 and is shown as a square filled with cross lines.

[0041] The reference block template 516 in the reference frame 514 and the current block template 506 in the current frame 504 are used to model a function, thereby reflecting the local illumination changes of the current block. In some examples, a linear function with a slope parameter α and an offset parameter β is used to simulate this relationship. More specifically, the predicted luminance sample p[x] of the current block is calculated using the following equation:

number

[0042] As described above, since the local illumination compensation model is obtained by minimizing a predefined error measure using luminance samples from neighboring templates, the model may not be optimal for predicting luminance samples of the current block. Although the neighboring templates and the current block are very close in space, the optimality of the model cannot be guaranteed. For example, the samples in the neighboring templates and the current block may be from different objects in the video scene and have different illumination characteristics. To solve these problems, the accuracy of the model may be improved by adjusting or updating the local illumination compensation model by taking into account the pixels in the current block.

[0043] JPEG2025508549000005.jpg69170

[0044] JPEG2025508549000006.jpg60170

[0045] Compared to the base model 602, the updated model 604 can adjust the slope parameter α and the offset parameter β. The adjustment to the slope (denoted as d) can be written into the bitstream and used to adjust the base local illumination compensation model at the decoder side. The updated slope α updated can be calculated using the following formula:

number

number

[0046] To find the tilt adjustments for the local illumination compensation model, the video encoder calculates a set of candidate tilt adjustments D={d1 ,d 2 ,…,d T} can be selected. In some examples, each candidate slope adjustment d i is a positive number, and D includes values ​​close to 0, e.g., {-3, -2, -1, 0, 1, 2, 3}. Candidate slope adjustment d i Given , the encoder calculates the updated slope and offset according to Equation 2 and Equation 3. The encoder can use the updated local illumination compensation model to calculate the prediction of the current block utilizing Equation 1. Since the encoder has access to the current block, it can calculate the distortion between the prediction and the original pixels in the current block. The candidate slope adjustment associated with the smallest distortion is selected as the final slope adjustment d of the current block and is thus coded into the bitstream. In some examples, the distortion may be the sum of absolute transform distortions (SATD). The SATD may be calculated as the sum of absolute values ​​of the transform sample differences between the current block and the prediction. The transform may be any type of transform applied to the current block and the prediction block, such as the Hadamard transform. Other types of distortions, such as mean square error (MSE), sum of absolute differences (SAD), etc. may also be used.

[0047] To effectively code the slope adjustment, first, a flag abs_delta_slope_greater0_flag may be coded to indicate whether the absolute value of the slope adjustment is equal to zero. If abs_delta_slope_greater0_flag is not zero, abs_delta_slope_greater1_flag may be coded to indicate whether the absolute value of the slope adjustment is greater than one. If abs_delta_slope_greater1_flag is greater than one, a syntax element abs_delta_slope_minus2 may be coded to indicate the absolute value of the slope adjustment minus two. Also, if the absolute value of the slope adjustment is not zero, a delta_slope_sign_flag may be coded to indicate whether the slope adjustment is positive or negative. Possible syntax representations are shown in Table 1 below. [Table 1] In another embodiment, both the tilt adjustment and the offset adjustment can be coded into the video bitstream and decoded at the decoder side to adjust the base model for local illumination compensation. The tilt adjustment and the offset adjustment are respectively denoted by d s and d o If we express it as, the updated slope α updated and offset β updated can be calculated using the following formula:

number

number

[0048] Since both the tilt adjustment and the offset adjustment are transmitted in the video bitstream, there is no need to constrain the updated local illumination compensation model. Thus, the adjustments to the tilt and offset parameters (or equivalently, the updated local illumination compensation model) may be determined using any method available at the decoder side that makes the local illumination compensation model more accurate. For example, the decoder may use pixels in the current block and reconstructed pixels in the reference block to obtain the parameters of the updated local illumination compensation model. The difference between the obtained parameters and the parameters of the base model may be coded into the video bitstream as an adjustment to the parameters of the local illumination compensation model.

[0049] In the above embodiment, the use of coding parameter adjustments can be written into the bitstream at different levels and can also be enabled or disabled. For example, at the Sequence Parameter Set (SPS) level, the following signaling can be added (additions are underlined): [Table 2] <<When sps_lic_delta_flag is equal to 1, it specifies that delta-based LIC is enabled in the Coding Layer Video Sequence (CLVS). When sps_lic_delta_flag is equal to 0, it specifies that delta-based LIC is disabled in the CLVS.>> [Table 3] <<When pps_lic_delta_flag is equal to 1, it specifies that delta-based LIC is enabled for the current picture, and when pps_lic_delta_flag is equal to 0, it specifies that delta-based LIC is disabled for the current picture. If pps_lic_delta_flag is not present, its value is inferred to be equal to 0.>> [Table 4] If sh_lic_delta_flag is equal to 1, it specifies that delta-based LIC is currently enabled for the slice, and if sh_lic_delta_flag is equal to 0, it specifies that delta-based LIC is currently disabled for the slice. If sh_lic_delta_flag is not present, its value is inferred to be equal to 0. 7 illustrates an example process 700 for performing inter prediction on a video frame using local illumination compensation, according to some embodiments of the present disclosure. One or more computing devices (e.g., computing devices implementing the video encoder 100) implement the operations illustrated in FIG. 7 by executing appropriate program code (e.g., program code implementing the inter prediction module 124). For illustrative purposes, the process 700 is described with reference to the illustrated example. However, other implementations may be employed.

[0050] At step 702, the process 700 includes accessing a set of frames of a video signal. Each frame in the set of frames is divided into blocks. A block may be a portion of a frame, such as the encoding unit 402 described in FIG. 4, or any type of block that is processed as a unit during encoding in a video encoder. In some examples, the set of frames includes at least one frame that has been coded and reconstructed, and one frame that is coded by inter prediction using the reconstructed frame as a reference frame.

[0051] In step 704, which includes steps 706 and 708, the process 700 includes performing inter prediction on a frame in the set of frames. In step 706, the process 700 includes determining parameters of a basic local illumination compensation model for a block of the frame based on reconstructed pixels in the frame and a reference frame. For example, as described in detail above with respect to FIG. 5, luminance samples in the current block template 506 and the reference block template 516 may be used to determine parameters of the basic local illumination compensation model using a least mean square (LMS) or similar method.

[0052] At step 708, process 700 includes determining adjustments to parameters of the local illumination compensation model. As described above, the encoder can determine the adjustments to the slope parameter and obtain the adjustments to the offset parameter or the adjusted offset parameter using the method described with respect to Figure 6. Alternatively, the encoder can determine adjustments to the slope and offset parameters of the model using any method that makes the local illumination compensation model more accurate.

[0053] At step 710, the process 700 includes encoding the adjustments to the parameters of the local illumination compensation model into the video bitstream. If the offset adjustment is derived based on the tilt adjustment, the tilt adjustment is encoded into the bitstream without encoding the offset adjustment. Otherwise, both the adjustments to the tilt and offset can be encoded into the bitstream based on the syntax representation shown in Table 1 and the signaling mechanism described above.

[0054] 8 illustrates an example process 800 for decoding an inter-predicted frame of a video encoded using local illumination compensation, according to some embodiments of the present disclosure. One or more computing devices may implement the operations illustrated in FIG. 8 by executing appropriate program code. For example, a computing device implementing the video decoder 200 may implement the operations illustrated in FIG. 8 by executing program code of the inter-prediction module 224. For illustrative purposes, the process 800 is described with reference to some illustrated examples. However, other implementations may be employed.

[0055] In step 802, the process 800 includes reconstructing one or more frames of the video from a video bitstream representing the video, for example using the decoding method described above with respect to FIG. 2. In step 804, which includes steps 806-818, the process 800 includes reconstructing a current frame based on a reference frame from the one or more reconstructed frames. In step 806, the process 800 includes identifying a reference block for a block of the current frame in the reference frame. The reference block may be identified based on a decoded motion vector associated with the block. In some examples, if the motion vector points to a block having a sub-pixel position in the reference frame, the sub-pixel position may be rounded to the nearest integer pixel position to identify the reference block.

[0056] In step 808, the process 800 includes determining parameters of a basic local illumination compensation model using reconstructed pixels of the current and reference frames. For example, as described in detail above with respect to FIG. 5, luminance samples in the current and reference block templates 506, 516 may be used to determine parameters of the basic local illumination compensation model using a least mean squares (LMS) or similar method.

[0057] In step 810, the process 800 includes decoding an adjustment to a slope parameter of the local illumination compensation model from the video bitstream. In step 812, the process 800 includes updating the slope parameter of the model, for example, according to Equation 2 or Equation 4. In step 814, the process 800 includes determining an adjusted offset parameter of the local illumination compensation model. For example, if the offset adjustment is not transmitted in the video bitstream, the adjusted offset can be obtained based on the adjusted slope according to Equation 3 above. If the offset adjustment is encoded in the video bitstream, the offset adjustment can be directly decoded from the video bitstream, or the adjusted offset can be calculated according to Equation 5 above. In step 816, the process 800 includes performing reconstruction on the block. The decoder can generate a prediction of the block by applying the updated local illumination compensation model to a reference block in a reference frame. The prediction can be used to reconstruct the block by combining it with other data associated with the block (such as a prediction residual). As described above with respect to Figures 1 and 2, the decoder may also use the reconstructed blocks to perform intra or inter prediction on other blocks or pictures in the video. In step 818, the reconstructed blocks may also be output for display along with other decoded blocks in the frame.

[0058] It should be understood that the above examples are illustrative and should not be construed as limiting. Different implementations can be adopted to generate and update the local illumination compensation model. For example, instead of the L-shaped template shown in FIG. 5, other shapes of templates can be used, such as a shape that includes only the left neighboring pixels or the upper neighboring pixels of the current block and the reference block. In another example, the left neighboring pixels of multiple columns and / or the upper neighboring pixels of multiple rows may be used as the template. Also, although the above focuses on a linear function for the local illumination compensation model, other types of functions can be used. For example, a general high-order polynomial function (such as a quadratic polynomial function) can be used to model the illumination change.

[0059] Example of a Computing System Any suitable computing system may be used to perform the operations described herein. For example, FIG. 9 illustrates an example of a computing device 900 capable of implementing the video encoder 100 of FIG. 1 or the video decoder 200 of FIG. 2. In some embodiments, the computing device 900 may include a processor 912 communicatively coupled to a memory 914 and configured to execute computer-executable program code and / or access information stored in the memory 914. The processor 912 may include a microprocessor, an application specific integrated circuit ("ASIC"), a state machine, or other processing device. The processor 912 may include any one or more of a plurality of processing devices. Such a processor may include or be in communication with a computer readable medium having instructions stored thereon. These instructions, when executed by the processor 912, cause the processor to perform the operations described herein.

[0060] The memory 914 may include any suitable non-transitory computer-readable medium. The computer-readable medium may include any electronic, optical, magnetic, or other storage device capable of providing computer-readable instructions or other program code to a processor. Non-limiting examples of computer-readable media include disks, memory chips, ROM, RAM, ASICs, configured processors, optical memory, magnetic tape or other magnetic storage devices, or other media from which a computer processor can read instructions. These instructions may include processor-specific instructions generated by a compiler and / or interpreter from code written in any suitable computer programming language (including, for example, C, C++, C#, Visual Basic, Java, Python, Perl, JavaScript, and ActionScript).

[0061] The computing device 900 may further include a bus 916. The bus 916 may be communicatively coupled to one or more components of the computing device 900. The computing device 900 may further include a number of external or internal devices, such as input or output devices. For example, the illustrated computing device 900 has an input / output ("I / O") interface 918 that may receive input from one or more input devices 920 or provide output to one or more output devices 922. The one or more input devices 920 and the one or more output devices 922 may be communicatively coupled to the I / O interface 918. The communicative coupling may be realized in any suitable manner (e.g., a connection via a printed circuit board, a connection via a cable, communication via wireless transmission, etc.). Non-limiting examples of input devices 920 include a touchscreen (e.g., one or more cameras for imaging the touch area, or a pressure sensor for detecting pressure changes due to a touch), a mouse, a keyboard, or any other device that can be used to generate input events in response to a user's physical manipulation of the computing device. Non-limiting examples of output devices 922 include an LCD screen, an external display, speakers, or any other device that can be used to display or otherwise present output generated by the computing device.

[0062] The computing device 900 may execute program code that configures the processor 912 to perform one or more operations described above with respect to Figures 1-8. The program code may include the video encoder 100 or the video decoder 200. The program code may reside in a memory 914 or any suitable computer readable medium and may be executed by the processor 912 or any other suitable processor.

[0063] The computing device 900 may further include at least one network interface device 924. The network interface devices 924 may include any device or group of devices suitable for establishing a wired or wireless data connection to one or more data networks 928. Non-limiting examples of the network interface devices 924 include Ethernet network adapters, modems, and / or similar devices. The computing device 900 may transmit messages in the form of electronic or optical signals through the network interface devices 924.

[0064] General Considerations Numerous details have been described herein to provide a thorough understanding of the claimed subject matter. However, those skilled in the art will understand that the claimed subject matter may be practiced without these details. In other instances, methods, apparatus, or systems known to those skilled in the art have not been described in detail so as not to obscure the claimed subject matter.

[0065] Unless otherwise indicated, it is to be understood that discussions herein using "processing," "computing," "determining," "identifying," or similar terms refer to operations or processes of a computing device (e.g., one or more computers or similar electronic computing devices) that manipulate or transform data represented as physical electronic or magnetic quantities in the memory, registers, or information storage, transmission, or display devices of the computing platform.

[0066] The systems discussed herein are not limited to any particular hardware architecture or configuration. A computing device may include any suitable arrangement of components conditioned to provide a result with respect to one or more inputs. Suitable computing devices include microprocessor-based general-purpose computer systems that can access stored software that programs or configures a computing system from a general-purpose computing device to a special-purpose computing device that implements one or more embodiments of the present subject matter. In the software used to program or configure a computing device, any suitable programming language, scripting language, or other type of language or combination of languages ​​may be used to implement the teachings contained herein.

[0067] Method embodiments of the present disclosure may be performed in operation of such a computing device. The order of steps presented in the above examples may be changed, for example, steps may be reordered, combined, and / or decomposed into sub-steps. Certain steps or processes may be performed in parallel.

[0068] As used herein, "adapted to" or "configured to" is open and inclusive language and does not exclude equipment adapted or configured to perform additional tasks or steps. Furthermore, use of the term "based on" is open and inclusive because a process, step, calculation, or other operation "based on" one or more stated conditions or values ​​may in fact be based on other conditions or values ​​other than those stated. Titles, lists, and numbers contained herein are for convenience of description only and are not meant to be limiting.

[0069] While the present subject matter has been described in detail with respect to specific embodiments thereof, it is to be understood that those skilled in the art, upon learning the above, may readily make modifications, variations, and equivalents to these embodiments. It is therefore to be understood that the present disclosure is presented for purposes of illustration and not limitation, and is not intended to exclude the inclusion of modifications, variations, and / or additions to the present subject matter as would become apparent to those skilled in the art.

Claims

1. 1. A method for decoding inter-predicted frames of video, the video being encoded using local illumination compensation, comprising: reconstructing one or more frames of the video from a video bitstream representing the video; reconstructing a current frame based on a reference frame from the reconstructed one or more frames; displaying the reconstructed one or more frames and the reconstructed current frame; The step of reconstructing the current frame for a block of the current frame includes: identifying a reference block in the reference frame based on a motion vector associated with the block; determining first and second parameters of a local illumination compensation model based on a first set of reconstructed pixels in the current frame and a second set of reconstructed pixels in the reference frame; decoding a first adjustment to the first parameter from the video bitstream; adjusting the first parameter by applying the first adjustment to the first parameter; determining an adjusted second parameter; and reconstructing pixels of the block by applying the local illumination compensation model having the adjusted first parameter and the adjusted second parameter to the reference block.

2. the local illumination compensation model is a linear model, the first parameter represents a slope and the second parameter represents an offset; The decoding method of claim 1 .

3. The decoding method comprises: further comprising calculating a representative average luminance value based on the reference frame or the current frame; determining the adjusted second parameter includes calculating the adjusted second parameter based on the adjusted first parameter, the second parameter, and the representative average luminance value; The decoding method according to claim 2.

4. the representative average luminance value is either an average value of reconstructed pixels of the reference block, an average value of a second set of reconstructed pixels in the reference frame, or an average value of a first set of reconstructed pixels in the current frame; The decoding method according to claim 3.

5. The step of determining the adjusted second parameter includes: extracting from the video bitstream a second adjustment to the second parameter; and determining the adjusted second parameter by applying the second adjustment to the second parameter. The decoding method of claim 1 .

6. The decoding method comprises: encoding one or more syntax elements into the video bitstream, the one or more syntax elements representing a first adjustment to the first parameter in the video bitstream. The decoding method of claim 1 .

7. the first set of reconstructed pixels in the current frame includes one or more of reconstructed pixels above the block in the current frame or reconstructed pixels to the left of the block in the current frame, and the second set of reconstructed pixels in the reference frame includes one or more of reconstructed pixels above the reference block in the reference frame or reconstructed pixels to the left of the reference block in the reference frame; The decoding method of claim 1 .

8. A non-transitory computer readable medium having stored thereon a program code and a video bitstream, the program code causing one or more processing devices to perform the decoding method of any one of claims 1 to 7 to decode the video bitstream.

9. 1. A method for performing inter prediction on a frame of a video using local illumination compensation, the inter prediction method comprising: accessing a plurality of frames of the video; performing inter prediction on a current frame among the plurality of frames, the step of performing inter prediction on a block of the current frame comprising: determining first and second parameters of a local illumination compensation model based on a first set of reconstructed pixels in the current frame and a second set of reconstructed pixels in a reference frame within the plurality of frames; determining an adjustment to the first parameter based at least in part on the first parameter, the second parameter, original pixels in the block, and reconstructed pixels in a reference block in the reference frame, the reference block being related to the block by a motion vector calculated for the block; encoding at least the adjustment to the first parameter into a bitstream representing the video.

10. the local illumination compensation model is a linear model, the first parameter represents a slope of the linear model, and the second parameter represents an offset of the linear model; The inter prediction method according to claim 9 .

11. The adjustment to the first parameter may further be determined based on a representative average luminance value, the representative average luminance value being determined based on the reference frame or the current frame, and determining the adjustment to the first parameter may include: For each candidate adjustment in the candidate adjustment list, adjusting the first parameter by applying the candidate adjustment to the first parameter; adjusting the second parameter by modifying the second parameter based on the adjusted first parameter and the representative average luminance value; calculating predicted pixels for the block by applying the local illumination compensation model having the adjusted first parameter and the adjusted second parameter to reconstructed pixels in the reference block; determining distortion between predicted pixels and original pixels in the block; the adjustment to the first parameter is determined as the candidate adjustment in the list of candidate adjustments that has the least distortion; The inter prediction method according to claim 10 .

12. The distortion is either the sum of absolute transformation distortions (SATD), the sum of absolute distortions (SAD), or the mean square error (MSE). The inter prediction method according to claim 11 .

13. the representative average luminance value is either an average value of reconstructed pixels of the reference block, an average value of a second set of reconstructed pixels in the reference frame, or an average value of a first set of reconstructed pixels in the current frame; The inter prediction method according to claim 12 .

14. The inter prediction method includes: determining an adjustment to the second parameter based at least in part on the first parameter, the second parameter, original pixels in the block, and reconstructed pixels in the reference block; encoding the adjustments to the second parameters into a bitstream representing the video. The inter prediction method according to claim 9 .

15. 15. A non-transitory computer-readable medium having stored thereon a program code and a video bitstream, the program code causing one or more processing devices to perform the inter prediction method of any one of claims 9 to 14 to generate the video bitstream.

Images