Image decoding device, image decoding method and program

Abstract

To improve coding efficiency.SOLUTION: According to the present invention, an image decoding device 200 includes an intra-frame prediction unit 204 for generating a first prediction pixel on the basis of a decoded pixel and control information, a first inter-frame prediction unit 205 for generating a second prediction pixel on the basis of a decoded pixel and control information stored in a storage unit 208, a second inter-frame prediction unit 206 for generating a weight on a per-pixel basis from related information on a reference pixel on the basis of the decoded pixel and the control information stored in the storage unit 208 to generate a third prediction pixel, and an adder 207 for adding a prediction residual to one of the first prediction pixel, the second prediction pixel and the third prediction pixel to acquire a decoded pixel.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image decoding device, an image decoding method, and a program. [Background technology]

[0002] Non-Patent Documents 1 and 2 disclose bi-prediction that uses two reference blocks as one type of inter prediction.

[0003] In bi-prediction, a simple average of the reference block L0 and the reference block L1 with weights of (1 / 2, 1 / 2) is used as a predicted pixel.

[0004] Furthermore, Non-Patent Documents 1 and 2 disclose coding unit (CU) level weighted bi-prediction (BCW) as an extension of bi-prediction.

[0005] In bi-prediction, BCW uses a weighted average of pixels weighted by a coefficient predetermined for each CU as a predicted pixel.

[0006] The BCW weights for the bi-predictive reference blocks L0 and L1 are selected by explicitly coded indices from (4 / 8, 4 / 8), (5 / 8, 3 / 8), (3 / 8, 5 / 8), (10 / 8, -2 / 8), and (-2 / 8, 10 / 8), respectively. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] ITU-T H.266 VVC [Non-patent document 2] M. Coban, et al., “Algorithm description of Enhanced Compression Model 12 (ECM 12)”, JVET-AG2025, 2024 Summary of the Invention [Problem to be solved by the invention]

[0008] However, the bi-prediction of Non-Patent Document 1 and Non-Patent Document 2 involves a simple average, which causes a problem that prediction accuracy is not necessarily high.

[0009] On the other hand, BCW allows different weights to be set, which can solve some of the problems with bi-prediction.

[0010] However, since the weights of BCW are only in units of CU, there is a problem that the prediction accuracy in units of pixels is not necessarily high.

[0011] Furthermore, since the index representing the weight of the BCW is explicitly coded, it is difficult to increase the number of types of weight.

[0012] Therefore, the present invention has been made in consideration of the above-mentioned problems, and an object of the present invention is to provide an image decoding device, an image decoding method, and a program with high coding efficiency. [Means for solving the problem]

[0013] a storage unit that stores the decoded pixels; a first inter-frame prediction unit that generates a second predicted pixel based on the decoded pixels stored in the storage unit and the control information; a second inter-frame prediction unit that generates a third predicted pixel by generating a pixel-by-pixel weight from related information of a reference pixel based on the decoded pixels stored in the storage unit and the control information; and an adder that adds the prediction residual and any one of the first predicted pixel, the second predicted pixel, or the third predicted pixel to obtain a decoded pixel.

[0014] A second feature of the present invention is summarized as an image decoding method comprising the steps of: decoding control information and quantized values; inversely quantizing the quantized values ​​to obtain transform coefficients; inversely transforming the transform coefficients to obtain prediction residuals; generating a first predicted pixel based on decoded pixels and the control information; storing the decoded pixels; generating a second predicted pixel based on the stored decoded pixels and the control information; generating a pixel-by-pixel weight from related information of a reference pixel based on the decoded pixels stored in the storage unit and the control information to generate a third predicted pixel; and adding the prediction residual to any one of the first predicted pixel, the second predicted pixel, or the third predicted pixel to obtain a decoded pixel.

[0015] a storage unit that stores the decoded pixels; a first inter-frame prediction unit that generates a second predicted pixel based on the decoded pixels stored in the storage unit and the control information; a second inter-frame prediction unit that generates a third predicted pixel by generating a pixel-by-pixel weight from related information of a reference pixel based on the decoded pixels stored in the storage unit and the control information; and an adder that adds the prediction residual and any one of the first predicted pixel, the second predicted pixel, and the third predicted pixel to obtain a decoded pixel. [Effects of the Invention]

[0016] According to the present invention, it is possible to provide an image decoding device, an image decoding method, and a program with high coding efficiency. [Brief explanation of the drawings]

[0017] [Figure 1]FIG. 1 is a diagram showing an example of functional blocks of an image decoding device 200 according to the first embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of a reference block in bi-prediction. [Figure 3] FIG. 3 is a diagram showing an example where a=2 and b=4 for a block of 8×8 pixels. [Figure 4] FIG. 4 is a flowchart showing an example of a method for setting a correction method on a sequence-by-sequence basis. [Figure 5] FIG. 5 is a flowchart showing an example of a method for applying pixel-wise bi-prediction on a block-by-block basis. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, etc., and various variations, including combinations with other existing components, are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

[0019] First Embodiment An image decoding device 200 according to this embodiment will be described below with reference to Figures 1 to 5. Figure 1 is a diagram showing an example of functional blocks of the image decoding device 200 according to this embodiment.

[0020] As shown in FIG. 1, the image decoding device 200 includes a decoding unit 201, an inverse quantization unit 202, an inverse transform unit 203, an intra-frame prediction unit 204, a first inter-frame prediction unit 205, a second inter-frame prediction unit 206, an adder 207, and an accumulation unit 208.

[0021] The decoding unit 201 is configured to decode control information and quantization values ​​from code information coded by the image coding device. For example, the decoding unit 201 is configured to output the control information and the quantization values ​​by performing variable length decoding on the code information.

[0022] Here, the quantized value is sent to the inverse quantization unit 202, and the control information is sent to the intra-frame prediction unit 204, the first inter-frame prediction unit 205, and the second inter-frame prediction unit 206. Note that this control information includes information necessary for controlling the inverse quantization unit 202, the intra-frame prediction unit 204, the first inter-frame prediction unit 205, the second inter-frame prediction unit 206, etc., and may also include header information such as a sequence parameter set, a picture parameter set, a picture header, and a slice header.

[0023] The inverse quantization unit 202 is configured to inverse quantize the quantized values ​​sent from the decoding unit 201 to generate transform coefficients. The transform coefficients are sent to the inverse transform unit 203.

[0024] The inverse transform unit 203 is configured to inversely transform the transform coefficients sent from the inverse quantization unit 202 to generate prediction residuals. The prediction residuals are sent to an adder 207.

[0025] The intra-frame prediction unit 204 is configured to generate a first predicted pixel to be added to the prediction residual by the adder 207, based on the decoded pixel obtained via the adder 207 and the control information decoded by the decoding unit 201. The first predicted pixel is sent to the adder 207.

[0026] The first inter-frame prediction unit 205 is configured to generate second predicted pixels to be added to the prediction residual in the adder 207, based on decoded pixels obtained by referring to the storage unit 208 and the control information decoded by the decoding unit 201. The second predicted pixels are sent to the adder 207.

[0027] The storage unit 208 is configured to cumulatively store the decoded pixels sent from the adder 207. These decoded pixels are referenced by the first inter-frame prediction unit 205 and the second inter-frame prediction unit 206 via the storage unit 208.

[0028] The adder 207 is configured to obtain decoded pixels by adding the prediction residual sent from the inverse transform unit 203 to first predicted pixels, second predicted pixels, and third predicted pixels (described later) input from the intra-frame prediction unit 204, first inter-frame prediction unit 205, and second inter-frame prediction unit 206, respectively. These decoded pixels are output as an image and sent to the storage unit 208 and the intra-frame prediction unit 204.

[0029] (Second inter-frame prediction unit 206) The function of the second inter-frame prediction unit 206 will be described below.

[0030] The second inter-frame prediction unit 206 is configured to generate pixel-by-pixel weights from related information of reference pixels based on the decoded pixels stored in the storage unit 208 and the control information decoded by the decoding unit 201, and to generate a third predicted pixel.

[0031] The role of the second inter-frame prediction unit 206 is to derive weights for bi-predictive reference pixels on a pixel-by-pixel basis and predict the pixel to be decoded so that the third predicted pixel can compensate for the block to be decoded with high precision in the subsequent adder 207.

[0032] Conventional bi-prediction is merely a simple average, and BCW is merely a limited weighted average on a block-by-block basis. Therefore, the image decoding device 200 according to this embodiment improves prediction accuracy by adaptive weighted averaging on a pixel-by-pixel basis.

[0033] The second inter-frame prediction unit 206 can limit the target to which pixel-based bi-prediction is applied to blocks of a specific size.

[0034] Furthermore, the second inter-frame predictor 206 can generate the weights described above by using one or more combinations of decoded pixels and control information, as shown in the following first to third embodiments.

[0035] First, as a first embodiment, an example will be described in which the second inter-frame prediction unit 206 generates a third predicted pixel by using the texture of a decoded pixel.

[0036] Specifically, the second inter-frame prediction unit 206 sets the ratio of the change between the reference pixels p0(x,y) and p1(x,y) of bi-prediction L0 and L1 and their neighboring pixels to weight w, and calculates the predicted value pred(x,y) by weighted averaging, as shown in the following equation.

[0037] pred(x,y)=w×p0(x,y)+(1-w)×p1(x,y) w=g1 / (g0+g1) g0=abs(4×p0(x,y)-(p0(x,y-1)+p0(x-1,y)+p0(x+1,y)+p0(x,y+1))) g1=abs(4×p1(x,y)-(p1(x,y-1)+p1(x-1,y)+p1(x+1,y)+p1(x,y+1))) Here, abs represents a function that outputs an absolute value. g0 and g1 can use any expression that represents the texture variation in pixel units or any neighboring pixels. For example, g0 and g1 may be set as follows:

[0038] g0=abs(p0(x-1,y-1)+2×p0(x,y-1)+p0(x+1,y-1)-(p0(x-1,y+1)+2×p0(x,y+1)+p0(x+1,y+1))) +abs(p0(x-1,y-1)+2×p0(x-1,y1)+p0(x-1,y+1)-((p0(x+1,y-1)+2×p0(x+1,y)+p0(x+1,y+1))) g1=abs(p1(x-1,y-1)+2×p1(x,y-1)+p1(x+1,y-1)-(p1(x-1,y+1)+2×p1(x,y+1)+p1(x+1,y+1))) +abs(p1(x-1,y-1)+2×p1(x-1,y1)+p1(x-1,y+1)-((p1(x+1,y-1)+2×p1(x+1,y)+p1(x+1,y+1))) The second inter-frame predictor 206 sets the weights so that they are inversely proportional to the variation in texture (the greater the variation, the smaller the weight).

[0039] That is, with this configuration, textures with large variations are regarded as errors, and the effect is reduced on a pixel-by-pixel basis, thereby improving prediction accuracy.

[0040] Secondly, as a second embodiment, an example will be described in which the second inter-frame predictor 206 generates a third predicted pixel by using a block boundary as control information.

[0041] In the above-described first embodiment, the additive noise of the original image is reduced, but in the second embodiment, the noise generated during encoding is reduced.

[0042] As a basic concept, the second inter-frame prediction unit 206 generates the weights for each pixel described above so that they are proportional to the prediction accuracy of the reference pixels.

[0043] As a premise, a reference block in bi-prediction may straddle multiple blocks in a reference frame as shown in FIG. 2, and therefore, a block boundary may be included within the reference block.

[0044] However, since the type and parameters of coding differ for each block, subjective image quality is likely to deteriorate at block boundaries due to differences in coding performance.

[0045] Therefore, if the pixel values ​​are referred to as they are, the influence of the block boundaries will be propagated. Therefore, a method of applying a filter to reduce the influence of the block boundaries has been conventionally used, but this affects prediction accuracy as pixel values ​​change.

[0046] That is, in bi-prediction, the second inter-frame prediction unit 206 can improve prediction accuracy by reducing the weights for pixels on block boundaries and increasing the weights for pixels that are not on block boundaries.

[0047] As an example of the control information used in the second embodiment, the second inter-frame prediction unit 206 uses the distance from the block boundary included in each of the L0 and L1 reference blocks in pixel units.

[0048] Specifically, first, the second inter-frame prediction unit 206 sets, for each pixel of the bi-predictive L0 and L1 reference blocks, a distance d from the block boundary included in the block.

[0049] If there are multiple block boundaries within one reference block, the second inter-frame prediction unit 206 calculates the distance d_i for each block boundary and sets the shortest distance. The second inter-frame prediction unit 206 may set an upper limit a for the distance.

[0050] Here, the distance d can be expressed by the following formula:

[0051] d=min(a,min(d_i)) Second, the second inter-frame predictor 206 sets a weight w based on the difference between the two distances (the distance d0 of L0 and the distance d1 of L1).

[0052] Specifically, the second inter-frame prediction unit 206 sets the amount of change in the distance d0 of L0 and the distance d1 of L1 in bi-prediction to a weight w, and calculates the predicted value pred(x, y) by weighted averaging, as shown in the following equation.

[0053] pred(x,y)=w×p(x,y)+(1-w)×q(x,y) w=(d1-d0+b) / (2b) Here, b is a value that controls the granularity of the weights. b can be set in advance, can be changed explicitly by signaling, or can be changed implicitly by control information such as block size.

[0054] FIG. 3 shows an example where a=2 and b=4 for a block of 8×8 pixels.

[0055] As shown in Figure 3, pixels that are far from the block boundary have a weight of w=4 / 8, but if one of them is adjacent to the block boundary, the weight is reduced to w=2 / 8, and if it is not adjacent, the weight is increased to w=6 / 8.

[0056] Therefore, this configuration reduces the influence of block boundaries and improves prediction accuracy.

[0057] Alternatively, the second inter-frame prediction unit 206 can use the inverse of the strength of a loop filter such as a deblocking filter as the distance d in addition to the block boundary.

[0058] That is, the second inter-frame prediction unit 206 can improve prediction accuracy by decreasing the weight as the filter strength increases and decreasing the weight as the filter strength decreases.

[0059] Thirdly, as a third embodiment, an example will be described in which the second inter-frame predictor 206 generates a third predicted pixel using a quantization parameter as control information.

[0060] As described in the second embodiment above, a reference block may contain multiple blocks.

[0061] Each can have a different QP (quantization parameter), and the QP is different even for L0 and L1, so the image quality is different for each.

[0062] That is, the second inter-frame predictor 206 can improve prediction accuracy by reducing the weights for reference pixels with large QPs and increasing the weights for reference pixels with small QPs in bi-prediction.

[0063] Specifically, the second inter-frame prediction unit 206 sets a weight w based on the ratio of two quantization parameters Q0(x,y) and Q1(x,y) of bi-prediction L0 and L1, as shown in the following equation, and calculates the predicted value pred(x,y) by weighted averaging.

[0064] pred(x,y)=w×p0(x,y)+(1-w)×p1(x,y) w=g1 / (g0+g1) g0=Q0(x,y)+c g1=Q1(x,y)+c Here, c is a value that controls the granularity of the weights. c can be set in advance, can be changed explicitly by signaling, or can be changed implicitly by control information such as block size.

[0065] In any of the embodiments, not only is an adaptive weight set for each pixel, but signaling indicating the weight is not required, so that the coding efficiency can be improved.

[0066] The control information decoded by the decoding unit 201 for the second inter-frame prediction will be described below.

[0067] The input coded information may include a sequence parameter set (SPS) that summarizes control information for each sequence. The input coded information may also include a picture parameter set (PPS) or a picture header (PH) that summarizes control information for each picture. Alternatively, the input coded information may include a slice header (SH) that summarizes control information for each slice.

[0068] Hereinafter, a method for setting the correction method for each sequence will be described with reference to FIG.

[0069] As shown in FIG. 4, in step S101, the decoding unit 201 determines whether sps_bpw_enabled_flag is 1 in the sequence parameter set.

[0070] sps_bpw_enabled_flag is a syntax that controls whether pixel-level bi-prediction is enabled or disabled. When sps_bpw_enabled_flag is 1, it indicates that pixel-level bi-prediction is enabled, and when sps_bpw_enabled_flag is 0, it indicates that pixel-level bi-prediction is disabled.

[0071] In step S102, the decoding unit 201 decodes sps_bpw_mode when sps_bpw_enabled_flag is 1. sps_bpw_mode is syntax that controls the pixel-by-pixel bi-prediction method.

[0072] By using sps_bpw_mode, the pixel-level bi-prediction method can be changed according to the image characteristics on a sequence-by-sequence basis, which is expected to maximize coding efficiency.

[0073] For example, for sequences composed of CG, pixel distributions are often composed of the same values, so it is possible to set the weight fluctuations for pixel-by-pixel bi-prediction to be small, while for sequences composed of natural images, pixel distributions are diverse, so it is possible to set the weight fluctuations for pixel-by-pixel bi-prediction to be large, thereby maximizing coding efficiency.

[0074] When the correction method is set on a picture-by-picture basis, the decoding unit 201 similarly decodes pps_bpw_enabled_flag and pps_bpw_mode in the picture parameter set or picture header.

[0075] By using pps_bpw_mode, the pixel-level bi-prediction method can be changed in accordance with the image characteristics on a picture-by-picture basis, which is expected to maximize coding efficiency.

[0076] For example, for pictures composed of CG, the pixel distribution is often composed of the same value, so the weight fluctuations for pixel-by-pixel bi-prediction can be set to be small, while for pictures composed of natural images, the pixel distribution is diverse, so the weight fluctuations for pixel-by-pixel bi-prediction can be set to be large, thereby maximizing coding efficiency.

[0077] When the correction method is set on a slice-by-slice basis, the decoding unit 201 similarly decodes sh_bpw_enabled_flag and sh_bpw_mode in the slice header.

[0078] By using sh_bpw_mode, the pixel-level bi-prediction method can be changed in accordance with the image characteristics in slice units, which is expected to maximize coding efficiency.

[0079] For example, for slices composed of CG, pixel distribution is often composed of the same value, so it is possible to set the weight fluctuations of pixel-by-pixel bi-prediction to be small, while for slices composed of natural images, pixel distribution is diverse, so it is possible to set the weight fluctuations of pixel-by-pixel bi-prediction to be large, thereby maximizing coding efficiency.

[0080] By setting only the upper layer, it is possible to suppress an increase in the amount of coding, and by setting the lower layer as well and then prioritizing the setting in the lower layer, adaptive control is possible.

[0081] Alternatively, if the pixel-by-pixel bi-prediction method is set in advance, decoding of the pixel-by-pixel bi-prediction method itself can be omitted.

[0082] In the above example, a method for setting the pixel-based bi-prediction method in units of sequences, pictures, or slices has been described, but a method may be directly selected in units of blocks, as described below, without setting these methods. In this case, the increase in header information described above can be avoided.

[0083] A method for applying pixel-wise bi-prediction on a block-by-block basis will be described with reference to FIG.

[0084] As shown in FIG. 5, in step S201, the decoding unit 201 determines whether any one of sps_bpw_enabled_flag, pps_bpw_enabled_flag, and sh_bpw_enabled_flag is 1.

[0085] If either of them is 1, in step S202, the decoding unit 201 decodes cu_bpw_mode, which controls the pixel-by-pixel bi-prediction method.

[0086] According to this embodiment, adaptive weights are set for each pixel in bi-prediction, and signaling indicating the weights is not required, so that coding efficiency can be improved.

[0087] The above-described image decoding device 200 may be realized as a program that causes a computer to execute each function (each step). [Industrial Applicability]

[0088] According to this embodiment, for example, it is possible to improve the overall service quality in video communication, which will contribute to the achievement of Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "Develop resilient infrastructure, promote sustainable industrialization and foster innovation." [Explanation of symbols]

[0089] 200...Image decoding device 201...Decoding unit 202...Inverse quantization section 203...Inverse conversion unit 204...Intra-frame prediction unit 205...First inter-frame prediction unit 206...Second inter-frame prediction unit 207...adder 208...Storage section

Claims

1. An image decoding device, a decoding unit for decoding the control information and the quantized value; an inverse quantization unit that inversely quantizes the quantized values ​​to generate transform coefficients; an inverse transform unit that inversely transforms the transform coefficients to generate prediction residuals; an intra-frame prediction unit that generates a first predicted pixel based on the decoded pixel and the control information; a storage unit that stores the decoded pixels; a first inter-frame prediction unit that generates a second predicted pixel based on the decoded pixel stored in the storage unit and the control information; a second inter-frame prediction unit that generates a weight for each pixel from related information of reference pixels based on the decoded pixels stored in the storage unit and the control information, and generates a third predicted pixel; an adder that adds the prediction residual and any one of the first predicted pixel, the second predicted pixel, or the third predicted pixel to obtain a decoded pixel.

2. The image decoding device according to claim 1 , wherein the second inter-frame prediction unit limits the application of pixel-wise bi-prediction to blocks of a specific size.

3. The image decoding device according to claim 1 , wherein the second inter-frame prediction unit generates the weights by using one or more combinations of the decoded pixels and the control information.

4. 4. The image decoding device according to claim 3, wherein the second inter-frame prediction section sets the weight to a ratio of an amount of change between the reference pixel and a pixel adjacent to the reference pixel.

5. 4. The image decoding device according to claim 3, wherein the second inter-frame prediction section generates the weight so that the weight is inversely proportional to a variation in the amount of change between the reference pixel and a pixel adjacent to the reference pixel.

6. 5. The image decoding device according to claim 4, wherein the second inter-frame prediction section generates the weights in proportion to prediction accuracy of the reference pixels.

7. 5. The image decoding device according to claim 4, wherein the second inter-frame prediction unit reduces the weights for pixels on a block boundary and increases the weights for pixels that are not on the block boundary.

8. 5. The image decoding device according to claim 4, wherein the second inter-frame prediction unit uses a distance from a block boundary included in each of the reference blocks in units of pixels.

9. The image decoding device according to claim 8 , wherein the second inter-frame prediction unit generates the weight based on a difference between two of the distances.

10. 5. The image decoding device according to claim 4, wherein the second inter-frame prediction unit reduces the weights for reference pixels having a large quantization parameter and increases the weights for reference pixels having a small quantization parameter.

11. 5. The image decoding device according to claim 4, wherein the second inter-frame predictor generates the weight based on a ratio of two quantization parameters.

12. An image decoding method, comprising: decoding the control information and the quantized values; dequantizing the quantized values ​​to form transform coefficients; inverse transforming the transform coefficients to obtain prediction residuals; generating a first predicted pixel based on the decoded pixel and the control information; storing the decoded pixels; generating a second predicted pixel based on the stored decoded pixel and the control information; generating a weight for each pixel from related information of reference pixels based on the decoded pixels stored in the storage unit and the control information, thereby generating a third predicted pixel; an image decoding method comprising: adding the prediction residual and any one of the first predicted pixel, the second predicted pixel, and the third predicted pixel to obtain a decoded pixel;

13. A program that causes a computer to function as an image decoding device, The image decoding device comprises: a decoding unit for decoding the control information and the quantized value; an inverse quantization unit that inversely quantizes the quantized values ​​to generate transform coefficients; an inverse transform unit that inversely transforms the transform coefficients to generate prediction residuals; an intra-frame prediction unit that generates a first predicted pixel based on the decoded pixel and the control information; a storage unit that stores the decoded pixels; a first inter-frame prediction unit that generates a second predicted pixel based on the decoded pixel stored in the storage unit and the control information; a second inter-frame prediction unit that generates a weight for each pixel from related information of reference pixels based on the decoded pixels stored in the storage unit and the control information, and generates a third predicted pixel; an adder that adds the prediction residual and any one of the first predicted pixel, the second predicted pixel, and the third predicted pixel to obtain a decoded pixel.

Images