Image decoding device and image decoding method

Abstract

To improve the coding efficiency.SOLUTION: An image coding device (1) encodes a target image in block units. The image coding device (1) includes: a prediction unit (109) that generates a prediction image corresponding to the target image by performing prediction using a plurality of reference images; an evaluation unit (111) that evaluates the similarity between the plurality of reference images on a pixel-by-pixel basis; a calculation unit (101) that calculates a predicted residual indicating a difference in pixel unit between the target image and the predicted image; a determination unit (112) that determines a part of the predicted residual area to which orthogonal transformation and quantization are applied, on the basis of the evaluation result by the evaluation unit; and a transformation / quantization unit (102) that performs orthogonal transformation and quantization only in the part of the predicted residual region.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present invention relates to an image decoding device and an image decoding method. [Background technology]

[0002] Conventionally, in an encoding device that encodes a target image (image block), a method is known in which motion compensation prediction is performed using a plurality of reference images, a predicted image corresponding to the target image is generated, and an orthogonal transform and quantization are performed on the prediction residual that indicates the difference between the target image and the predicted image on a pixel-by-pixel basis.

[0003] The image coding device also encodes the quantized transform coefficients obtained by the orthogonal transform and quantization using entropy coding, and restores the prediction residual by performing inverse quantization and inverse orthogonal transform on the quantized transform coefficients.The image coding device then combines the restored prediction residual with a predicted image to reconstruct the target image, and uses the reconstructed image for subsequent predictions.

[0004] On the other hand, an image decoding device decodes the entropy-coded quantized transform coefficients and performs motion compensation prediction using multiple reference images to generate a predicted image corresponding to the target image. The image decoding device performs inverse quantization and inverse orthogonal transform on the quantized transform coefficients to restore a prediction residual, and combines the restored prediction residual with the predicted image to reconstruct the target image.

[0005] In motion compensation prediction using a plurality of reference images, such image encoding devices and image decoding devices generate a predicted image by, for example, averaging the plurality of reference images (see Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Recommendation ITU-T H.265,(12 / 2016), “High efficiency video coding”, International Telecommunication Union Summary of the Invention

[0007] However, if there is a part where the similarity between multiple reference images used to generate a predicted image is significantly low, the accuracy of the predicted image in that part (i.e., prediction accuracy) will decrease, and the prediction residual in that part will be larger than in other parts.

[0008] When a prediction residual containing both large and small prediction residuals is orthogonally transformed, power is not concentrated in low-frequency components, and the degree of power concentration in the transform coefficients is reduced. Coarsely quantizing the high-frequency components of such transform coefficients results in degradation of the transform coefficients in the high-frequency components. This degradation of the transform coefficients propagates throughout the image (block) through inverse quantization and inverse orthogonal transform.

[0009] When the degraded restored prediction residual is synthesized with a predicted image to reconstruct the target image block in this way, the degradation of image quality propagates to parts where high-precision prediction has been performed. In other words, due to parts where prediction accuracy is low, the effect of quantization of orthogonal transform coefficients propagates to parts where prediction accuracy is high, degrading image quality and causing a problem of reduced coding efficiency.

[0010] Therefore, an object of the present invention is to provide an image decoding device and an image decoding method that can improve coding efficiency when prediction is performed using multiple reference images.

[0011] An image decoding device according to a first feature is an image decoding device that decodes a target image in units of blocks from encoded data, and includes: a prediction unit that generates blocks of a predicted image corresponding to blocks of the target image by performing bi-prediction using a plurality of reference images; an evaluation unit that calculates an evaluation value that indicates the similarity between the plurality of reference images; an acquisition unit that decodes the encoded data and acquires prediction residuals; and a synthesis unit that synthesizes the prediction residuals with blocks of the predicted image to reconstruct blocks of the target image, wherein the evaluation value is used to correct the synthesis target of the synthesis unit in units of partial blocks that are smaller than the blocks and consist of a plurality of pixels.

[0012] Here, the prediction performed by the prediction unit can be any prediction method that uses multiple reference images, regardless of the type of prediction method. A typical example of the prediction method is bidirectional motion compensation prediction, but prediction using the IntraBC mode (intra block copy mode) used in the HEVC system may also be used.

[0013] An image decoding device according to a second feature is an image decoding device that decodes a target image in units of blocks from an encoded bitstream, and includes: a prediction unit that generates blocks of a predicted image corresponding to blocks of the target image by performing inter-prediction; an entropy decoding unit that decodes the encoded bitstream and obtains quantized transform coefficients obtained by the image encoding device performing transform and quantization on only a partial region of a block of a prediction residual; and an identification unit that identifies the partial region, wherein the entropy decoding unit obtains information indicating a mode selected from a plurality of modes related to the partial region, the plurality of modes including a mode in which a vertical size of the partial region is equal to a size of the block while a horizontal size of the partial region is smaller than the size of the block, and a mode in which a vertical size of the partial region is smaller than the size of the block while a horizontal size of the partial region is equal to the size of the block.

[0014] A program according to the third aspect is a program for causing a computer to function as the image encoding device according to the first aspect.

[0015] A program according to the fourth feature is a program for causing a computer to function as the image decoding device according to the second feature.

[0016] According to the present invention, it is possible to provide an image decoding device and an image decoding method that can improve coding efficiency when prediction is performed using a plurality of reference images. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an image encoding device according to an embodiment. [Figure 2] FIG. 1 is a diagram illustrating a configuration of an image decoding device according to an embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of motion compensation prediction. [Figure 4] FIG. 10 is a diagram showing an example of a predicted image generated by motion compensation prediction. [Figure 5] FIG. 2 illustrates an example of the configuration of an evaluation unit. [Figure 6] FIG. 10 is a diagram illustrating an example of an operation of a determination unit according to the embodiment. [Figure 7] FIG. 1 is a diagram illustrating an example of an operation flow in an image encoding device according to an embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of an operation flow in an image decoding device according to an embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of operation in the second modification of the embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of operation in a third modified example of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] An image encoding device and an image decoding device according to an embodiment will be described with reference to the drawings. The image encoding device and the image decoding device according to the embodiment encode and decode moving images, as typified by MPEG. In the following description of the drawings, identical or similar parts are denoted by identical or similar reference numerals.

[0019] <1. Configuration of the image encoding device> Fig. 1 is a diagram showing the configuration of an image encoding device 1 according to an embodiment. As shown in Fig. 1, the image encoding device 1 includes a block dividing unit 100, a subtraction unit (calculation unit) 101, a transformation unit 102a, a quantization unit 102b, an entropy encoding unit 103, an inverse quantization unit 104a, an inverse transformation unit 104b, a synthesis unit 105, an intra prediction unit 106, a loop filter 107, a frame memory 108, a motion compensation prediction unit 109, a switching unit 110, an evaluation unit 111, and a determination unit 112.

[0020] The block division unit 100 divides an input image in units of frames (or pictures) into small block-shaped regions and outputs the image blocks to the subtraction unit 101 (and the motion compensation prediction unit 109). The size of the image blocks is, for example, 32×32 pixels, 16×16 pixels, 8×8 pixels, or 4×4 pixels. However, the shape of the image blocks is not limited to squares and may be rectangular. An image block is a unit for encoding by the image encoding device 1 and a unit for decoding by the image decoding device 2, and such an image block is referred to as a target image block. Such an image block may also be referred to as a coding unit (CU) or coding block (CB).

[0021] The subtraction unit 101 calculates a prediction residual indicating a difference in pixel units between the current image block input from the block division unit 100 and a predicted image (predicted image block) corresponding to the current image block. Specifically, the subtraction unit 101 calculates the prediction residual by subtracting each pixel value of the predicted image from each pixel value of the current image block, and outputs the calculated prediction residual to the conversion unit 102a. The predicted image is input to the subtraction unit 101 via a switching unit 110 from an intra prediction unit 106 or a motion compensation prediction unit 109, which will be described later.

[0022] The transform unit 102a and the quantization unit 102b constitute a transform / quantization unit 102 that performs orthogonal transform processing and quantization processing on the prediction residuals. The transform / quantization unit 102 does not perform orthogonal transform and quantization on the prediction residuals of all pixel positions in the target image block, but performs orthogonal transform and quantization on the prediction residuals of a partial region of the target image block. This partial region is an region where the prediction residual is expected to be large, and is determined by the determination unit 112. Hereinafter, this partial region will be referred to as a "partial transform block." The partial transform block is a part of the target image block.

[0023] The transform unit 102a identifies an area corresponding to the partial transform block among the prediction residuals of the target image block input from the subtraction unit 101, based on the information on the partial transform block input from the determination unit 112. The transform unit 102a extracts the prediction residual of the partial transform block from the prediction residuals input from the subtraction unit 101, performs an orthogonal transform on the extracted prediction residual, calculates transform coefficients indicating the intensity (power) of spatial frequency components, and outputs the calculated transform coefficients to the quantization unit 102b. Note that orthogonal transform refers to, for example, a discrete cosine transform (DCT), a discrete sine transform (DST), or a Karhunen Loeve transform (KLT).

[0024] The quantization unit 102b quantizes the transform coefficients of the partial transform block input from the transform unit 102a using a quantization parameter (Qp) and a quantization matrix to generate quantized transform coefficients. The quantization parameter (Qp) is a parameter commonly applied to each transform coefficient in the block and determines the coarseness of quantization. The quantization matrix is ​​a matrix whose elements are quantization values ​​used when quantizing each transform coefficient. The quantization unit 102b outputs quantization control information, generated quantized transform coefficient information, etc. to the entropy coding unit 103 and the inverse quantization unit 104a.

[0025] The entropy coding unit 103 performs entropy coding on the partial transform block quantized transform coefficients input from the quantization unit 102b, compresses the data, generates coded data (bit stream), and outputs the coded data to the outside of the image coding device 1. For entropy coding, Huffman coding, CABAC (Context-based Adaptive Binary Arithmetic Coding), etc. may be used. Note that the entropy coding unit 103 receives as input information related to prediction from the intra prediction unit 106 and the motion compensation prediction unit 109, as input information related to filtering from the loop filter 107, and as input information related to the partial transform block from the determination unit 112. The entropy coding unit 103 also performs entropy coding on these pieces of information.

[0026] The inverse quantization unit 104a and the inverse transform unit 104b constitute the inverse quantization and inverse transform unit 104. The inverse quantization and inverse transform unit 104 performs inverse quantization and inverse orthogonal transform on the quantized transform coefficients of the partial transform block input from the quantization unit 102b, thereby restoring a partial prediction residual corresponding to the partial transform block.

[0027] The inverse quantization unit 104a performs inverse quantization processing corresponding to the quantization processing performed by the quantization unit 102b. Specifically, the inverse quantization unit 104a restores the transform coefficients by inverse quantizing the quantized transform coefficients of the partial transform block input from the quantization unit 102b using a quantization parameter (Qp) and a quantization matrix, and outputs the restored transform coefficients to the inverse transform unit 104b.

[0028] The inverse transform unit 104b performs an inverse orthogonal transform process corresponding to the orthogonal transform process performed by the transform unit 102a. For example, if the transform unit 102a performs a discrete cosine transform, the inverse transform unit 104b performs an inverse discrete cosine transform. The inverse transform unit 104b performs an inverse orthogonal transform on the transform coefficients of the partial transform block input from the inverse quantization unit 104a to restore partial prediction residuals, and outputs the restored partial prediction residuals to the synthesis unit 105.

[0029] The synthesis unit 105 identifies a region of the predicted image input from the switching unit 110 that corresponds to the partial transform block, based on the information on the partial transform block input from the determination unit 112. The synthesis unit 105 then synthesizes the restored prediction residual (restored partial prediction residual) input from the inverse transform unit 104b with the region of the predicted image that corresponds to the partial transform block, to reconstruct a current image block. Specifically, the synthesis unit 105 adds each pixel value of the restored partial prediction residual to each pixel value of the region of the predicted image that corresponds to the partial transform block, to reconstruct the current image block. The synthesis unit 105 outputs a reconstructed image, which is the reconstructed current image block, to the intra prediction unit 106 and the loop filter 107.

[0030] The intra prediction unit 106 performs intra prediction using the reconstructed image input from the synthesis unit 105 to generate an intra prediction image, and outputs the intra prediction image to the switching unit 110. The intra prediction unit 106 also outputs information about the selected intra prediction mode, etc. to the entropy coding unit 103.

[0031] The loop filter 107 performs filtering as post-processing on the reconstructed image input from the synthesis unit 105, and outputs the filtered reconstructed image to the frame memory 108. The loop filter 107 also outputs information related to the filtering to the entropy coding unit 103.

[0032] The frame memory 108 stores the reconstructed image input from the loop filter 107 in units of frames.

[0033] The motion compensation prediction unit 109 performs inter prediction using one or more reconstructed images stored in the frame memory 108 as reference images. Specifically, the motion compensation prediction unit 109 calculates a motion vector using a technique such as block matching, generates a motion compensation prediction image based on the motion vector, and outputs the motion compensation prediction image to the switching unit 110. The motion compensation prediction unit 109 also outputs information related to the motion vector to the entropy coding unit 103.

[0034] The switching unit 110 switches between the intra-predicted image input from the intra-prediction unit 106 and the motion-compensated predicted image input from the motion-compensated prediction unit 109, and outputs the predicted image (intra-predicted image or motion-compensated predicted image) to the subtraction unit 101 and the synthesis unit 105.

[0035] When the motion compensation prediction unit 109 performs motion compensation prediction using multiple reference images, the evaluation unit 111 evaluates the similarity between the multiple reference images on a pixel-by-pixel basis, and outputs information on the evaluation result to the determination unit 112. Note that the evaluation unit 111 may also evaluate the similarity between the multiple reference images when using intra prediction using multiple reference images (for example, intra block copy mode) or the like and the intra prediction unit 106 performs prediction using multiple reference images.

[0036] The determination unit 112 determines a partial transform block to which orthogonal transform and quantization are applied, based on the result of the evaluation by the evaluation unit 111. Specifically, the determination unit 112 determines the center position of the partial transform block and the size of the partial transform block based on this center position, based on the result of the evaluation by the evaluation unit 111. The determination unit 112 outputs information indicating the determined partial transform block to the transform unit 102a and the synthesis unit 105, and outputs information indicating the size of the determined partial transform block (size information) to the entropy coding unit 103. The size information is coded in the entropy coding unit 103 and output as part of control information included in the coded data.

[0037] <2. Configuration of the image decoding device> Fig. 2 is a diagram showing the configuration of an image decoding device 2 according to an embodiment. As shown in Fig. 2, the image decoding device 2 includes an entropy decoding unit 200, an inverse quantization unit 201a, an inverse transform unit 201b, a synthesis unit 202, an intra prediction unit 203, a loop filter 204, a frame memory 205, a motion compensation prediction unit 206, a switching unit 207, an evaluation unit 208, and a specification unit 209.

[0038] The entropy decoding unit 200 decodes the coded data generated by the coding device 1 and outputs quantized transform coefficients corresponding to partial transform blocks to which orthogonal transform and quantization have been applied in the image coding device 1 to the inverse quantization unit 201a. The entropy decoding unit 200 also decodes the coded data and obtains information related to prediction (intra prediction and motion compensation prediction) and information related to filtering. The entropy decoding unit 200 outputs the information related to prediction to the intra prediction unit 203 and the motion compensation prediction unit 206, and outputs the information related to filtering to the loop filter 204. The entropy decoding unit 200 outputs size information of the partial transform blocks to the identification unit 209 and the inverse transform unit 201b.

[0039] The inverse quantization unit 201a and the inverse transform unit 201b constitute an inverse quantization and inverse transform unit 201. The inverse quantization and inverse transform unit 201 performs inverse quantization and inverse orthogonal transform on the quantized transform coefficients to reconstruct partial prediction residuals corresponding to partial transform blocks.

[0040] The inverse quantization unit 201a performs inverse quantization processing corresponding to the quantization processing performed by the quantization unit 102b of the image encoding device 1. The inverse quantization unit 201a restores the transform coefficients by inverse quantizing the quantized transform coefficients of the partial transform block input from the entropy decoding unit 200 using a quantization parameter (Qp) and a quantization matrix, and outputs the restored transform coefficients to the inverse transform unit 201b.

[0041] The inverse transform unit 201b performs inverse orthogonal transform processing corresponding to the orthogonal transform processing performed by the transform unit 102a of the image encoding device 1. The inverse transform unit 201b performs inverse orthogonal transform on the transform coefficients input from the inverse quantization unit 201a based on the size information of the partial transform block input from the entropy decoding unit 200 to restore the prediction residual, and outputs the restored prediction residual (restored prediction residual) to the synthesis unit 202.

[0042] The synthesis unit 202 identifies an area of ​​the predicted image input from the switching unit 207 that corresponds to the partial transform block, based on the information on the partial transform block input from the identification unit 209. The synthesis unit 202 reconstructs the original target image block by combining, on a pixel-by-pixel basis, the partial prediction residual restored by the inverse transform unit 201b with the area of ​​the predicted image that corresponds to the partial transform block, and outputs the reconstructed image block to the intra prediction unit 203 and the loop filter 204.

[0043] The intra prediction unit 203 references the reconstructed image block input from the synthesis unit 202 and performs intra prediction according to the intra prediction information input from the entropy decoding unit 200 to generate an intra prediction image, and outputs the intra prediction image to the switching unit 207.

[0044] Based on the filter processing information input from the entropy decoding unit 200, the loop filter 204 performs a filter processing on the reconstructed image input from the synthesis unit 202 similar to the filter processing performed by the loop filter 107 of the image encoding device 1, and outputs the reconstructed image after the filter processing to the frame memory 205.

[0045] The frame memory 205 stores the reconstructed images input from the loop filter 204 in units of frames. The frame memory 205 outputs the stored reconstructed images to the outside of the image decoding device 2 in the order in which they are to be displayed.

[0046] The motion compensation prediction unit 206 uses one or more reconstructed images stored in the frame memory 205 as reference images and performs motion compensation prediction (inter-prediction) according to the motion vector information input from the entropy decoding unit 200 to generate a motion compensation prediction image, and outputs the motion compensation prediction image to the switching unit 207.

[0047] The switching unit 207 switches between the intra-predicted image input from the intra-prediction unit 203 and the motion-compensated predicted image input from the motion-compensated prediction unit 206, and outputs the predicted image (intra-predicted image or motion-compensated predicted image) to the synthesis unit 202.

[0048] The evaluation unit 208 performs the same operation as the evaluation unit 111 of the image encoding device 1. Specifically, when the motion compensation prediction unit 206 performs motion compensation prediction using multiple reference images, the evaluation unit 208 evaluates the similarity between the multiple reference images on a pixel-by-pixel basis, and outputs information on the evaluation result to the identification unit 209. Note that the evaluation unit 208 may also evaluate the similarity between the multiple reference images when using intra prediction using multiple reference images (for example, intra block copy mode) or the like and when the intra prediction unit 203 performs prediction using multiple reference images.

[0049] The identification unit 209 identifies a partial transform block to which orthogonal transform and quantization have been applied in the image encoding device 1, based on the result of the evaluation by the evaluation unit 208. Specifically, the identification unit 209 identifies a center position of the partial transform block based on the result of the evaluation by the evaluation unit 208. The identification unit 209 then identifies the partial transform block based on the identified center position and the size information input from the entropy decoding unit 200. The identification unit 209 outputs information indicating the identified partial transform block to the synthesis unit 202.

[0050] 3. Motion Compensated Prediction Fig. 3 is a diagram showing an example of motion compensation prediction. Fig. 4 is a diagram showing an example of a predicted image generated by motion compensation prediction. As a simple example of motion compensation prediction, a case where bi-prediction used in HEVC, particularly forward and backward prediction (bi-directional prediction), is used will be described.

[0051] As shown in Figure 3, motion compensation prediction refers to frames temporally before and after the target frame (current frame). In the example of Figure 3, motion compensation prediction of a block in an image of the tth frame is performed by referring to the t-1th frame and the t+1th frame. Motion compensation detects a location (block) similar to the target image block from within the reference frames of the t-1st and t+1st frames within a search range set by the system.

[0052] The detected location is the reference image. Information indicating the relative position of the reference image with respect to the target image block is indicated by the arrow in the figure and is called a motion vector. The motion vector information is coded by entropy coding together with frame information of the reference image in the image coding device 1. Meanwhile, the image decoding device 2 detects the reference image based on the information of the motion vector generated by the image coding device 1.

[0053] As shown in Figures 3 and 4, reference images 1 and 2 detected by motion compensation prediction are similar partial images aligned with the target image block within the reference frame, and are therefore similar to the target image block (image to be encoded). In the example of Figure 4, the target image block includes a star pattern and a partial circle pattern. Reference image 1 includes a star pattern and a full circle pattern. Reference image 2 includes a star pattern but does not include a circle pattern.

[0054] A predicted image is generated from these reference images 1 and 2. Because prediction processing places a high processing load on the system, it is common to generate a predicted image by averaging reference images 1 and 2. However, a predicted image may also be generated by using more advanced processing, such as signal enhancement processing using a low-pass filter or high-pass filter. Here, reference image 1 contains a circular pattern, and reference image 2 does not contain a circular pattern. Therefore, when reference images 1 and 2 are averaged to generate a predicted image, the signal of the circular pattern in the predicted image is reduced by half compared to reference image 1.

[0055] The difference between the predicted image obtained from reference images 1 and 2 and the target image block (target image to be encoded) is the prediction residual. In the prediction residual shown in Figure 4, a large difference occurs only in the misaligned portions of the edges of the star pattern and the misaligned portions of the circle pattern (hatched areas), but for other parts, predictions are performed with high accuracy and the difference is small (no difference occurs in the example of Figure 4).

[0056] The areas where no difference occurs (non-edge areas of the star pattern and background areas) are areas where the similarity between reference image 1 and reference image 2 is high and where highly accurate predictions have been made. On the other hand, the areas where a large difference occurs are areas specific to each reference image, that is, areas where the similarity between reference image 1 and reference image 2 is significantly low. Therefore, it can be seen that areas where the similarity between reference image 1 and reference image 2 is significantly low have low prediction accuracy and cause large differences (residuals).

[0057] When a prediction residual having both large and small differences is orthogonally transformed, power is not concentrated in the low-frequency components, and the degree of power concentration in the transform coefficients is reduced. As a result, the degradation of the transform coefficients due to quantization becomes significant, and this degradation propagates throughout the image (target image block) through inverse quantization and inverse orthogonal transform.

[0058] When the prediction residual (restored prediction residual) restored by inverse quantization and inverse orthogonal transform is combined with the predicted image to reconstruct the target image block, the degradation of image quality propagates to areas where high-precision predictions have been made, such as the non-edge and background parts of the star pattern shown in Figure 4.

[0059] In an embodiment, by performing orthogonal transformation and quantization of the prediction residual only on a partial transform block, which is a part where the similarity between reference image 1 and reference image 2 is low (i.e., a part where the prediction residual is estimated to be large), it becomes possible to suppress the propagation of the degradation of the transform coefficients caused by quantization to areas within the target image block where the prediction residual is small, thereby improving image quality.

[0060] <4. Evaluation Unit in Image Encoding Device and Image Decoding Device> In the image encoding device 1, the evaluation unit 111 evaluates the prediction accuracy of the predicted image on a pixel-by-pixel basis by calculating the similarity between multiple reference images. Similarly, in the image decoding device 2, the evaluation unit 208 evaluates the prediction accuracy of the predicted image on a pixel-by-pixel basis by calculating the similarity between multiple reference images on a pixel-by-pixel basis.

[0061] Fig. 5 is a diagram showing an example of the configuration of the evaluation unit 111 in the image encoding device 1. As shown in Fig. 5, the evaluation unit 111 includes a difference calculation unit (subtraction unit) 111a, a normalization unit 111b, and an adjustment unit 111c.

[0062] The difference calculation unit 111a calculates a difference value between reference image 1 and reference image 2 on a pixel-by-pixel basis, and outputs the calculated difference value to the normalization unit 111b. The difference value is an example of a value indicating similarity. It can be said that the smaller the difference value, the higher the similarity, and the larger the difference value, the lower the similarity. The difference calculation unit 111a may calculate the difference value after performing a filter process on each reference image. The difference calculation unit 111a may calculate a statistic such as a squared error, and use this statistic as the similarity. An example of using the difference value as the similarity will be described below.

[0063] The normalization unit 111b normalizes the difference value input from the similarity calculation unit 111a by the largest difference value in the block (i.e., the maximum value of the difference values ​​in the block) and outputs the normalized difference value. The normalized difference value is used as the importance for determining the partial transform block. The smaller the difference value, the higher the similarity and the higher the prediction accuracy, and therefore the lower the importance in encoding. On the other hand, the larger the difference value, the lower the similarity and the lower the prediction accuracy, and therefore the higher the importance in encoding.

[0064] The normalization unit 111b normalizes the difference value of each pixel input from the difference calculation unit 111a by the difference value of the pixel with the largest difference value in the block (i.e., the maximum value of the difference value in the block), and outputs the normalized difference value (importance).

[0065] The adjustment unit 111c adjusts the normalized difference value input from the normalization unit 111b based on a quantization parameter (Qp) that determines the coarseness of quantization, and outputs the adjusted normalized difference value. Since the degree of degradation of the restored prediction residual increases as the coarseness of quantization increases, the adjustment unit 111c adjusts the normalized difference value (weight) based on the quantization parameter (Qp).

[0066] The importance Rij of each pixel position (ij) output by the evaluation unit 111 can be expressed, for example, as in the following equation (1).

[0067] Rij = (abs(Xij-Yij) / maxD ×Scale(Qp)) ···(1)

[0068] In equation (1), Xij is the pixel value of pixel ij in reference image 1, Yij is the pixel value of pixel ij in reference image 2, and abs is a function for obtaining the absolute value. The difference calculation unit 111a shown in FIG. 5 outputs abs(Xij-Yij).

[0069] In addition, in equation (1), maxD is the maximum value of the difference values ​​abs(Xij-Yij) within the block. To calculate maxD, it is necessary to calculate the difference values ​​for all pixels within the block. However, to omit this process, the maximum value of adjacent blocks that have already been coded may be used instead. Alternatively, maxD may be calculated from the quantization parameter (Qp) using a table that defines the correspondence between the quantization parameter (Qp) and maxD. Alternatively, a fixed value defined in advance in the specifications may be used as maxD. The normalization unit 111b outputs abs(Xij-Yij) / maxD.

[0070] In addition, in equation (1), Scale(Qp) is a coefficient multiplied according to the quantization parameter (Qp). Scale(Qp) is designed to approach 1.0 when Qp is large and to approach 0 when Qp is small, and the degree of this is adjusted depending on the system. Alternatively, a fixed value defined in advance in the specifications may be used as Scale(Qp). Furthermore, to simplify processing, Scale(Qp) may be a fixed value, such as 1.0, designed according to the system.

[0071] The adjustment unit 111c outputs abs(Xij-Yij) / maxD×Scale(Qp) as the importance Rij. Furthermore, this Rij may be output as a weighting adjusted by a sensitivity function designed according to the system. For example, the sensitivity may be adjusted by setting abs(Xij-Yij) / maxD×Scale(Qp)=Rij and setting Rij=Clip(Rij,1.0,0.0), or by adding an offset such as Rij=Clip(Rij+offset,1.0,0.0). Note that Clip(x,max,min) indicates a process of clipping at max if x exceeds max, and at min if x is below min.

[0072] The importance Rij calculated in this way is a value within the range from 0 to 1.0. Basically, the importance Rij approaches 1.0 when the difference value of pixel position ij between reference images is large (i.e., prediction accuracy is low), and approaches 0 when the difference value of pixel position ij between reference images is small (i.e., prediction accuracy is high). The evaluation unit 111 outputs two-dimensional map information consisting of the importance Rij of each pixel position ij in the block to the determination unit 112 on a block-by-block basis.

[0073] Alternatively, the evaluation unit 111 may calculate Rij according to the following equation (2).

[0074] Rij = 1.0 - (abs(Xij-Yij) / maxD × Scale(Qp)) ···(2)

[0075] When formula (2) is used, Rij approaches 0 when the difference value of pixel position ij is large (i.e., low prediction accuracy), and approaches 1 when the difference value is small (i.e., high prediction accuracy). Therefore, Rij can be regarded as a value representing the prediction accuracy (accuracy) of pixel position ij. In the following, an example will be described in which the evaluation unit 111 uses formula (2) to output two-dimensional map information consisting of the accuracy Rij of each pixel position ij in a block, and such map information will be referred to as an "accuracy map" as appropriate.

[0076] Furthermore, the evaluation unit 208 in the image decoding device 2 has the same configuration as the evaluation unit 111 in the image encoding device 1. Specifically, the evaluation unit 208 in the image decoding device 2 includes a similarity calculation unit 208a, a normalization unit 208b, and an adjustment unit 208c. The evaluation unit 208 in the image decoding device 2 outputs to the identification unit 209 an accuracy map consisting of the accuracy Rij of each pixel position ij in the block.

[0077] Note that the evaluation unit 111 and the evaluation unit 208 perform evaluation (calculation of Rij) only when prediction using multiple reference images is applied, and do not need to perform evaluation in other modes, such as unidirectional prediction or intra prediction processing that does not use multiple reference images.

[0078] The evaluation unit 111 may calculate an error map using the following method and output it to the identification unit 209.

[0079] If the luminance signals of the two reference images (reference blocks) used to generate a predicted image in bi-prediction mode are L0[i,j] and L1[i,j] (where [i,j] are coordinates within the target image block), the error map map[i,j] and its maximum value max_map are calculated using the following equation (3).

[0080] map [i,j] = abs (L0 [i,j] - L1 [i,j]) max#map = max (map [i,j]) ···(3)

[0081] If max_map in equation (3) exceeds 6-bit precision (exceeds 64), the error map and maximum value are updated using the shift set so that max_map falls within 6-bit precision using equation (4) below.

[0082] max#map = max#map >> shift map [i,j] = map [i,j] >> shift ···(4)

[0083] The error map calculated by equation (4) differs from the likelihood map in that the values ​​in the area where the prediction residual is estimated to be large are large, and the values ​​in the area where the prediction residual is estimated to be small are small. That is, the normalized likelihood map is Accuracy map[i,j] = 1 - (map[i,j] / max#map) Hereinafter, the embodiment of the present invention will be described using the error map, but the prediction accuracy may be estimated using the above-mentioned accuracy map.

[0084] <5. Determination unit in image encoding device and specification unit in image decoding device> In the image encoding device 1, the determination unit 112 determines the center position of the partial transform block and the size of the partial transform block based on this center position, based on the result of the evaluation by the evaluation unit 111. Figure 6 is a diagram showing an example of the operation of the determination unit 112.

[0085] First, as shown in Fig. 6(A), based on the error map input from the evaluation unit 111, the determination unit 112 detects the least certain position, which is the pixel position in the error map where map[i, j] is maximum, and determines the detected least certain position as the center position. Note that, instead of detecting the least certain position using the error map, as described above, the determination unit 112 may be configured to determine the pixel position in the accuracy map where rij is minimum as the least certain position. Alternatively, instead of determining the least certain position as the center position, the determination unit 112 may calculate the center of gravity of the error map and determine the calculated center of gravity as the center position.

[0086] Second, as shown in FIG. 6B, the determination unit 112 averages the error map in the horizontal and vertical directions to generate a horizontal error map (horizontal error map) and a vertical error map (vertical error map). Then, the determination unit 112 expands the range of the horizontal error map and the vertical error map by one pixel at a time, based on the least likely position, and determines horizontal and vertical lengths that cover X% (e.g., 25% or 50%) of the total pixel positions with low probability. The determination unit 112 may determine the value of X based on the remaining code amount of the coded data that has been coded before the target image block, relative to the target code amount, or may use a system-fixed value as X. Furthermore, for the purpose of speeding up processing, the determination unit 112 may determine horizontal and vertical lengths that satisfy X% of the length that is a multiple of 2 (e.g., 2, 4, 8, 16, etc.). By determining the horizontal and vertical lengths, the partial transform blocks can be rectangular blocks, not just square blocks.

[0087] The determination unit 112 outputs center position information and size information (horizontal and vertical lengths) of the determined partial transform block to the transformation unit 102a and the synthesis unit 105, and outputs the size information to the entropy coding unit 103. The size information is coded in the entropy coding unit 103 and output as part of the control information included in the coded data. Note that the center position of the partial transform block can be identified by the identification unit 209 in the image decoding device 2 based on the error map, and there is no need to code the center position information, thereby reducing the amount of coding.

[0088] Meanwhile, in the image encoding device 1, the identification unit 209 identifies the center position of the partial transform block based on the result of the evaluation by the evaluation unit 208, and identifies the partial transform block based on the identified center position and the decoded size information.

[0089] First, the identifying unit 209 identifies the center position based on the error map in the same way as the determining unit 112 in the image encoding device 1 .

[0090] Second, the determination unit 209 determines the size of the target image block based on the size information (horizontal and vertical lengths) input from the entropy decoding unit 200. In this way, the determination unit 209 can determine the center position and size of the partial transform block.

[0091] The specifying unit 209 outputs the center position information and size information (horizontal and vertical lengths) of the specified partial transform block to the synthesis unit 202.

[0092] Alternatively, candidates for the center position may be defined in advance in the system. In such a case, the determining unit 112 and the identifying unit 209 may determine and identify the detected candidate position with the lowest probability or the candidate position closest to the center of gravity as the center position.

[0093] <6. Image Encoding Operation> FIG. 7 is a diagram showing an example of an operation flow in the image encoding device 1 according to the embodiment.

[0094] 7, in step S101, the motion compensation prediction unit 109 predicts a current image block by performing motion compensation prediction using multiple reference images, and generates a predicted image corresponding to the current image block. The motion compensation prediction information is generated as part of the coded data by the entropy coding unit 103, and the entropy coding unit 103 outputs coded data including the motion compensation prediction information.

[0095] In step S102, the evaluation unit 111 evaluates the prediction accuracy of the predicted image on a pixel-by-pixel basis by calculating the similarity between multiple reference images, and generates an error map that represents the low prediction accuracy of each pixel position within the target image block.

[0096] In step S103, the determination unit 112 determines a partial transform block, which is a partial region in the target image block to which orthogonal transform and quantization are applied, based on the error map, and notifies and sets the determined partial transform block to the transform / quantization unit 102 (specifically, the transform unit 102a) and the synthesis unit 105. In addition, size information of the partial transform block is coded as part of the coded data by the entropy coding unit 103.

[0097] In step S104, the subtraction unit 101 calculates a prediction residual that indicates the difference between the current image block and the predicted image in pixel units.

[0098] In step S105, the transform / quantization unit 102 performs orthogonal transform and quantization on the prediction residual of the partial transform block among the prediction residuals calculated by the subtraction unit 101, thereby generating quantized transform coefficients.

[0099] In step S106, the entropy coding unit 103 entropy codes the quantized transform coefficients of the partial transform block and outputs coded data.

[0100] In step S107, the inverse quantization / inverse transform unit 104 performs inverse quantization and inverse orthogonal transform on the quantized transform coefficients of the partial transform block to restore the prediction residual of the partial transform block, thereby generating a restored prediction residual of the partial transform block.

[0101] In step S108, the synthesis unit 105 reconstructs the target image block by synthesizing the restored prediction residual of the partial transform block with a region of the predicted image corresponding to the partial transform block on a pixel-by-pixel basis, thereby generating a reconstructed image block.

[0102] In step S109, the loop filter 107 performs a filter process on the reconstructed image block. Information about the loop filter (such as the offset and category information to which the offset is applied) is coded by the entropy coding unit 103 as part of the coded data.

[0103] In step S110, the frame memory 108 stores the reconstructed image blocks after the filtering process on a frame-by-frame basis.

[0104] <7. Image Decoding Operation> FIG. 8 is a diagram showing an example of an operation flow in the image decoding device 2 according to the embodiment.

[0105] As shown in FIG. 8, in step S201, the entropy decoding unit 200 decodes the coded data to obtain quantized transform coefficients, motion vector information, information on the loop filter, and size information of the partial transform block.

[0106] In step S202, the motion compensation prediction unit 206 predicts the current image block by performing motion compensation prediction using a plurality of reference images based on the motion vector information, and generates a predicted image corresponding to the current image block.

[0107] In step S203, the evaluation unit 208 evaluates the prediction accuracy of the predicted image on a pixel-by-pixel basis by calculating the similarity between multiple reference images, and generates an error map that indicates the low accuracy of the prediction at each pixel position within the target image block.

[0108] In step S204, the identification unit 209 identifies a partial transform block to which orthogonal transformation and quantization have been applied within the target image block based on the error map and size information of the partial transform block, and notifies and sets the identified partial transform block to the synthesis unit 202.

[0109] In step S205, the inverse quantization and inverse transform unit 201 performs inverse quantization and inverse orthogonal transform on the quantized transform coefficients of the partial transform block to restore the prediction residual of the partial transform block, thereby generating a restored prediction residual of the partial transform block.

[0110] In step S206, the synthesis unit 202 reconstructs the target image block by synthesizing the restored prediction residual of the partial transform block with a region of the predicted image corresponding to the partial transform block on a pixel-by-pixel basis, thereby generating a reconstructed image block.

[0111] In step S207, the loop filter 204 performs a filtering process on the reconstructed image block.

[0112] In step S208, the frame memory 205 stores and outputs the reconstructed image blocks after the filtering process in units of frames.

[0113] <8. Summary of embodiments> The image coding device 1 according to the embodiment includes a motion compensation prediction unit 109 that generates a predicted image corresponding to a current image block by making predictions using a plurality of reference images, an evaluation unit 111 that evaluates the similarity between the plurality of reference images on a pixel-by-pixel basis, a subtraction unit 101 that calculates a prediction residual that indicates the difference on a pixel-by-pixel basis between the current image block and the predicted image, a determination unit 112 that determines partial transform blocks of the prediction residual to which orthogonal transform and quantization are to be applied based on the evaluation result by the evaluation unit 111, and a transform / quantization unit 102 that performs orthogonal transform and quantization only on the partial transform blocks of the prediction residual.

[0114] The image decoding device 2 according to the embodiment includes a motion compensation prediction unit 206 that generates a predicted image corresponding to a target image block by making predictions using a plurality of reference images, an evaluation unit 208 that evaluates the similarity between the plurality of reference images on a pixel-by-pixel basis, an entropy decoding unit 200 that decodes encoded data and obtains quantized transform coefficients of a partial transform block to which orthogonal transform and quantization have been applied in the image encoding device 1, and an identification unit 209 that identifies the partial transform block based on the results of the evaluation by the evaluation unit 208.

[0115] According to the embodiment, it is possible to perform orthogonal transform and quantization of the prediction residuals only on the part where the similarity between the reference images used for generating the prediction image is low (i.e., the part where the prediction residuals are estimated to be large), thereby improving the image quality and reducing the amount of code of the encoded data, thereby improving the coding efficiency.

[0116] <9. Change Example 1> In the above embodiment, an example has been described in which one partial transform block is determined for one target image block, but multiple partial transform blocks may be determined for one target image block.

[0117] For example, in the image coding device 1, the determination unit 112 detects the pixel position with the second lowest probability and the pixel position with the third lowest probability in addition to the lowest probability position, and determines partial transform blocks centered on each of the detected pixel positions. In this case, the determination unit 112 outputs number information indicating the number of determined partial transform blocks (for example, 3) to the entropy coding unit 103, and the entropy coding unit 103 encodes the number information.

[0118] Meanwhile, in the image decoding device 2, the entropy decoding unit 200 decodes the number information from the encoded data and outputs the number information to the identification unit 209. The identification unit 209 identifies the number of partial transform blocks based on the number information, and identifies each partial transform block.

[0119] The determination unit 112 evaluates the distance between the detected pixel positions and determines each partial transform block so that the partial transform blocks do not overlap each other. Alternatively, if the distance between the detected pixel positions is less than a certain value, the determination unit 112 may combine multiple partial transform blocks corresponding to these pixel positions. For example, the determination unit 112 may combine the partial transform block centered on the second lowest pixel position with the partial transform block centered on the least likely position. In such a case, the determination unit 112 may further detect the pixel position with the fourth lowest likelihood and add a partial transform block centered on the detected pixel position.

[0120] <10. Change Example 2> The motion compensation prediction unit 109 of the image encoding device 1 and the motion compensation prediction unit 206 of the image decoding device 2 may divide a current image block (CU) into multiple small blocks, apply a different motion vector to each small block, and switch between unidirectional prediction and bidirectional prediction for each small block. In such a case, the evaluation unit 111 of the image encoding device 1 and the evaluation unit 208 of the image decoding device 2 may not calculate an error map for a CU that generates a predicted image using both unidirectional prediction and bidirectional prediction. On the other hand, if predicted images are generated using bidirectional prediction for all small blocks, the evaluation unit 111 of the image encoding device 1 and the evaluation unit 208 of the image decoding device 2 generate an error map.

[0121] Furthermore, the motion compensation prediction unit 109 of the image encoding device 1 and the motion compensation prediction unit 208 of the image decoding device 2 may perform overlapped block motion compensation (OBMC) to reduce discontinuity in predicted images at block boundaries with different motion vectors. The evaluation unit 111 of the image encoding device 1 and the evaluation unit 208 of the image decoding device 2 may take into account correction of reference pixels by OBMC when generating an error map.

[0122] For example, when the prediction mode of the surrounding blocks used for OBMC correction is bi-predictive, the evaluation unit 111 of the image encoding device 1 and the evaluation unit 208 of the image decoding device 2 correct the error map for the region of the predicted image affected by the OBMC correction using the motion vectors of the reference images (L0 and L1) used for generating the bi-predictive predicted image of the surrounding blocks. Specifically, for the block boundary region of map[i,j], when the motion vectors of the adjacent blocks are bi-predictive, a weighted average is taken according to the position with respect to the error map of the adjacent block. When the adjacent blocks are in intra mode or uni-predictive, the error map is not corrected. In the case of FIG. 9, for the upper block boundary, L0 a and L1 aAn error map is generated using the above, and a weighted average is taken with the error map of the CU for the area below it (for the area overlapping with the CU). Because the prediction mode for the CUs below, to the right, and to the left is unidirectional prediction, no correction is made to the error map for the area overlapping with those CUs.

[0123] <11. Change Example 3> The determination unit 112 of the image encoding device 1 may determine the partial transform block in the following manner.

[0124] The determination unit 112 determines a partial transform block when, for example, a predictive image is generated for a current image block (CU: Coding Unit) by bi-prediction and the transform skip mode is not applied to the current CU (i.e., when an orthogonal transform is applied). Note that a flag indicating whether the transform skip mode is applied is included in the encoded data.

[0125] FIG. 10 is a diagram showing a method for determining partial transform blocks in this modified example.

[0126] First, as shown in Fig. 10(A), the determination unit 112 calculates the center of gravity of the error map by the following equation (5). Here, it is assumed that the above-mentioned error map map[i,j] is used as the error map.

[0127]

number

[0128] Here, gx is the horizontal center of gravity position, and gy is the vertical center of gravity position.

[0129] Second, as shown in FIG. 10(B), the determination unit 112 selects an optimal size from candidate sizes (size modes) of the partial transform block. Specifically, the determination unit 112 calculates the RD cost when each size mode is used, and determines the optimal size mode based on the RD cost. The RD cost calculation takes into account the signaling cost of the transform coefficients and the distortion of the entire CU. There are, for example, the following four size modes:

[0130] Mode 1: A mode in which the size of the partial transform block is equal to the CU size (i.e., a mode in which the entire CU is orthogonally transformed) Mode 2: A mode in which the vertical and horizontal sizes of the partial transform block are half the CU size. Mode 3: A mode in which the vertical size of the partial transform block is equal to the CU size, and the horizontal size of the partial transform block is half the CU size. Mode 4: A mode in which the vertical size of the partial transform block is half the CU size and the horizontal size of the partial transform block is equal to the CU size. Here, the explanation will proceed assuming that a mode other than mode 1 is determined and a partial transform block is set. Note that the determination unit 112 sets the partial transform block centered at the center position (gx, gy) so that it does not cross the CU boundary.

[0131] Third, as shown in FIG. 10(C), the transform unit 102a performs a partial orthogonal transform on the prediction residual (residual signal) of the partial transform block based on the determined center position and size.

[0132] Fourth, as shown in FIG. 10(D), the quantization unit 102b quantizes the transform coefficients of the partial transform block, and the entropy coding unit 103 entropy codes the quantized transform coefficients.

[0133] Furthermore, the entropy coding unit 103 includes in the coded data bPartialTransform, which is a flag indicating whether or not to apply partial orthogonal transform, and sizeMode, which is information indicating the size of the partial transform block. bPartialTransform is "1" when partial orthogonal transform is applied, and "0" when partial orthogonal transform is not applied. sizeMode is, for example, "0" for mode 2, "10" for mode 3, and "11" for mode 4. Note that the signaling cost is also taken into consideration when calculating the RD cost.

[0134] Meanwhile, in the image decoding device 2, the entropy decoding unit 200 decodes encoded data and acquires a flag indicating whether or not transform skip mode is applied. When generating a predicted image for a CU to be decoded by bi-prediction and when transform skip mode is not applied to this CU (i.e., when orthogonal transform is applied), the entropy decoding unit 200 acquires bPartialTransform and sizeMode. Then, the identification unit 209 identifies a partial transform block based on the error map generated by the evaluation unit 208 and sizeMode. Subsequent operations are the same as those in the above-described embodiment.

[0135] <12. Other embodiments> In the above-described embodiments, inter prediction has been mainly described as a type of motion compensation prediction. In inter prediction, a reference image in a frame different from the current frame is used to predict a target image block of the current frame. However, a technique called intra block copy can also be applied as motion compensation prediction. In intra block copy, a reference image in the same frame as the current frame is used to predict a target image block of the current frame.

[0136] The above-described specific examples of the present invention may be provided by a program causing a computer to execute each process performed by the image encoding device 1 and a program causing a computer to execute each process performed by the image decoding device 2. The program may also be recorded on a computer-readable medium. The computer-readable medium allows the program to be installed on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM. Furthermore, circuits that execute each process performed by the image encoding device 1 may be integrated, and the image encoding device 1 may be configured as a semiconductor integrated circuit (chip set, SoC). Similarly, circuits that execute each process performed by the image decoding device 2 may be integrated, and the image decoding device 2 may be configured as a semiconductor integrated circuit (chip set, SoC).

[0137] The above describes the embodiments in detail with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes can be made within the scope that does not deviate from the gist of the invention.

[0138] The entire contents of Japanese Patent Application No. 2018-65780 (filed March 29, 2018) are incorporated herein by reference.

Claims

1. An image decoding device that decodes a target image in units of blocks from encoded data, a prediction unit that performs bi-prediction using a plurality of reference images to generate a block of a predicted image corresponding to a block of the target image; an evaluation unit that calculates an evaluation value indicating a similarity between the plurality of reference images; an acquisition unit that decodes the encoded data and acquires a prediction residual; a synthesis unit that synthesizes the prediction residual with a block of the predicted image to reconstruct a block of the target image, the evaluation unit calculates absolute differences between the plurality of reference images in pixel units, and calculates the evaluation value in accordance with the absolute differences in pixel units; An image decoding device in which the evaluation value is used to correct the synthesis target of the synthesis unit in partial block units consisting of multiple pixels, which are units smaller than the block, only when the bi-prediction is performed.

2. An image decoding method for decoding a target image in units of blocks from encoded data, comprising: a prediction step of generating a block of a predicted image corresponding to a block of the target image by performing bi-prediction using a plurality of reference images; an evaluation step of calculating an evaluation value indicating a similarity between the plurality of reference images; an obtaining step of decoding the encoded data to obtain a prediction residual; a combining step of combining the prediction residual with a block of the predicted image to reconstruct a block of the target image, the evaluating step includes a step of calculating an absolute difference between the plurality of reference images in pixel units, and calculating the evaluation value in accordance with the absolute difference in pixel units; An image decoding method in which the evaluation value is used to correct the synthesis target in the synthesis step in partial block units smaller than the block and consisting of multiple pixels only when the bi-prediction is performed.

Images