Image decoding device and image decoding method

Abstract

To improve the coding efficiency.SOLUTION: An image coding device (1) encodes a target image in block units. The image coding device (1) includes: a prediction unit (109) that generates a prediction image corresponding to the target image by performing prediction using a plurality of reference images; an evaluation unit (111) that evaluates the similarity between the plurality of reference images on a pixel-by-pixel basis; a calculation unit (101) that calculates a predicted residual indicating a difference in pixel unit between the target image and the predicted image; a determination unit (112) that determines a part of the predicted residual area to which orthogonal transformation and quantization are applied, on the basis of the evaluation result by the evaluation unit; and a transformation / quantization unit (102) that performs orthogonal transformation and quantization only in the part of the predicted residual region.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present invention relates to an image decoding apparatus and an image decoding method.

Background Art

[0002] Conventionally, in an encoding apparatus that encodes a target image (image block), motion compensation prediction is performed using a plurality of reference images to generate a prediction image corresponding to the target image, and orthogonal transformation and quantization are performed on a prediction residual indicating a pixel-by-pixel difference between the target image and the prediction image.

[0003] Further, the image encoding apparatus encodes the quantization transform coefficients obtained by orthogonal transformation and quantization by entropy encoding, and performs inverse quantization and inverse orthogonal transformation on the quantization transform coefficients to restore the prediction residual. Then, the image encoding apparatus synthesizes the restored prediction residual with the prediction image to reconstruct the target image, and uses the reconstructed image for subsequent prediction.

[0004] On the other hand, the image decoding apparatus decodes the entropy-encoded quantization transform coefficients, performs motion compensation prediction using a plurality of reference images, and generates a prediction image corresponding to the target image. The image decoding apparatus performs inverse quantization and inverse orthogonal transformation on the quantization transform coefficients to restore the prediction residual, and synthesizes the restored prediction residual with the prediction image to reconstruct the target image.

[0005] In such an image encoding apparatus and image decoding apparatus, in motion compensation prediction using a plurality of reference images, for example, a prediction image is generated by averaging a plurality of reference images (see Non-Patent Document 1).

Prior Art Documents

Non-Patent Documents

[0006]

Non-Patent Document 1

[0007] However, if there are areas where the similarity between multiple reference images used to generate a predicted image is significantly low, the accuracy of the predicted image in those areas (i.e., the prediction accuracy) will decrease, and the prediction residual in those areas will be larger than in other areas.

[0008] When prediction residuals containing both large and small prediction residuals are subjected to an orthogonal transformation, power is not concentrated in the low-frequency components, and the power concentration of the transformation coefficients decreases. If the high-frequency components of such transformation coefficients are coarsely quantized, the transformation coefficients in the high-frequency components deteriorate. This deterioration of the transformation coefficients propagates throughout the image (block) via inverse quantization and inverse orthogonal transformation.

[0009] When the degraded reconstructed prediction residuals are combined with the predicted image to reconstruct the target image block, the degradation of image quality propagates even to the parts where high-precision predictions were made. In other words, the degradation of image quality originates in the parts with low prediction accuracy, and the effects of quantization of orthogonal transformation coefficients propagate to the parts with high prediction accuracy, leading to a decrease in coding efficiency.

[0010] Therefore, the present invention aims to provide an image decoding device and an image decoding method that can improve coding efficiency when making predictions using multiple reference images.

[0011] The first feature of the image decoding device is an image decoding device that decodes a target image in block units from encoded data, and comprises: a prediction unit that generates blocks of a predicted image corresponding to blocks of the target image by performing dual prediction using a plurality of reference images; an evaluation unit that calculates an evaluation value indicating the similarity between the plurality of reference images; an acquisition unit that decodes the encoded data and obtains prediction residuals; and a synthesis unit that synthesizes the prediction residuals with the blocks of the predicted image to reconstruct the blocks of the target image, wherein the evaluation value is used to correct the synthesis target of the synthesis unit in units smaller than the blocks, consisting of a plurality of pixels in partial block units.

[0012] Here, the prediction performed by the prediction unit can utilize various prediction methods, regardless of the specific prediction method used, as long as it uses multiple reference images. A typical example of a prediction method is bidirectional motion compensation prediction, but prediction using the IntraBC mode (intrablock copy mode) used in the HEVC method may also be used.

[0013] The image decoding device according to the second feature is an image decoding device that decodes a block-unit target image from an encoded bitstream, and comprises: a prediction unit that generates blocks of a predicted image corresponding to blocks of the target image by performing interpretation; an entropy decoding unit that decodes the encoded bitstream and obtains quantization conversion coefficients obtained by the image encoding device performing transformation and quantization limited to a portion of the blocks of the predicted residual; and a specification unit that identifies the portion of the region, wherein the entropy decoding unit obtains information indicating a mode selected from a plurality of modes relating to the portion of the region, and the plurality of modes include a mode in which the vertical size of the portion of the region is equal to the size of the block while the horizontal size of the portion of the region is smaller than the size of the block, and a mode in which the vertical size of the portion of the region is smaller than the size of the block while the horizontal size of the portion of the region is equal to the size of the block.

[0014] The program according to the third feature is a program for causing a computer to function as the image encoding device according to the first feature.

[0015] The program according to the fourth feature is a program for causing a computer to function as the image decoding device according to the second feature.

[0016] According to the present invention, it is possible to provide an image decoding device and an image decoding method capable of improving encoding efficiency when performing prediction using a plurality of reference images.

Brief Description of the Drawings

[0017] [Figure 1] It is a diagram showing the configuration of the image encoding device according to the embodiment. [Figure 2] It is a diagram showing the configuration of the image decoding device according to the embodiment. [Figure 3] It is a diagram showing an example of motion compensation prediction. [Figure 4] It is a diagram showing an example of a predicted image generated by motion compensation prediction. [Figure 5] It is a diagram showing an example of the configuration of the evaluation unit. [Figure 6] It is a diagram showing an example of the operation of the determination unit according to the embodiment. [Figure 7] It is a diagram showing an example of an operation flow in the image encoding device according to the embodiment. [Figure 8] It is a diagram showing an example of an operation flow in the image decoding device according to the embodiment. [Figure 9] It is a diagram showing an operation example in Modification Example 2 of the embodiment. [Figure 10] It is a diagram showing an operation example in Modification Example 3 of the embodiment.

Embodiments of the Invention

[0018] Referring to the drawings, an image encoding apparatus and an image decoding apparatus according to an embodiment will be described. The image encoding apparatus and the image decoding apparatus according to the embodiment perform encoding and decoding of moving images represented by MPEG. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

[0019] <1. Configuration of Image Encoding Apparatus> FIG. 1 is a diagram showing the configuration of an image encoding apparatus 1 according to an embodiment. As shown in FIG. 1, the image encoding apparatus 1 includes a block division unit 100, a subtraction unit (calculation unit) 101, a conversion unit 102a, a quantization unit 102b, an entropy encoding unit 103, an inverse quantization unit 104a, an inverse conversion unit 104b, a synthesis unit 105, an intra prediction unit 106, a loop filter 107, a frame memory 108, a motion compensation prediction unit 109, a switching unit 110, an evaluation unit 111, and a determination unit 112.

[0020] The block division unit 100 divides an input image in units of frames (or pictures) into small block-shaped regions, and outputs the image blocks to the subtraction unit 101 (and the motion compensation prediction unit 109). The size of the image blocks is, for example, 32×32 pixels, 16×16 pixels, 8×8 pixels, or 4×4 pixels, etc. However, the shape of the image blocks is not limited to a square, and may be a rectangular shape. The image blocks are units for the image encoding apparatus 1 to perform encoding and units for the image decoding apparatus 2 to perform decoding, and such image blocks are referred to as target image blocks. Such image blocks may be referred to as coding units (CUs) or coding blocks (CBs).

[0021] The subtraction unit 101 calculates the predicted residual, which is the pixel-level difference between the target image block input from the block division unit 100 and the predicted image (predicted image block) corresponding to the target image block. Specifically, the subtraction unit 101 calculates the predicted residual by subtracting each pixel value of the predicted image from each pixel value of the target image block, and outputs the calculated predicted residual to the conversion unit 102a. The predicted image is input to the subtraction unit 101 via the switching unit 110 from the intra prediction unit 106 or the motion compensation prediction unit 109, which will be described later.

[0022] The transformation unit 102a and the quantization unit 102b constitute a transformation / quantization unit 102 that performs orthogonal transformation and quantization processing on the predicted residuals. The transformation / quantization unit 102 does not perform orthogonal transformation and quantization on the predicted residuals of all pixel positions in the target image block, but rather performs orthogonal transformation and quantization on the predicted residuals of a portion of the target image block. This portion of the block is a portion of the block where the predicted residuals are expected to be large, and is determined by the determination unit 112. Hereinafter, this portion of the block will be referred to as a "partial transformation block". A partial transformation block is a part of the target image block.

[0023] The transformation unit 102a identifies the region corresponding to the partial transformation block from the predicted residuals of the target image block input from the subtraction unit 101, based on the information of the partial transformation block input from the determination unit 112. The transformation unit 102a extracts the predicted residuals of the partial transformation block from the predicted residuals input from the subtraction unit 101, performs an orthogonal transformation on the extracted predicted residuals, calculates transformation coefficients that indicate the intensity (power) of the spatial frequency components, and outputs the calculated transformation coefficients to the quantization unit 102b. Orthogonal transformations include, for example, the discrete cosine transform (DCT), discrete sine transform (DST), and Karhunen-Loeve transform (KLT).

[0024] The quantization unit 102b quantizes the transformation coefficients of the partial transformation block input from the transformation unit 102a using the quantization parameter (Qp) and the quantization matrix to generate quantized transformation coefficients. The quantization parameter (Qp) is a parameter that is applied in common to each transformation coefficient within the block and determines the coarseness of the quantization. The quantization matrix is ​​a matrix whose elements are the quantized values ​​obtained when quantizing each transformation coefficient. The quantization unit 102b outputs quantization control information, generated quantized transformation coefficient information, etc., to the entropy coding unit 103 and the inverse quantization unit 104a.

[0025] The entropy coding unit 103 performs entropy coding on the partial transformation block quantization conversion coefficients input from the quantization unit 102b, compresses the data to generate encoded data (bitstream), and outputs the encoded data to the outside of the image coding device 1. For entropy coding, Huffman coding or CABAC (Context-based Adaptive Binary Arithmetic Coding) can be used. The entropy coding unit 103 receives prediction information from the intra prediction unit 106 and the motion compensation prediction unit 109, information on filtering from the loop filter 107, and information on partial transformation blocks from the determination unit 112. The entropy coding unit 103 also performs entropy coding on this information.

[0026] The inverse quantization unit 104a and the inverse transformation unit 104b constitute the inverse quantization / inverse transformation unit 104. The inverse quantization / inverse transformation unit 104 recovers the partial prediction residuals corresponding to the partial transformation block by performing inverse quantization and inverse orthogonal transformation on the quantization transformation coefficients of the partial transformation block input from the quantization unit 102b.

[0027] The inverse quantization unit 104a performs inverse quantization processing corresponding to the quantization processing performed by the quantization unit 102b. Specifically, the inverse quantization unit 104a recovers the quantization coefficients of the partial transformation block input from the quantization unit 102b by inverse quantization using the quantization parameter (Qp) and the quantization matrix, and outputs the recovered transformation coefficients to the inverse transformation unit 104b.

[0028] The inverse transform unit 104b performs an inverse orthogonal transform process corresponding to the orthogonal transform process performed by the transform unit 102a. For example, if the transform unit 102a performs a discrete cosine transform, the inverse transform unit 104b performs an inverse discrete cosine transform. The inverse transform unit 104b performs an inverse orthogonal transform on the transformation coefficients of the partial transformation block input from the inverse quantization unit 104a to reconstruct the partial predicted residuals, and outputs the reconstructed partial predicted residuals to the synthesis unit 105.

[0029] The synthesis unit 105 identifies the region of the predicted image input from the switching unit 110 that corresponds to the partial transformation block, based on the information of the partial transformation block input from the determination unit 112. Then, the synthesis unit 105 reconstructs the target image block by combining the restored predicted residual (restored partial predicted residual) input from the inverse transformation unit 104b with the region of the predicted image that corresponds to the partial transformation block. Specifically, the synthesis unit 105 reconstructs the target image block by adding each pixel value of the restored partial predicted residual to each pixel value of the region of the predicted image that corresponds to the partial transformation block. The synthesis unit 105 outputs the reconstructed image, which is the reconstructed target image block, to the intra-prediction unit 106 and the loop filter 107.

[0030] The intra-prediction unit 106 performs intra-prediction using the reconstructed image input from the synthesis unit 105 to generate an intra-predicted image, and outputs the intra-predicted image to the switching unit 110. The intra-prediction unit 106 also outputs information such as the selected intra-prediction mode to the entropy coding unit 103.

[0031] The loop filter 107 performs a post-processing filter on the reconstructed image input from the synthesis unit 105, and outputs the reconstructed image after filtering to the frame memory 108. The loop filter 107 also outputs information related to the filtering process to the entropy coding unit 103.

[0032] The frame memory 108 stores the reconstructed images input from the loop filter 107 on a frame-by-frame basis.

[0033] The motion compensation prediction unit 109 performs interpretation using one or more reconstructed images stored in the frame memory 108 as reference images. Specifically, the motion compensation prediction unit 109 calculates motion vectors using methods such as block matching, generates motion compensation prediction images based on the motion vectors, and outputs the motion compensation prediction images to the switching unit 110. The motion compensation prediction unit 109 also outputs information regarding the motion vectors to the entropy coding unit 103.

[0034] The switching unit 110 switches between the intra-prediction image input from the intra-prediction unit 106 and the motion-compensated prediction image input from the motion-compensated prediction unit 109, and outputs the prediction image (intra-prediction image or motion-compensated prediction image) to the subtraction unit 101 and the synthesis unit 105.

[0035] The evaluation unit 111 evaluates the similarity between multiple reference images on a pixel-by-pixel basis when the motion compensation prediction unit 109 performs motion compensation prediction using multiple reference images, and outputs the evaluation result information to the determination unit 112. The evaluation unit 111 may also evaluate the similarity between multiple reference images when using intra prediction using multiple reference images (for example, intra block copy mode) and when the intra prediction unit 106 performs prediction using multiple reference images.

[0036] The determination unit 112 determines the partial transformation blocks to which orthogonal transformation and quantization will be applied, based on the evaluation results by the evaluation unit 111. Specifically, the determination unit 112 determines the center position of the partial transformation block and the size of the partial transformation block relative to this center position, based on the evaluation results by the evaluation unit 111. The determination unit 112 outputs information indicating the determined partial transformation block to the transformation unit 102a and the synthesis unit 105, and outputs information indicating the size of the determined partial transformation block (size information) to the entropy coding unit 103. This size information is encoded in the entropy coding unit 103 and output as part of the control information included in the encoded data.

[0037] <2. Configuration of the image decoding device> Figure 2 shows the configuration of the image decoding device 2 according to the embodiment. As shown in Figure 2, the image decoding device 2 includes an entropy decoding unit 200, an inverse quantization unit 201a, an inverse transformation unit 201b, a synthesis unit 202, an intra prediction unit 203, a loop filter 204, a frame memory 205, a motion compensation prediction unit 206, a switching unit 207, an evaluation unit 208, and a identification unit 209.

[0038] The entropy decoding unit 200 decodes the encoded data generated by the encoding device 1 and outputs the quantization transformation coefficients corresponding to the partial transformation blocks to which orthogonal transformation and quantization have been applied in the image encoding device 1 to the inverse quantization unit 201a. The entropy decoding unit 200 also decodes the encoded data and obtains information regarding prediction (intra prediction and motion compensation prediction) and information regarding filtering. The entropy decoding unit 200 outputs the prediction information to the intra prediction unit 203 and the motion compensation prediction unit 206, and outputs the filtering information to the loop filter 204. The entropy decoding unit 200 outputs the size information of the partial transformation blocks to the identification unit 209 and the inverse transformation unit 201b.

[0039] The inverse quantization unit 201a and the inverse transformation unit 201b constitute the inverse quantization / inverse transformation unit 201. The inverse quantization / inverse transformation unit 201 recovers the partial prediction residuals corresponding to the partial transformation block by performing inverse quantization and inverse orthogonal transformation on the quantization transformation coefficients.

[0040] The inverse quantization unit 201a performs inverse quantization processing corresponding to the quantization processing performed by the quantization unit 102b of the image coding device 1. The inverse quantization unit 201a recovers the quantization coefficients by inverse quantization of the quantization conversion coefficients of the partial conversion block input from the entropy decoding unit 200 using the quantization parameter (Qp) and the quantization matrix, and outputs the recovered conversion coefficients to the inverse conversion unit 201b.

[0041] The inverse transform unit 201b performs an inverse orthogonal transform process corresponding to the orthogonal transform process performed by the transform unit 102a of the image coding device 1. Based on the size information of the partial transform block input from the entropy decoding unit 200, the inverse transform unit 201b performs an inverse orthogonal transform on the transform coefficients input from the inverse quantization unit 201a to restore the predicted residuals, and outputs the restored predicted residuals (restored predicted residuals) to the synthesis unit 202.

[0042] The synthesis unit 202 identifies the region of the predicted image input from the switching unit 207 that corresponds to the partial transformation block, based on the information of the partial transformation block input from the identification unit 209. The synthesis unit 202 reconstructs the original target image block by synthesizing the partial prediction residual restored by the inverse transformation unit 201b with the region of the predicted image that corresponds to the partial transformation block, pixel by pixel, and outputs the reconstructed image block to the intra-prediction unit 203 and the loop filter 204.

[0043] The intra-prediction unit 203 refers to the reconstructed image block input from the synthesis unit 202 and generates an intra-predicted image by performing intra-prediction according to the intra-prediction information input from the entropy decoding unit 200, and outputs the intra-predicted image to the switching unit 207.

[0044] The loop filter 204 performs a filtering process on the reconstructed image input from the synthesis unit 202 based on the filtering information input from the entropy decoding unit 200, similar to the filtering process performed by the loop filter 107 of the image encoding device 1, and outputs the reconstructed image after filtering to the frame memory 205.

[0045] The frame memory 205 stores the reconstructed images input from the loop filter 204 on a frame-by-frame basis. The frame memory 205 outputs the stored reconstructed images to the outside of the image decoding device 2 in the order they are displayed.

[0046] The motion compensation prediction unit 206 uses one or more reconstructed images stored in the frame memory 205 as reference images and performs motion compensation prediction (interpretation) according to the motion vector information input from the entropy decoding unit 200 to generate a motion compensation prediction image, which is then output to the switching unit 207.

[0047] The switching unit 207 switches between the intra-prediction image input from the intra-prediction unit 203 and the motion-compensated prediction image input from the motion-compensated prediction unit 206, and outputs the prediction image (intra-prediction image or motion-compensated prediction image) to the synthesis unit 202.

[0048] The evaluation unit 208 operates similarly to the evaluation unit 111 of the image coding device 1. Specifically, when the motion compensation prediction unit 206 performs motion compensation prediction using multiple reference images, the evaluation unit 208 evaluates the similarity between the multiple reference images on a pixel-by-pixel basis and outputs the evaluation result information to the identification unit 209. The evaluation unit 208 may also evaluate the similarity between the multiple reference images when using intra-prediction using multiple reference images (for example, intra-block copy mode) and when the intra-prediction unit 203 performs prediction using multiple reference images.

[0049] The identification unit 209 identifies the partial transformation block to which orthogonal transformation and quantization have been applied in the image coding device 1, based on the evaluation results by the evaluation unit 208. Specifically, the identification unit 209 identifies the center position of the partial transformation block based on the evaluation results by the evaluation unit 208. Then, the identification unit 209 identifies the partial transformation block based on the identified center position and the size information input from the entropy decoding unit 200. The identification unit 209 outputs information indicating the identified partial transformation block to the synthesis unit 202.

[0050] <3. Motion Compensation Prediction> Figure 3 shows an example of motion-compensated prediction. Figure 4 shows an example of a predicted image generated by motion-compensated prediction. As a simple example of motion-compensated prediction, we will explain the case using bidirectional prediction, particularly forward and backward prediction (bidirectional prediction), which is used in HEVC.

[0051] As shown in Figure 3, motion compensation prediction refers to frames that are temporally preceding and succeeding the target frame (current frame). In the example in Figure 3, motion compensation prediction for a block in the image of frame t is performed by referring to frames t-1 and t+1. Motion compensation detects areas (blocks) similar to the target image block from within the reference frames of frames t-1 and t+1, within a search range set by the system.

[0052] The detected area is the reference image. The arrows shown in the figure indicate the relative position of the reference image to the target image block, and are called motion vectors. The motion vector information is encoded by entropy coding together with the frame information of the reference image in the image encoding device 1. Meanwhile, the image decoding device 2 detects the reference image based on the motion vector information generated by the image encoding device 1.

[0053] As shown in Figures 3 and 4, the reference images 1 and 2 detected by motion compensation prediction are similar partial images aligned within the reference frame to the target image block, and therefore become images similar to the target image block (the image to be encoded). In the example in Figure 4, the target image block contains a star pattern and a partial circle pattern. Reference image 1 contains a star pattern and a full circle pattern. Reference image 2 contains a star pattern but does not contain a circle pattern.

[0054] A prediction image is generated from the reference images 1 and 2. Since the prediction process is computationally intensive, it is common to generate the prediction image by averaging reference images 1 and 2. However, more advanced processing, such as signal enhancement processing using low-pass or high-pass filters, may also be used to generate the prediction image. Here, reference image 1 contains a circle pattern, and reference image 2 does not contain a circle pattern. Therefore, when a prediction image is generated by averaging reference images 1 and 2, the signal of the circle pattern in the prediction image is halved compared to reference image 1.

[0055] The difference between the predicted image obtained from reference images 1 and 2 and the target image block (image to be encoded) is the prediction residual. In the prediction residual shown in Figure 4, large differences occur only in the shifted edges of the star pattern and the shifted parts of the circle pattern (shaded areas), but in other areas, the prediction is made with good accuracy and the difference is small (no difference occurs in the example in Figure 4).

[0056] The areas where no difference is observed (the non-edge parts of the star pattern and the background) are areas where the similarity between reference image 1 and reference image 2 is high, and where accurate predictions were made. On the other hand, the areas where large differences are observed are areas specific to each reference image, i.e., areas where the similarity between reference image 1 and reference image 2 is significantly low. Therefore, it can be seen that areas where the similarity between reference image 1 and reference image 2 is significantly low result in low prediction accuracy and produce large differences (residuals).

[0057] When a prediction residual containing both large and small differences is subjected to an orthogonal transformation, power does not concentrate in the low-frequency components, and the power concentration of the transformation coefficients decreases. As a result, the degradation of the transformation coefficients due to quantization becomes greater, and this degradation of the transformation coefficients propagates throughout the image (target image block) via inverse quantization and inverse orthogonal transformation.

[0058] Then, when the predicted residuals (restored predicted residuals) recovered by inverse quantization and inverse orthogonal transformation are combined with the predicted image to reconstruct the target image block, the degradation of image quality propagates even to areas where high-precision predictions were made, such as the non-edge and background parts of the star pattern shown in Figure 4.

[0059] In this embodiment, by performing orthogonal transformation and quantization of the predicted residual only on a partial transformation block that is a portion of the image block where the similarity between reference image 1 and reference image 2 is low (i.e., a portion where the predicted residual is estimated to be large), it is possible to suppress the propagation of the degradation of the transformation coefficient caused by quantization to regions within the target image block where the predicted residual is small, thereby improving image quality.

[0060] <4. Evaluation Unit in Image Encoding Device and Image Decoding Device> In the image encoding device 1, the evaluation unit 111 evaluates the prediction accuracy of the predicted image on a pixel-by-pixel basis by calculating the similarity between multiple reference images on a pixel-by-pixel basis. Similarly, in the image decoding device 2, the evaluation unit 208 evaluates the prediction accuracy of the predicted image on a pixel-by-pixel basis by calculating the similarity between multiple reference images on a pixel-by-pixel basis.

[0061] Figure 5 shows an example of the configuration of the evaluation unit 111 in the image encoding device 1. As shown in Figure 5, the evaluation unit 111 comprises a difference calculation unit (subtraction unit) 111a, a normalization unit 111b, and an adjustment unit 111c.

[0062] The difference calculation unit 111a calculates the difference value between reference image 1 and reference image 2 on a pixel-by-pixel basis and outputs the calculated difference value to the normalization unit 111b. The difference value is an example of a value that indicates similarity. A smaller difference value indicates higher similarity, and a larger difference value indicates lower similarity. The difference calculation unit 111a may also calculate the difference value after performing a filter process on each reference image. The difference calculation unit 111a may also calculate a statistical quantity such as the squared error and use such a statistical quantity as the similarity. An example of using the difference value as the similarity will be described below.

[0063] The normalization unit 111b normalizes the difference values ​​input from the similarity calculation unit 111a by the maximum difference value within the block (i.e., the maximum value of the difference values ​​within the block) and outputs it. The normalized difference values ​​are used as importance for determining the partial transformation block. The smaller the difference value, the higher the similarity and the higher the prediction accuracy, and therefore the lower the importance in encoding. On the other hand, the larger the difference value, the lower the similarity and the lower the prediction accuracy, and therefore the higher the importance in encoding.

[0064] The normalization unit 111b normalizes the difference value of each pixel input from the difference calculation unit 111a by the difference value of the pixel with the largest difference value within the block (i.e., the maximum difference value within the block), and outputs the normalized difference value (importance).

[0065] The adjustment unit 111c adjusts the normalized difference values ​​input from the normalization unit 111b based on the quantization parameter (Qp) that determines the coarseness of quantization, and outputs the adjusted normalized difference values. Since the degree of degradation of the reconstructed predicted residual is higher the coarser the quantization, the adjustment unit 111c adjusts the normalized difference values ​​(weights) based on the quantization parameter (Qp).

[0066] The importance Rij of each pixel position (ij) output by the evaluation unit 111 can be expressed, for example, as shown in equation (1) below.

[0067] Rij = (abs(Xij-Yij) / maxD ×Scale(Qp)) ···(1)

[0068] In equation (1), Xij is the pixel value of pixel ij in reference image 1, Yij is the pixel value of pixel ij in reference image 2, and abs is a function that obtains the absolute value. The difference calculation unit 111a shown in Figure 5 outputs abs(Xij-Yij).

[0069] Furthermore, in equation (1), maxD is the maximum value of the difference value abs(Xij-Yij) within the block. To find maxD, it is necessary to find the difference value for all pixels in the block, but to omit this process, the maximum value of an adjacent block that has already been encoded may be used as a substitute. Alternatively, maxD may be found from the quantization parameter (Qp) using a table that defines the correspondence between the quantization parameter (Qp) and maxD. Alternatively, a fixed value specified in the specifications beforehand may be used as maxD. The normalization unit 111b outputs abs(Xij-Yij) / maxD.

[0070] Furthermore, in equation (1), Scale(Qp) is a coefficient multiplied according to the quantization parameter (Qp). Scale(Qp) is designed to approach 1.0 when Qp is large and approach 0 when Qp is small, and the degree of this adjustment is to be adjusted by the system. Alternatively, a fixed value specified in the specifications may be used as Scale(Qp). In addition, to simplify the process, Scale(Qp) may be set to a fixed value such as 1.0, which is designed according to the system.

[0071] The adjustment unit 111c outputs abs(Xij-Yij) / maxD×Scale(Qp) as the importance Rij. Alternatively, this Rij may output a weight adjusted by a sensitivity function designed according to the system. For example, abs(Xij-Yij) / maxD×Scale(Qp)=Rij and Rij=Clip(Rij,1.0,0.0), or Rij=Clip(Rij+offset,1.0,0.0) to adjust the sensitivity with an offset. Note that Clip(x,max,min) indicates the process of clipping at max if x exceeds max, and at min if x falls below min.

[0072] The importance score Rij calculated in this way will be a value within the range of 0 to 1.0. Basically, the importance score Rij approaches 1.0 when the difference in pixel position ij between reference images is large (i.e., prediction accuracy is low), and approaches 0 when the difference in pixel position ij between reference images is small (i.e., prediction accuracy is high). The evaluation unit 111 outputs a two-dimensional map information consisting of the importance score Rij of each pixel position ij within a block to the determination unit 112 on a block-by-block basis.

[0073] Alternatively, the evaluation unit 111 may calculate Rij as shown in equation (2) below.

[0074] Rij = 1.0 - (abs(Xij-Yij) / maxD × Scale(Qp)) ···(2)

[0075] When using equation (2), Rij approaches 0 when the difference value of the pixel position ij is large (i.e., the prediction accuracy is low), and approaches 1 when it is small (i.e., the prediction accuracy is high). Therefore, Rij can be considered a value that represents the prediction accuracy (accuracy) of the pixel position ij. Below, we will explain an example in which the evaluation unit 111 outputs two-dimensional map information consisting of the accuracy Rij of each pixel position ij in a block using equation (2), and such map information will be appropriately referred to as an "accuracy map".

[0076] Furthermore, the evaluation unit 208 in the image decoding device 2 is configured similarly to the evaluation unit 111 in the image coding device 1. Specifically, the evaluation unit 208 in the image decoding device 2 comprises a similarity calculation unit 208a, a normalization unit 208b, and an adjustment unit 208c. The evaluation unit 208 in the image decoding device 2 outputs an accuracy map consisting of the accuracy Rij of each pixel position ij within a block to the identification unit 209.

[0077] Furthermore, the evaluation unit 111 and evaluation unit 208 perform evaluation (calculation of Rij) only when applying prediction using multiple reference images, and do not need to perform evaluation in other modes, such as unidirectional prediction or intra-prediction processing that does not use multiple reference images.

[0078] The evaluation unit 111 may calculate the error map using the method shown below and output it to the identification unit 209.

[0079] If the luminance signals of the two reference images (reference blocks) used to generate the predicted image in dual prediction mode are L0[i,j] and L1[i,j] (where [i,j] are coordinates within the target image block), then the error map map[i,j] and its maximum value max_map are calculated using the following equation (3).

[0080] map [i,j] = abs (L0 [i,j] - L1 [i,j]) max#map = max (map [i,j]) ···(3)

[0081] If the max#map in equation (3) exceeds 6 bits of precision (exceeds 64), the error map and maximum value are updated by a shift set to keep max#map within 6 bits of precision, according to equation (4) below.

[0082] max#map = max#map >> shift map [i,j] = map [i,j] >> shift ···(4)

[0083] Unlike the accuracy map, the error map calculated by equation (4) shows that the values ​​in areas where the predicted residual is estimated to be large are larger, and the values ​​in areas where the predicted residual is estimated to be small are smaller. In other words, the normalized accuracy map mentioned above is Accuracy map[i,j] = 1 - (map[i,j] / max#map) This can be calculated by the following. The embodiments of the present invention will be described below using an error map, but the system may also be configured to estimate prediction accuracy using the accuracy map described above.

[0084] <5. Determination unit in image encoding device and identification unit in image decoding device> In the image encoding device 1, the determination unit 112 determines the center position of the partial transformation block and the size of the partial transformation block relative to this center position, based on the evaluation results from the evaluation unit 111. Figure 6 shows an example of the operation of the determination unit 112.

[0085] Firstly, as shown in Figure 6(A), the determination unit 112 detects the lowest accuracy position, which is the pixel position where map[i, j] is maximized in the error map, based on the error map input from the evaluation unit 111, and determines the detected lowest accuracy position as the center position. Alternatively, instead of detecting the lowest accuracy position using the error map as described above, the determination unit 112 may be configured to use the pixel position where rij is minimized in the accuracy map as the lowest accuracy position. Or, instead of using the lowest accuracy position as the center position, the determination unit 112 may calculate the centroid of the error map and determine the calculated centroid as the center position.

[0086] Secondly, as shown in Figure 6(B), the determination unit 112 averages the error map horizontally and vertically to generate a horizontal error map and a vertical error map. Then, in the horizontal and vertical error maps, the determination unit 112 expands the range by one pixel at a time, starting from the lowest accuracy position, and determines the horizontal and vertical lengths that cover X% (e.g., 25% or 50%) of the low-accuracy pixel positions. The determination unit 112 may determine the value of X based on the remaining code amount of the encoded data encoded before the target image block relative to the target code amount, or it may use a system-fixed value as X. Furthermore, for the purpose of speeding up processing, the horizontal and vertical lengths that satisfy X% may be determined in lengths that are multiples of 2 (2, 4, 8, 16, etc.). By determining the horizontal and vertical lengths respectively, a rectangular block shape, not limited to a square block shape, can be used as a partial transformation block.

[0087] The determination unit 112 outputs the center position information and size information (horizontal and vertical lengths) of the determined partial transformation block to the transformation unit 102a and the synthesis unit 105, and outputs the size information to the entropy coding unit 103. The size information is encoded in the entropy coding unit 103 and output as part of the control information included in the encoded data. Note that the center position of the partial transformation block can be determined by the identification unit 209 in the image decoding device 2 based on the error map, and there is no need to encode the center position information, thus reducing the amount of code.

[0088] Meanwhile, in the image encoding device 1, the identification unit 209 identifies the center position of a partial transformation block based on the evaluation results by the evaluation unit 208, and identifies the partial transformation block based on the identified center position and the decoded size information.

[0089] Firstly, the identification unit 209 identifies the center position based on the error map in the same way as the determination unit 112 in the image encoding device 1.

[0090] Secondly, the identification unit 209 determines the size of the target image block based on the size information (horizontal and vertical lengths) input from the entropy decoding unit 200. In this way, the identification unit 209 can determine the center position and size of the partial transformation block.

[0091] The identification unit 209 outputs the center position information and size information (horizontal and vertical lengths) of the identified partial transformation block to the synthesis unit 202.

[0092] In addition, the system may pre-define a list of candidate center positions. In such cases, the determination unit 112 and the identification unit 209 may determine and identify the candidate position closest to the detected lowest accuracy position or the center of gravity as the center position.

[0093] <6. Image encoding operation> Figure 7 shows an example of the operation flow in the image encoding device 1 according to the embodiment.

[0094] As shown in Figure 7, in step S101, the motion compensation prediction unit 109 predicts the target image block by performing motion compensation prediction using multiple reference images and generates a predicted image corresponding to the target image block. The motion compensation prediction information is generated as part of the encoded data by the entropy encoding unit 103, and the entropy encoding unit 103 outputs encoded data including the motion compensation prediction information.

[0095] In step S102, the evaluation unit 111 calculates the similarity between multiple reference images on a pixel-by-pixel basis to evaluate the prediction accuracy of the predicted image on a pixel-by-pixel basis and generates an error map that represents the low prediction accuracy of each pixel position within the target image block.

[0096] In step S103, the determination unit 112 determines a partial transformation block, which is a portion of the target image block to which orthogonal transformation and quantization are applied, based on the error map, and notifies and sets the determined partial transformation block to the transformation / quantization unit 102 (specifically, the transformation unit 102a) and the synthesis unit 105. In addition, the size information of the partial transformation block is encoded as part of the encoded data by the entropy encoding unit 103.

[0097] In step S104, the subtraction unit 101 calculates the predicted residual, which represents the difference in pixels between the target image block and the predicted image.

[0098] In step S105, the transformation / quantization unit 102 generates quantization transformation coefficients by performing orthogonal transformation and quantization on the predicted residuals of the partial transformation block among the predicted residuals calculated by the subtraction unit 101.

[0099] In step S106, the entropy coding unit 103 entropy codes the quantization conversion coefficients of the partial conversion block and outputs the coded data.

[0100] In step S107, the inverse quantization / inverse transformation unit 104 performs inverse quantization and inverse orthogonal transformation on the quantization transformation coefficients of the partial transformation block to restore the predicted residual of the partial transformation block and generate the restored predicted residual of the partial transformation block.

[0101] In step S108, the synthesis unit 105 reconstructs the target image block by synthesizing the restored predicted residual of the partial transformation block with the region of the predicted image corresponding to the partial transformation block on a pixel-by-pixel basis, thereby generating the reconstructed image block.

[0102] In step S109, the loop filter 107 performs filtering on the reconstructed image block. Information about the loop filter (such as the offset and the category information to which the offset is applied) is encoded as part of the encoded data by the entropy encoding unit 103.

[0103] In step S110, the frame memory 108 stores the reconstructed image blocks after filtering on a frame-by-frame basis.

[0104] <7. Image Decoding Operation> Figure 8 shows an example of the operation flow in the image decoding device 2 according to the embodiment.

[0105] As shown in Figure 8, in step S201, the entropy decoding unit 200 decodes the encoded data to obtain quantization transformation coefficients, motion vector information, loop filter information, and partial transformation block size information.

[0106] In step S202, the motion compensation prediction unit 206 predicts the target image block by performing motion compensation prediction using multiple reference images based on motion vector information, and generates a predicted image corresponding to the target image block.

[0107] In step S203, the evaluation unit 208 calculates the similarity between multiple reference images on a pixel-by-pixel basis to evaluate the prediction accuracy of the predicted image on a pixel-by-pixel basis and generates an error map that represents the low accuracy of the prediction for each pixel position within the target image block.

[0108] In step S204, the identification unit 209 identifies the partial transformation blocks within the target image block to which orthogonal transformation and quantization have been applied, based on the error map and the size information of the partial transformation blocks, and notifies and sets the identified partial transformation blocks to the synthesis unit 202.

[0109] In step S205, the inverse quantization / inverse transformation unit 201 performs inverse quantization and inverse orthogonal transformation on the quantization transformation coefficients of the partial transformation block to restore the predicted residual of the partial transformation block and generate the restored predicted residual of the partial transformation block.

[0110] In step S206, the synthesis unit 202 reconstructs the target image block by synthesizing the restored predicted residual of the partial transformation block with the region of the predicted image corresponding to the partial transformation block on a pixel-by-pixel basis, thereby generating the reconstructed image block.

[0111] In step S207, the loop filter 204 performs filtering on the reconstructed image block.

[0112] In step S208, the frame memory 205 stores and outputs the reconstructed image blocks after filtering on a frame-by-frame basis.

[0113] <8. Summary of Embodiments> The image coding apparatus 1 according to this embodiment includes: a motion compensation prediction unit 109 that generates a predicted image corresponding to a target image block by performing prediction using a plurality of reference images; an evaluation unit 111 that evaluates the similarity between these plurality of reference images on a pixel-by-pixel basis; a subtraction unit 101 that calculates a predicted residual that shows the difference between the target image block and the predicted image on a pixel-by-pixel basis; a determination unit 112 that determines the partial transformation blocks of the predicted residual to which orthogonal transformation and quantization are applied based on the evaluation results by the evaluation unit 111; and a transformation / quantization unit 102 that performs orthogonal transformation and quantization only on the partial transformation blocks in the predicted residual.

[0114] The image decoding device 2 according to this embodiment includes a motion compensation prediction unit 206 that generates a predicted image corresponding to a target image block by performing prediction using a plurality of reference images, an evaluation unit 208 that evaluates the similarity between these plurality of reference images on a pixel-by-pixel basis, an entropy decoding unit 200 that decodes the encoded data and obtains the quantization transformation coefficients of the partial transformation block to which orthogonal transformation and quantization have been applied in the image encoding device 1, and an identification unit 209 that identifies the partial transformation block based on the evaluation results by the evaluation unit 208.

[0115] According to this embodiment, it becomes possible to perform orthogonal transformation and quantization of prediction residuals only in the parts where the similarity between the reference images used to generate the prediction image is low (i.e., the parts where the prediction residual is estimated to be large), thereby improving image quality and reducing the amount of code in the encoded data. Therefore, encoding efficiency can be improved.

[0116] <9. Example of change 1> In the embodiment described above, an example was shown in which one partial transformation block is determined for one target image block. However, multiple partial transformation blocks may be determined for one target image block.

[0117] For example, in the image coding device 1, the determination unit 112 detects the pixel position with the second lowest accuracy and the pixel position with the third lowest accuracy, in addition to the lowest accuracy position, and determines partial transformation blocks centered on each detected pixel position. In this case, the determination unit 112 outputs count information (for example, 3) indicating the number of determined partial transformation blocks to the entropy coding unit 103, and the entropy coding unit 103 encodes the count information.

[0118] Meanwhile, in the image decoding device 2, the entropy decoding unit 200 decodes the number information from the encoded data and outputs the number information to the identification unit 209. The identification unit 209 identifies the number of partial transformation blocks based on the number information and identifies each partial transformation block.

[0119] The determination unit 112 evaluates the distance between the detected pixel positions and determines each partial transformation block so that the partial transformation blocks do not overlap with each other. Alternatively, if the distance between the detected pixel positions is less than a certain value, the determination unit 112 may merge multiple partial transformation blocks corresponding to these pixel positions. For example, a partial transformation block centered on the second lowest pixel position may be combined with a partial transformation block centered on the lowest accuracy position. In such a case, the fourth lowest accuracy pixel position may be detected, and a partial transformation block centered on the detected pixel position may be added.

[0120] <10. Example of change 2> The motion compensation prediction unit 109 of the image coding device 1 and the motion compensation prediction unit 206 of the image decoding device 2 may divide the target image block (CU) into multiple subblocks, apply a different motion vector to each subblock, and switch between unidirectional and bidirectional prediction for each subblock. In such a case, for CUs that generate predicted images using both unidirectional and bidirectional prediction, the evaluation unit 111 of the image coding device 1 and the evaluation unit 208 of the image decoding device 2 may not calculate an error map. On the other hand, if predicted images are generated by bidirectional prediction for all subblocks, the evaluation unit 111 of the image coding device 1 and the evaluation unit 208 of the image decoding device 2 generate an error map.

[0121] Furthermore, the motion compensation prediction unit 109 of the image coding device 1 and the motion compensation prediction unit 208 of the image decoding device 2 may perform overlapped block motion compensation (OBMC) to reduce discontinuities in the predicted image at block boundaries with different motion vectors. The evaluation unit 111 of the image coding device 1 and the evaluation unit 208 of the image decoding device 2 may consider correction of reference pixels by OBMC when generating the error map.

[0122] For example, if the prediction mode of the surrounding blocks used for OBMC correction is biprediction, the evaluation unit 111 of the image coding device 1 and the evaluation unit 208 of the image decoding device 2 correct the error map for the region of the predicted image affected by OBMC correction, using the motion vectors of the reference images (L0 and L1) used for generating the biprediction predicted image of the surrounding blocks. Specifically, for the block boundary region of map[i,j], if the motion vectors of adjacent blocks are biprediction, a weighted average is performed according to the position of the error map of the adjacent block. If the adjacent block is in intra-mode or unidirectional prediction, no error map correction is performed. In the case of Figure 9, L0 is used for the upper block boundary. a and L1 aAn error map is generated using this method, and a weighted average is performed on the lower region (the region overlapping with the CU) with the error map of the CU in question. Since the prediction mode of the lower, right, and left CUs is unidirectional, no correction is performed on the error map in the region overlapping with those CUs.

[0123] <11. Example of change 3> The determination unit 112 of the image encoding device 1 may determine the partial transformation block in the following manner.

[0124] The determination unit 112 determines a partial transformation block, for example, when generating a predicted image by dual prediction for a target image block (CU: Coding Unit), and when the transformation skip mode is not applied to this CU (i.e., when an orthogonal transformation is applied). A flag indicating whether or not the transformation skip mode is applied is included in the encoded data.

[0125] Figure 10 shows how the partial transformation block is determined in this example of modification.

[0126] Firstly, as shown in Figure 10(A), the determination unit 112 calculates the centroid of the error map using the following equation (5). Here, the error map map[i,j] described above is used as the error map.

[0127]

number

[0128] Here, gx is the horizontal center of gravity, and gy is the vertical center of gravity.

[0129] Secondly, as shown in Figure 10(B), the determination unit 112 selects the optimal size from among the candidate sizes (size modes) of the partial transformation block. Specifically, the determination unit 112 calculates the RD cost for each size mode and determines the optimal size mode based on the RD cost. In calculating the RD cost, the signaling cost of the transformation coefficients and the distortion of the entire CU are taken into consideration. The size modes are, for example, the following four:

[0130] Mode 1: A mode in which the size of the partial transformation block is equal to the CU size (i.e., a mode in which the entire CU is orthogonally transformed). Mode 2: A mode in which the vertical and horizontal sizes of partial transformation blocks are each halved to the CU size. Mode 3: A mode in which the vertical size of the partial transformation block is equal to the CU size, and the horizontal size of the partial transformation block is half the CU size. Mode 4: A mode in which the vertical size of a partial transformation block is set to half the CU size, and the horizontal size of a partial transformation block is set to be equal to the CU size. Here, we will proceed with the explanation assuming that a mode other than mode 1 is determined and a partial transformation block is set. The determination unit 112 is set so that the partial transformation block centered on the central position (gx, gy) does not cross the CU boundary.

[0131] Thirdly, as shown in Figure 10(C), the transformation unit 102a performs a partial orthogonal transformation on the predicted residual (residual signal) of the partial transformation block based on the determined center position and size.

[0132] Fourth, as shown in Figure 10(D), the quantization unit 102b quantizes the transformation coefficients of the partial transformation block, and the entropy coding unit 103 entropy codes the quantized transformation coefficients.

[0133] Furthermore, the entropy coding unit 103 includes a flag, bPartialTransform, which indicates whether or not to apply a partial orthogonal transform, and sizeMode, which is information indicating the size of the partial transform block, in the encoded data. bPartialTransform is "1" when a partial orthogonal transform is applied, and "0" when a partial orthogonal transform is not applied. sizeMode is, for example, "0" for mode 2, "10" for mode 3, and "11" for mode 4. Note that the cost of such signaling is also taken into consideration when calculating the RD cost described above.

[0134] Meanwhile, in the image decoding device 2, the entropy decoding unit 200 decodes the encoded data and obtains a flag indicating whether or not the transformation skip mode is applied. The entropy decoding unit 200 obtains bPartialTransform and sizeMode when it generates a predicted image for the CU to be decoded by dual prediction and the transformation skip mode is not applied to this CU (i.e., when an orthogonal transformation is applied). The identification unit 209 then identifies the partial transformation block based on the error map generated by the evaluation unit 208 and sizeMode. The subsequent operation is the same as in the embodiment described above.

[0135] <12. Other Embodiments> In the embodiments described above, inter-prediction was primarily explained as motion-compensated prediction. In inter-prediction, a reference image in a frame different from the current frame is used to predict the target image block in the current frame. However, a technique called intra-block copying can also be applied as motion-compensated prediction. In intra-block copying, a reference image in the same frame as the current frame is used to predict the target image block in the current frame.

[0136] The specific examples of the present invention described above may be provided by a program that causes a computer to execute each process performed by the image encoding device 1 and a program that causes a computer to execute each process performed by the image decoding device 2. The program may also be recorded on a computer-readable medium. Using a computer-readable medium makes it possible to install the program on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, but may be a recording medium such as a CD-ROM or DVD-ROM. Furthermore, the circuits that execute each process performed by the image encoding device 1 may be integrated and the image encoding device 1 may be configured as a semiconductor integrated circuit (chipset, SoC). Similarly, the circuits that execute each process performed by the image decoding device 2 may be integrated and the image decoding device 2 may be configured as a semiconductor integrated circuit (chipset, SoC).

[0137] Although the embodiments have been described in detail above with reference to the drawings, the specific configuration is not limited to those described above, and various design changes can be made without departing from the gist of the invention.

[0138] Furthermore, the entire contents of Japanese Patent Application No. 2018-65780 (filed on March 29, 2018) are incorporated into this specification by reference.

Claims

1. An image decoding device that decodes a target image in block units from encoded data, A prediction unit that generates blocks of predicted images corresponding to blocks of the target image by performing predictions using multiple reference images, An evaluation unit that calculates an evaluation value indicating the similarity between the plurality of reference images, An acquisition unit that decodes the encoded data and obtains the predicted residual, The system includes a synthesis unit that synthesizes the predicted residual with the blocks of the predicted image to reconstruct the blocks of the target image, The evaluation unit calculates the absolute difference value at the pixel level between the plurality of reference images, and calculates the evaluation value according to the absolute difference value at the pixel level. The aforementioned evaluation value is used to correct the synthesis target of the synthesis unit in units smaller than the block, consisting of multiple pixels in a partial block unit. The evaluation unit is an image decoding device that controls the calculation of the evaluation value depending on whether or not bidirectional prediction using forward and backward prediction is applied by the prediction unit.

2. An image decoding method for decoding a target image in block units from encoded data, A prediction step that generates blocks of predicted images corresponding to blocks of the target image by performing prediction using multiple reference images, An evaluation step of calculating an evaluation value that indicates the similarity between the plurality of reference images, The acquisition step involves decoding the encoded data to obtain the predicted residual, The process includes a synthesis step of synthesizing the predicted residual with the blocks of the predicted image to reconstruct the blocks of the target image, The evaluation step includes calculating the absolute difference value at the pixel level between the plurality of reference images and calculating the evaluation value according to the absolute difference value at the pixel level. The aforementioned evaluation value is used to correct the object to be synthesized in the synthesis step in units smaller than the block, consisting of multiple pixels in a partial block unit. The image decoding method includes a step of controlling the calculation of the evaluation value depending on whether or not a bidirectional prediction using forward and backward prediction is applied by the prediction unit.

Images