Prediction image correction device, image coding device, and device and program for decoding image

Abstract

To enable appropriate correction of a prediction image.SOLUTION: A prediction image correction device comprises: an inter-prediction unit for predicting a block of an object image obtained by dividing an image in a frame unit through inter-prediction to generate an inter-prediction block; a correction unit for performing correction processing on the inter-prediction block, and a calculation unit. Only when the inter-prediction unit performs the inter-prediction by using two reference image blocks temporally preceding and following the current frame, the calculation unit calculates a statistic of an error between the two reference image blocks for each image part which is a part of the block and comprises a plurality of pixels, and the correction unit performs correction processing only on image parts for which the statistic of the error exceeds a prescribed threshold, out of the blocks of prediction images.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a predictive image correction device, an image encoding device, an image decoding device, and a program. [Background technology]

[0002] In video coding technology, efficient coding (compression) is achieved by dividing the current image into frames (pictures) and performing transformation and prediction on the resulting block units. There are two types of prediction methods: intra-prediction and inter-prediction.

[0003] Intra prediction is a method of generating a predicted image by referring to a decoded adjacent block that is adjacent to a block to be coded or decoded (hereinafter referred to as a "target image block"), whereas inter prediction is a method of generating a predicted image by referring to a decoded frame that is different from the current frame to which the target image block belongs.

[0004] Patent document 1 describes a predicted image correction device that predicts a target image block using inter-prediction to generate an inter-predicted image, predicts the target image block using intra-prediction to generate an intra-predicted image, and performs a weighted average of the inter-predicted image and the intra-predicted image.

[0005] Specifically, the predictive image correction device described in Patent Document 1 evaluates the continuity between an inter-predictive image corresponding to a target image block and a decoded adjacent block adjacent to the target image block, and if it evaluates that there is discontinuity, corrects the edge region of the inter-predictive image by filtering using the decoded adjacent block. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] US Patent Application Publication No. 2013 / 051467 Summary of the Invention

[0007] The predicted image correction device described in Patent Document 1 applies filtering in accordance with an evaluation of the continuity between the inter-predicted image and the decoded adjacent blocks, regardless of the prediction accuracy of the inter-prediction.

[0008] Therefore, when there is an object boundary at the boundary between the inter-predicted image and the decoded adjacent block, even if the prediction accuracy of the edge region of the inter-predicted image is high, filtering is applied to this edge region, which can result in a problem that the accuracy of the predicted image may decrease due to filtering.

[0009] Therefore, an object of the present invention is to provide a prediction image correction device, an image encoding device, an image decoding device, and a program that are capable of appropriately correcting a prediction image.

[0010] The predictive image correction device of the present invention comprises an inter-prediction unit that predicts a target image block obtained by dividing an image on a frame-by-frame basis using inter-prediction and generates an inter-prediction block corresponding to the target image block, an intra-prediction unit that predicts the target image block using intra-prediction and generates an intra-prediction block corresponding to the target image block, and a predictive image correction unit that performs a weighted averaging process between the inter-prediction block and the intra-prediction block as a correction process, wherein the predictive image correction unit adjusts weights in the weighted averaging process depending on a prediction mode that controls the intra-prediction process for a decoded adjacent block that is adjacent to the target image block. Furthermore, a predictive image correction device according to a first feature includes a prediction unit that generates a predictive image corresponding to a target image block by predicting the target image block obtained by dividing a current image on a frame-by-frame basis using a plurality of reference images; a prediction accuracy evaluation unit that evaluates prediction accuracy of the predicted image based on similarities between the plurality of reference images used to generate the predicted image; and a correction unit that performs a correction process on the predicted image using decoded adjacent blocks that are adjacent to the target image block, and the correction unit controls the correction process based on at least a result of evaluation by the prediction accuracy evaluation unit.

[0011] Note that prediction using multiple reference images is typically bi-prediction in inter-prediction, but is not limited to this. For example, a similar method can be applied when generating a predicted image using multiple references, such as the IntraBC mode (intra block copy mode) of images used in the HEVC encoding method.

[0012] The image encoding device according to the second feature is summarized as including the predictive image correction device according to the first feature.

[0013] The third feature of the present invention relates to an image decoding device including the predictive image correction device according to the first feature.

[0014] A fourth aspect of the present invention relates to a program for causing a computer to function as the predictive image correction device according to the first aspect.

[0015] According to the present invention, it is possible to provide a predicted image correction device, an image encoding device, an image decoding device, and a program that are capable of appropriately correcting a predicted image. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram illustrating a configuration of an image encoding device according to an embodiment. [Figure 2] FIG. 1 is a diagram illustrating a configuration of an image decoding device according to an embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of inter prediction. [Figure 4] FIG. 10 is a diagram showing an example of a predicted image generated by inter prediction. [Figure 5] FIG. 2 is a diagram illustrating an example of a configuration of a prediction accuracy evaluation unit according to the embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of a configuration of a predicted image correction unit according to the embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of the operation of a continuity evaluation unit according to the embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of a predicted image correction operation according to the embodiment. [Figure 9] FIG. 10 is a diagram illustrating a configuration of a predicted image correction unit according to a modified example of the embodiment. [Figure 10] 10A and 10B are diagrams illustrating an example of the operation of a continuity evaluation unit and a filter processing unit according to a modified example of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0017] An image encoding device and an image decoding device according to an embodiment will be described with reference to the drawings. The image encoding device and the image decoding device according to the embodiment encode and decode moving images, as typified by MPEG. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

[0018] (1. Configuration of the Image Encoding Device) Fig. 1 is a diagram showing the configuration of an image encoding device 1 according to this embodiment. As shown in Fig. 1, the image encoding device 1 includes a block division unit 100, a subtraction unit 101, a transformation unit 102a, a quantization unit 102b, an entropy encoding unit 103, an inverse quantization unit 104a, an inverse transformation unit 104b, a synthesis unit 105, a memory 106, an intra prediction unit 107, an inter prediction unit 108, a prediction accuracy evaluation unit 109, and a prediction image correction unit (correction unit) 110. In this embodiment, the intra prediction unit 107, the inter prediction unit 108, the prediction accuracy evaluation unit 109, and the prediction image correction unit 110 constitute a prediction image correction device.

[0019] The block division unit 100 divides an input image in units of frames (or pictures) into small block-shaped regions and outputs the image blocks to the subtraction unit 101 (and the inter prediction unit 108). The size of the image blocks is, for example, 32×32 pixels, 16×16 pixels, 8×8 pixels, or 4×4 pixels. An image block is a unit for encoding by the image encoding device 1 and a unit for decoding by the image decoding device 2, and such an image block is referred to as a target image block. Note that the shape of the image block is not limited to a square and may be rectangular.

[0020] The subtraction unit 101 calculates a prediction residual indicating a difference in pixel units between a current image block input from the block division unit 100 and a predicted image (predicted image block) corresponding to the current image block. Specifically, the subtraction unit 101 calculates the prediction residual by subtracting each pixel value of the predicted image from each pixel value of the current block to be coded, and outputs the calculated prediction residual to the conversion unit 102a. In this embodiment, the predicted image is corrected by a predicted image correction unit 110 (described later) and is input from the predicted image correction unit 110 to the subtraction unit 101.

[0021] The transform unit 102a and the quantization unit 102b constitute a transform / quantization unit 102 that performs orthogonal transform processing and quantization processing on a block-by-block basis.

[0022] The transform unit 102a calculates transform coefficients by performing an orthogonal transform on the prediction residuals input from the subtraction unit 101, and outputs the calculated transform coefficients to the quantization unit 102b. Examples of orthogonal transform include a discrete cosine transform (DCT), a discrete sine transform (DST), and a Karhunen-Loeve transform (KLT).

[0023] The quantization unit 102b quantizes the transform coefficients input from the transform unit 102a using a quantization parameter (Qp) and a quantization matrix to generate quantized transform coefficients. The quantization parameter (Qp) is a parameter commonly applied to each transform coefficient in a block and determines the coarseness of quantization. The quantization matrix is ​​a matrix having, as elements, quantization values ​​used when quantizing each transform coefficient. The quantization unit 102b outputs quantization control information, generated quantized transform coefficient information, and the like to the entropy coding unit 103 and the inverse quantization unit 104a.

[0024] The entropy coding unit 103 performs entropy coding on the quantized transform coefficients input from the quantization unit 102b, compresses the data, generates coded data (bit stream), and outputs the coded data to the outside of the image coding device 1. For entropy coding, Huffman coding, CABAC (Context-based Adaptive Binary Arithmetic Coding), etc. may be used. Note that information related to prediction is input to the entropy coding unit 103 from the intra prediction unit 107 and the inter prediction unit 108. The entropy coding unit 103 also performs entropy coding on this information.

[0025] The inverse quantization unit 104a and the inverse transformation unit 104b constitute an inverse quantization and inverse transformation unit 104 that performs inverse quantization processing and inverse orthogonal transformation processing on a block-by-block basis.

[0026] The inverse quantization unit 104a performs inverse quantization processing corresponding to the quantization processing performed by the quantization unit 102b. Specifically, the inverse quantization unit 104a restores the transform coefficients by inverse quantizing the quantized transform coefficients input from the quantization unit 102b using a quantization parameter (Qp) and a quantization matrix, and outputs the restored transform coefficients to the inverse transform unit 104b.

[0027] The inverse transform unit 104b performs an inverse orthogonal transform process corresponding to the orthogonal transform process performed by the transform unit 102a. For example, if the transform unit 102a performs a discrete cosine transform, the inverse transform unit 104b performs an inverse discrete cosine transform. The inverse transform unit 104b performs an inverse orthogonal transform on the transform coefficients input from the inverse quantization unit 104a to restore prediction residuals, and outputs the restored prediction residuals to the synthesis unit 105.

[0028] The synthesis unit 105 synthesizes, on a pixel-by-pixel basis, the restored prediction residual input from the inverse transform unit 104b with the predicted image input from the predicted image correction unit 110. The synthesis unit 105 adds each pixel value of the restored prediction residual to each pixel value of the predicted image to reconstruct (decode) a target image block, and outputs a reconstructed image block, which is the reconstructed target image block, to the memory 106. Note that such a reconstructed image block may also be referred to as a decoded block.

[0029] The memory 106 stores the reconstructed image blocks input from the synthesis unit 105. The memory 106 stores the reconstructed image blocks in frame units.

[0030] The intra prediction unit 107 generates an intra prediction image by referring to a decoded neighboring block adjacent to the current image block among the reconstructed image blocks (decoded blocks) stored in the memory 106. The intra prediction unit 107 selects an optimal intra prediction mode and performs intra prediction using the selected intra prediction mode. A plurality of intra prediction modes corresponding to a plurality of intra prediction directions are predefined. The intra prediction direction refers to the direction of neighboring reference pixels relative to the current pixel when predicting a current pixel in the current image block by referring to neighboring pixels adjacent to the current image block. In other words, the intra prediction mode (intra prediction direction) determines the neighboring reference pixels to be used for predicting each pixel in the current image block. The intra prediction unit 107 outputs the intra prediction image (and / or neighboring reference pixels) to the predicted image correction unit 110 and outputs information about the selected intra prediction mode to the entropy coding unit 103.

[0031] The inter prediction unit 108 performs inter prediction, which predicts a current image block using a frame-by-frame reconstructed image (decoded image) stored in the memory 106 as a reference image. Specifically, the inter prediction unit 108 calculates a motion vector using a technique such as block matching and generates an inter prediction image based on the motion vector. The inter prediction unit 108 selects an optimal inter prediction method from inter prediction using multiple reference images (typically, bi-prediction) and inter prediction using one reference image (unidirectional prediction), and performs inter prediction using the selected inter prediction method. In this embodiment, the case where the inter prediction unit 108 uses bi-prediction will be mainly described, and the inter prediction unit 108 corresponds to a prediction unit that performs prediction using multiple reference images. The inter prediction unit 108 outputs the generated inter prediction image to the prediction image correction unit 110 and outputs information regarding the selected inter prediction method and motion vector to the entropy coding unit 103. The inter prediction unit 108 also outputs multiple reference images used for inter prediction to the prediction accuracy evaluation unit 109.

[0032] The prediction accuracy evaluation unit 109 evaluates the prediction accuracy in edge regions of the inter-predicted image based on the similarity between multiple reference images used to generate the inter-predicted image. Specifically, when the inter prediction unit 108 performs inter prediction using multiple reference images, the prediction accuracy evaluation unit 109 calculates the similarity between the multiple reference images used for inter prediction for each image portion consisting of one or more pixels, evaluates the prediction accuracy of the inter-predicted image for each image portion based on this similarity, and outputs information on the evaluation result to the predicted image correction unit 110. In this embodiment, an example will be mainly described in which the prediction accuracy evaluation unit 109 calculates the similarity between the multiple reference images used for prediction on a pixel-by-pixel basis and evaluates the prediction accuracy of the inter-predicted image on a pixel-by-pixel basis. Details of the prediction accuracy evaluation unit 109 will be described later.

[0033] The predicted image correction unit 110 corrects the inter-predicted image input from the inter prediction unit 108 and the intra-predicted image input from the intra prediction unit 107 by performing a weighted average. Specifically, the predicted image correction unit 110 corrects edge regions of the inter-predicted image by filtering, using decoded neighboring blocks (neighboring reference pixels) corresponding to the intra-predicted image. The predicted image correction unit 110 also controls filtering based at least on the evaluation result by the prediction accuracy evaluation unit 109. "Controlling filtering" includes controlling whether or not to perform filtering and controlling the filter strength. In this embodiment, an example will be mainly described in which the predicted image correction unit 110 controls whether or not to perform filtering based at least on the evaluation result by the prediction accuracy evaluation unit 109. For example, the predicted image correction unit 110 performs filtering when the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 is equal to or less than a first threshold, and does not perform filtering when the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 exceeds the first threshold. When filtering is performed, the predicted image correction unit 110 outputs the inter predicted image after filtering to the subtraction unit 101 and the synthesis unit 105 as a predicted image, and when filtering is not performed, the predicted image is output as is to the subtraction unit 101 and the synthesis unit 105 as a predicted image. Note that "prediction accuracy" may be expressed as "prediction inaccuracy." Regarding "prediction inaccuracy," for example, prediction accuracy equal to or less than a first threshold may mean that the prediction is highly inaccurate. Prediction accuracy exceeding the first threshold may mean that the prediction is not highly inaccurate. Details of the predicted image correction unit 110 will be described later.

[0034] (2. Configuration of Image Decoding Device) Fig. 2 is a diagram showing the configuration of an image decoding device 2 according to this embodiment. As shown in Fig. 2, the image decoding device 2 includes an entropy decoding unit 200, an inverse quantization unit 201a, an inverse transform unit 201b, a synthesis unit 202, a memory 203, an intra prediction unit 204, an inter prediction unit 205, a prediction accuracy evaluation unit 206, and a prediction image correction unit 207. In this embodiment, the intra prediction unit 204, the inter prediction unit 205, the prediction accuracy evaluation unit 206, and the prediction image correction unit 207 configure a prediction image correction device.

[0035] The entropy decoding unit 200 decodes the coded data generated by the coding device 1 and outputs the quantized transform coefficients to the inverse quantization unit 201a. The entropy decoding unit 200 also decodes the coded data, obtains information related to prediction (intra prediction and inter prediction), and outputs the information related to prediction to the intra prediction unit 204 and the inter prediction unit 205.

[0036] The inverse quantization unit 201a and the inverse transform unit 201b constitute an inverse quantization and inverse transform unit 201 that performs inverse quantization processing and inverse orthogonal transform processing on a block-by-block basis.

[0037] The inverse quantization unit 201a performs inverse quantization processing corresponding to the quantization processing performed by the quantization unit 102b of the image encoding device 1. The inverse quantization unit 201a restores the transform coefficients by inverse quantizing the quantized transform coefficients input from the entropy decoding unit 200 using a quantization parameter (Qp) and a quantization matrix, and outputs the restored transform coefficients to the inverse transform unit 201b.

[0038] The inverse transform unit 201b performs inverse orthogonal transform processing corresponding to the orthogonal transform processing performed by the transform unit 102a of the image encoding device 1. The inverse transform unit 201b performs inverse orthogonal transform on the transform coefficients input from the inverse quantization unit 201a to restore prediction residuals, and outputs the restored prediction residuals (restored prediction residuals) to the synthesis unit 202.

[0039] The synthesis unit 202 reconstructs (decodes) the original target image block by synthesizing the prediction residual input from the inverse transformation unit 201b and the prediction image input from the prediction image correction unit 207 on a pixel-by-pixel basis, and outputs the reconstructed image block to the memory 203.

[0040] The memory 203 stores the reconstructed image blocks input from the synthesis unit 202. The memory 203 stores the reconstructed image blocks in units of frames. The memory 203 outputs the reconstructed images (decoded images) in units of frames to the outside of the image decoding device 2 in display order.

[0041] The intra prediction unit 204 generates an intra prediction image by referring to the reconstructed image block stored in the memory 203 and performing intra prediction in accordance with the intra prediction information (intra prediction mode) input from the entropy decoding unit 200. Specifically, the intra prediction unit 204 generates the intra prediction image by referring to neighboring reference pixels determined according to the intra prediction mode among the reconstructed image blocks (decoded blocks) stored in the memory 203. The intra prediction unit 204 outputs the intra prediction image (and / or neighboring reference pixels) to the prediction image correction unit 207.

[0042] The inter prediction unit 205 performs inter prediction to predict a current image block using a frame-by-frame reconstructed image (decoded image) stored in the memory 106 as a reference image. The inter prediction unit 205 generates an inter predicted image by performing inter prediction in accordance with the inter prediction information (motion vector information, etc.) input from the entropy decoding unit 200, and outputs the inter predicted image to the prediction image correction unit 207. The inter prediction unit 205 also outputs a plurality of reference images used for inter prediction to the prediction accuracy evaluation unit 206.

[0043] The prediction accuracy evaluation unit 206 performs the same operation as the prediction accuracy evaluation unit 109 of the image encoding device 1. When the inter prediction unit 205 performs inter prediction using multiple reference images, the prediction accuracy evaluation unit 206 calculates the similarity between the multiple reference images on a pixel-by-pixel basis to evaluate the prediction accuracy of the predicted image, and outputs information on the evaluation result to the predicted image correction unit 207.

[0044] The predicted image correction unit 207 performs the same operation as the predicted image correction unit 110 of the image encoding device 1. The predicted image correction unit 207 corrects edge regions of the inter predicted image by filtering, using decoded adjacent blocks (adjacent reference pixels) corresponding to the intra predicted image. The predicted image correction unit 207 performs filtering when the prediction accuracy evaluated by the prediction accuracy evaluation unit 206 is equal to or less than a first threshold, and does not perform filtering when the prediction accuracy evaluated by the prediction accuracy evaluation unit 206 exceeds the first threshold. When filtering is performed, the predicted image correction unit 207 outputs the inter predicted image after filtering to the synthesis unit 202 as a predicted image, and when filtering is not performed, outputs the inter predicted image as is to the synthesis unit 202 as a predicted image. Details of the predicted image correction unit 207 will be described later.

[0045] (3. Inter-prediction) Fig. 3 is a diagram showing an example of inter prediction. Fig. 4 is a diagram showing an example of a predicted image generated by inter prediction. As a simple example of inter prediction, a case where bi-prediction used in HEVC, particularly forward and backward prediction (bi-directional prediction), is used will be described.

[0046] As shown in Figure 3, bi-prediction refers to frames temporally before and after a target frame (current frame). In the example of Figure 3, prediction of a block in an image of the tth frame is performed by referring to the t-1th frame and the t+1th frame. In motion estimation, a portion (block) similar to the target image block is detected within a search range set by the system from the reference frames of the t-1st and t+1st frames.

[0047] The detected location is the reference image. Information indicating the relative position of the reference image with respect to the target image block is indicated by the arrow in the figure and is called a motion vector. The motion vector information is coded by entropy coding together with frame information of the reference image in the image coding device 1. Meanwhile, the image decoding device 2 detects the reference image based on the information of the motion vector generated by the image coding device 1.

[0048] As shown in Figures 3 and 4, reference images 1 and 2 detected by motion detection are similar partial images aligned with the target image block within the reference frame, and are therefore similar to the target image block (image to be encoded). In the example of Figure 4, the target image block includes a star pattern and a partial circle pattern. Reference image 1 includes a star pattern and a full circle pattern. Reference image 2 includes a star pattern but does not include a circle pattern.

[0049] A predicted image is generated from these reference images 1 and 2. Note that prediction processing generally generates a predicted image with the characteristics of each reference image by averaging reference images 1 and 2, which have different characteristics but are partially similar. However, a predicted image may also be generated by using more advanced processing, for example, signal enhancement processing using a low-pass filter or a high-pass filter. Here, since reference image 1 contains a circular pattern and reference image 2 does not contain a circular pattern, when reference images 1 and 2 are averaged to generate a predicted image, the signal of the circular pattern in the predicted image is reduced by half compared to reference image 1.

[0050] The difference between the predicted image obtained from reference images 1 and 2 and the target image block (target image to be encoded) is the prediction residual. In the prediction residual shown in Figure 4, a large difference occurs only in the misaligned portions of the edges of the star pattern and the misaligned portions of the circle pattern (hatched areas), but for other parts, predictions are performed with high accuracy and the difference is small (no difference occurs in the example of Figure 4).

[0051] The areas where no difference occurs (non-edge areas of the star pattern and background areas) are areas where the similarity between reference image 1 and reference image 2 is high and where highly accurate predictions have been made. On the other hand, the areas where a large difference occurs are areas specific to each reference image, that is, areas where the similarity between reference image 1 and reference image 2 is significantly low. Therefore, it can be seen that areas where the similarity between reference image 1 and reference image 2 is significantly low have low prediction accuracy and cause large differences (residuals).

[0052] When a prediction residual containing a mixture of large and small differences is orthogonally transformed and quantization causes degradation of the transform coefficients, the degradation of the transform coefficients propagates throughout the image (block) through inverse quantization and inverse orthogonal transform. When the prediction residual (restored prediction residual) restored by inverse quantization and inverse orthogonal transform is synthesized with a predicted image to reconstruct the target image block, the degradation of image quality propagates to areas where high-precision predictions were made, such as the non-edge and background areas of the star pattern shown in Figure 4.

[0053] (4. Prediction Accuracy Evaluation Section) FIG. 5 is a diagram showing an example of the configuration of the prediction accuracy evaluation unit 109 in the image encoding device 1. The prediction accuracy evaluation unit 109 evaluates the prediction accuracy of a predicted image on a pixel-by-pixel basis by calculating the similarity between multiple reference images used in inter prediction. Here, an example will be described in which the prediction accuracy evaluation unit 109 evaluates the prediction accuracy for all pixels of the predicted image, but the prediction accuracy evaluation unit 109 does not necessarily have to evaluate the prediction accuracy for all pixels of the predicted image. The prediction accuracy evaluation unit 109 only needs to evaluate the prediction accuracy for at least pixels in the edge regions of the predicted image.

[0054] As shown in FIG. 5, the prediction accuracy evaluation unit 109 includes a difference calculation unit (subtraction unit) 109a, a normalization unit 109b, and an adjustment unit 109c.

[0055] The difference calculation unit 109a calculates the absolute value of the difference between reference image 1 and reference image 2 on a pixel-by-pixel basis, and outputs the calculated absolute value of the difference to the normalization unit 109b. The absolute value of the difference is an example of a value indicating the similarity. It can be said that the smaller the absolute value of the difference, the higher the similarity, and the larger the absolute value of the difference, the lower the similarity. The difference calculation unit 109a may calculate the absolute value of the difference after performing a filter process on each reference image. The difference calculation unit 109a may also calculate a statistic such as a squared error and use the statistic as the similarity.

[0056] The normalization unit 109b normalizes the difference value of each pixel input from the difference calculation unit 109a by the absolute value of the difference value of the pixel whose absolute value of the difference value is maximum in the block (i.e., the maximum absolute value of the difference values ​​in the block), and outputs a normalized difference value, which is the absolute value of the normalized difference value, to the adjustment unit 109c. In this embodiment, the normalized difference value is used as a weight for weighting, on a pixel-by-pixel basis, the restored prediction residual to be combined with the predicted image in the combination unit 105.

[0057] The adjustment unit 109c adjusts the normalized difference value (weight) input from the normalization unit 109b based on a quantization parameter (Qp) that determines the coarseness of quantization, and outputs the weight. Since the greater the coarseness of quantization, the higher the degree of degradation of the restored prediction residual, the adjustment unit 109c can weight the restored prediction residual in consideration of the degree of degradation by adjusting the normalized difference value (weight) based on the quantization parameter (Qp).

[0058] The estimated prediction accuracy Rij of each pixel (ij) output by the prediction accuracy evaluation unit 109 can be expressed, for example, as in the following equation (1).

[0059] Rij = 1 - (abs(Xij-Yij) / maxD × Scale(Qp)) ···(1)

[0060] In equation (1), Xij is the pixel value of pixel ij in reference image 1, Yij is the pixel value of pixel ij in reference image 2, and abs is a function for obtaining the absolute value.

[0061] In addition, in equation (1), maxD is the maximum value of the difference values ​​abs(Xij-Yij) within the block. To calculate maxD, it is necessary to calculate the difference values ​​for all pixels within the block. However, to omit this process, the maximum value of an adjacent block that has already been processed may be used instead. For example, if there is a value equal to or greater than this value, maxD may be normalized by clipping it with the maximum value used. Alternatively, maxD may be calculated from the quantization parameter (Qp) using a table that defines the correspondence between the quantization parameter (Qp) and maxD. Alternatively, a fixed value defined in advance in the specifications may be used as maxD.

[0062] In addition, in equation (1), Scale(Qp) is a coefficient multiplied according to the quantization parameter (Qp). Scale(Qp) is designed to approach 1.0 when Qp is large and approach 0 when Qp is small, and the degree of this is adjusted depending on the system. Alternatively, a fixed value defined in advance in the specifications may be used as Scale(Qp). Furthermore, to simplify processing, Scale(Qp) may be a fixed value, such as 1.0, designed according to the system.

[0063] The adjustment unit 109c outputs the estimated prediction accuracy Rij. Furthermore, this Rij may be output as a weight adjusted by a sensitivity function designed according to the system. For example, abs(Xij-Yij) / maxD×Scale(Qp)=Rij may be used, and the sensitivity may be adjusted by adding an offset according to control information such as QP, such as Rij=Clip(rij+offset,1.0,0.0), rather than Rij=Clip(rij,1.0,0.0). Note that Clip(x,max,min) indicates a process of clipping at max if x exceeds max, and at min if x is below min.

[0064] The estimated prediction accuracy Rij calculated in this way is a value within the range from 0 to 1.0. Basically, the estimated prediction accuracy Rij approaches 0 when the absolute value of the difference value of pixel ij between reference images is large (i.e., prediction accuracy is low), and approaches 1 when the absolute value of the difference value of pixel ij between reference images is small (i.e., prediction accuracy is high). The prediction accuracy evaluation unit 109 outputs map information consisting of the estimated prediction accuracy Rij of each pixel ij in the block to the predicted image correction unit 110 on a block-by-block basis.

[0065] Note that the prediction accuracy evaluation unit 109 performs evaluation (calculation of estimated prediction accuracy Rij) only when inter prediction using multiple reference images is applied, and does not need to perform evaluation in other modes, such as unidirectional prediction or intra prediction processing that does not use multiple reference images.

[0066] Furthermore, although the prediction accuracy evaluation unit 109 in the image encoding device 1 has been described above, the prediction accuracy evaluation unit 206 in the image decoding device 2 has the same configuration as the prediction accuracy evaluation unit 109 in the image encoding device 1. Specifically, the prediction accuracy evaluation unit 206 in the image decoding device 2 includes a difference calculation unit 206a, a normalization unit 206b, and an adjustment unit 206c.

[0067] (5. Prediction Image Correction Unit) Fig. 6 is a diagram showing an example of the configuration of the predicted image correction unit 110 in the image encoding device 1. As shown in Fig. 6, the predicted image correction unit 110 includes a continuity evaluation unit 110a, a filter determination unit 110b, and a filter processing unit 110c.

[0068] The continuity evaluation unit 110a evaluates the continuity between the inter-predicted image input from the inter prediction unit 108 and the decoded adjacent block input from the intra prediction unit 107. The decoded adjacent block is a decoded block adjacent to the current image block. The continuity evaluation unit 110a outputs the evaluation result of the continuity between the inter-predicted image and the decoded adjacent block to the filter determination unit 110b.

[0069] Fig. 7 is a diagram showing an example of the operation of the continuity evaluation unit 110a. In the example of Fig. 7, it is assumed that decoded blocks exist to the left and above the image block to be coded. The continuity evaluation unit 110a evaluates the continuity of the block boundary between the inter-predicted image (target image block) and the decoded block in pixel-by-pixel order from top to bottom (or bottom to top). The continuity evaluation unit 110a calculates, for example, the continuity Cont of the block boundary spanning pixels p0 and q0 in Fig. 7 using the following equation (2):

[0070] Cont=|p2-2×p1+p0|+|q2-2×q1+q0| ···(2)

[0071] The units and evaluation indices used to evaluate continuity are not limited to those shown in formula (2), and can be changed to other units and evaluation indices as long as they are predefined by the system. For example, the unit for evaluating continuity may be the side unit of the block boundary (continuity evaluation is performed on the left boundary and on the upper boundary), or, to simplify processing, the pixel value difference across the boundary (abs(p0-q0)) may be used as the evaluation indices for continuity.

[0072] The filter determination unit 110b determines whether to perform filtering based on the prediction accuracy evaluation result input from the prediction accuracy evaluation unit 109 and the continuity evaluation result input from the continuity evaluation unit 110a. The filter determination unit 110b may perform the determination on a pixel-by-pixel basis at the block boundary between the inter-predicted image and the decoded block, or on a side-by-side basis at the block boundary. The filter determination unit 110b may determine to change the filter strength (e.g., the number of filter taps or frequency response) based on the prediction accuracy evaluation result and the continuity evaluation result. Note that when the prediction accuracy evaluation unit 109 evaluates the prediction accuracy on a pixel-by-pixel basis, the filter determination unit 110b may use the average value of the estimated prediction accuracy R of each pixel in the edge region (region targeted for filtering) of the inter-predicted image as the prediction accuracy evaluation result.

[0073] The filter determination unit 110b determines to perform filtering when the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 is equal to or less than the first threshold α and the continuity evaluated by the continuity evaluation unit 110a is equal to or less than the second threshold β. On the other hand, when the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 exceeds the first threshold α and / or the continuity evaluated by the continuity evaluation unit 110a exceeds the second threshold β, the filter determination unit 110b determines not to perform filtering. Here, the thresholds α and β may be set to system-fixed values, or may be calculated and set to variable values ​​using a function that uses a quantization parameter (Qp). Furthermore, when the estimated prediction accuracy R is not normalized, the threshold α may be calculated and set to a variable value using a function that uses the maximum estimated prediction accuracy within the target image block.

[0074] When the filter determination unit 110b determines that filtering should be performed, the filter processing unit 110c performs filtering on the edge region of the inter-predicted image. As shown in Fig. 7, the filter processing unit 110c performs correction using the following equation (3) using predicted pixels q0, q1, and q2 included in the edge region of the inter-predicted image (block to be coded) and adjacent decoded reference pixels p0, p1, and p2 input from the intra prediction unit 107, and outputs corrected predicted pixels q'0, q'1, and q'2.

[0075]

number

[0076] Here, clip(x, max, min) indicates a process of clipping at max when x exceeds max, and at min when x is below min. Also, tc in equation (3) is an adjustment offset used in the clipping process, and a variable value can be calculated and set using a function that uses the quantization parameter (Qp).

[0077] Although the predicted image correction unit 110 in the image encoding device 1 has been described above, the predicted image correction unit 207 in the image decoding device 2 has the same configuration as the predicted image correction unit 110 in the image encoding device 1. Specifically, the predicted image correction unit 207 in the image decoding device 2 includes a continuity evaluation unit 207a, a filter determination unit 207b, and a filter processing unit 207c.

[0078] (6. Predictive Image Correction Operation) Fig. 8 is a diagram showing an example of a prediction image correction operation. Here, the prediction image correction operation in the image encoding device 1 will be described, but a similar prediction image correction operation is also performed in the image decoding device 2. The operation flow in Fig. 8 may be performed in units of pixels at the block boundary between the inter-predicted image and the decoded block.

[0079] As shown in FIG. 8, in step S1, the prediction accuracy evaluation unit 109 evaluates the prediction accuracy in the edge region of the inter-predicted image based on the similarity between multiple reference images used by the inter-prediction unit 108 to generate the inter-predicted image.

[0080] In step S2, the continuity evaluation unit 110a evaluates the continuity between the inter-predicted image output by the inter-prediction unit 108 and the decoded neighboring blocks that are adjacent to the inter-predicted image.

[0081] In step S3, the filter determination unit 110b compares the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 with a first threshold value α, and compares the continuity evaluated by the continuity evaluation unit 110a with a second threshold value β.

[0082] If the prediction accuracy is equal to or less than the first threshold α and the continuity is equal to or less than the second threshold β (step S3: YES), the filter determination unit 110b determines to perform filtering. In this case, in step S4, the filtering processing unit 110c corrects the edge regions of the inter-predicted image by filtering using decoded adjacent blocks (adjacent reference pixels).

[0083] On the other hand, if the prediction accuracy exceeds the first threshold α and / or the continuity exceeds the second threshold β (step S3: NO), the filter determination unit 110b determines not to perform filtering. In this case, the filtering processing unit 110c does not perform filtering.

[0084] (7. Summary of embodiments) The image encoding device 1 and the image decoding device 2 according to this embodiment control the filter process based on the evaluation result of the prediction accuracy of the inter prediction. As a result, when the prediction accuracy of the edge region of the inter prediction image is high, it is possible to control so that the filter process is not applied to this edge region, thereby preventing a decrease in the accuracy of the predicted image due to the filter process. In other words, only when the prediction accuracy of the edge region of the inter prediction image is low, it is possible to apply the filter process to this edge region, so that the filter process can be performed appropriately.

[0085] Furthermore, in this embodiment, the image encoding device 1 and the image decoding device 2 determine whether or not to apply filter processing by a common operation. Specifically, the image encoding device 1 and the image decoding device 2 each evaluate the prediction accuracy of inter prediction, and determine whether or not to apply filter processing based on the evaluation results. This eliminates the need to transmit a flag indicating whether or not to apply filter processing from the image encoding device 1 to the image decoding device 2, thereby preventing an increase in the amount of code due to the flag.

[0086] (8. Modification 1 of the embodiment) In the continuity evaluation and filtering process according to the above-described embodiment, the intra prediction mode (intra prediction direction) applied to the decoded neighboring block may be taken into consideration. This can improve the accuracy of the continuity evaluation and filtering process. FIG. 9 is a diagram showing the configuration of the predicted image correction unit 110 according to this modified example. As shown in FIG. 9, the intra prediction mode applied to the decoded neighboring block is input from the intra prediction unit 107 to the continuity evaluation unit 110a and the filtering process unit 110c. The continuity evaluation unit 110a evaluates continuity according to the intra prediction mode applied to the decoded neighboring block. Furthermore, the filtering process unit 110c performs filtering according to the intra prediction mode applied to the decoded neighboring block.

[0087] FIG. 10 is a diagram illustrating an example of the operation of the continuity evaluation unit 110a and the filtering unit 110c according to this modified example. As shown in FIG. 10, when evaluating continuity and performing filtering on an edge pixel q0 of an inter-predicted image corresponding to a current block, the continuity evaluation unit 110a and the filtering unit 110c calculate p1, p0, q1, and q2 on a line passing through q0 by weighted averaging along the intra-prediction mode (intra-prediction direction) applied to the decoded neighboring block. Specifically, p1 and p0 on a line passing through q0 are calculated using pixel values ​​in the decoded neighboring block, and q1 and q2 on a line passing through q0 are calculated using pixel values ​​in the inter-predicted image. The continuity evaluation unit 110a evaluates the continuity Cont using the above formula (2) using p1, p0, q1, and q2 calculated in this way. The filtering unit 110c calculates q'0 using the above formula (3) using p1, p0, q1, and q2 calculated in this way.

[0088] (9. Modification 2 of the embodiment) In the above-described embodiment, an example has been described in which the predicted image correction unit 110 performs filtering on edge regions of the inter predicted image using decoded adjacent blocks as correction processing on the inter predicted image.

[0089] Furthermore, the correction process is not limited to such filtering, and weighted averaging of inter-predicted images and intra-predicted images may also be used as the correction process.

[0090] In this modified example, the intra prediction unit 107 and the predicted image correction unit 110 of the image encoding device 1 shown in FIG. 1 constitute a correction unit that performs correction processing on an inter predicted image using decoded adjacent blocks that are adjacent to the current image block.

[0091] The intra prediction unit 107 generates an intra prediction image corresponding to the current image block using decoded neighboring blocks. For example, the intra prediction unit 107 generates an intra prediction image from the decoded neighboring blocks using a predetermined intra prediction mode. The prediction image correction unit 110 performs a weighted averaging process on the inter prediction image input from the inter prediction unit 108 and the intra prediction image input from the intra prediction unit 107 as a correction process. Note that the transmitting side may be configured to select one prediction mode from multiple intra prediction mode candidates and generate an intra prediction image. In this case, the entropy coding unit 103 entropy codes an intra prediction mode flag indicating which intra prediction mode was used to generate the intra prediction image on the transmitting side.

[0092] Similarly, the intra prediction unit 204 and the predicted image correction unit 207 of the image decoding device 2 shown in Fig. 2 constitute a correction unit that performs correction processing on an inter predicted image using decoded adjacent blocks that are adjacent to a current image block. The intra prediction unit 204 generates an intra predicted image corresponding to the current image block using the decoded adjacent blocks.

[0093] For example, the intra prediction unit 204 generates an intra prediction image from decoded neighboring blocks using a predetermined intra prediction mode. As a correction process, the prediction image correction unit 207 performs weighted averaging of the inter prediction image input from the inter prediction unit 205 and the intra prediction image input from the intra prediction unit 204. Note that the intra prediction mode to be used for generating the intra prediction image may be determined based on an intra prediction mode flag sent from the transmitting side.

[0094] When applying the configuration of this modified example, the “filtering process” in the above-described embodiment can be read as “weighted averaging process.” In addition to determining whether or not to perform such weighted averaging process, the weights in the weighted averaging process may be adjusted based on the evaluation results of the prediction accuracy and the continuity.

[0095] (10. Other Embodiments) In the above-described embodiment, bi-prediction has been mainly described as a prediction method using multiple reference images. However, a technique called intra block copy can also be applied as a prediction method using multiple reference images. In intra block copy, a reference image in the same frame as the current frame is used to predict a target image block of the current frame.

[0096] The image encoding device 1 may be provided by a program causing a computer to execute each process performed by the image encoding device 1, and a program causing a computer to execute each process performed by the image decoding device 2. The program may also be recorded on a computer-readable medium. The computer-readable medium can be used to install the program on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM. Furthermore, circuits that execute each process performed by the image encoding device 1 may be integrated, and the image encoding device 1 may be configured as a semiconductor integrated circuit (chip set, SoC). Similarly, circuits that execute each process performed by the image decoding device 2 may be integrated, and the image decoding device 2 may be configured as a semiconductor integrated circuit (chip set, SoC).

[0097] The above describes the embodiments in detail with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes can be made within the scope that does not deviate from the gist of the invention.

[0098] The entire contents of Japanese Patent Application No. 2018-72451 (filed April 4, 2018) are incorporated herein by reference.

Claims

1. an inter prediction unit that predicts blocks of a target image obtained by dividing an image in units of frames by inter prediction to generate inter prediction blocks; a correction unit that performs correction processing on the inter-predicted block; a calculation unit, When the inter prediction unit performs the inter prediction using blocks of two reference images temporally before and after the current frame, the calculation unit calculates a statistic of an error between the blocks of the two reference images for each image portion that is a part of the block and is made up of a plurality of pixels; The correction unit performs a correction process only on the image portion of the block of the predicted image where the statistical amount of the error exceeds a threshold. Predictive image correction device.

2. A predictive image correction device according to claim 1, Image encoding device.

3. A predictive image correction device according to claim 1, Image decoding device.

4. A computer is caused to function as the predictive image correction device according to claim 1. program.

Images