Prediction image correction device, image coding device, image decoding device, and program

Abstract

To enable an appropriate correction of a prediction image.SOLUTION: A prediction image correction device comprises: an inter-prediction part that predicts an object image block obtained by dividing an image in a frame unit by an inter-prediction, and generates an inter-prediction block corresponding to the object image block; an intra-prediction part that predicts the object image block by a predetermined intra-prediction mode, and generates an intra-prediction block corresponding to the object image block; an evaluation part that performs an evaluation on the basis of the prediction mode for controlling an intra-prediction processing against a decoded adjacent block that is adjacent to the object image block; and a prediction image correction part that executes a weighted average processing of the inter-prediction block and the intra-prediction block as a correction processing. The prediction image correction part adjusts the weight in the weighted average processing in accordance with a result of the evaluation.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a predictive image correction device, an image encoding device, an image decoding device, and a program. [Background technology]

[0002] In video coding technology, efficient coding (compression) is achieved by dividing the current image into frames (pictures) and performing conversion and prediction on a block-by-block basis. There are two types of prediction methods: intra-prediction and inter-prediction.

[0003] Intra prediction is a method of generating a predicted image by referring to a decoded adjacent block adjacent to a block to be coded or decoded (hereinafter referred to as a "target image block"), whereas inter prediction is a method of generating a predicted image by referring to a decoded frame different from the current frame to which the target image block belongs.

[0004] Patent document 1 describes a predictive image correction device that predicts a target image block using inter prediction to generate an inter predicted image, predicts the target image block using intra prediction to generate an intra predicted image, and performs a weighted average of the inter predicted image and the intra predicted image.

[0005] Specifically, the predictive image correction device described in Patent Document 1 evaluates the continuity between an inter-predictive image corresponding to a target image block and a decoded adjacent block adjacent to the target image block, and if it evaluates that there is discontinuity, corrects the edge region of the inter-predictive image by filtering using the decoded adjacent block. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] US Patent Application Publication No. 2013 / 051467 Summary of the Invention

[0007] The predicted image correction device described in Patent Document 1 applies filtering in accordance with an evaluation of the continuity between the inter-predicted image and the decoded adjacent blocks, regardless of the prediction accuracy of the inter-prediction.

[0008] Therefore, when there is an object boundary at the boundary between the inter-predicted image and the decoded adjacent block, even if the prediction accuracy of the edge region of the inter-predicted image is high, filtering is applied to this edge region, which may result in a problem that the accuracy of the predicted image may be reduced by the filtering.

[0009] Therefore, an object of the present invention is to provide a predictive image correction device, an image encoding device, an image decoding device, and a program capable of appropriately correcting a predictive image.

[0010] The predictive image correction device of the present invention comprises an inter prediction unit that predicts a target image block obtained by dividing an image on a frame-by-frame basis by inter prediction and generates an inter prediction block corresponding to the target image block, an intra prediction unit that predicts the target image block by intra prediction and generates an intra prediction block corresponding to the target image block, and a predictive image correction unit that performs a weighted averaging process of the inter prediction block and the intra prediction block as a correction process, and the predictive image correction unit adjusts weights in the weighted averaging process depending on a prediction mode that controls the intra prediction process for a decoded adjacent block adjacent to the target image block. Moreover, a predictive image correction device according to a first feature includes a prediction unit that generates a predictive image corresponding to a target image block obtained by dividing a current image on a frame-by-frame basis, by predicting the target image block using a plurality of reference images, a prediction accuracy evaluation unit that evaluates prediction accuracy of the predicted image based on a similarity between the plurality of reference images used for generating the predicted image, and a correction unit that performs a correction process on the predicted image using a decoded adjacent block that is adjacent to the target image block, and the correction unit controls the correction process based on at least a result of evaluation by the prediction accuracy evaluation unit.

[0011] Note that prediction using multiple reference images is typically performed using bi-prediction in inter prediction, but is not limited to this. A similar method can also be applied when generating a predicted image by using multiple references, such as the IntraBC mode (intra block copy mode) of images used in the HEVC encoding method.

[0012] The second feature relates to an image encoding device including the predictive image correction device according to the first feature.

[0013] The third feature relates to an image decoding device including the predictive image correction device according to the first feature.

[0014] A fourth aspect of the present invention relates to a program for causing a computer to function as the predictive image correction device according to the first aspect.

[0015] According to the present invention, it is possible to provide a prediction image correction device, an image encoding device, an image decoding device, and a program capable of appropriately correcting a prediction image. [Brief description of the drawings]

[0016] [Figure 1] FIG. 1 is a diagram showing a configuration of an image encoding device according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing a configuration of an image decoding device according to an embodiment. [Diagram 3] FIG. 1 is a diagram illustrating an example of inter prediction. [Figure 4] FIG. 1 is a diagram showing an example of a predicted image generated by inter prediction. [Diagram 5] FIG. 2 is a diagram illustrating an example of a configuration of a prediction accuracy evaluation unit according to the embodiment. [Figure 6] FIG. 4 is a diagram illustrating an example of a configuration of a predicted image correction unit according to the embodiment. [Figure 7] 11A and 11B are diagrams illustrating an example of the operation of a continuity evaluation unit according to the embodiment. [Figure 8] 11A and 11B are diagrams illustrating an example of a predicted image correction operation according to the embodiment. [Figure 9] FIG. 13 is a diagram illustrating a configuration of a predicted image correction unit according to a modified example of the embodiment. [Figure 10] 13A and 13B are diagrams illustrating an example of the operation of a continuity evaluation unit and a filter processing unit according to a modified example of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] An image encoding device and an image decoding device according to an embodiment will be described with reference to the drawings. The image encoding device and the image decoding device according to the embodiment encode and decode moving images as represented by MPEG. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

[0018] (1. Configuration of the Image Encoding Device) Fig. 1 is a diagram showing the configuration of an image encoding device 1 according to this embodiment. As shown in Fig. 1, the image encoding device 1 includes a block division unit 100, a subtraction unit 101, a transformation unit 102a, a quantization unit 102b, an entropy encoding unit 103, an inverse quantization unit 104a, an inverse transformation unit 104b, a synthesis unit 105, a memory 106, an intra prediction unit 107, an inter prediction unit 108, a prediction accuracy evaluation unit 109, and a prediction image correction unit (correction unit) 110. In this embodiment, the intra prediction unit 107, the inter prediction unit 108, the prediction accuracy evaluation unit 109, and the prediction image correction unit 110 constitute a prediction image correction device.

[0019] The block division unit 100 divides an input image in units of frames (or pictures) into small block-shaped regions, and outputs the image blocks to the subtraction unit 101 (and the inter prediction unit 108). The size of the image blocks is, for example, 32×32 pixels, 16×16 pixels, 8×8 pixels, or 4×4 pixels. An image block is a unit for encoding by the image encoding device 1 and a unit for decoding by the image decoding device 2, and such an image block is referred to as a target image block. Note that the shape of the image block is not limited to a square, and may be a rectangle.

[0020] The subtraction unit 101 calculates a prediction residual indicating a difference in pixel units between a current image block input from the block division unit 100 and a predicted image (predicted image block) corresponding to the current image block. Specifically, the subtraction unit 101 calculates a prediction residual by subtracting each pixel value of the predicted image from each pixel value of the current block to be coded, and outputs the calculated prediction residual to the conversion unit 102a. In this embodiment, the predicted image is corrected by a predicted image correction unit 110 (described later) and is input from the predicted image correction unit 110 to the subtraction unit 101.

[0021] The transform unit 102a and the quantization unit 102b constitute a transform / quantization unit 102 that performs orthogonal transform processing and quantization processing on a block-by-block basis.

[0022] The transform unit 102a performs an orthogonal transform on the prediction residual input from the subtraction unit 101 to calculate a transform coefficient, and outputs the calculated transform coefficient to the quantization unit 102b. The orthogonal transform refers to, for example, a discrete cosine transform (DCT), a discrete sine transform (DST), a KarhunenLoeve transform (KLT), or the like.

[0023] The quantization unit 102b quantizes the transform coefficients input from the transform unit 102a using a quantization parameter (Qp) and a quantization matrix to generate quantized transform coefficients. The quantization parameter (Qp) is a parameter commonly applied to each transform coefficient in a block, and is a parameter that determines the roughness of quantization. The quantization matrix is ​​a matrix having, as elements, quantization values ​​used when quantizing each transform coefficient. The quantization unit 102b outputs quantization control information, generated quantized transform coefficient information, and the like to the entropy coding unit 103 and the inverse quantization unit 104a.

[0024] The entropy coding unit 103 performs entropy coding on the quantized transform coefficients input from the quantization unit 102b, performs data compression to generate coded data (bit stream), and outputs the coded data to the outside of the image coding device 1. For the entropy coding, Huffman codes, CABAC (Context-based Adaptive Binary Arithmetic Coding), etc. can be used. Note that information related to prediction is input to the entropy coding unit 103 from the intra prediction unit 107 and the inter prediction unit 108. The entropy coding unit 103 also performs entropy coding on this information.

[0025] The inverse quantization unit 104a and the inverse transformation unit 104b constitute an inverse quantization / inverse transformation unit 104 that performs inverse quantization processing and inverse orthogonal transformation processing on a block-by-block basis.

[0026] The inverse quantization unit 104a performs an inverse quantization process corresponding to the quantization process performed by the quantization unit 102b. Specifically, the inverse quantization unit 104a restores the transform coefficients by inverse quantizing the quantized transform coefficients input from the quantization unit 102b using a quantization parameter (Qp) and a quantization matrix, and outputs the restored transform coefficients to the inverse transform unit 104b.

[0027] The inverse transform unit 104b performs an inverse orthogonal transform process corresponding to the orthogonal transform process performed by the transform unit 102a. For example, when the transform unit 102a performs a discrete cosine transform, the inverse transform unit 104b performs an inverse discrete cosine transform. The inverse transform unit 104b performs an inverse orthogonal transform on the transform coefficients input from the inverse quantization unit 104a to restore the prediction residual, and outputs the restored prediction residual, which is the restored prediction residual, to the synthesis unit 105.

[0028] The synthesis unit 105 synthesizes, on a pixel-by-pixel basis, the restored prediction residual input from the inverse transform unit 104b with the predicted image input from the predicted image correction unit 110. The synthesis unit 105 adds each pixel value of the restored prediction residual to each pixel value of the predicted image to reconstruct (decode) a target image block, and outputs a reconstructed image block, which is the reconstructed target image block, to the memory 106. Note that such a reconstructed image block may be referred to as a decoded block.

[0029] The memory 106 stores the reconstructed image blocks input from the synthesis unit 105. The memory 106 stores the reconstructed image blocks on a frame-by-frame basis.

[0030] The intra prediction unit 107 generates an intra prediction image by referring to a decoded adjacent block adjacent to a target image block among reconstructed image blocks (decoded blocks) stored in the memory 106. The intra prediction unit 107 selects an optimal intra prediction mode and performs intra prediction using the selected intra prediction mode. A plurality of intra prediction modes corresponding to a plurality of intra prediction directions are predefined. The intra prediction direction refers to the direction of adjacent reference pixels based on a target pixel when predicting a target pixel in a target image block by referring to adjacent pixels adjacent to the target image block. That is, the intra prediction mode (intra prediction direction) determines adjacent reference pixels to be used for predicting each pixel in the target image block. The intra prediction unit 107 outputs the intra prediction image (and / or adjacent reference pixels) to the prediction image correction unit 110, and outputs information on the selected intra prediction mode to the entropy coding unit 103.

[0031] The inter prediction unit 108 performs inter prediction, which predicts a target image block using a frame-by-frame reconstructed image (decoded image) stored in the memory 106 as a reference image. Specifically, the inter prediction unit 108 calculates a motion vector by a method such as block matching, and generates an inter prediction image based on the motion vector. The inter prediction unit 108 selects an optimal inter prediction method from among inter prediction using multiple reference images (typically bi-prediction) and inter prediction using one reference image (unidirectional prediction), and performs inter prediction using the selected inter prediction method. In this embodiment, the case where the inter prediction unit 108 uses bi-prediction is mainly described, and the inter prediction unit 108 corresponds to a prediction unit that performs prediction using multiple reference images. The inter prediction unit 108 outputs the generated inter prediction image to the prediction image correction unit 110, and outputs information on the selected inter prediction method and motion vector to the entropy coding unit 103. The inter prediction unit 108 also outputs multiple reference images used for inter prediction to the prediction accuracy evaluation unit 109.

[0032] The prediction accuracy evaluation unit 109 evaluates the prediction accuracy in the edge region of the inter prediction image based on the similarity between the multiple reference images used to generate the inter prediction image. Specifically, when the inter prediction unit 108 performs inter prediction using multiple reference images, the similarity between the multiple reference images used for inter prediction is calculated for each image portion consisting of one or multiple pixels, the prediction accuracy of the inter prediction image is evaluated for each image portion based on the similarity, and the evaluation result information is output to the prediction image correction unit 110. In this embodiment, an example in which the prediction accuracy evaluation unit 109 calculates the similarity between the multiple reference images used for prediction on a pixel-by-pixel basis and evaluates the prediction accuracy of the inter prediction image on a pixel-by-pixel basis will be mainly described. The prediction accuracy evaluation unit 109 will be described in detail later.

[0033] The predicted image correction unit 110 corrects the inter-predicted image input from the inter prediction unit 108 and the intra-predicted image input from the intra prediction unit 107 by weighted averaging. Specifically, the predicted image correction unit 110 corrects the edge region of the inter-predicted image by filtering using decoded adjacent blocks (adjacent reference pixels) corresponding to the intra-predicted image. The predicted image correction unit 110 also controls filtering based at least on the evaluation result by the prediction accuracy evaluation unit 109. "Controlling filtering" includes control of whether filtering is performed and control of filter strength. In this embodiment, an example in which the predicted image correction unit 110 controls whether filtering is performed based at least on the evaluation result by the prediction accuracy evaluation unit 109 will be mainly described. For example, the predicted image correction unit 110 performs filtering when the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 is equal to or less than a first threshold, and does not perform filtering when the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 exceeds the first threshold. When performing filtering, the predicted image correction unit 110 outputs the inter-predicted image after filtering as a predicted image to the subtraction unit 101 and the synthesis unit 105, and when not performing filtering, the inter-predicted image is directly output as a predicted image to the subtraction unit 101 and the synthesis unit 105. Note that the "prediction accuracy" may be expressed as "prediction inaccuracy". Regarding the "prediction inaccuracy", for example, the prediction accuracy being equal to or less than a first threshold may mean that the prediction inaccuracy is high. The prediction accuracy exceeding the first threshold may mean that the prediction inaccuracy is low. Details of the predicted image correction unit 110 will be described later.

[0034] (2. Configuration of the Image Decoding Device) Fig. 2 is a diagram showing the configuration of an image decoding device 2 according to this embodiment. As shown in Fig. 2, the image decoding device 2 includes an entropy decoding unit 200, an inverse quantization unit 201a, an inverse transform unit 201b, a synthesis unit 202, a memory 203, an intra prediction unit 204, an inter prediction unit 205, a prediction accuracy evaluation unit 206, and a prediction image correction unit 207. In this embodiment, the intra prediction unit 204, the inter prediction unit 205, the prediction accuracy evaluation unit 206, and the prediction image correction unit 207 configure a prediction image correction device.

[0035] The entropy decoding unit 200 decodes the coded data generated by the coding device 1, and outputs the quantized transform coefficients to the inverse quantization unit 201a. The entropy decoding unit 200 also decodes the coded data, obtains information related to prediction (intra prediction and inter prediction), and outputs the information related to prediction to the intra prediction unit 204 and the inter prediction unit 205.

[0036] The inverse quantization unit 201a and the inverse transform unit 201b constitute an inverse quantization / inverse transform unit 201 that performs inverse quantization processing and inverse orthogonal transform processing on a block-by-block basis.

[0037] The inverse quantization unit 201a performs an inverse quantization process corresponding to the quantization process performed by the quantization unit 102b of the image encoding device 1. The inverse quantization unit 201a restores the transform coefficients by inverse quantizing the quantized transform coefficients input from the entropy decoding unit 200 using a quantization parameter (Qp) and a quantization matrix, and outputs the restored transform coefficients to the inverse transform unit 201b.

[0038] The inverse transform unit 201b performs inverse orthogonal transform processing corresponding to the orthogonal transform processing performed by the transform unit 102a of the image encoding device 1. The inverse transform unit 201b performs inverse orthogonal transform on the transform coefficients input from the inverse quantization unit 201a to restore the prediction residual, and outputs the restored prediction residual (restored prediction residual) to the synthesis unit 202.

[0039] The synthesis unit 202 reconstructs (decodes) the original target image block by synthesizing the prediction residual input from the inverse transformation unit 201b and the prediction image input from the prediction image correction unit 207 on a pixel-by-pixel basis, and outputs the reconstructed image block to the memory 203.

[0040] The memory 203 stores the reconstructed image block input from the synthesis unit 202. The memory 203 stores the reconstructed image block on a frame-by-frame basis. The memory 203 outputs the reconstructed image (decoded image) on a frame-by-frame basis to the outside of the image decoding device 2 in display order.

[0041] The intra prediction unit 204 generates an intra prediction image by referring to the reconstructed image block stored in the memory 203 and performing intra prediction according to the intra prediction information (intra prediction mode) input from the entropy decoding unit 200. Specifically, the intra prediction unit 204 generates an intra prediction image by referring to adjacent reference pixels determined according to the intra prediction mode among the reconstructed image blocks (decoded blocks) stored in the memory 203. The intra prediction unit 204 outputs the intra prediction image (and / or adjacent reference pixels) to the prediction image correction unit 207.

[0042] The inter prediction unit 205 performs inter prediction to predict a current image block using a frame-by-frame reconstructed image (decoded image) stored in the memory 106 as a reference image. The inter prediction unit 205 generates an inter prediction image by performing inter prediction according to inter prediction information (motion vector information, etc.) input from the entropy decoding unit 200, and outputs the inter prediction image to the prediction image correction unit 207. The inter prediction unit 205 also outputs a plurality of reference images used for inter prediction to the prediction accuracy evaluation unit 206.

[0043] The prediction accuracy evaluation unit 206 performs an operation similar to that of the prediction accuracy evaluation unit 109 of the image encoding device 1. When the inter prediction unit 205 performs inter prediction using multiple reference images, the prediction accuracy evaluation unit 206 calculates the similarity between the multiple reference images on a pixel-by-pixel basis to evaluate the prediction accuracy of the predicted image and outputs information on the evaluation result to the predicted image correction unit 207.

[0044] The predicted image correction unit 207 performs the same operation as the predicted image correction unit 110 of the image encoding device 1. The predicted image correction unit 207 corrects the edge area of ​​the inter predicted image by filtering using the decoded adjacent block (adjacent reference pixels) corresponding to the intra predicted image. The predicted image correction unit 207 performs filtering when the prediction accuracy evaluated by the prediction accuracy evaluation unit 206 is equal to or less than a first threshold, and does not perform filtering when the prediction accuracy evaluated by the prediction accuracy evaluation unit 206 exceeds the first threshold. When filtering is performed, the predicted image correction unit 207 outputs the inter predicted image after filtering to the synthesis unit 202 as a predicted image, and when filtering is not performed, outputs the inter predicted image as it is to the synthesis unit 202 as a predicted image. Details of the predicted image correction unit 207 will be described later.

[0045] (3. Inter Prediction) Fig. 3 is a diagram showing an example of inter prediction. Fig. 4 is a diagram showing an example of a predicted image generated by inter prediction. As a simple example of inter prediction, a case where bi-prediction used in HEVC, particularly forward and backward prediction (bi-directional prediction), is used will be described.

[0046] As shown in Fig. 3, bi-prediction refers to frames temporally before and after a target frame (current frame). In the example of Fig. 3, prediction of a block in an image of the tth frame is performed by referring to the t-1th frame and the t+1th frame. In motion detection, a portion (block) similar to the target image block is detected within a search range set in the system from within the reference frames of the t-1st and t+1st frames.

[0047] The detected location is the reference image. Information indicating the relative position of the reference image with respect to the target image block is the arrow shown in the figure, and is called a motion vector. The motion vector information is encoded by the image encoding device 1 together with frame information of the reference image by entropy encoding. Meanwhile, the image decoding device 2 detects the reference image based on the motion vector information generated by the image encoding device 1.

[0048] As shown in Figures 3 and 4, reference images 1 and 2 detected by motion detection are similar partial images aligned in a reference frame with respect to a target image block, and therefore are images similar to the target image block (image to be encoded). In the example of Figure 4, the target image block includes a star pattern and a partial circle pattern. Reference image 1 includes a star pattern and a full circle pattern. Reference image 2 includes a star pattern but does not include a circle pattern.

[0049] A predicted image is generated from these reference images 1 and 2. Note that prediction processing generally averages reference images 1 and 2, which have different characteristics but are partially similar, to generate a predicted image with the characteristics of each reference image. However, a predicted image may be generated by combining more advanced processing, for example, signal enhancement processing using a low-pass filter or a high-pass filter. Here, since reference image 1 includes a circular pattern and reference image 2 does not include a circular pattern, when reference images 1 and 2 are averaged to generate a predicted image, the signal of the circular pattern in the predicted image is reduced by half compared to reference image 1.

[0050] The difference between the predicted image obtained from reference images 1 and 2 and the target image block (target image to be encoded) is the prediction residual. In the prediction residual shown in Fig. 4, a large difference occurs only in the shifted portion of the edge of the star pattern and the shifted portion of the circle pattern (hatched portion), but for other parts, prediction is performed with high accuracy and the difference is small (no difference occurs in the example of Fig. 4).

[0051] The parts where no difference occurs (non-edge parts of the star pattern and background parts) are parts where the similarity between reference image 1 and reference image 2 is high, and where a highly accurate prediction has been performed. On the other hand, the parts where a large difference occurs are parts that are specific to each reference image, that is, parts where the similarity between reference image 1 and reference image 2 is significantly low. Therefore, it can be seen that the parts where the similarity between reference image 1 and reference image 2 is significantly low have low prediction accuracy, resulting in a large difference (residual).

[0052] When the prediction residuals, which include a mixture of large and small differences, are orthogonally transformed and the transformation coefficients are degraded by quantization, the degradation of the transformation coefficients propagates throughout the image (block) through inverse quantization and inverse orthogonal transformation. When the prediction residuals (restored prediction residuals) restored by inverse quantization and inverse orthogonal transformation are synthesized with a predicted image to reconstruct the target image block, the degradation of image quality propagates to the non-edge and background parts of the star pattern shown in FIG. 4 where a highly accurate prediction was performed.

[0053] (4. Prediction Accuracy Evaluation Unit) 5 is a diagram showing an example of the configuration of the prediction accuracy evaluation unit 109 in the image encoding device 1. The prediction accuracy evaluation unit 109 evaluates the prediction accuracy of a predicted image on a pixel-by-pixel basis by calculating the similarity between a plurality of reference images used in inter prediction on a pixel-by-pixel basis. Here, an example will be described in which the prediction accuracy evaluation unit 109 evaluates the prediction accuracy for all pixels of the predicted image, but the prediction accuracy evaluation unit 109 does not necessarily need to evaluate the prediction accuracy for all pixels of the predicted image. The prediction accuracy evaluation unit 109 only needs to evaluate the prediction accuracy for at least pixels in the edge regions of the predicted image.

[0054] As shown in FIG. 5, the prediction accuracy evaluation unit 109 includes a difference calculation unit (subtraction unit) 109a, a normalization unit 109b, and an adjustment unit 109c.

[0055] The difference calculation unit 109a calculates the absolute value of the difference value between reference image 1 and reference image 2 in pixel units, and outputs the calculated absolute value of the difference value to the normalization unit 109b. The absolute value of the difference value is an example of a value indicating the similarity. It can be said that the smaller the absolute value of the difference value, the higher the similarity, and the larger the absolute value of the difference value, the lower the similarity. The difference calculation unit 109a may perform a filter process on each reference image and then calculate the absolute value of the difference value. The difference calculation unit 109a may calculate a statistic such as a squared error, and use the statistic as the similarity.

[0056] The normalization unit 109b normalizes the difference value of each pixel input from the difference calculation unit 109a by the absolute value of the difference value of the pixel having the maximum absolute difference value in the block (i.e., the maximum absolute value of the difference values ​​in the block), and outputs a normalized difference value, which is the absolute value of the normalized difference value, to the adjustment unit 109c. In this embodiment, the normalized difference value is used as a weight for weighting the restored prediction residual to be synthesized with the predicted image in the synthesis unit 105, on a pixel-by-pixel basis.

[0057] The adjustment unit 109c adjusts the normalized difference value (weight) input from the normalization unit 109b based on a quantization parameter (Qp) that determines the coarseness of quantization, and outputs the weight. Since the degree of degradation of the restored prediction residual increases as the coarseness of quantization increases, the adjustment unit 109c can weight the restored prediction residual in consideration of the degree of degradation by adjusting the normalized difference value (weight) based on the quantization parameter (Qp).

[0058] The estimated prediction accuracy Rij of each pixel (ij) output by the prediction accuracy evaluation unit 109 can be expressed, for example, as in the following equation (1).

[0059] Rij = 1 - (abs(Xij-Yij) / maxD ×Scale(Qp)) ···(1)

[0060] In equation (1), Xij is the pixel value of pixel ij in reference image 1, Yij is the pixel value of pixel ij in reference image 2, and abs is a function for obtaining the absolute value.

[0061] In addition, in formula (1), maxD is the maximum value of the difference value abs(Xij-Yij) in the block. To obtain maxD, it is necessary to obtain the difference value for all pixels in the block, but to omit this process, the maximum value of an adjacent block that has already been processed may be used instead. For example, if there is a value equal to or greater than this value, maxD may be normalized by clipping it with the maximum value used. Alternatively, maxD may be obtained from the quantization parameter (Qp) using a table that defines the correspondence between the quantization parameter (Qp) and maxD. Alternatively, a fixed value previously specified in the specifications may be used as maxD.

[0062] In addition, in formula (1), Scale(Qp) is a coefficient multiplied according to the quantization parameter (Qp). Scale(Qp) is designed to approach 1.0 when Qp is large and approach 0 when Qp is small, and the degree of approach is adjusted by the system. Alternatively, a fixed value defined in advance in the specifications may be used as Scale(Qp). Furthermore, to simplify processing, Scale(QP) may be a fixed value designed according to the system, such as 1.0.

[0063] The adjustment unit 109c outputs the estimated prediction accuracy Rij. In addition, this Rij may output a weighting adjusted by a sensitivity function designed according to the system. For example, it is possible to adjust the sensitivity by setting abs(Xij-Yij) / maxD×Scale(Qp)=Rij, and adding an offset according to control information such as QP, for example, Rij=Clip(rij+offset,1.0,0.0), instead of Rij=Clip(rij,1.0,0.0). Note that Clip(x,max,min) indicates a process of clipping at max when x exceeds max, and at min when x falls below min.

[0064] The estimated prediction accuracy Rij calculated in this way is a value within the range from 0 to 1.0. Basically, the estimated prediction accuracy Rij approaches 0 when the absolute value of the difference value of pixel ij between reference images is large (i.e., prediction accuracy is low), and approaches 1 when the absolute value of the difference value of pixel ij between reference images is small (i.e., prediction accuracy is high). The prediction accuracy evaluation unit 109 outputs map information consisting of the estimated prediction accuracy Rij of each pixel ij in the block to the predicted image correction unit 110 on a block-by-block basis.

[0065] In addition, the prediction accuracy evaluation unit 109 performs evaluation (calculation of estimated prediction accuracy Rij) only when inter prediction using multiple reference images is applied, and does not need to perform evaluation in other modes, such as unidirectional prediction or intra prediction processing that does not use multiple reference images.

[0066] Moreover, while the prediction accuracy assessment unit 109 in the image encoding device 1 has been described, the prediction accuracy assessment unit 206 in the image decoding device 2 is configured similarly to the prediction accuracy assessment unit 109 in the image encoding device 1. Specifically, the prediction accuracy assessment unit 206 in the image decoding device 2 includes a difference calculation unit 206a, a normalization unit 206b, and an adjustment unit 206c.

[0067] (5. Predictive Image Correction Unit) Fig. 6 is a diagram showing an example of the configuration of the predicted image correction unit 110 in the image encoding device 1. As shown in Fig. 6, the predicted image correction unit 110 includes a continuity evaluation unit 110a, a filter determination unit 110b, and a filter processing unit 110c.

[0068] The continuity evaluation unit 110a evaluates the continuity between the inter-predicted image input from the inter prediction unit 108 and the decoded adjacent block input from the intra prediction unit 107. The decoded adjacent block is a decoded block adjacent to the target image block. The continuity evaluation unit 110a outputs the evaluation result of the continuity between the inter-predicted image and the decoded adjacent block to the filter determination unit 110b.

[0069] Fig. 7 is a diagram showing an example of the operation of the continuity evaluation unit 110a. In the example of Fig. 7, it is assumed that decoded blocks exist on the left and above of a target image block to be encoded. The continuity evaluation unit 110a evaluates the continuity of the block boundary between the inter-predicted image (target image block) and the decoded block in the order from top to bottom (or bottom to top) in pixel units. The continuity evaluation unit 110a calculates the continuity Cont of the block boundary spanning pixels p0 and q0 in Fig. 7, for example, by the following formula (2).

[0070] Cont=|p2-2×p1+p0|+|q2-2×q1+q0| ···(2)

[0071] The units and evaluation indices used in evaluating continuity are not limited to those shown in formula (2), and can be changed to other units and evaluation indices as long as they are predefined in the system. For example, the unit of evaluation of continuity may be the side unit of the block boundary (evaluating the continuity of the left boundary and the continuity of the upper boundary), or the pixel value difference across the boundary (abs(p0-q0)) may be used as the evaluation index of continuity to simplify processing.

[0072] The filter determination unit 110b determines whether or not to perform a filter process based on the prediction accuracy evaluation result input from the prediction accuracy evaluation unit 109 and the continuity evaluation result input from the continuity evaluation unit 110a. The filter determination unit 110b may perform the determination on a pixel-by-pixel basis of the block boundary between the inter-predicted image and the decoded block, or may perform the determination on a side-by-side basis of the block boundary. The filter determination unit 110b may determine to change the strength of the filter (for example, the number of taps or frequency response of the filter) based on the prediction accuracy evaluation result and the continuity evaluation result. Note that when the prediction accuracy evaluation unit 109 evaluates the prediction accuracy on a pixel-by-pixel basis, the filter determination unit 110b may use the average value of the estimated prediction accuracy R of each pixel of the edge region (target region of the filter process) of the inter-predicted image as the prediction accuracy evaluation result.

[0073] The filter determination unit 110b determines to perform a filter process when the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 is equal to or less than the first threshold value α and the continuity evaluated by the continuity evaluation unit 110a is equal to or less than the second threshold value β. On the other hand, when the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 exceeds the first threshold value α and / or the continuity evaluated by the continuity evaluation unit 110a exceeds the second threshold value β, the filter determination unit 110b determines not to perform a filter process. Here, as the threshold values ​​α and β, a system-fixed value may be set, or a variable value may be calculated and set by a function using a quantization parameter (Qp). In addition, when the estimated prediction accuracy R is not normalized, the threshold value α may be calculated and set by a variable value using a function using the maximum value of the estimated prediction accuracy in the target image block.

[0074] When the filter determination unit 110b determines that the filter process is to be performed, the filter processing unit 110c performs the filter process on the edge region of the inter-predicted image. As shown in Fig. 7, the filter processing unit 110c performs correction according to the following formula (3) using the predicted pixels q0, q1, and q2 included in the edge region of the inter-predicted image (block to be coded) and the adjacent decoded reference pixels p0, p1, and p2 input from the intra prediction unit 107, and outputs the corrected predicted pixels q'0, q'1, and q'2.

[0075]

number

[0076] Here, clip(x, max, min) indicates a process of clipping at max when x exceeds max, and at min when x falls below min. Also, tc in formula (3) is an adjustment offset used in the clipping process, and a variable value can be calculated and set by a function using the quantization parameter (Qp).

[0077] Although the prediction image correction unit 110 in the image encoding device 1 has been described above, the prediction image correction unit 207 in the image decoding device 2 is configured similarly to the prediction image correction unit 110 in the image encoding device 1. Specifically, the prediction image correction unit 207 in the image decoding device 2 includes a continuity evaluation unit 207a, a filter determination unit 207b, and a filter processing unit 207c.

[0078] (6. Predictive Image Correction Operation) Fig. 8 is a diagram showing an example of a predictive image correction operation. Here, the predictive image correction operation in the image encoding device 1 will be described, but a similar predictive image correction operation is also performed in the image decoding device 2. The operation flow in Fig. 8 may be performed in units of pixels on the block boundary between the inter-predictive image and the decoded block.

[0079] As shown in FIG. 8, in step S1, the prediction accuracy evaluation unit 109 evaluates the prediction accuracy in edge regions of the inter-predicted image based on the similarity between multiple reference images used by the inter-prediction unit 108 to generate the inter-predicted image.

[0080] In step S2, the continuity evaluation unit 110a evaluates the continuity between the inter predicted image output by the inter prediction unit 108 and adjacent decoded blocks adjacent to the inter predicted image.

[0081] In step S3, the filter determination unit 110b compares the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 with a first threshold value α, and compares the continuity evaluated by the continuity evaluation unit 110a with a second threshold value β.

[0082] If the prediction accuracy is equal to or less than the first threshold α and the continuity is equal to or less than the second threshold β (step S3: YES), the filter determination unit 110b determines to perform filtering. In this case, in step S4, the filtering unit 110c corrects the edge region of the inter-predicted image by filtering using the decoded adjacent blocks (adjacent reference pixels).

[0083] On the other hand, when the prediction accuracy exceeds the first threshold α and / or the continuity exceeds the second threshold β (step S3: NO), the filter determination unit 110b determines not to perform the filtering process. In this case, the filtering process unit 110c does not perform the filtering process.

[0084] (7. Summary of the embodiment) The image encoding device 1 and the image decoding device 2 according to the present embodiment control the filter process based on the evaluation result of the prediction accuracy of the inter prediction. As a result, when the prediction accuracy of the edge region of the inter prediction image is high, it is possible to control not to apply the filter process to this edge region, so that it is possible to prevent the accuracy of the predicted image from being reduced by the filter process. In other words, only when the prediction accuracy of the edge region of the inter prediction image is low, it is possible to apply the filter process to this edge region, so that the filter process can be performed appropriately.

[0085] Furthermore, in this embodiment, the image encoding device 1 and the image decoding device 2 determine whether or not to apply filter processing by a common operation. Specifically, the image encoding device 1 and the image decoding device 2 each evaluate the prediction accuracy of inter prediction, and determine whether or not to apply filter processing based on the evaluation result. This eliminates the need to transmit a flag indicating whether or not to apply filter processing from the image encoding device 1 to the image decoding device 2, thereby preventing an increase in the amount of code due to the flag.

[0086] (8. Modification 1 of the embodiment) In the continuity evaluation and filtering process according to the above embodiment, the intra prediction mode (intra prediction direction) applied to the decoded adjacent block may be taken into consideration. This can improve the accuracy of the continuity evaluation and filtering process. FIG. 9 is a diagram showing a configuration of a predicted image correction unit 110 according to this modified example. As shown in FIG. 9, the intra prediction mode applied to the decoded adjacent block is input from the intra prediction unit 107 to the continuity evaluation unit 110a and the filtering process unit 110c. The continuity evaluation unit 110a evaluates the continuity according to the intra prediction mode applied to the decoded adjacent block. Moreover, the filtering process unit 110c performs filtering process according to the intra prediction mode applied to the decoded adjacent block.

[0087] FIG. 10 is a diagram showing an example of the operation of the continuity evaluation unit 110a and the filter processing unit 110c according to this modified example. As shown in FIG. 10, when the continuity evaluation unit 110a and the filter processing unit 110c perform continuity evaluation and filtering on the edge pixel q0 of the inter-prediction image corresponding to the encoding target block, they calculate p1, p0, q1, and q2 on a line passing through q0 by weighted average along the intra-prediction mode (intra-prediction direction) applied to the decoded adjacent block. Specifically, p1 and p0 on a line passing through q0 are calculated using pixel values ​​in the decoded adjacent block, and q1 and q2 on a line passing through q0 are calculated using pixel values ​​in the inter-prediction image. The continuity evaluation unit 110a evaluates the continuity Cont using the above formula (2) using p1, p0, q1, and q2 calculated in this way. The filter processing unit 110c calculates q'0 using the above formula (3) using p1, p0, q1, and q2 calculated in this way.

[0088] (9. Modification 2 of the embodiment) In the above-described embodiment, an example has been described in which the predicted image correction unit 110 performs filtering on edge regions of an inter predicted image by using decoded adjacent blocks as a correction process for the inter predicted image.

[0089] Furthermore, the correction process is not limited to such filtering, and a weighted average process of an inter-predicted image and an intra-predicted image may be used as the correction process.

[0090] In this modified example, the intra prediction unit 107 and the predicted image correction unit 110 of the image encoding device 1 shown in FIG. 1 constitute a correction unit that performs correction processing on an inter predicted image using a decoded adjacent block that is adjacent to a current image block.

[0091] The intra prediction unit 107 generates an intra prediction image corresponding to the target image block using the decoded adjacent blocks. For example, the intra prediction unit 107 generates an intra prediction image from the decoded adjacent blocks using a predetermined intra prediction mode. The prediction image correction unit 110 performs a weighted average process of the inter prediction image input from the inter prediction unit 108 and the intra prediction image input from the intra prediction unit 107 as a correction process. Note that the transmission side may be configured to select one prediction mode from a plurality of intra prediction mode candidates and generate an intra prediction image. In this case, the entropy coding unit 103 entropy codes an intra prediction mode flag indicating which intra prediction mode was used to generate the intra prediction image on the transmission side.

[0092] Similarly, the intra prediction unit 204 and the predicted image correction unit 207 of the image decoding device 2 shown in Fig. 2 constitute a correction unit that performs a correction process on an inter predicted image using a decoded adjacent block adjacent to a current image block. The intra prediction unit 204 generates an intra predicted image corresponding to the current image block using the decoded adjacent block.

[0093] For example, the intra prediction unit 204 generates an intra prediction image from a decoded adjacent block in a predetermined intra prediction mode. The prediction image correction unit 207 performs a weighted average process as a correction process on the inter prediction image input from the inter prediction unit 205 and the intra prediction image input from the intra prediction unit 204. Note that the intra prediction mode used for generating the intra prediction image may be determined based on an intra prediction mode flag sent from the transmitting side.

[0094] When applying the configuration of this modified example, the “filter process” in the above-described embodiment may be read as “weighted average process.” In addition to determining whether or not to perform such weighted average process, the weights in the weighted average process may be adjusted based on the evaluation results of the prediction accuracy and the evaluation results of the continuity.

[0095] (10. Other embodiments) In the above-mentioned embodiment, bi-prediction has been mainly described as a prediction using multiple reference images. However, a technique called intra block copy can also be applied as a prediction using multiple reference images. In intra block copy, a reference image in the same frame as the current frame is used to predict a target image block of the current frame.

[0096] The image encoding device 1 may be provided by a program that causes a computer to execute each process performed by the image encoding device 1 and a program that causes a computer to execute each process performed by the image decoding device 2. The program may be recorded in a computer-readable medium. The computer-readable medium can be used to install the program in a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM. Furthermore, the image encoding device 1 may be configured as a semiconductor integrated circuit (chip set, SoC) by integrating circuits that execute each process performed by the image decoding device 2. Similarly, the image decoding device 2 may be configured as a semiconductor integrated circuit (chip set, SoC) by integrating circuits that execute each process performed by the image decoding device 2.

[0097] The above describes the embodiments in detail with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes, etc. are possible without departing from the spirit of the invention.

[0098] The entire contents of Japanese Patent Application No. 2018-72451 (filed on April 4, 2018) are incorporated by reference into this specification.

Claims

1. an inter prediction unit that predicts a current image block obtained by dividing an image in units of frames by inter prediction, and generates an inter prediction block corresponding to the current image block; an intra prediction unit that predicts the current image block in a predetermined intra prediction mode and generates an intra prediction block corresponding to the current image block; an evaluation unit that performs evaluation based on a prediction mode for controlling an intra prediction process for a decoded adjacent block that is adjacent to the current image block; a prediction image correction unit that executes a weighted average process of the inter prediction block and the intra prediction block as a correction process, the predicted image correction unit adjusts weights in the weighted averaging process in accordance with a result of the evaluation. Predictive image correction device.

2. A predictive image correction device according to claim 1, Image encoding device.

3. A predictive image correction device according to claim 1, Image decoding device.

4. A method for making a computer function as the predictive image correction device according to claim 1, program.

Images