Prediction Image Correction Device, Image Encoding Device, Image Decoding Device, and Program

Abstract

To enable an appropriate correction of a prediction image.SOLUTION: A prediction image correction device comprises: an inter-prediction part that predicts an object image block obtained by dividing an image in a frame unit by an inter-prediction, and generates an inter-prediction block corresponding to the object image block; an intra-prediction part that predicts the object image block by a predetermined intra-prediction mode, and generates an intra-prediction block corresponding to the object image block; an evaluation part that performs an evaluation on the basis of the prediction mode for controlling an intra-prediction processing against a decoded adjacent block that is adjacent to the object image block; and a prediction image correction part that executes a weighted average processing of the inter-prediction block and the intra-prediction block as a correction processing. The prediction image correction part adjusts the weight in the weighted average processing in accordance with a result of the evaluation.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a predicted image correction device, an image encoding device, an image decoding device, and a program.

Background Art

[0002] In video encoding technology, efficient encoding (compression) is achieved by performing conversion, prediction, etc. in units of blocks obtained by dividing a current image in units of frames (pictures). As prediction methods, there are two types: intra prediction and inter prediction.

[0003] Intra prediction is a method of generating a predicted image by referring to decoded adjacent blocks adjacent to a block to be encoded or decoded (hereinafter referred to as "target image block"). Inter prediction is a method of generating a predicted image by referring to a decoded frame different from the current frame to which the target image block belongs.

[0004] Patent Document 1 describes a predicted image correction device that predicts a target image block by inter prediction to generate an inter prediction image, predicts the target image block by intra prediction to generate an intra prediction image, and performs weighted averaging of the inter prediction image and the intra prediction image.

[0005] Specifically, the predicted image correction device described in Patent Document 1 evaluates the continuity between the inter prediction image corresponding to the target image block and the decoded adjacent blocks adjacent to the target image block, and when it is evaluated as discontinuous, corrects the end region of the inter prediction image by filtering using the decoded adjacent blocks.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Summary of the Invention

[0007] The prediction image correction device described in Patent Document 1 applies filter processing according to the evaluation of the continuity between the inter prediction image and the decoded adjacent block regardless of the prediction accuracy of the inter prediction.

[0008] Therefore, when there is an object boundary at the boundary between the inter prediction image and the decoded adjacent block, even if the prediction accuracy of the end region of the inter prediction image is high, filter processing is applied to this end region, so there is a problem that the accuracy of the prediction image may be reduced by the filter processing.

[0009] Therefore, an object of the present invention is to provide a prediction image correction device, an image encoding device, an image decoding device, and a program that can appropriately correct a prediction image.

[0010] The prediction image correction device according to the present invention includes an inter prediction unit that predicts a target image block obtained by dividing an image in frame units by inter prediction and generates an inter prediction block corresponding to the target image block, an intra prediction unit that predicts the target image block by intra prediction and generates an intra prediction block corresponding to the target image block, and a prediction image correction unit that performs weighted average processing of the inter prediction block and the intra prediction block as correction processing. The gist of the prediction image correction unit is to adjust the weight in the weighted average processing according to a prediction mode that controls intra prediction processing for decoded adjacent blocks adjacent to the target image block. Further, the prediction image correction device according to the first feature predicts a target image block obtained by dividing a current image in frame units using a plurality of reference images to generate a prediction image corresponding to the target image block, a prediction accuracy evaluation unit that evaluates the prediction accuracy of the prediction image based on the similarity between the plurality of reference images used for generating the prediction image, and a correction unit that performs correction processing on the prediction image using decoded adjacent blocks adjacent to the target image block. The gist of the correction unit is to control the correction processing based at least on the evaluation result by the prediction accuracy evaluation unit.

[0011] Note that the prediction using a plurality of reference images is typically represented by dual prediction in inter prediction, but is not limited thereto. For example, the same method is applicable when generating a prediction image by referring to a plurality of images such as the IntraBC mode (intra block copy mode) of an image used in the HEVC encoding method.

[0012] The gist of the image encoding device according to the second feature is to include the prediction image correction device according to the first feature.

[0013] The gist of the image decoding device according to the third feature is to include the prediction image correction device according to the first feature.

[0014] The gist of the program according to the fourth feature is to cause a computer to function as the prediction image correction device according to the first feature.

[0015] According to the present invention, it is possible to provide a prediction image correction device, an image encoding device, an image decoding device, and a program that can appropriately correct a prediction image.

Brief Description of Drawings

[0016]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0017] With reference to the drawings, the image encoding device and the image decoding device according to the embodiment will be described. The image encoding device and the image decoding device according to the embodiment perform encoding and decoding of moving images typified by MPEG. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

[0018] (1. Configuration of Image Encoding Device) FIG. 1 is a diagram showing the configuration of an image encoding device 1 according to the present embodiment. As shown in FIG. 1, the image encoding device 1 includes a block division unit 100, a subtraction unit 101, a conversion unit 102a, a quantization unit 102b, an entropy encoding unit 103, an inverse quantization unit 104a, an inverse conversion unit 104b, a synthesis unit 105, a memory 106, an intra prediction unit 107, an inter prediction unit 108, a prediction accuracy evaluation unit 109, and a prediction image correction unit (correction unit) 110. In the present embodiment, the intra prediction unit 107, the inter prediction unit 108, the prediction accuracy evaluation unit 109, and the prediction image correction unit 110 constitute a prediction image correction device.

[0019] The block division unit 100 divides an input image in units of frames (or pictures) into small block-shaped regions, and outputs the image blocks to the subtraction unit 101 (and the inter prediction unit 108). The size of the image blocks is, for example, 32×32 pixels, 16×16 pixels, 8×8 pixels, or 4×4 pixels, etc. The image blocks are the units for the image encoding device 1 to perform encoding and the units for the image decoding device 2 to perform decoding, and such image blocks are referred to as target image blocks. Note that the shape of the image blocks is not limited to a square, and may be a rectangular shape.

[0020] The subtraction unit 101 calculates a prediction residual indicating the difference in pixel units between the target image block input from the block division unit 100 and the prediction image (prediction image block) corresponding to the target image block. Specifically, the subtraction unit 101 calculates the prediction residual by subtracting the pixel values of the prediction image from the pixel values of each pixel of the block to be encoded, and outputs the calculated prediction residual to the conversion unit 102a. In the present embodiment, the prediction image is corrected by the prediction image correction unit 110 described later, and is input from the prediction image correction unit 110 to the subtraction unit 101.

[0021] The conversion unit 102a and the quantization unit 102b constitute a conversion / quantization unit 102 that performs orthogonal conversion processing and quantization processing in units of blocks.

[0022] The conversion unit 102a performs an orthogonal transformation on the prediction residual input from the subtraction unit 101 to calculate conversion coefficients, and outputs the calculated conversion coefficients to the quantization unit 102b. The orthogonal transformation refers to, for example, the discrete cosine transform (DCT), the discrete sine transform (DST), the Karhunen-Loeve transform (KLT), etc.

[0023] The quantization unit 102b quantizes the conversion coefficients input from the conversion unit 102a using a quantization parameter (Qp) and a quantization matrix to generate quantized conversion coefficients. The quantization parameter (Qp) is a parameter that is commonly applied to each conversion coefficient within a block and determines the coarseness of quantization. The quantization matrix is a matrix having quantization values as elements when quantizing each conversion coefficient. The quantization unit 102b outputs quantization control information, the generated quantized conversion coefficient information, etc. to the entropy encoding unit 103 and the inverse quantization unit 104a.

[0024] The entropy encoding unit 103 performs entropy encoding on the quantized conversion coefficients input from the quantization unit 102b, performs data compression to generate encoded data (bitstream), and outputs the encoded data to the outside of the image encoding device 1. For entropy encoding, Huffman coding, CABAC (Context-based Adaptive Binary Arithmetic Coding), etc. can be used. Note that information regarding prediction is input to the entropy encoding unit 103 from the intra prediction unit 107 and the inter prediction unit 108. The entropy encoding unit 103 also performs entropy encoding of this information.

[0025] The inverse quantization unit 104a and the inverse conversion unit 104b constitute an inverse quantization and inverse conversion unit 104 that performs inverse quantization processing and inverse orthogonal transformation processing in block units.

[0026] The inverse quantization unit 104a performs an inverse quantization process corresponding to the quantization process performed by the quantization unit 102b. Specifically, the inverse quantization unit 104a restores the transform coefficients by inverse quantizing the quantized transform coefficients input from the quantization unit 102b using the quantization parameter (Qp) and the quantization matrix, and outputs the restored transform coefficients to the inverse transform unit 104b.

[0027] The inverse transform unit 104b performs an inverse orthogonal transform process corresponding to the orthogonal transform process performed by the transform unit 102a. For example, when the transform unit 102a performs a discrete cosine transform, the inverse transform unit 104b performs an inverse discrete cosine transform. The inverse transform unit 104b performs an inverse orthogonal transform on the transform coefficients input from the inverse quantization unit 104a to restore the prediction residual, and outputs the restored prediction residual, which is the restored prediction residual, to the synthesis unit 105.

[0028] The synthesis unit 105 synthesizes the restored prediction residual input from the inverse transform unit 104b and the prediction image input from the prediction image correction unit 110 on a pixel-by-pixel basis. The synthesis unit 105 adds the pixel values of the restored prediction residual and the pixel values of the prediction image to reconstruct (decode) the target image block, and outputs the reconstructed image block, which is the reconstructed target image block, to the memory 106. Note that such a reconstructed image block may be referred to as a decoded block.

[0029] The memory 106 stores the reconstructed image block input from the synthesis unit 105. The memory 106 stores the reconstructed image blocks in units of frames.

[0030] The intra prediction unit 107 generates an intra prediction image by referring to decoded adjacent blocks adjacent to the target image block among the reconstructed image blocks (decoded blocks) stored in the memory 106. The intra prediction unit 107 selects an optimal intra prediction mode and performs intra prediction using the selected intra prediction mode. A plurality of intra prediction modes corresponding to a plurality of intra prediction directions are defined in advance. The intra prediction direction refers to the direction of adjacent reference pixels with respect to the target pixel when predicting the target pixel in the target image block by referring to adjacent pixels adjacent to the target image block. That is, the adjacent reference pixels to be used for predicting each pixel in the target image block are determined by the intra prediction mode (intra prediction direction). The intra prediction unit 107 outputs the intra prediction image (and / or adjacent reference pixels) to the prediction image correction unit 110 and outputs information on the selected intra prediction mode to the entropy encoding unit 103.

[0031] The inter prediction unit 108 performs inter prediction for predicting the target image block using the frame-based reconstructed image (decoded image) stored in the memory 106 as a reference image. Specifically, the inter prediction unit 108 calculates a motion vector by a method such as block matching and generates an inter prediction image based on the motion vector. The inter prediction unit 108 selects an optimal inter prediction method from among inter prediction using a plurality of reference images (typically, bidirectional prediction) and inter prediction using one reference image (unidirectional prediction), and performs inter prediction using the selected inter prediction method. In the present embodiment, the case where the inter prediction unit 108 uses bidirectional prediction is mainly described, and the inter prediction unit 108 corresponds to a prediction unit that performs prediction using a plurality of reference images. The inter prediction unit 108 outputs the generated inter prediction image to the prediction image correction unit 110 and outputs information on the selected inter prediction method and the motion vector to the entropy encoding unit 103. Further, the inter prediction unit 108 outputs a plurality of reference images used for inter prediction to the prediction accuracy evaluation unit 109.

[0032] The prediction accuracy evaluation unit 109 evaluates the prediction accuracy in the end region of the inter-predicted image based on the similarity between a plurality of reference images used for generating the inter-predicted image. Specifically, when the inter-prediction unit 108 performs inter-prediction using a plurality of reference images, the similarity between the plurality of reference images used for inter-prediction is calculated for each image portion composed of one or more pixels, and the prediction accuracy of the inter-predicted image is evaluated for each image portion based on such similarity, and information on the evaluation result is output to the predicted image correction unit 110. In the present embodiment, an example will be mainly described in which the prediction accuracy evaluation unit 109 calculates the similarity between a plurality of reference images used for prediction in units of one pixel and evaluates the prediction accuracy of the inter-predicted image in units of one pixel. Details of the prediction accuracy evaluation unit 109 will be described later.

[0033] The prediction image correction unit 110 corrects by weighted-averaging the inter-prediction image input from the inter-prediction unit 108 and the intra-prediction image input from the intra-prediction unit 107. Specifically, the prediction image correction unit 110 corrects the end region of the inter-prediction image by filter processing using the decoded adjacent blocks (adjacent reference pixels) corresponding to the intra-prediction image. Also, the prediction image correction unit 110 controls the filter processing based at least on the evaluation result by the prediction accuracy evaluation unit 109. "Controlling the filter processing" includes controlling whether to perform the filter processing and controlling the filter strength. In the present embodiment, an example in which the prediction image correction unit 110 controls whether to perform the filter processing based at least on the evaluation result by the prediction accuracy evaluation unit 109 will be mainly described. For example, when the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 is equal to or lower than a first threshold value, the prediction image correction unit 110 performs the filter processing, and when the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 exceeds the first threshold value, the prediction image correction unit 110 does not perform the filter processing. When performing the filter processing, the prediction image correction unit 110 outputs the inter-prediction image after the filter processing as the prediction image to the subtraction unit 101 and the synthesis unit 105, and when not performing the filter processing, the prediction image correction unit 110 outputs the inter-prediction image as the prediction image as it is to the subtraction unit 101 and the synthesis unit 105. Note that "prediction accuracy" may be expressed as "inaccuracy of prediction". Regarding "inaccuracy of prediction", for example, the fact that the prediction accuracy is equal to or lower than the first threshold value may mean that the inaccuracy of prediction is high. The fact that the prediction accuracy exceeds the first threshold value may mean that the inaccuracy of prediction is low. Details of the prediction image correction unit 110 will be described later.

[0034] (2. Configuration of Image Decoding Device) FIG. 2 is a diagram showing the configuration of the image decoding apparatus 2 according to the present embodiment. As shown in FIG. 2, the image decoding apparatus 2 includes an entropy decoding unit 200, an inverse quantization unit 201a, an inverse transformation unit 201b, a synthesis unit 202, a memory 203, an intra prediction unit 204, an inter prediction unit 205, a prediction accuracy evaluation unit 206, and a predicted image correction unit 207. In the present embodiment, the intra prediction unit 204, the inter prediction unit 205, the prediction accuracy evaluation unit 206, and the predicted image correction unit 207 constitute a predicted image correction apparatus.

[0035] The entropy decoding unit 200 decodes the encoded data generated by the encoding apparatus 1 and outputs the quantized transform coefficients to the inverse quantization unit 201a. Further, the entropy decoding unit 200 decodes the encoded data, acquires information regarding prediction (intra prediction and inter prediction), and outputs the information regarding prediction to the intra prediction unit 204 and the inter prediction unit 205.

[0036] The inverse quantization unit 201a and the inverse transformation unit 201b constitute an inverse quantization / inverse transformation unit 201 that performs inverse quantization processing and inverse orthogonal transformation processing in block units.

[0037] The inverse quantization unit 201a performs an inverse quantization process corresponding to the quantization process performed by the quantization unit 102b of the image encoding apparatus 1. The inverse quantization unit 201a inverse quantizes the quantized transform coefficients input from the entropy decoding unit 200 using a quantization parameter (Qp) and a quantization matrix to restore the transform coefficients, and outputs the restored transform coefficients to the inverse transformation unit 201b.

[0038] The inverse transformation unit 201b performs an inverse orthogonal transformation process corresponding to the orthogonal transformation process performed by the transformation unit 102a of the image encoding apparatus 1. The inverse transformation unit 201b performs an inverse orthogonal transformation on the transform coefficients input from the inverse quantization unit 201a to restore the prediction residue, and outputs the restored prediction residue (restored prediction residue) to the synthesis unit 202.

[0039] The synthesis unit 202 reconstructs (decodes) the original target image block by synthesizing, on a pixel-by-pixel basis, the prediction residual input from the inverse conversion unit 201b and the prediction image input from the prediction image correction unit 207, and outputs the reconstructed image block to the memory 203.

[0040] The memory 203 stores the reconstructed image block input from the synthesis unit 202. The memory 203 stores the reconstructed image blocks in units of frames. The memory 203 outputs the frame-based reconstructed image (decoded image) to the outside of the image decoding device 2 in the display order.

[0041] The intra prediction unit 204 generates an intra prediction image by performing intra prediction with reference to the reconstructed image blocks stored in the memory 203 according to the intra prediction information (intra prediction mode) input from the entropy decoding unit 200. Specifically, the intra prediction unit 204 generates an intra prediction image by referring to the adjacent reference pixels determined according to the intra prediction mode among the reconstructed image blocks (decoded blocks) stored in the memory 203. The intra prediction unit 204 outputs the intra prediction image (and / or adjacent reference pixels) to the prediction image correction unit 207.

[0042] The inter prediction unit 205 performs inter prediction to predict the target image block using the frame-based reconstructed image (decoded image) stored in the memory 106 as a reference image. The inter prediction unit 205 generates an inter prediction image by performing inter prediction according to the inter prediction information (motion vector information, etc.) input from the entropy decoding unit 200, and outputs the inter prediction image to the prediction image correction unit 207. Also, the inter prediction unit 205 outputs a plurality of reference images used for inter prediction to the prediction accuracy evaluation unit 206.

[0043] The prediction accuracy evaluation unit 206 performs the same operation as the prediction accuracy evaluation unit 109 of the image encoding apparatus 1. When the inter prediction unit 205 performs inter prediction using a plurality of reference images, the prediction accuracy evaluation unit 206 calculates the similarity between the plurality of reference images on a pixel-by-pixel basis, evaluates the prediction accuracy of the predicted image on a pixel-by-pixel basis, and outputs information on the evaluation result to the predicted image correction unit 207.

[0044] The predicted image correction unit 207 performs the same operation as the predicted image correction unit 110 of the image encoding apparatus 1. The predicted image correction unit 207 corrects an end region of the inter predicted image by filtering using a decoded adjacent block (adjacent reference pixel) corresponding to the intra predicted image. When the prediction accuracy evaluated by the prediction accuracy evaluation unit 206 is equal to or less than a first threshold value, the predicted image correction unit 207 performs filtering. When the prediction accuracy evaluated by the prediction accuracy evaluation unit 206 exceeds the first threshold value, the predicted image correction unit 207 does not perform filtering. When performing filtering, the predicted image correction unit 207 outputs the inter predicted image after filtering to the synthesis unit 202 as a predicted image. When not performing filtering, the predicted image correction unit 207 outputs the inter predicted image to the synthesis unit 202 as a predicted image as it is. Details of the predicted image correction unit 207 will be described later.

[0045] (3. Inter Prediction) FIG. 3 is a diagram showing an example of inter prediction. FIG. 4 is a diagram showing an example of a predicted image generated by inter prediction. As a simple example of inter prediction, the case of using dual prediction used in HEVC, particularly forward and backward prediction (bidirectional prediction), will be described.

[0046] As shown in FIG. 3, dual prediction refers to frames temporally before and after the target frame (current frame). In the example of FIG. 3, prediction of a block in the image of the t-th frame is performed with reference to the (t - 1)-th frame and the (t + 1)-th frame. In motion detection, a location (block) similar to the target image block is detected from within the search range set by the system in the reference frames of the (t - 1)-th and (t + 1)-th frames.

[0047] The detected location is the reference image. Information indicating the relative position of the reference image with respect to the target image block is the arrow shown in the figure and is called a motion vector. The information of the motion vector is encoded by entropy encoding together with the frame information of the reference image in the image encoding device 1. On the other hand, the image decoding device 2 detects the reference image based on the information of the motion vector generated by the image encoding device 1.

[0048] As shown in FIGS. 3 and 4, since the reference images 1 and 2 detected by motion detection are similar partial images aligned within the frame to be referred to with respect to the target image block, they become images similar to the target image block (image to be encoded). In the example of FIG. 4, the target image block includes a star pattern and a partial circle pattern. The reference image 1 includes a star pattern and an overall circle pattern. The reference image 2 includes a star pattern but does not include a circle pattern.

[0049] A predicted image is generated from such reference images 1 and 2. Note that the prediction process generally generates a predicted image having the characteristics of each reference image by averaging the reference images 1 and 2 that are partially similar but have different characteristics. However, more advanced processing, for example, signal enhancement processing using a low-pass filter, a high-pass filter, etc. may be used in combination to generate the predicted image. Here, since the reference image 1 includes a circle pattern and the reference image 2 does not include a circle pattern, when the reference images 1 and 2 are averaged to generate a predicted image, the signal of the circle pattern in the predicted image is halved compared to the reference image 1.

[0050] The difference between the predicted image obtained from the reference images 1 and 2 and the target image block (image to be encoded) is the prediction residual. In the prediction residual shown in FIG. 4, a large difference occurs only in the displaced part of the edge of the star pattern and the displaced part of the round pattern (hatched part), but for the other parts, the prediction can be made accurately and the difference is reduced (no difference occurs in the example of FIG. 4).

[0051] The parts where there is no difference (the non-edge parts and the background parts of the star pattern) are the parts with a high similarity between reference image 1 and reference image 2, and are the parts where high-precision prediction has been performed. On the other hand, the parts where a large difference occurs are the parts specific to each reference image, that is, the parts with a significantly low similarity between reference image 1 and reference image 2. Therefore, it can be seen that the parts with a significantly low similarity between reference image 1 and reference image 2 have low prediction accuracy and cause a large difference (residual).

[0052] When the prediction residual in which parts with a large difference and parts with no difference are mixed is orthogonally transformed in this way and deterioration of the transform coefficient due to quantization occurs, such deterioration of the transform coefficient propagates throughout the image (block) through inverse quantization and inverse orthogonal transformation. Then, when the prediction residual (restored prediction residual) restored by inverse quantization and inverse orthogonal transformation is synthesized with the prediction image to reconstruct the target image block, the deterioration of the image quality also propagates to the parts where high-precision prediction has been performed, such as the non-edge parts and the background parts of the star pattern shown in FIG. 4.

[0053] (4. Prediction accuracy evaluation unit) FIG. 5 is a diagram showing an example of the configuration of the prediction accuracy evaluation unit 109 in the image encoding apparatus 1. The prediction accuracy evaluation unit 109 evaluates the prediction accuracy of the prediction image in pixel units by calculating the similarity between a plurality of reference images used for inter prediction in pixel units. Here, an example will be described in which the prediction accuracy evaluation unit 109 evaluates the prediction accuracy for all the pixels of the prediction image, but the prediction accuracy evaluation unit 109 does not necessarily have to evaluate the prediction accuracy for all the pixels of the prediction image. The prediction accuracy evaluation unit 109 may evaluate the prediction accuracy for at least the pixels in the end region of the prediction image.

[0054] As shown in FIG. 5, the prediction accuracy evaluation unit 109 includes a difference calculation unit (subtraction unit) 109a, a normalization unit 109b, and an adjustment unit 109c.

[0055] The difference calculation unit 109a calculates the absolute value of the difference value between the reference image 1 and the reference image 2 in pixel units, and outputs the calculated absolute value of the difference value to the normalization unit 109b. The absolute value of the difference value is an example of a value indicating the similarity. It can be said that the smaller the absolute value of the difference value, the higher the similarity, and the larger the absolute value of the difference value, the lower the similarity. The difference calculation unit 109a may calculate the absolute value of the difference value after performing a filtering process on each reference image. The difference calculation unit 109a may calculate a statistic such as the mean squared error and use such a statistic as the similarity.

[0056] The normalization unit 109b normalizes the difference value of each pixel input from the difference calculation unit 109a by the absolute value of the difference value of the pixel with the largest absolute value of the difference value within the block (that is, the maximum value of the absolute values of the difference values within the block), and outputs the normalized difference value, which is the absolute value of the normalized difference value, to the adjustment unit 109c. In the present embodiment, the normalized difference value is used as a weight for weighting the restoration prediction residual to be synthesized with the predicted image in the synthesis unit 105 in pixel units.

[0057] The adjustment unit 109c adjusts the normalized difference value (weight) input from the normalization unit 109b based on the quantization parameter (Qp) that determines the coarseness of quantization, and outputs this weight. Since the higher the coarseness of quantization, the higher the degree of degradation of the restoration prediction residual, the adjustment unit 109c can perform weighting of the restoration prediction residual in consideration of the degree of degradation by adjusting the normalized difference value (weight) based on the quantization parameter (Qp).

[0058] The estimated prediction accuracy Rij of each pixel (ij) output by the prediction accuracy evaluation unit 109 can be expressed, for example, by the following formula (1).

[0059] Rij = 1 - (abs(Xij - Yij) / maxD × Scale(Qp)) ···(1)

[0060] In formula (1), Xij is the pixel value of pixel ij of the reference image 1, Yij is the pixel value of pixel ij of the reference image 2, and abs is a function for obtaining the absolute value.

[0061] Also, in Equation (1), maxD is the maximum value of the difference values abs(Xij - Yij) within the block. To obtain maxD, it is necessary to calculate the difference values for all pixels within the block. However, to omit this process, it may be substituted with the maximum value of adjacent blocks that have already been processed. For example, if a value greater than that exists, normalization of maxD may be performed by clipping with the maximum value used. Alternatively, maxD may be obtained from the quantization parameter (Qp) using a table that defines the correspondence between the quantization parameter (Qp) and maxD. Alternatively, a fixed value specified in advance in the specification may be used as maxD.

[0062] Also, in Equation (1), Scale(Qp) is a coefficient that is multiplied according to the quantization parameter (Qp). Scale(Qp) is designed to approach 1.0 when Qp is large and approach 0 when Qp is small, and the degree is adjusted by the system. Alternatively, a fixed value specified in advance in the specification may be used as Scale(Qp). Furthermore, to simplify the process, Scale(QP) may be set to a fixed value designed according to the system, such as 1.0.

[0063] The adjustment unit 109c outputs the estimated prediction accuracy Rij. Also, this Rij may output weighted values adjusted by a sensitivity function designed according to the system. For example, instead of simply setting abs(Xij - Yij) / maxD × Scale(Qp) = Rij and Rij = Clip(rij, 1.0, 0.0), the sensitivity may be adjusted by adding an offset according to control information such as QP, e.g., Rij = Clip(rij + offset, 1.0, 0.0). Note that Clip(x, max, min) indicates a process of clipping x to max if x exceeds max and to min if x is less than min.

[0064] The estimated prediction accuracy Rij calculated in this way is a value within the range from 0 to 1.0. Basically, the estimated prediction accuracy Rij approaches 0 when the absolute value of the difference value of the pixel ij between the reference images is large (that is, the prediction accuracy is low), and approaches 1 when the absolute value of the difference value of the pixel ij between the reference images is small (that is, the prediction accuracy is high). The prediction accuracy evaluation unit 109 outputs map information composed of the estimated prediction accuracy Rij of each pixel ij in the block to the prediction image correction unit 110 in block units.

[0065] Note that the prediction accuracy evaluation unit 109 performs evaluation (calculation of the estimated prediction accuracy Rij) only when applying inter prediction using a plurality of reference images, and in other modes, for example, in uni - directional prediction or intra prediction processing without using a plurality of reference images, evaluation may not be performed.

[0066] Also, although the prediction accuracy evaluation unit 109 in the image encoding apparatus 1 has been described, the prediction accuracy evaluation unit 206 in the image decoding apparatus 2 is configured in the same manner as the prediction accuracy evaluation unit 109 in the image encoding apparatus 1. Specifically, the prediction accuracy evaluation unit 206 in the image decoding apparatus 2 includes a difference calculation unit 206a, a normalization unit 206b, and an adjustment unit 206c.

[0067] (5. Prediction Image Correction Unit) FIG. 6 is a diagram showing an example of the configuration of the prediction image correction unit 110 in the image encoding apparatus 1. As shown in FIG. 6, the prediction image correction unit 110 includes a continuity evaluation unit 110a, a filter determination unit 110b, and a filter processing unit 110c.

[0068] The continuity evaluation unit 110a evaluates the continuity between the inter - prediction image input from the inter - prediction unit 108 and the decoded adjacent block input from the intra - prediction unit 107. The decoded adjacent block is a decoded block adjacent to the target image block. The continuity evaluation unit 110a outputs the evaluation result of the continuity between the inter - prediction image and the decoded adjacent block to the filter determination unit 110b.

[0069] FIG. 7 is a diagram showing an operation example of the continuity evaluation unit 110a. In the example of FIG. 7, it is assumed that there are decoded blocks on the left side and the upper side of the image block to be encoded. The continuity evaluation unit 110a evaluates the continuity pixel by pixel in the order from top to bottom (or from bottom to top) for the block boundaries between the inter-predicted image (target image block) and the decoded blocks. The continuity evaluation unit 110a calculates, for example, the continuity Cont of the block boundary straddling pixels p0 and q0 in FIG. 7 by the following formula (2).

[0070] Cont = |p2 - 2×p1 + p0| + |q2 - 2×q1 + q0| ···(2)

[0071] Note that the unit and evaluation index for evaluating continuity are not limited to those shown in formula (2), and can be changed to other units and evaluation indexes if they are predefined in the system. For example, the unit of continuity evaluation may be the side unit of the block boundary (performed for the continuity evaluation of the left boundary and the continuity evaluation of the upper boundary), or the pixel value difference across the boundary (abs(p0 - q0)) may be used as the evaluation index of continuity for simplifying the processing.

[0072] The filter determination unit 110b determines whether to perform filter processing based on the prediction accuracy evaluation result input from the prediction accuracy evaluation unit 109 and the continuity evaluation result input from the continuity evaluation unit 110a. The filter determination unit 110b may perform the determination pixel by pixel for the block boundary between the inter-predicted image and the decoded block, or may perform the determination in the side unit of the block boundary. The filter determination unit 110b may determine to change the strength of the filter (for example, the number of taps or the frequency response of the filter) based on the prediction accuracy evaluation result and the continuity evaluation result. When the prediction accuracy evaluation unit 109 evaluates the prediction accuracy pixel by pixel, the filter determination unit 110b may use the average value of the estimated prediction accuracy R of each pixel in the end region (the region to be filtered) of the inter-predicted image as the prediction accuracy evaluation result.

[0073] When the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 is less than or equal to the first threshold α and the continuity evaluated by the continuity evaluation unit 110a is less than or equal to the second threshold β, the filter determination unit 110b determines that filter processing is to be performed. On the other hand, when the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 exceeds the first threshold α and / or the continuity evaluated by the continuity evaluation unit 110a exceeds the second threshold β, the filter determination unit 110b determines that no filter processing is to be performed. Here, as the thresholds α and β, system-fixed values may be set, or variable values may be calculated and set by a function using the quantization parameter (Qp). Further, when the estimated prediction accuracy R is not normalized, the threshold α may be calculated and set as a variable value by a function using the maximum value of the estimated prediction accuracy within the target image block.

[0074] When it is determined by the filter determination unit 110b that filter processing is to be performed, the filter processing unit 110c performs filter processing on the end region of the inter-predicted image. As shown in FIG. 7, the filter processing unit 110c corrects the prediction pixels q0, q1, q2 included in the end region of the inter-predicted image (encoding target block) and the adjacent decoded reference pixels p0, p1, p2 input from the intra-prediction unit 107 using the following formula (3), and outputs the corrected prediction pixels q'0, q'1, q'2.

[0075]

Equation

[0076] Here, clip(x, max, min) indicates a process of clipping x to max when x exceeds max and to min when x is less than min. Also, tc in formula (3) is an adjustment offset for the process used for clipping, and variable values can be calculated and set by a function using the quantization parameter (Qp).

[0077] Although the prediction image correction unit 110 in the image encoding device 1 has been described, the prediction image correction unit 207 in the image decoding device 2 is configured in the same manner as the prediction image correction unit 110 in the image encoding device 1. Specifically, the prediction image correction unit 207 in the image decoding device 2 includes a continuity evaluation unit 207a, a filter determination unit 207b, and a filter processing unit 207c.

[0078] (6. Prediction Image Correction Operation) FIG. 8 is a diagram showing an example of the prediction image correction operation. Here, the prediction image correction operation in the image encoding device 1 will be described, but the same prediction image correction operation is also performed in the image decoding device 2. The operation flow in FIG. 8 may be performed in units of pixels at the block boundary between the inter prediction image and the decoded block.

[0079] As shown in FIG. 8, in step S1, the prediction accuracy evaluation unit 109 evaluates the prediction accuracy in the end region of the inter prediction image based on the similarity between a plurality of reference images used by the inter prediction unit 108 to generate the inter prediction image.

[0080] In step S2, the continuity evaluation unit 110a evaluates the continuity between the inter prediction image output by the inter prediction unit 108 and the decoded adjacent block adjacent to the inter prediction image.

[0081] In step S3, the filter determination unit 110b compares the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 with a first threshold value α, and compares the continuity evaluated by the continuity evaluation unit 110a with a second threshold value β.

[0082] When the prediction accuracy is less than or equal to the first threshold value α and the continuity is less than or equal to the second threshold value β (step S3: YES), the filter determination unit 110b determines to perform filter processing. In such a case, in step S4, the filter processing unit 110c corrects the end region of the inter prediction image by filter processing using the decoded adjacent block (adjacent reference pixel).

[0083] On the other hand, when the prediction accuracy exceeds the first threshold α and / or the continuity exceeds the second threshold β (step S3: NO), the filter determination unit 110b determines not to perform the filter process. In such a case, the filter processing unit 110c does not perform the filter process.

[0084] (7. Summary of the Embodiment) The image encoding device 1 and the image decoding device 2 according to the present embodiment control the filter process based on the evaluation result of the prediction accuracy of the inter prediction. Thereby, when the prediction accuracy of the end region of the inter prediction image is high, it is possible to control not to apply the filter process to this end region, so that a decrease in the accuracy of the prediction image due to the filter process can be prevented. In other words, since the filter process can be applied only when the prediction accuracy of the end region of the inter prediction image is low, the filter process can be appropriately performed.

[0085] Also, in the present embodiment, the presence or absence of applying the filter process is determined by a common operation in the image encoding device 1 and the image decoding device 2. Specifically, the evaluation of the prediction accuracy of the inter prediction is performed in each of the image encoding device 1 and the image decoding device 2, and the presence or absence of applying the filter process is determined based on the evaluation result. Thereby, since it is not necessary to transmit a flag indicating the presence or absence of applying the filter process from the image encoding device 1 to the image decoding device 2, an increase in the code amount due to the flag can be prevented.

[0086] (8. First Modification Example of the Embodiment) In the continuity evaluation and filtering process according to the above-described embodiment, the intra prediction mode (intra prediction direction) applied to the decoded adjacent blocks may be considered. Thereby, the accuracy of the continuity evaluation and the filtering process can be improved. FIG. 9 is a diagram showing the configuration of the prediction image correction unit 110 according to this modification example. As shown in FIG. 9, the intra prediction mode applied to the decoded adjacent blocks is input from the intra prediction unit 107 to the continuity evaluation unit 110a and the filtering process unit 110c. The continuity evaluation unit 110a evaluates the continuity according to the intra prediction mode applied to the decoded adjacent blocks. Further, the filtering process unit 110c performs a filtering process according to the intra prediction mode applied to the decoded adjacent blocks.

[0087] FIG. 10 is a diagram showing an operation example of the continuity evaluation unit 110a and the filtering process unit 110c according to this modification example. As shown in FIG. 10, when evaluating the continuity and performing the filtering process on the end pixel q0 of the inter prediction image corresponding to the block to be encoded, the continuity evaluation unit 110a and the filtering process unit 110c calculate p1, p0, q1, and q2 on the straight line passing through q0 by weighted average along the intra prediction mode (intra prediction direction) applied to the decoded adjacent blocks. Specifically, using the pixel values in the decoded adjacent blocks, p1 and p0 on the straight line passing through q0 are calculated, and using the pixel values in the inter prediction image, q1 and q2 on the straight line passing through q0 are calculated. The continuity evaluation unit 110a evaluates the continuity Cont by the above formula (2) using p1, p0, q1, and q2 calculated in this way. The filtering process unit 110c calculates q'0 by the above formula (3) using p1, p0, q1, and q2 calculated in this way.

[0088] (9. Modification Example 2 of the Embodiment) In the above-described embodiment, an example in which the prediction image correction unit 110 performs a filtering process on the end region of the inter prediction image using the decoded adjacent blocks as a correction process for the inter prediction image has been described.

[0089] Furthermore, the correction process is not limited to such filtering processes, and weighted average processing of the inter-predicted image and the intra-predicted image may be used as the correction process.

[0090] In this modification example, the intra prediction unit 107 and the prediction image correction unit 110 of the image encoding apparatus 1 shown in FIG. 1 constitute a correction unit that performs correction processing on the inter-predicted image using the decoded adjacent blocks adjacent to the target image block.

[0091] The intra prediction unit 107 generates an intra-predicted image corresponding to the target image block using the decoded adjacent blocks. For example, the intra prediction unit 107 generates an intra-predicted image from the decoded adjacent blocks in a predetermined intra prediction mode. The prediction image correction unit 110 performs weighted average processing on the inter-predicted image input from the inter prediction unit 108 and the intra-predicted image input from the intra prediction unit 107 as the correction process. Note that, on the transmission side, one prediction mode may be selected from a plurality of intra prediction mode candidates to generate an intra-predicted image. In that case, the entropy encoding unit 103 entropy-encodes an intra prediction mode flag indicating which intra prediction mode was used to generate the intra-predicted image on the transmission side.

[0092] Similarly, the intra prediction unit 204 and the prediction image correction unit 207 of the image decoding apparatus 2 shown in FIG. 2 constitute a correction unit that performs correction processing on the inter-predicted image using the decoded adjacent blocks adjacent to the target image block. The intra prediction unit 204 generates an intra-predicted image corresponding to the target image block using the decoded adjacent blocks.

[0093] For example, the intra prediction unit 204 generates an intra prediction image from the decoded adjacent blocks in a predetermined intra prediction mode. The prediction image correction unit 207 performs weighted average processing on the inter prediction image input from the inter prediction unit 205 and the intra prediction image input from the intra prediction unit 204 as correction processing. Note that the intra prediction mode used for generating the intra prediction image may be determined based on the intra prediction mode flag sent from the transmission side.

[0094] When applying the configuration of this modification example, the "filter processing" in the above-described embodiment may be read as "weighted average processing". Further, not only the determination of whether or not to perform such weighted average processing, but also the weight in the weighted average processing may be adjusted based on the evaluation result of the prediction accuracy and the evaluation result of the continuity.

[0095] (10. Other Embodiments) In the above-described embodiment, bi-prediction has been mainly described as prediction using a plurality of reference images. However, a technique called intra block copy can also be applied as prediction using a plurality of reference images. In intra block copy, a reference image within the same frame as the current frame is used for predicting the target image block of the current frame.

[0096] A program for causing a computer to execute each process performed by the image encoding device 1 and a program for causing a computer to execute each process performed by the image decoding device 2 may be provided. Further, the program may be recorded on a computer-readable medium. By using a computer-readable medium, it is possible to install the program in a computer. Here, the computer-readable medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM. Further, a circuit that executes each process performed by the image encoding device 1 may be integrated, and the image encoding device 1 may be configured as a semiconductor integrated circuit (chipset, SoC). Similarly, a circuit that executes each process performed by the image decoding device 2 may be integrated, and the image decoding device 2 may be configured as a semiconductor integrated circuit (chipset, SoC).

[0097] As described above, the embodiments have been described in detail with reference to the drawings. However, the specific configuration is not limited to the above, and various design changes and the like can be made without departing from the gist.

[0098] Note that the entire contents of Japanese Patent Application No. 2018-72451 (filed on April 4, 2018) are incorporated herein by reference.

Claims

1. An inter prediction unit that predicts a target image block obtained by dividing an image in frame units by inter prediction and generates an inter prediction block corresponding to the target image block; An intra prediction unit that predicts the target image block by a predetermined intra prediction mode and generates an intra prediction block corresponding to the target image block; An evaluation unit that performs an evaluation based on a prediction mode for controlling an intra prediction process for a decoded adjacent block adjacent to the target image block; A predicted image correction unit that executes weighted average processing of the inter prediction block and the intra prediction block as correction processing, and The predicted image correction unit adjusts weights in the weighted average processing according to the result of the evaluation. A predicted image correction apparatus.

2. An image encoding apparatus comprising the predicted image correction apparatus according to claim 1.

3. An image decoding apparatus comprising the predicted image correction apparatus according to claim 1.

4. A program for causing a computer to function as the predicted image correction apparatus according to claim 1. ​ ​ ​

Images