Predictive image correction device, image coding device, image decoding device, and program

Abstract

To enable appropriate correction of a prediction image.SOLUTION: A prediction image correction device comprises: an inter-prediction unit for predicting a block of an object image obtained by dividing an image in a frame unit through inter-prediction to generate an inter-prediction block; a correction unit for performing correction processing on the inter-prediction block, and a calculation unit. Only when the inter-prediction unit performs the inter-prediction by using two reference image blocks temporally preceding and following the current frame, the calculation unit calculates a statistic of an error between the two reference image blocks for each image part which is a part of the block and comprises a plurality of pixels, and the correction unit performs correction processing only on image parts for which the statistic of the error exceeds a prescribed threshold, out of the blocks of prediction images.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a predicted image correction apparatus, an image encoding apparatus, an image decoding apparatus, and a program.

Background Art

[0002] In video encoding technology, efficient encoding (compression) is achieved by performing conversion, prediction, etc. in units of blocks obtained by dividing a current image in units of frames (pictures). As prediction methods, there are two types: intra prediction and inter prediction.

[0003] Intra prediction is a method of generating a predicted image by referring to decoded adjacent blocks adjacent to a block to be encoded or decoded (hereinafter referred to as "target image block"). Inter prediction is a method of generating a predicted image by referring to a decoded frame different from the current frame to which the target image block belongs.

[0004] Patent Document 1 describes a predicted image correction apparatus that predicts a target image block by inter prediction to generate an inter prediction image, predicts the target image block by intra prediction to generate an intra prediction image, and weights and averages the inter prediction image and the intra prediction image.

[0005] Specifically, the predicted image correction apparatus described in Patent Document 1 evaluates the continuity between an inter prediction image corresponding to a target image block and decoded adjacent blocks adjacent to the target image block, and when it is evaluated as discontinuous, corrects an end region of the inter prediction image by filtering using the decoded adjacent blocks.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Summary of the Invention

[0007] The predictive image correction device described in Patent Document 1 applies filtering processing according to the evaluation of the continuity between the interpredicted image and the decoded adjacent block, regardless of the prediction accuracy of the interprediction.

[0008] Therefore, when there is an object boundary at the boundary between the interpredicted image and the decoded adjacent block, even if the prediction accuracy is high in the edge regions of the interpredicted image, applying filtering to these edge regions can lead to a decrease in the accuracy of the predicted image.

[0009] Therefore, the present invention aims to provide a predictive image correction device, an image encoding device, an image decoding device, and a program that can appropriately correct predicted images.

[0010] The predictive image correction device according to the present invention comprises: an inter-prediction unit that predicts target image blocks obtained by dividing an image on a frame-by-frame basis using inter-prediction and generates inter-prediction blocks corresponding to the target image blocks; an intra-prediction unit that predicts the target image blocks using intra-prediction and generates intra-prediction blocks corresponding to the target image blocks; and a predictive image correction unit that performs a weighted averaging process of the inter-prediction blocks and the intra-prediction blocks as a correction process. The gist of the predictive image correction unit is that it adjusts the weights in the weighted averaging process according to a prediction mode that controls the intra-prediction process for decoded adjacent blocks adjacent to the target image blocks. Furthermore, the predictive image correction device according to the first feature comprises: a prediction unit that generates a predicted image corresponding to a target image block by predicting the target image block obtained by dividing the current image in frame units using a plurality of reference images; a prediction accuracy evaluation unit that evaluates the prediction accuracy of the predicted image based on the similarity between the plurality of reference images used to generate the predicted image; and a correction unit that performs correction processing on the predicted image using a decoded adjacent block adjacent to the target image block, wherein the correction unit controls the correction processing based at least on the evaluation result by the prediction accuracy evaluation unit.

[0011] While biprediction in interpretation is a typical example of prediction using multiple reference images, it is not limited to this. For example, similar methods can be applied when generating prediction images by referencing multiple images, such as in the IntraBC mode (intrablock copy mode) of images used in the HEVC coding scheme.

[0012] The image encoding device relating to the second feature is characterized by comprising a predictive image correction device relating to the first feature.

[0013] The image decoding device relating to the third feature is characterized by comprising a predictive image correction device relating to the first feature.

[0014] The program relating to the fourth feature essentially aims to make the computer function as a predictive image correction device relating to the first feature.

[0015] According to the present invention, it is possible to provide a predictive image correction device, an image encoding device, an image decoding device, and a program that can appropriately correct predicted images. [Brief explanation of the drawing]

[0016] [Figure 1] This diagram shows the configuration of an image encoding device according to an embodiment. [Figure 2] This figure shows the configuration of an image decoding device according to an embodiment. [Figure 3] This figure shows an example of interpretation prediction. [Figure 4] This figure shows an example of a predicted image generated by interpretation. [Figure 5] This figure shows an example of the configuration of the prediction accuracy evaluation unit according to the embodiment. [Figure 6] This figure shows an example of the configuration of the predictive image correction unit according to the embodiment. [Figure 7] This figure shows an example of the operation of the continuity evaluation unit according to the embodiment. [Figure 8] This figure shows an example of the predictive image correction operation according to the embodiment. [Figure 9] This diagram shows the configuration of the predictive image correction unit according to an example of a modified embodiment. [Figure 10] This figure shows an example of the operation of the continuity evaluation unit and filter processing unit according to a modified embodiment. [Modes for carrying out the invention]

[0017] An image encoding apparatus and an image decoding apparatus according to an embodiment will be described with reference to the drawings. The image encoding apparatus and image decoding apparatus according to the embodiment perform encoding and decoding of video, such as MPEG. In the following drawings, identical or similar parts are denoted by the same or similar reference numerals.

[0018] (1. Configuration of Image Encoding Device) FIG. 1 is a diagram showing the configuration of an image encoding device 1 according to the present embodiment. As shown in FIG. 1, the image encoding device 1 includes a block division unit 100, a subtraction unit 101, a conversion unit 102a, a quantization unit 102b, an entropy encoding unit 103, an inverse quantization unit 104a, an inverse conversion unit 104b, a synthesis unit 105, a memory 106, an intra prediction unit 107, an inter prediction unit 108, a prediction accuracy evaluation unit 109, and a prediction image correction unit (correction unit) 110. In the present embodiment, the intra prediction unit 107, the inter prediction unit 108, the prediction accuracy evaluation unit 109, and the prediction image correction unit 110 constitute a prediction image correction device.

[0019] The block division unit 100 divides an input image in units of frames (or pictures) into small block-shaped regions, and outputs the image blocks to the subtraction unit 101 (and the inter prediction unit 108). The size of the image blocks is, for example, 32×32 pixels, 16×16 pixels, 8×8 pixels, or 4×4 pixels, etc. The image blocks are units for the image encoding device 1 to perform encoding and units for the image decoding device 2 to perform decoding, and such image blocks are referred to as target image blocks. Note that the shape of the image blocks is not limited to a square, and may be a rectangular shape.

[0020] The subtraction unit 101 calculates a prediction residual indicating the difference in pixel units between the target image block input from the block division unit 100 and the prediction image (prediction image block) corresponding to the target image block. Specifically, the subtraction unit 101 calculates the prediction residual by subtracting the pixel values of the prediction image from the pixel values of each pixel of the encoding target block, and outputs the calculated prediction residual to the conversion unit 102a. In the present embodiment, the prediction image is corrected by a prediction image correction unit 110 described later, and is input from the prediction image correction unit 110 to the subtraction unit 101.

[0021] The conversion unit 102a and the quantization unit 102b constitute a conversion / quantization unit 102 that performs orthogonal conversion processing and quantization processing in block units.

[0022] The transformation unit 102a performs an orthogonal transformation on the predicted residual input from the subtraction unit 101 to calculate transformation coefficients, and outputs the calculated transformation coefficients to the quantization unit 102b. Orthogonal transformations include, for example, the discrete cosine transform (DCT), discrete sine transform (DST), and Karhunen-Loeve transform (KLT).

[0023] The quantization unit 102b quantizes the conversion coefficients input from the conversion unit 102a using a quantization parameter (Qp) and a quantization matrix to generate quantized conversion coefficients. The quantization parameter (Qp) is a parameter applied in common to each conversion coefficient within a block and determines the coarseness of the quantization. The quantization matrix is ​​a matrix whose elements are the quantized values ​​obtained when quantizing each conversion coefficient. The quantization unit 102b outputs quantization control information, generated quantized conversion coefficient information, etc., to the entropy coding unit 103 and the inverse quantization unit 104a.

[0024] The entropy coding unit 103 performs entropy coding on the quantization conversion coefficients input from the quantization unit 102b, compresses the data to generate coded data (bitstream), and outputs the coded data to the outside of the image coding device 1. For entropy coding, Huffman coding or CABAC (Context-based Adaptive Binary Arithmetic Coding) can be used. The entropy coding unit 103 also receives prediction information from the intra-prediction unit 107 and the inter-prediction unit 108. The entropy coding unit 103 also performs entropy coding on this information.

[0025] The inverse quantization unit 104a and the inverse transformation unit 104b constitute the inverse quantization / inverse transformation unit 104, which performs inverse quantization processing and inverse orthogonal transformation processing on a block-by-block basis.

[0026] The inverse quantization unit 104a performs inverse quantization processing corresponding to the quantization processing performed by the quantization unit 102b. Specifically, the inverse quantization unit 104a recovers the quantization conversion coefficients input from the quantization unit 102b by inverse quantization using the quantization parameter (Qp) and the quantization matrix, and outputs the recovered conversion coefficients to the inverse conversion unit 104b.

[0027] The inverse transform unit 104b performs an inverse orthogonal transform process corresponding to the orthogonal transform process performed by the transform unit 102a. For example, if the transform unit 102a performs a discrete cosine transform, the inverse transform unit 104b performs an inverse discrete cosine transform. The inverse transform unit 104b performs an inverse orthogonal transform on the transformation coefficients input from the inverse quantization unit 104a to restore the predicted residuals, and outputs the restored predicted residuals to the synthesis unit 105.

[0028] The synthesis unit 105 synthesizes the restored predicted residuals input from the inverse transform unit 104b with the predicted image input from the predicted image correction unit 110 on a pixel-by-pixel basis. The synthesis unit 105 adds each pixel value of the restored predicted residuals to each pixel value of the predicted image to reconstruct (decode) the target image block, and outputs the reconstructed image block, which is the reconstructed target image block, to the memory 106. Such a reconstructed image block is sometimes referred to as a decoded block.

[0029] Memory 106 stores the reconstructed image blocks input from the synthesis unit 105. Memory 106 stores the reconstructed image blocks on a frame-by-frame basis.

[0030] The intra-prediction unit 107 generates an intra-prediction image by referring to the decoded adjacent blocks adjacent to the target image block from among the reconstructed image blocks (decoded blocks) stored in the memory 106. The intra-prediction unit 107 selects the optimal intra-prediction mode and performs intra-prediction using the selected intra-prediction mode. Multiple intra-prediction modes corresponding to multiple intra-prediction directions are predetermined. The intra-prediction direction refers to the direction of the adjacent reference pixels relative to the target pixel when predicting the target pixel in the target image block by referring to adjacent pixels adjacent to the target image block. In other words, the intra-prediction mode (intra-prediction direction) determines the adjacent reference pixels to be used for predicting each pixel in the target image block. The intra-prediction unit 107 outputs the intra-prediction image (and / or adjacent reference pixels) to the prediction image correction unit 110 and outputs information of the selected intra-prediction mode to the entropy coding unit 103.

[0031] The interpretation unit 108 performs interpretation, predicting target image blocks using frame-by-frame reconstructed images (decoded images) stored in memory 106 as reference images. Specifically, the interpretation unit 108 calculates motion vectors using methods such as block matching and generates interpretation images based on the motion vectors. The interpretation unit 108 selects the optimal interpretation method from among interpretation using multiple reference images (typically biprediction) and interpretation using one reference image (unidirectional prediction), and performs interpretation using the selected interpretation method. In this embodiment, the case in which the interpretation unit 108 uses biprediction will be mainly described, and the interpretation unit 108 corresponds to a prediction unit that performs prediction using multiple reference images. The interpretation unit 108 outputs the generated interpretation image to the prediction image correction unit 110, and also outputs information regarding the selected interpretation method and motion vectors to the entropy coding unit 103. The interpretation unit 108 also outputs the multiple reference images used for interpretation to the prediction accuracy evaluation unit 109.

[0032] The prediction accuracy evaluation unit 109 evaluates the prediction accuracy in the edge region of the inter-predicted image based on the similarity between multiple reference images used to generate the inter-predicted image. Specifically, when the inter-prediction unit 108 performs inter-prediction using multiple reference images, it calculates the similarity between the multiple reference images used for inter-prediction for each image portion consisting of one or more pixels, evaluates the prediction accuracy of the inter-predicted image for each image portion based on this similarity, and outputs the evaluation result information to the prediction image correction unit 110. In this embodiment, an example in which the prediction accuracy evaluation unit 109 calculates the similarity between multiple reference images used for prediction on a pixel-by-pixel basis and evaluates the prediction accuracy of the inter-predicted image on a pixel-by-pixel basis will be mainly described. Details of the prediction accuracy evaluation unit 109 will be described later.

[0033] The predictive image correction unit 110 corrects the image by weighting and averaging the inter-predicted image input from the inter-prediction unit 108 and the intra-predicted image input from the intra-prediction unit 107. Specifically, the predictive image correction unit 110 corrects the edge regions of the inter-predicted image by filtering using the decoded adjacent blocks (adjacent reference pixels) corresponding to the intra-predicted image. The predictive image correction unit 110 also controls the filtering process based at least on the evaluation results of the prediction accuracy evaluation unit 109. "Controlling the filtering process" includes controlling whether or not to perform filtering and controlling the filter intensity. In this embodiment, an example of how the predictive image correction unit 110 controls whether or not to perform filtering based at least on the evaluation results of the prediction accuracy evaluation unit 109 will be mainly described. For example, the predictive image correction unit 110 performs filtering if the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 is below a first threshold, and does not perform filtering if the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 exceeds the first threshold. The predictive image correction unit 110 outputs the filtered interpredicted image as the predicted image to the subtraction unit 101 and the synthesis unit 105 when filtering is performed, and outputs the interpredicted image as the predicted image as is to the subtraction unit 101 and the synthesis unit 105 when filtering is not performed. Note that "prediction accuracy" may also be expressed as "inaccuracy of prediction". Regarding "inaccuracy of prediction", for example, a prediction accuracy of less than or equal to the first threshold may mean that the prediction is inaccurate. A prediction accuracy exceeding the first threshold may mean that the prediction is inaccurate. Details of the predictive image correction unit 110 will be described later.

[0034] (2. Configuration of the image decoding device) Figure 2 shows the configuration of the image decoding device 2 according to this embodiment. As shown in Figure 2, the image decoding device 2 includes an entropy decoding unit 200, an inverse quantization unit 201a, an inverse transformation unit 201b, a synthesis unit 202, a memory 203, an intra prediction unit 204, an inter prediction unit 205, a prediction accuracy evaluation unit 206, and a prediction image correction unit 207. In this embodiment, the intra prediction unit 204, the inter prediction unit 205, the prediction accuracy evaluation unit 206, and the prediction image correction unit 207 constitute a prediction image correction device.

[0035] The entropy decoding unit 200 decodes the encoded data generated by the encoding device 1 and outputs the quantization conversion coefficients to the inverse quantization unit 201a. The entropy decoding unit 200 also decodes the encoded data, obtains information regarding predictions (intra-prediction and inter-prediction), and outputs the prediction information to the intra-prediction unit 204 and the inter-prediction unit 205.

[0036] The inverse quantization unit 201a and the inverse transformation unit 201b constitute the inverse quantization / inverse transformation unit 201, which performs inverse quantization processing and inverse orthogonal transformation processing on a block-by-block basis.

[0037] The inverse quantization unit 201a performs inverse quantization processing corresponding to the quantization processing performed by the quantization unit 102b of the image coding device 1. The inverse quantization unit 201a recovers the quantization conversion coefficients by inverse quantization of the quantization conversion coefficients input from the entropy decoding unit 200 using the quantization parameter (Qp) and the quantization matrix, and outputs the recovered conversion coefficients to the inverse conversion unit 201b.

[0038] The inverse transform unit 201b performs an inverse orthogonal transform process corresponding to the orthogonal transform process performed by the transform unit 102a of the image encoding device 1. The inverse transform unit 201b performs an inverse orthogonal transform on the transformation coefficients input from the inverse quantization unit 201a to restore the predicted residuals, and outputs the restored predicted residuals (restored predicted residuals) to the synthesis unit 202.

[0039] The synthesis unit 202 reconstructs (decodes) the original target image block by combining the predicted residual input from the inverse transform unit 201b and the predicted image input from the predicted image correction unit 207 on a pixel-by-pixel basis, and outputs the reconstructed image block to the memory 203.

[0040] Memory 203 stores the reconstructed image blocks input from the synthesis unit 202. Memory 203 stores the reconstructed image blocks on a frame-by-frame basis. Memory 203 outputs the reconstructed images (decoded images) on a frame-by-frame basis to the outside of the image decoding device 2 in the order they are displayed.

[0041] The intra-prediction unit 204 generates an intra-predicted image by referring to the reconstructed image block stored in the memory 203 and performing intra-prediction according to the intra-prediction information (intra-prediction mode) input from the entropy decoding unit 200. Specifically, the intra-prediction unit 204 generates an intra-predicted image by referring to adjacent reference pixels determined according to the intra-prediction mode from among the reconstructed image blocks (decoded blocks) stored in the memory 203. The intra-prediction unit 204 outputs the intra-predicted image (and / or adjacent reference pixels) to the prediction image correction unit 207.

[0042] The interpretation unit 205 performs interpretation to predict target image blocks using frame-by-frame reconstructed images (decoded images) stored in memory 106 as reference images. The interpretation unit 205 generates an interpretation image by performing interpretation according to the interpretation information (motion vector information, etc.) input from the entropy decoding unit 200, and outputs the interpretation image to the prediction image correction unit 207. The interpretation unit 205 also outputs multiple reference images used for interpretation to the prediction accuracy evaluation unit 206.

[0043] The prediction accuracy evaluation unit 206 operates in the same manner as the prediction accuracy evaluation unit 109 of the image coding device 1. When the interpretation unit 205 performs interpretation using multiple reference images, the prediction accuracy evaluation unit 206 calculates the similarity between the multiple reference images on a pixel-by-pixel basis, evaluates the prediction accuracy of the predicted image on a pixel-by-pixel basis, and outputs the evaluation result information to the predicted image correction unit 207.

[0044] The predictive image correction unit 207 operates similarly to the predictive image correction unit 110 of the image encoding device 1. The predictive image correction unit 207 corrects the edge regions of the inter-predicted image by filtering using the decoded adjacent blocks (adjacent reference pixels) corresponding to the intra-predicted image. The predictive image correction unit 207 performs filtering if the prediction accuracy evaluated by the prediction accuracy evaluation unit 206 is below a first threshold, and does not perform filtering if the prediction accuracy evaluated by the prediction accuracy evaluation unit 206 exceeds the first threshold. If filtering is performed, the predictive image correction unit 207 outputs the filtered inter-predicted image as the predicted image to the synthesis unit 202, and if filtering is not performed, outputs the inter-predicted image as the predicted image to the synthesis unit 202. Details of the predictive image correction unit 207 will be described later.

[0045] (3. Interpretation) Figure 3 shows an example of interpretation. Figure 4 shows an example of a predicted image generated by interpretation. As a simple example of interpretation, we will explain the case using bidirectional prediction, particularly forward and backward prediction (bidirectional prediction), which is used in HEVC.

[0046] As shown in Figure 3, dual prediction refers to frames that are temporally preceding and succeeding the target frame (current frame). In the example in Figure 3, the prediction of a block in the image of frame t is performed by referring to frames t-1 and t+1. In motion detection, similar areas (blocks) to the target image block are detected from within the reference frames of frames t-1 and t+1 within the search range set by the system.

[0047] The detected area is the reference image. The arrows shown in the figure indicate the relative position of the reference image to the target image block, and are called motion vectors. The motion vector information is encoded by entropy coding together with the frame information of the reference image in the image encoding device 1. Meanwhile, the image decoding device 2 detects the reference image based on the motion vector information generated by the image encoding device 1.

[0048] As shown in Figures 3 and 4, reference images 1 and 2 detected by motion detection are similar partial images aligned within the reference frame to the target image block, and therefore become images similar to the target image block (the image to be encoded). In the example in Figure 4, the target image block contains a star pattern and a partial circle pattern. Reference image 1 contains a star pattern and a full circle pattern. Reference image 2 contains a star pattern but does not contain a circle pattern.

[0049] A prediction image is generated from the reference images 1 and 2. Generally, the prediction process generates a prediction image that possesses the characteristics of each reference image by averaging reference images 1 and 2, which have different but partially similar features. However, more advanced processing, such as signal enhancement processing using low-pass or high-pass filters, may also be used to generate the prediction image. Here, since reference image 1 contains a circle pattern and reference image 2 does not, when reference images 1 and 2 are averaged to generate a prediction image, the circle pattern in the prediction image will have half the signal compared to reference image 1.

[0050] The difference between the predicted image obtained from reference images 1 and 2 and the target image block (image to be encoded) is the prediction residual. In the prediction residual shown in Figure 4, large differences occur only in the shifted edges of the star pattern and the shifted parts of the circle pattern (shaded areas), but in other areas, the prediction is made with good accuracy and the difference is small (no difference occurs in the example in Figure 4).

[0051] The areas where no difference is observed (the non-edge parts of the star pattern and the background) are areas where the similarity between reference image 1 and reference image 2 is high, and where accurate predictions were made. On the other hand, the areas where large differences are observed are areas specific to each reference image, i.e., areas where the similarity between reference image 1 and reference image 2 is significantly low. Therefore, it can be seen that areas where the similarity between reference image 1 and reference image 2 is significantly low result in low prediction accuracy and produce large differences (residuals).

[0052] When the predicted residuals, which contain both large and small differences, are orthogonal transformed, and degradation of the transformation coefficients occurs due to quantization, this degradation of the transformation coefficients propagates throughout the image (block) via inverse quantization and inverse orthogonal transformation. Then, when the predicted residuals restored by inverse quantization and inverse orthogonal transformation (restored predicted residuals) are combined with the predicted image to reconstruct the target image block, the degradation of image quality propagates even to areas where high-precision predictions were made, such as the non-edge and background parts of the star pattern shown in Figure 4.

[0053] (4. Prediction Accuracy Evaluation Unit) Figure 5 shows an example of the configuration of the prediction accuracy evaluation unit 109 in the image coding device 1. The prediction accuracy evaluation unit 109 evaluates the prediction accuracy of the predicted image on a pixel-by-pixel basis by calculating the similarity between multiple reference images used for interpretation on a pixel-by-pixel basis. Here, we describe an example in which the prediction accuracy evaluation unit 109 evaluates the prediction accuracy for all pixels of the predicted image, but the prediction accuracy evaluation unit 109 does not necessarily have to evaluate the prediction accuracy for all pixels of the predicted image. The prediction accuracy evaluation unit 109 only needs to evaluate the prediction accuracy for pixels in the edge regions of the predicted image.

[0054] As shown in Figure 5, the prediction accuracy evaluation unit 109 includes a difference calculation unit (subtraction unit) 109a, a normalization unit 109b, and an adjustment unit 109c.

[0055] The difference calculation unit 109a calculates the absolute value of the difference between reference image 1 and reference image 2 on a pixel-by-pixel basis and outputs the calculated absolute value of the difference to the normalization unit 109b. The absolute value of the difference is an example of a value that indicates similarity. The smaller the absolute value of the difference, the higher the similarity, and the larger the absolute value of the difference, the lower the similarity. The difference calculation unit 109a may also calculate the absolute value of the difference after performing a filtering process on each reference image. The difference calculation unit 109a may also calculate a statistical quantity such as the squared error and use such a statistical quantity as the similarity.

[0056] The normalization unit 109b normalizes the difference value of each pixel input from the difference calculation unit 109a by the absolute value of the difference value of the pixel with the largest absolute value of the difference value within the block (i.e., the maximum absolute value of the difference value within the block), and outputs the normalized difference value, which is the absolute value of the normalized difference value, to the adjustment unit 109c. In this embodiment, the normalized difference value is used as a weight to weight the restored predicted residual, which is synthesized with the predicted image in the synthesis unit 105, on a pixel-by-pixel basis.

[0057] The adjustment unit 109c adjusts the normalized difference values ​​(weights) input from the normalization unit 109b based on the quantization parameter (Qp) that determines the coarseness of quantization, and outputs these weights. Since the degree of degradation of the reconstructed predicted residual is higher the coarser the quantization, the adjustment unit 109c can weight the reconstructed predicted residual considering the degree of degradation by adjusting the normalized difference values ​​(weights) based on the quantization parameter (Qp).

[0058] The estimated prediction accuracy Rij for each pixel (ij) output by the prediction accuracy evaluation unit 109 can be expressed, for example, as shown in equation (1) below.

[0059] Rij = 1 - (abs(Xij-Yij) / maxD × Scale(Qp)) ···(1)

[0060] In equation (1), Xij is the pixel value of pixel ij in reference image 1, Yij is the pixel value of pixel ij in reference image 2, and abs is the function that obtains the absolute value.

[0061] Furthermore, in equation (1), maxD is the maximum value of the difference value abs(Xij-Yij) within the block. To find maxD, it is necessary to find the difference value for all pixels in the block, but to omit this process, the maximum value of an already processed adjacent block may be used as a substitute. For example, if there are values ​​greater than or equal to that value, maxD may be normalized by clipping with the maximum value used. Alternatively, maxD may be found from the quantization parameter (Qp) using a table that defines the correspondence between the quantization parameter (Qp) and maxD. Alternatively, a fixed value specified in the specifications beforehand may be used as maxD.

[0062] Furthermore, in equation (1), Scale(Qp) is a coefficient multiplied according to the quantization parameter (Qp). Scale(Qp) is designed to approach 1.0 when Qp is large and approach 0 when Qp is small, and the degree of this adjustment is to be adjusted by the system. Alternatively, a fixed value specified in the specifications may be used as Scale(Qp). In addition, to simplify the process, Scale(Qp) may be set to a fixed value such as 1.0, which is designed according to the system.

[0063] The adjustment unit 109c outputs the estimated prediction accuracy Rij. This Rij may also output a weight adjusted by a sensitivity function designed according to the system. For example, if abs(Xij-Yij) / maxD×Scale(Qp)=Rij, then Rij=Clip(rij,1.0,0.0) or Rij=Clip(rij+offset,1.0,0.0) may be used to adjust the sensitivity with an offset according to control information such as QP. Note that Clip(x,max,min) indicates the process of clipping at max if x exceeds max, and at min if x falls below min.

[0064] The estimated prediction accuracy Rij calculated in this way will be a value within the range of 0 to 1.0. Basically, the estimated prediction accuracy Rij approaches 0 when the absolute value of the difference in pixel ij between reference images is large (i.e., the prediction accuracy is low), and approaches 1 when the absolute value of the difference in pixel ij between reference images is small (i.e., the prediction accuracy is high). The prediction accuracy evaluation unit 109 outputs map information consisting of the estimated prediction accuracy Rij of each pixel ij within a block to the prediction image correction unit 110 in block units.

[0065] Furthermore, the prediction accuracy evaluation unit 109 performs evaluation (calculation of estimated prediction accuracy Rij) only when applying interpretation using multiple reference images. In other modes, such as unidirectional prediction or intraprediction processing that does not use multiple reference images, evaluation is not required.

[0066] Furthermore, while the prediction accuracy evaluation unit 109 in the image coding device 1 has been described, the prediction accuracy evaluation unit 206 in the image decoding device 2 is configured similarly to the prediction accuracy evaluation unit 109 in the image coding device 1. Specifically, the prediction accuracy evaluation unit 206 in the image decoding device 2 comprises a difference calculation unit 206a, a normalization unit 206b, and an adjustment unit 206c.

[0067] (5. Predictive Image Correction Unit) Figure 6 shows an example of the configuration of the predictive image correction unit 110 in the image encoding device 1. As shown in Figure 6, the predictive image correction unit 110 comprises a continuity evaluation unit 110a, a filter determination unit 110b, and a filter processing unit 110c.

[0068] The continuity evaluation unit 110a evaluates the continuity between the inter-predicted image input from the inter-prediction unit 108 and the decoded adjacent blocks input from the intra-prediction unit 107. The decoded adjacent blocks are the decoded blocks adjacent to the target image block. The continuity evaluation unit 110a outputs the evaluation result of the continuity between the inter-predicted image and the decoded adjacent blocks to the filter determination unit 110b.

[0069] Figure 7 shows an example of the operation of the continuity evaluation unit 110a. In the example in Figure 7, it is assumed that decoded blocks exist to the left and above the image block to be encoded. The continuity evaluation unit 110a evaluates the continuity of the block boundary between the interpredicted image (target image block) and the decoded block on a pixel-by-pixel basis, in the order from top to bottom (or bottom to top). For example, the continuity evaluation unit 110a calculates the continuity Cont of the block boundary spanning pixels p0 and q0 in Figure 7 using the following equation (2).

[0070] Cont=|p2-2×p1+p0|+|q2-2×q1+q0| ···(2)

[0071] Note that the units and evaluation metrics used to evaluate continuity are not limited to those shown in equation (2), and can be changed to other units or evaluation metrics if they are predetermined in the system. For example, the unit for evaluating continuity may be the edge unit of the block boundary (performed in the continuity evaluation of the left boundary and the continuity evaluation of the upper boundary), or the pixel value difference across the boundary (abs(p0-q0)) may be used as the continuity evaluation metric to simplify the process.

[0072] The filter determination unit 110b determines whether or not to perform filtering based on the prediction accuracy evaluation result input from the prediction accuracy evaluation unit 109 and the continuity evaluation result input from the continuity evaluation unit 110a. The filter determination unit 110b may perform the determination on a pixel-by-pixel basis at the block boundary between the interpredicted image and the decoded block, or on an edge-by-edge basis at the block boundary. The filter determination unit 110b may also determine to change the filter strength (for example, the number of filter taps or the frequency response) based on the prediction accuracy evaluation result and the continuity evaluation result. If the prediction accuracy evaluation unit 109 evaluates the prediction accuracy on a pixel-by-pixel basis, the filter determination unit 110b may use the average value of the estimated prediction accuracy R of each pixel in the edge region (the region targeted for filtering) of the interpredicted image as the prediction accuracy evaluation result.

[0073] The filter determination unit 110b determines to perform filtering if the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 is less than or equal to the first threshold α, and the continuity evaluated by the continuity evaluation unit 110a is less than or equal to the second threshold β. On the other hand, if the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 exceeds the first threshold α, and / or the continuity evaluated by the continuity evaluation unit 110a exceeds the second threshold β, the filter determination unit 110b determines not to perform filtering. Here, the thresholds α and β may be set to system-fixed values, or variable values ​​may be calculated and set using a function that uses a quantization parameter (Qp). Furthermore, if the estimated prediction accuracy R is not normalized, the threshold α may be calculated and set to a variable value using a function that uses the maximum value of the estimated prediction accuracy within the target image block.

[0074] The filter processing unit 110c performs a filter on the edge region of the inter-prediction image if the filter determination unit 110b determines that a filter should be performed. As shown in Figure 7, the filter processing unit 110c corrects the predicted pixels q0, q1, q2 included in the edge region of the inter-prediction image (encoded block) and the adjacent decoded reference pixels p0, p1, p2 input from the intra-prediction unit 107 using the following equation (3), and outputs the corrected predicted pixels q'0, q'1, q'2.

[0075]

number

[0076] Here, clip(x, max, min) indicates the process of clipping at max if x exceeds max, and at min if x falls below min. Also, tc in equation (3) is an adjustment offset for the clipping process, and its variable value can be calculated and set using a function that uses the quantization parameter (Qp).

[0077] Although the predictive image correction unit 110 in the image coding device 1 has been described, the predictive image correction unit 207 in the image decoding device 2 is configured similarly to the predictive image correction unit 110 in the image coding device 1. Specifically, the predictive image correction unit 207 in the image decoding device 2 comprises a continuity evaluation unit 207a, a filter determination unit 207b, and a filter processing unit 207c.

[0078] (6. Predictive image correction operation) Figure 8 shows an example of predictive image correction operation. Here, the predictive image correction operation in the image encoding device 1 is described, but a similar predictive image correction operation is performed in the image decoding device 2. The operation flow in Figure 8 may also be performed at the pixel level of the block boundary between the interpredicted image and the decoded block.

[0079] As shown in Figure 8, in step S1, the prediction accuracy evaluation unit 109 evaluates the prediction accuracy in the edge region of the inter-predicted image based on the similarity between the multiple reference images used by the inter-prediction unit 108 to generate the inter-predicted image.

[0080] In step S2, the continuity evaluation unit 110a evaluates the continuity between the interprediction image output by the interprediction unit 108 and the decoded adjacent blocks adjacent to the interprediction image.

[0081] In step S3, the filter determination unit 110b compares the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 with a first threshold α, and compares the continuity evaluated by the continuity evaluation unit 110a with a second threshold β.

[0082] If the prediction accuracy is less than or equal to the first threshold α and the continuity is less than or equal to the second threshold β (step S3: YES), the filter determination unit 110b determines that filtering should be performed. In this case, in step S4, the filter processing unit 110c corrects the edge region of the interpredicted image by filtering using the decoded adjacent block (adjacent reference pixel).

[0083] On the other hand, if the prediction accuracy exceeds the first threshold α and / or the continuity exceeds the second threshold β (step S3: NO), the filter determination unit 110b determines not to perform filtering. In this case, the filter processing unit 110c does not perform filtering.

[0084] (7. Summary of the Embodiments) The image encoding device 1 and image decoding device 2 according to this embodiment control the filtering process based on the evaluation result of the prediction accuracy of the interpretation. This allows the system to control the filtering process so that it is not applied to the edge regions of the interpretation image when the prediction accuracy is high, thereby preventing a decrease in the accuracy of the prediction image due to filtering. In other words, filtering can be applied to the edge regions of the interpretation image only when the prediction accuracy is low, thus enabling appropriate filtering.

[0085] Furthermore, in this embodiment, the application of filtering is determined by a common operation between the image encoding device 1 and the image decoding device 2. Specifically, the prediction accuracy of interpretation is evaluated in both the image encoding device 1 and the image decoding device 2, and the application of filtering is determined based on the evaluation results. As a result, there is no need to transmit a flag indicating whether filtering is applied from the image encoding device 1 to the image decoding device 2, thus preventing an increase in the amount of code due to the flag.

[0086] (8. Example of a modified embodiment 1) In the continuity evaluation and filtering process according to the above-described embodiment, the intra-prediction mode (intra-prediction direction) applied to the decoded adjacent block may be taken into consideration. This improves the accuracy of the continuity evaluation and filtering process. Figure 9 shows the configuration of the prediction image correction unit 110 according to this modified example. As shown in Figure 9, the intra-prediction mode applied to the decoded adjacent block is input from the intra-prediction unit 107 to the continuity evaluation unit 110a and the filtering unit 110c. The continuity evaluation unit 110a evaluates the continuity according to the intra-prediction mode applied to the decoded adjacent block. The filtering unit 110c performs filtering according to the intra-prediction mode applied to the decoded adjacent block.

[0087] Figure 10 shows an example of the operation of the continuity evaluation unit 110a and the filter processing unit 110c according to this modified example. As shown in Figure 10, when the continuity evaluation unit 110a and the filter processing unit 110c evaluate continuity and perform filtering on the edge pixels q0 of the inter-prediction image corresponding to the block to be encoded, they calculate p1, p0, q1, and q2 on a straight line passing through q0 along the intra-prediction mode (intra-prediction direction) applied to the decoded adjacent block by weighted average. Specifically, they calculate p1 and p0 on a straight line passing through q0 using the pixel values ​​in the decoded adjacent block, and calculate q1 and q2 on a straight line passing through q0 using the pixel values ​​in the inter-prediction image. The continuity evaluation unit 110a evaluates the continuity Cont using the p1, p0, q1, and q2 calculated in this way according to equation (2) above. The filter processing unit 110c calculates q'0 using the p1, p0, q1, and q2 calculated in this way according to equation (3) above.

[0088] (9. Example of a modified embodiment 2) In the embodiment described above, an example was explained in which the prediction image correction unit 110 performs a filter process on the edge region of the interprediction image using decoded adjacent blocks as a correction process for the interprediction image.

[0089] Furthermore, the correction process is not limited to such filtering; weighted averaging of the inter-predicted image and intra-predicted image may also be used as the correction process.

[0090] In this modified example, the intra-prediction unit 107 and the predicted image correction unit 110 of the image encoding device 1 shown in Figure 1 constitute a correction unit that performs correction processing on the inter-predicted image using the decoded adjacent block adjacent to the target image block.

[0091] The intra-prediction unit 107 generates an intra-prediction image corresponding to the target image block using the decoded adjacent blocks. For example, the intra-prediction unit 107 generates an intra-prediction image from the decoded adjacent blocks using a predetermined intra-prediction mode. The prediction image correction unit 110 performs a weighted averaging process on the inter-prediction image input from the inter-prediction unit 108 and the intra-prediction image input from the intra-prediction unit 107 as a correction process. Alternatively, the transmitting side may be configured to select one prediction mode from among multiple intra-prediction mode candidates and generate an intra-prediction image. In this case, the entropy encoding unit 103 entropy encodes an intra-prediction mode flag indicating which intra-prediction mode was used to generate the intra-prediction image on the transmitting side.

[0092] Similarly, the intra-prediction unit 204 and the predicted image correction unit 207 of the image decoding device 2 shown in Figure 2 constitute a correction unit that performs correction processing on the intra-predicted image using the decoded adjacent blocks adjacent to the target image block. The intra-prediction unit 204 generates an intra-predicted image corresponding to the target image block using the decoded adjacent blocks.

[0093] For example, the intra-prediction unit 204 generates an intra-prediction image from decoded adjacent blocks using a predetermined intra-prediction mode. The prediction image correction unit 207 performs a weighted averaging process on the inter-prediction image input from the inter-prediction unit 205 and the intra-prediction image input from the intra-prediction unit 204 as a correction process. Alternatively, the system may be configured to determine the intra-prediction mode to be used for generating the intra-prediction image based on an intra-prediction mode flag sent from the transmitting side.

[0094] When applying the configuration of this modified example, one should replace "filtering" in the above-described embodiment with "weighted averaging." Furthermore, in addition to determining whether or not to perform such weighted averaging, the weights in the weighted averaging may be adjusted based on the evaluation results of prediction accuracy and continuity.

[0095] (10. Other Embodiments) In the embodiments described above, biprediction was primarily explained as a prediction method using multiple reference images. However, a technique called intrablock copying can also be applied as a prediction method using multiple reference images. In intrablock copying, reference images within the same frame as the current frame are used to predict the target image block of the current frame.

[0096] The image encoding device 1 may be provided by a program that causes a computer to execute each process performed by the image encoding device 1, and by a program that causes a computer to execute each process performed by the image decoding device 2. The program may also be recorded on a computer-readable medium. Using a computer-readable medium makes it possible to install the program on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, but may be a recording medium such as a CD-ROM or DVD-ROM. Furthermore, the circuits that execute each process performed by the image encoding device 1 may be integrated, and the image encoding device 1 may be configured as a semiconductor integrated circuit (chipset, SoC). Similarly, the circuits that execute each process performed by the image decoding device 2 may be integrated, and the image decoding device 2 may be configured as a semiconductor integrated circuit (chipset, SoC).

[0097] Although the embodiments have been described in detail above with reference to the drawings, the specific configuration is not limited to those described above, and various design changes can be made without departing from the gist of the invention.

[0098] Furthermore, the entire contents of Japanese Patent Application No. 2018-72451 (filed on April 4, 2018) are incorporated into this specification by reference.

Claims

1. An interpretation unit that divides an image frame by frame, predicts the blocks of the target image obtained by interpretation, and generates interpretation-predicted blocks, A correction unit that performs correction processing on the aforementioned inter prediction block, It comprises a calculation unit, When the interpretation unit performs the interpretation using two reference image blocks, one preceding and one succeeding the current frame in time, the calculation unit calculates a statistical amount of the error between the two reference image blocks for each image portion that is a part of the block and consists of multiple pixels. The correction unit performs the correction process only on the portion of the image within the interpretation block where the statistical amount of the error exceeds a threshold. Predictive image correction device.

2. A predictive image correction device according to claim 1, Image encoding device.

3. A predictive image correction device according to claim 1, Image decoding device.

4. The computer is made to function as the predictive image correction device described in claim 1. program.

Images