Predictive image correction device, image encoding device, image decoding device, and storage medium

Abstract

This application discloses a predictive image correction apparatus, comprising: a prediction unit (108) that generates a predictive image corresponding to an object image block obtained by segmenting the current image of a frame unit using multiple reference images; a prediction accuracy evaluation unit (109) that evaluates the prediction accuracy of the predictive image based on the similarity between the multiple reference images used to generate the predictive image; and a correction unit (110) that performs correction processing on the predictive image, wherein the correction unit controls the correction processing based at least on the evaluation result of the prediction accuracy evaluation unit.

Description

Technical Field

[0001] This invention relates to a predictive image correction apparatus, an image encoding apparatus, an image decoding apparatus, and a program. Background Technology

[0002] In video coding technology, efficient encoding (compression) is achieved by transforming and predicting block units obtained from the current image, which are divided into frames (pictures). For prediction, there are two methods: intra-frame prediction and inter-frame prediction.

[0003] Intra-frame prediction is a method of generating a predicted image by referencing decoded neighboring blocks that are adjacent to the block being encoded or decoded (hereinafter referred to as the "object image block"). Inter-frame prediction is a method of generating a predicted image by referencing decoded frames that are different from the current frame to which the object image block belongs.

[0004] Patent Document 1 describes a predictive image correction device that predicts an object image block and generates an inter-frame predicted image by inter-frame prediction, predicts an object image block and generates an intra-frame predicted image by intra-frame prediction, and performs a weighted average of the inter-frame predicted image and the intra-frame predicted image.

[0005] Specifically, the predictive image correction device described in Patent Document 1 evaluates the continuity between the inter-frame predictive image corresponding to the target image block and the decoded neighboring blocks adjacent to the target image block. If the evaluation is discontinuous, the end regions of the inter-frame predictive image are corrected by filtering using the decoded neighboring blocks.

[0006] Existing technical documents

[0007] Patent technology documents

[0008] Patent document 1: U.S. Patent Application Publication No. 2013 / 051467 Summary of the Invention

[0009] The predictive image correction device described in Patent Document 1 is independent of the prediction accuracy of inter-frame prediction, and applies filtering processing based on the evaluation of the continuity between the inter-frame predicted image and the decoded adjacent blocks.

[0010] Therefore, when the target boundary exists at the boundary between the inter-frame prediction image and the decoded adjacent block, even if the prediction accuracy of the end region of the inter-frame prediction image is high, filtering will be applied to that end region. As a result, the accuracy of the prediction image may be reduced due to the filtering.

[0011] Therefore, the object of the present invention is to provide a predictive image correction device, an image encoding device, an image decoding device, and a program capable of appropriately correcting a predicted image.

[0012] The intra-frame prediction apparatus according to the first feature is characterized by comprising: a prediction unit that generates a predicted image corresponding to an object image patch obtained by segmenting the current image of a frame unit using multiple reference images; a prediction accuracy evaluation unit that evaluates the prediction accuracy of the predicted image based on the similarity between the multiple reference images used to generate the predicted image; and a correction unit that performs correction processing on the predicted image using decoded neighbor blocks adjacent to the object image patch, wherein the correction unit controls the correction processing based at least on the evaluation result of the prediction accuracy evaluation unit.

[0013] Furthermore, the prediction using multiple reference images refers to prediction represented by double prediction in inter-frame prediction, but it is not limited to this. For example, the same method can be applied in cases where multiple reference images are used to generate prediction images, such as in the inter-frame BC mode (intra-block copy mode) of the image used in HEVC encoding.

[0014] The purpose of the image encoding device based on the second feature is to include a predictive image correction device based on the first feature.

[0015] The purpose of the image decoding device based on the third feature is to include a predictive image correction device based on the first feature.

[0016] The purpose of the program according to the fourth feature is to enable the computer to function as a predictive image correction device based on the first feature.

[0017] According to the present invention, a predictive image correction apparatus, an image encoding apparatus, an image decoding apparatus, and a program capable of appropriately correcting a predicted image can be provided. Attached Figure Description

[0018] Figure 1 This is a diagram showing the structure of an image encoding apparatus according to an embodiment.

[0019] Figure 2 This is a diagram showing the structure of an image decoding apparatus according to an embodiment.

[0020] Figure 3 This is a diagram illustrating an example of inter-frame prediction.

[0021] Figure 4 This is a diagram showing an example of a predicted image generated through inter-frame prediction.

[0022] Figure 5 This is a diagram illustrating an example of the structure of the prediction accuracy evaluation unit according to an embodiment.

[0023] Figure 6 This is a diagram illustrating an example of the structure of the predictive image correction unit according to an embodiment.

[0024] Figure 7 This is a diagram illustrating an example of the operation of the continuity evaluation unit according to the implementation method.

[0025] Figure 8 This is a diagram illustrating an example of a predictive image correction action according to an implementation method.

[0026] Figure 9 This is a diagram showing the structure of the predictive image correction unit according to a modified example of the embodiment.

[0027] Figure 10 This is a diagram illustrating an example of the operation of the continuity evaluation unit and the filtering processing unit according to a modified embodiment. Detailed Implementation

[0028] Referring to the accompanying drawings, an image encoding apparatus and an image decoding apparatus according to embodiments will be described. The image encoding apparatus and image decoding apparatus according to embodiments perform encoding and decoding of animation, such as MPEG. In the following description of the drawings, the same or similar parts are labeled with the same or similar symbols.

[0029] (1. Structure of an image encoding device)

[0030] Figure 1 This is a diagram showing the structure of the image encoding apparatus 1 according to this embodiment. Figure 1 As shown, the image encoding apparatus 1 includes a block segmentation unit 100, a subtraction unit 101, a transform unit 102a, a quantization unit 102b, an entropy encoding unit 103, an inverse quantization unit 104a, an inverse transform unit 104b, a synthesis unit 105, a memory 106, an intra-frame prediction unit 107, an inter-frame prediction unit 108, a prediction accuracy evaluation unit 109, and a prediction image correction unit (correction unit) 110. In this embodiment, the intra-frame prediction unit 107, the inter-frame prediction unit 108, the prediction accuracy evaluation unit 109, and the prediction image correction unit 110 constitute a prediction image correction apparatus.

[0031] The block segmentation unit 100 divides the input image (frame or image unit) into small block-shaped regions and outputs the image blocks to the subtraction unit 101 (and the inter-frame prediction unit 108). The size of the image block is, for example, 32×32 pixels, 16×16 pixels, 8×8 pixels, or 4×4 pixels. The image block is the unit encoded by the image encoding device 1 and the unit decoded by the image decoding device 2, and this image block is called the object image block. In addition, the shape of the image block is not limited to a square, but can also be rectangular.

[0032] The subtraction unit 101 calculates a prediction residual, which represents the pixel-based difference between the object image block input from the block segmentation unit 100 and the prediction image (prediction image block) corresponding to the object image block. Specifically, the subtraction unit 101 calculates the prediction residual by subtracting the pixel values ​​of the prediction image from the pixel values ​​of the encoded object block, and outputs the calculated prediction residual to the transformation unit 102a. In this embodiment, the prediction image is an image corrected by the prediction image correction unit 110 (described later), which is input from the prediction image correction unit 110 to the subtraction unit 101.

[0033] The transformation unit 102a and the quantization unit 102b constitute a transformation and quantization unit 102 that performs orthogonal transformation and quantization processing on a block-by-block basis.

[0034] The transformation unit 102a performs an orthogonal transformation on the prediction residual input from the subtraction unit 101 to calculate the transformation coefficients, and outputs the calculated transformation coefficients to the quantization unit 102b. Orthogonal transformations include, for example, Discrete Cosine Transform (DCT), Discrete Sine Transform (DST), and Karhunen-Loeve Transform (KLT).

[0035] The quantization unit 102b uses quantization parameters (Qp) and a quantization matrix to quantize the transform coefficients input from the transform unit 102a and generates quantized transform coefficients. The quantization parameters (Qp) are parameters applied to all transform coefficients within a block and determine the roughness of the quantization. The quantization matrix is ​​a matrix containing the quantized values ​​as elements when each transform coefficient is quantized. The quantization unit 102b outputs quantization control information, generated quantized transform coefficient information, etc., to the entropy encoding unit 103 and the inverse quantization unit 104a.

[0036] The entropy coding unit 103 performs entropy coding on the quantization transform coefficients input from the quantization unit 102b, compresses the data to generate coded data (bitstream), and outputs the coded data to the outside of the image coding device 1. Huffman coding, CABAC (Context-based Adaptive Binary Arithmetic Coding), etc., can be used in the entropy coding. Furthermore, prediction information is input to the entropy coding unit 103 from the intra-frame prediction unit 107 and the inter-frame prediction unit 108. The entropy coding unit 103 also performs entropy coding on this information.

[0037] The inverse quantization unit 104a and the inverse transformation unit 104b constitute the inverse quantization and inverse orthogonal transformation unit 104, which performs inverse quantization and inverse orthogonal transformation in block units.

[0038] The inverse quantization unit 104a performs inverse quantization processing corresponding to the quantization processing performed by the quantization unit 102b. Specifically, the inverse quantization unit 104a performs inverse quantization on the quantization transform coefficients input from the quantization unit 102b using quantization parameters (Qp) and a quantization matrix, thereby restoring the transform coefficients, and outputs the restored transform coefficients to the inverse transform unit 104b.

[0039] The inverse transform unit 104b performs an inverse orthogonal transform process corresponding to the orthogonal transform process performed by the transform unit 102a. For example, if the transform unit 102a performs a discrete cosine transform, the inverse transform unit 104b performs an inverse discrete cosine transform. The inverse transform unit 104b performs an inverse orthogonal transform on the transform coefficients input from the inverse quantization unit 104a to restore the prediction residual, and outputs the restored prediction residual, i.e., the restored prediction residual, to the synthesis unit 105.

[0040] The compositing unit 105 combines the restored prediction residual input from the inverse transform unit 104b with the predicted image input from the prediction image correction unit 110 on a pixel-by-pixel basis. The compositing unit 105 reconstructs (decodes) the target image block by adding the pixel values ​​of the restored prediction residual to the pixel values ​​of the predicted image, and outputs the reconstructed target image block, i.e., the reconstructed image block, to the memory 106. This reconstructed image block is sometimes referred to as a decoded block.

[0041] The memory 106 stores the reconstructed image blocks input from the compositing unit 105. The memory 106 stores the reconstructed image blocks in frame units.

[0042] The intra-prediction unit 107 generates an intra-prediction image by referencing decoded neighboring blocks adjacent to the target image block in the reconstructed image block (decoded block) stored in the memory 106. The intra-prediction unit 107 selects the optimal intra-prediction mode and performs intra-prediction using the selected mode. Multiple intra-prediction modes corresponding to multiple intra-prediction directions are predefined. An intra-prediction direction refers to the direction in which the neighboring reference pixels are referenced to the target pixel when predicting the target pixel in the target image block. That is, the neighboring reference pixels that should be used to predict each pixel in the target image block are determined based on the intra-prediction mode (prediction direction). The intra-prediction unit 107 outputs the intra-prediction image (and / or neighboring reference pixels) to the prediction image correction unit 110 and outputs information about the selected intra-prediction mode to the entropy coding unit 103.

[0043] The inter-frame prediction unit 108 uses the reconstructed image (decoded image) of the frame unit stored in the memory 106 as a reference image to perform inter-frame prediction of the target image block. Specifically, the inter-frame prediction unit 108 calculates motion vectors using methods such as block matching and generates an inter-frame prediction image based on the motion vectors. The inter-frame prediction unit 108 selects the optimal inter-frame prediction method from inter-frame prediction using multiple reference images (typically dual prediction) and inter-frame prediction using one reference image (unidirectional prediction), and performs inter-frame prediction using the selected inter-frame prediction method. In this embodiment, the case where the inter-frame prediction unit 108 uses dual prediction will be mainly described, and the inter-frame prediction unit 108 is equivalent to a prediction unit that performs prediction using multiple reference images. The inter-frame prediction unit 108 outputs the generated inter-frame prediction image to the prediction image correction unit 110, and outputs information related to the selected inter-frame prediction method and motion vectors to the entropy coding unit 103. In addition, the inter-frame prediction unit 108 outputs multiple reference images used for inter-frame prediction to the prediction accuracy evaluation unit 109.

[0044] The prediction accuracy evaluation unit 109 evaluates the prediction accuracy in the end regions of the inter-frame prediction image based on the similarity between multiple reference images used to generate the inter-frame prediction image. Specifically, when the inter-frame prediction unit 108 uses multiple reference images for inter-frame prediction, it calculates the similarity between the multiple reference images used for inter-frame prediction for each image portion consisting of one or more pixels, evaluates the prediction accuracy of the inter-frame prediction image for each image portion based on this similarity, and outputs the evaluation result information to the prediction image correction unit 110. In this embodiment, an example of the prediction accuracy evaluation unit 109 calculating the similarity between multiple reference images used for prediction on a pixel-by-pixel basis and evaluating the prediction accuracy of the inter-frame prediction image on a pixel-by-pixel basis will be mainly described. Details of the prediction accuracy evaluation unit 109 will be described later.

[0045] The prediction image correction unit 110 performs correction by weighted averaging the inter-frame prediction image input from the inter-frame prediction unit 108 and the intra-frame prediction image input from the intra-frame prediction unit 107. Specifically, the prediction image correction unit 110 corrects the end regions of the inter-frame prediction image using filtered processing with decoded neighboring blocks (neighboring reference pixels) corresponding to the intra-frame prediction image. Furthermore, the prediction image correction unit 110 controls the filtering processing at least based on the evaluation result of the prediction accuracy evaluation unit 109. "Controlling the filtering processing" includes controlling whether filtering processing is performed and controlling the filtering intensity. In this embodiment, an example will be described primarily based on an example where the prediction image correction unit 110 controls whether filtering processing is performed at least based on the evaluation result of the prediction accuracy evaluation unit 109. For example, the prediction image correction unit 110 performs filtering processing when the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 is below a first threshold, and does not perform filtering processing when the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 exceeds the first threshold. When filtering is performed, the prediction image correction unit 110 outputs the filtered inter-frame prediction image as the prediction image to the subtraction unit 101 and the synthesis unit 105. When no filtering is performed, the prediction image correction unit 110 directly outputs the inter-frame prediction image as the prediction image to the subtraction unit 101 and the synthesis unit 105. Alternatively, "prediction accuracy" can be expressed as "prediction inappropriateness." For example, "prediction inappropriateness" can be expressed as a prediction accuracy below a first threshold indicating high prediction inappropriateness, and a prediction accuracy exceeding the first threshold indicating low prediction inappropriateness. Details of the prediction image correction unit 110 will be described later.

[0046] (2. Structure of the image decoding device)

[0047] Figure 2 This is a diagram showing the structure of the image decoding apparatus 2 according to this embodiment. Figure 2 As shown, the image decoding apparatus 2 includes an entropy decoding unit 200, an inverse quantization unit 201a, an inverse transform unit 201b, a synthesis unit 202, a memory 203, an intra-frame prediction unit 204, an inter-frame prediction unit 205, a prediction accuracy evaluation unit 206, and a prediction image correction unit 207. In this embodiment, the intra-frame prediction unit 204, the inter-frame prediction unit 205, the prediction accuracy evaluation unit 206, and the prediction image correction unit 207 constitute a prediction image correction apparatus.

[0048] The entropy decoding unit 200 decodes the encoded data generated by the encoding device 1 and outputs the quantization transform coefficients to the inverse quantization unit 201a. In addition, the entropy decoding unit 200 decodes the encoded data, obtains information related to prediction (intra-frame prediction and inter-frame prediction), and outputs the prediction-related information to the intra-frame prediction unit 204 and the inter-frame prediction unit 205.

[0049] The inverse quantization unit 201a and the inverse transformation unit 201b constitute the inverse quantization and inverse orthogonal transformation unit 201, which performs inverse quantization and inverse orthogonal transformation in block units.

[0050] The inverse quantization unit 201a performs inverse quantization processing corresponding to the quantization processing performed by the quantization unit 102b of the image encoding apparatus 1. The inverse quantization unit 201a recovers the transform coefficients by inverse quantization of the quantization transform coefficients input from the entropy decoding unit 200 using quantization parameters (Qp) and quantization matrix, and outputs the recovered transform coefficients to the inverse transform unit 201b.

[0051] The inverse transform unit 201b performs an inverse orthogonal transform process corresponding to the orthogonal transform process performed by the transform unit 102a of the image encoding device 1. The inverse transform unit 201b performs an inverse orthogonal transform on the transform coefficients input from the inverse quantization unit 201a to restore the prediction residual, and outputs the restored prediction residual (restored prediction residual) to the synthesis unit 202.

[0052] The synthesis unit 202 reconstructs (decodes) the original object image block by synthesizing the prediction residual input from the inverse transform unit 201b and the prediction image input from the prediction image correction unit 207 in pixel units, and outputs the reconstructed image block to the memory 203.

[0053] The memory 203 stores the reconstructed image blocks input from the compositing unit 202. The memory 203 stores the reconstructed image blocks in frame units. The memory 203 outputs the reconstructed images (decoded images) in frame units to the outside of the image decoding device 2 in display order.

[0054] The intra-prediction unit 204 performs intra-prediction based on the intra-prediction information (intra-prediction mode) input from the entropy decoding unit 200, with reference to the reconstructed image block stored in the memory 203, thereby generating an intra-predicted image. Specifically, the intra-prediction unit 204 generates an intra-prediction image with reference to neighboring reference pixels in the reconstructed image block (decoded block) stored in the memory 203, determined according to the intra-prediction mode. The intra-prediction unit 204 outputs the intra-prediction image (and / or neighboring reference pixels) to the prediction image correction unit 207.

[0055] The inter-frame prediction unit 205 uses the reconstructed image (decoded image) of the frame unit stored in the memory 106 as a reference image to perform inter-frame prediction of the target image block. The inter-frame prediction unit 205 generates an inter-frame prediction image by performing inter-frame prediction based on the inter-frame prediction information (motion vector information, etc.) input from the entropy decoding unit 200, and outputs the inter-frame prediction image to the prediction image correction unit 207. Furthermore, the inter-frame prediction unit 205 outputs multiple reference images used for inter-frame prediction to the prediction accuracy evaluation unit 206.

[0056] The prediction accuracy evaluation unit 206 performs the same operation as the prediction accuracy evaluation unit 109 of the image coding apparatus 1. When the inter-frame prediction unit 205 performs inter-frame prediction using multiple reference images, the prediction accuracy evaluation unit 206 calculates the similarity between the multiple reference images in pixel units, thereby evaluating the prediction accuracy of the predicted image in pixel units, and outputs the evaluation result information to the prediction image correction unit 207.

[0057] The prediction image correction unit 207 performs the same operation as the prediction image correction unit 110 of the image encoding apparatus 1. The prediction image correction unit 207 corrects the end regions of the inter-frame prediction image using the decoded neighboring blocks (neighboring reference pixels) corresponding to the intra-frame prediction image through filtering. The prediction image correction unit 207 performs filtering if the prediction accuracy evaluated by the prediction accuracy evaluation unit 206 is below a first threshold, and does not perform filtering if the prediction accuracy evaluated by the prediction accuracy evaluation unit 206 exceeds the first threshold. When filtering is performed, the prediction image correction unit 207 outputs the filtered inter-frame prediction image as a prediction image to the synthesis unit 202; when no filtering is performed, the inter-frame prediction image is directly output as a prediction image to the synthesis unit 202. Details of the prediction image correction unit 207 will be described later.

[0058] (3. Inter-frame prediction)

[0059] Figure 3 This is a diagram illustrating an example of inter-frame prediction. Figure 4 This is a diagram illustrating an example of a predicted image generated through inter-frame prediction. As a simple example of inter-frame prediction, it illustrates the use of dual prediction as used in HEVC, specifically the use of forward and backward prediction (bidirectional prediction).

[0060] like Figure 3 As shown, the dual prediction reference is relative to the object frame (the current frame) in time, encompassing both the frames preceding and following it. Figure 3 In the example, the prediction of blocks in the image of frame t is made with reference to frames t-1 and t+1. In motion detection, regions (blocks) similar to the object image blocks are detected from within the reference frames t-1 and t+1, within a search range set in the system.

[0061] The detected region is a reference image. The information representing the relative position of the reference image with respect to the object image patch is shown by the arrows in the diagram, and is called a motion vector. The motion vector information is encoded in image encoding device 1 along with the frame information of the reference image through entropy coding. On the other hand, image decoding device 2 detects the reference image based on the motion vector information generated by image encoding device 1.

[0062] like Figure 3 and Figure 4 As shown, reference image 1 and reference image 2, detected by motion detection, are similar partial images that overlap with the object image block position within the reference frame, and are therefore images similar to the object image block (encoded object image). Figure 4 In the example, the object image block contains a pattern of stars and a partial circle pattern. Reference image 1 contains a pattern of stars and a complete circle pattern. Reference image 2 contains a pattern of stars but does not contain a circle pattern.

[0063] A predicted image is generated from reference image 1 and reference image 2. The prediction process typically generates a predicted image that incorporates the features of each reference image by averaging reference images 1 and 2, which have different but partially similar features. However, more advanced processing, such as signal enhancement using low-pass filters or high-pass filters, can also be used to generate the predicted image. Here, reference image 1 contains a circular pattern, while reference image 2 does not. Therefore, if reference images 1 and 2 are averaged to generate the predicted image, the signal of the circular pattern in the predicted image is halved compared to reference image 1.

[0064] The difference between the predicted image obtained from reference image 1 and reference image 2 and the object image patch (encoded object image) is the prediction residual. Figure 4 In the predicted residuals shown, large differences only occur at the edges of the star pattern and the edges of the circle pattern (the diagonal lines). However, for the remaining areas, high-precision predictions are possible with fewer differences. Figure 4 No difference was generated in the example.

[0065] The parts that did not produce a difference (the non-edge parts of the star pattern and the background) are those with high similarity between reference image 1 and reference image 2, and are the parts that were predicted with high accuracy. On the other hand, the parts that produce a large difference are those unique to each reference image, that is, the parts with significantly low similarity between reference image 1 and reference image 2. Therefore, it can be seen that the parts with significantly low similarity between reference image 1 and reference image 2 have low prediction accuracy and produce large differences (residuals).

[0066] Thus, when an orthogonal transformation is performed on the prediction residuals that mix the parts with large differences and the parts without differences, if the transformation coefficients deteriorate due to quantization, this deterioration is propagated throughout the image (patch) through inverse quantization and inverse orthogonal transformation. Furthermore, if the prediction residuals restored through inverse quantization and inverse orthogonal transformation (the restored prediction residuals) are combined with the prediction image to reconstruct the object image patch, the image quality degradation is also propagated to... Figure 4 The non-edge portions and background portions of the star pattern shown were predicted with high precision.

[0067] (4. Prediction Accuracy Evaluation Department)

[0068] Figure 5 This diagram illustrates an example of the structure of the prediction accuracy evaluation unit 109 in the image coding apparatus 1. The prediction accuracy evaluation unit 109 calculates the similarity between multiple reference images used for inter-frame prediction on a pixel-by-pixel basis, thereby evaluating the prediction accuracy of the predicted image on a pixel-by-pixel basis. Here, an example is described where the prediction accuracy evaluation unit 109 evaluates the prediction accuracy for all pixels of the predicted image; however, the prediction accuracy evaluation unit 109 may not necessarily evaluate the prediction accuracy for all pixels of the predicted image. The prediction accuracy evaluation unit 109 may evaluate the prediction accuracy for at least the pixels in the end regions of the predicted image.

[0069] like Figure 5 As shown, the prediction accuracy evaluation unit 109 includes a difference calculation unit (subtraction unit) 109a, a standardization unit 109b, and an adjustment unit 109c.

[0070] The difference calculation unit 109a calculates the absolute value of the difference between reference image 1 and reference image 2 in pixels and outputs the calculated absolute value of the difference to the normalization unit 109b. The absolute value of the difference is an example of a value representing similarity. The smaller the absolute value of the difference, the higher the similarity; the larger the absolute value of the difference, the lower the similarity. The difference calculation unit 109a may also calculate the absolute value of the difference after filtering each reference image. The difference calculation unit 109a may also calculate statistics such as squared error and use these statistics as the similarity.

[0071] The standardization unit 109b uses the absolute value of the difference value of the pixel with the largest absolute value of the intra-block difference value (i.e., the maximum value of the absolute value of the intra-block difference value) to standardize the difference values ​​of each pixel input from the difference calculation unit 109a, and outputs the absolute value of the standardized difference value, i.e., the standardized difference value, to the adjustment unit 109c. In this embodiment, the standardized difference value is used as a weight to weight the restored prediction residual synthesized with the prediction image in the synthesis unit 105 in pixel units.

[0072] The adjustment unit 109c adjusts the standardized difference value (weight) input from the standardization unit 109b based on the quantization parameter (Qp) that determines the quantized roughness, and outputs the weight. Since the larger the quantized roughness, the higher the degree of degradation of the restored prediction residual, the adjustment unit 109c can take into account the degree of degradation when adjusting the standardized difference value (weight) based on the quantization parameter (Qp) to perform weighting of the restored prediction residual.

[0073] The predicted prediction accuracy Rij of each pixel (ij) output by the prediction accuracy evaluation unit 109 can be expressed, for example, by the following formula (1).

[0074]

[0075] In equation (1), Xij is the pixel value of pixel ij in reference image 1, Yij is the pixel value of pixel ij in reference image 2, and abs is a function to obtain the absolute value.

[0076] Additionally, in equation (1), maxD is the maximum value of the difference value abs(Xij-Yij) within the block. To obtain maxD, the difference value needs to be calculated for all pixels within the block. However, to omit this process, the maximum value of already processed adjacent blocks can be used instead. For example, if a value higher than this exists, maxD can be normalized by cropping the previously used maximum value. Alternatively, maxD can be obtained from the quantization parameter (Qp) using a table that determines the correspondence between the quantization parameter (Qp) and maxD. Alternatively, a fixed value pre-defined in the specification can be used as maxD.

[0077] Furthermore, in equation (1), Scale(Qp) is a coefficient multiplied by the quantization parameter (Qp). Scale(Qp) is designed to be close to 1.0 when Qp is large and close to 0 when Qp is small, and its degree is adjusted by the system. Alternatively, a fixed value specified in the specification can be used as Scale(Qp). In addition, to simplify processing, Scale(Qp) can be set to a fixed value according to the system design, such as 1.0.

[0078] The adjustment unit 109c outputs the predicted accuracy Rij. Alternatively, this Rij can also be a weighted average adjusted according to the sensitivity function designed for the system. For example, it can be set not only to abs(Xij-Yij) / maxD×Scale(Qp)=Rij, Rij=Clip(rij, 1.0, 0.0), but also to Rij=Clip(rij+offset, 1.0, 0.0) to adjust the sensitivity by adding compensation corresponding to control information such as QP. Furthermore, Clip(x, max, min) indicates that if x exceeds max, it is clipped to max; if x is below min, it is clipped to min.

[0079] The calculated inferred prediction accuracy Rij is a value in the range of 0 to 1.0. Basically, the inferred prediction accuracy Rij is close to 0 when the absolute value of the difference between pixel ij in the reference images is large (i.e., low prediction accuracy), and close to 1 when the absolute value of the difference between pixel ij in the reference images is small (i.e., high prediction accuracy). The prediction accuracy evaluation unit 109 outputs the image information composed of the inferred prediction accuracy Rij of each pixel ij within the block to the prediction image correction unit 110 in block units.

[0080] Furthermore, the prediction accuracy evaluation unit 109 can only perform evaluation (calculation of the predicted accuracy Rij) when inter-frame prediction using multiple reference images is applied. In other modes, such as unidirectional prediction or intra-frame prediction processing that does not use multiple reference images, no evaluation is performed.

[0081] Furthermore, while the prediction accuracy evaluation unit 109 in the image encoding apparatus 1 has been described, the prediction accuracy evaluation unit 206 in the image decoding apparatus 2 is configured in the same way as the prediction accuracy evaluation unit 109 in the image encoding apparatus 1. Specifically, the prediction accuracy evaluation unit 206 in the image decoding apparatus 2 includes a difference calculation unit 206a, a normalization unit 206b, and an adjustment unit 206c.

[0082] (5. Predictive Image Correction Unit)

[0083] Figure 6 This is a diagram illustrating an example of the structure of the predictive image correction unit 110 in the image encoding apparatus 1. (See diagram for example.) Figure 6 As shown, the predicted image correction unit 110 includes a continuity evaluation unit 110a, a filter determination unit 110b, and a filter processing unit 110c.

[0084] The continuity evaluation unit 110a evaluates the continuity of the inter-frame prediction image input from the inter-frame prediction unit 108 and the decoded neighboring blocks input from the intra-frame prediction unit 107. The decoded neighboring blocks are the decoded blocks adjacent to the target image block. The continuity evaluation unit 110a outputs the evaluation result of the continuity of the inter-frame prediction image and the decoded neighboring blocks to the filtering determination unit 110b.

[0085] Figure 7 This is a diagram illustrating an example of the operation of the continuity evaluation unit 110a. Figure 7 In the example, it is assumed that there are decoded blocks on the left and top sides of the encoded object image block. The continuity evaluation unit 110a evaluates the continuity of the block boundaries of the inter-frame predicted image (object image block) and the decoded blocks in pixel units in a top-down (or bottom-up) order. The continuity evaluation unit 110a calculates, for example, by the following formula (2). Figure 7 Continuity of the block boundary between pixels p0 and q0 (Cont).

[0086]

[0087] Furthermore, the units and evaluation metrics used to evaluate continuity are not limited to those shown in equation (2). They can be changed to other units and evaluation metrics as long as they are predefined in the system. For example, the unit for evaluating continuity can be the edge unit of the block boundary (based on the continuity evaluation of the left boundary and the continuity evaluation of the upper boundary). To simplify the processing, the pixel value difference across the boundary (abs(p0-q0)) can also be used as the continuity evaluation metric.

[0088] The filtering determination unit 110b determines whether to perform filtering processing based on the prediction accuracy evaluation result input from the prediction accuracy evaluation unit 109 and the continuity evaluation result input from the continuity evaluation unit 110a. The filtering determination unit 110b can make the determination on a pixel-by-pixel basis between the inter-frame prediction image and the block boundary of the decoded block, or on an edge-by-edge basis between the block boundaries. The filtering determination unit 110b can also determine whether to change the strength of the filter (e.g., the number of filter taps or the frequency response) based on the prediction accuracy evaluation result and the continuity evaluation result. Furthermore, when the prediction accuracy evaluation unit 109 evaluates the prediction accuracy on a pixel-by-pixel basis, the filtering determination unit 110b can also use the average value of the predicted prediction accuracy R of each pixel in the end region (the region to be filtered) of the inter-frame prediction image as the prediction accuracy evaluation result.

[0089] The filtering determination unit 110b determines to perform filtering processing when the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 is below a first threshold α and the continuity evaluated by the continuity evaluation unit 110a is below a second threshold β. On the other hand, the filtering determination unit 110b determines not to perform filtering processing when the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 exceeds the first threshold α and / or the continuity evaluated by the continuity evaluation unit 110a exceeds the second threshold β. Here, the thresholds α and β can be set to fixed system values, or they can be calculated and set as variable values ​​using a function of the quantization parameter (Qp). Furthermore, when the inferred prediction accuracy R is not standardized, the threshold α can also be calculated and set as a variable value using a function of the maximum inferred prediction accuracy within the object image patch.

[0090] When the filtering determination unit 110b determines that filtering processing is required, the filtering processing unit 110c performs filtering processing on the end regions of the inter-frame prediction image. For example... Figure 7 As shown, the filtering processing unit 110c uses the predicted pixels q0, q1, q2 contained in the end region of the inter-frame prediction image (coded object block) and the adjacent decoded reference pixels p0, p1, p2 input by the intra-frame prediction unit 107 to perform correction by the following formula (3) and outputs the corrected predicted pixels q'0, q'1, q'2.

[0091] [Number 1]

[0092]

[0093] Here, clip(x, max, min) represents the process of clipping using max when x exceeds max and clipping using min when x is below min. In addition, tc in equation (3) is the adjustment compensation for the clipping process, which can be calculated and set to a variable value using a function of the quantization parameter (Qp).

[0094] Furthermore, the prediction image correction unit 110 in the image encoding apparatus 1 has been described, and the prediction image correction unit 207 in the image decoding apparatus 2 is configured in the same way as the prediction image correction unit 110 in the image encoding apparatus 1. Specifically, the prediction image correction unit 207 in the image decoding apparatus 2 includes a continuity evaluation unit 207a, a filter determination unit 207b, and a filter processing unit 207c.

[0095] (6. Predicting image correction actions)

[0096] Figure 8 This diagram illustrates an example of a predictive image correction operation. Here, the predictive image correction operation in image encoding device 1 will be explained, while the same predictive image correction operation is performed in image decoding device 2. Figure 8 The action flow can also be performed in pixel units of inter-frame prediction images and block boundaries of decoded blocks.

[0097] like Figure 8 As shown, in step S1, the prediction accuracy evaluation unit 109 evaluates the prediction accuracy in the end region of the inter-frame prediction image based on the similarity between multiple reference images used by the inter-frame prediction unit 108 to generate the inter-frame prediction image.

[0098] In step S2, the continuity evaluation unit 110a evaluates the continuity of the inter-frame prediction image output by the inter-frame prediction unit 108 and the decoded neighboring blocks adjacent to the inter-frame prediction image.

[0099] In step S3, the filtering determination unit 110b compares the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 with the first threshold α, and compares the continuity evaluated by the continuity evaluation unit 110a with the second threshold β.

[0100] If the prediction accuracy is below the first threshold α and the continuity is below the second threshold β (the case where "Yes" is in step S3), the filtering determination unit 110b determines to perform filtering processing. In this case, in step S4, the filtering processing unit 110c uses the decoded neighboring blocks (neighboring reference pixels) to correct the end regions of the inter-frame prediction image through filtering processing.

[0101] On the other hand, if the prediction accuracy exceeds the first threshold α and / or the continuity exceeds the second threshold β (the case where "No" is stated in step S3), the filtering determination unit 110b determines that no filtering processing will be performed. In this case, the filtering processing unit 110c does not perform filtering processing.

[0102] (7. Summary of implementation methods)

[0103] According to this embodiment, the image encoding apparatus 1 and the image decoding apparatus 2 control the filtering process based on the evaluation result of the prediction accuracy of the inter-frame prediction. Therefore, when the prediction accuracy of the end regions of the inter-frame prediction image is high, control can be implemented to prevent filtering from being applied to those end regions, thus preventing a decrease in the accuracy of the prediction image caused by filtering. In other words, since it can be set to apply filtering only to the end regions of the inter-frame prediction image when the prediction accuracy is low, appropriate filtering can be performed.

[0104] Furthermore, in this embodiment, the determination of whether to apply filtering processing is performed through a joint operation in both the image encoding device 1 and the image decoding device 2. Specifically, the prediction accuracy of inter-frame prediction is evaluated in both the image encoding device 1 and the image decoding device 2, and the determination of whether to apply filtering processing is based on the evaluation result. Therefore, it is unnecessary to transmit a flag indicating whether filtering processing is applied from the image encoding device 1 to the image decoding device 2, thus preventing an increase in coding volume caused by the flag.

[0105] (8. Variation of the implementation method 1)

[0106] In the continuity evaluation and filtering process according to the above embodiments, the intra-prediction mode (intra-prediction direction) applied to decoded adjacent blocks can be considered. This improves the accuracy of the continuity evaluation and filtering process. Figure 9 This is a diagram showing the structure of the predictive image correction unit 110 according to this modified example. Figure 9 As shown, an intra-prediction mode applied to decoded neighboring blocks is input from the intra-prediction unit 107 to the continuity evaluation unit 110a and the filtering processing unit 110c. The continuity evaluation unit 110a evaluates continuity based on the intra-prediction mode applied to the decoded neighboring blocks. Additionally, the filtering processing unit 110c performs filtering processing based on the intra-prediction mode applied to the decoded neighboring blocks.

[0107] Figure 10 This is a diagram illustrating an example of the operation of the continuity evaluation unit 110a and the filtering processing unit 110c according to this modified example. Figure 10As shown, when the continuity evaluation unit 110a and the filtering processing unit 110c perform continuity evaluation and filtering processing on the end pixel q0 of the inter-frame prediction image corresponding to the coded target block, they calculate p1, p0, q1, and q2 on the line passing through q0 by weighted averaging according to the intra-frame prediction mode (intra-frame prediction direction) applied to the decoded adjacent blocks. Specifically, p1 and p0 on the line passing through q0 are calculated using the pixel values ​​in the decoded adjacent blocks, and q1 and q2 on the line passing through q0 are calculated using the pixel values ​​in the inter-frame prediction image. The continuity evaluation unit 110a uses the p1, p0, q1, and q2 calculated in this way to evaluate the continuity Cont using the above equation (2). The filtering processing unit 110c uses the p1, p0, q1, and q2 calculated in this way to calculate q'0 using the above equation (3).

[0108] (9. Variation 2 of the implementation method)

[0109] In the above embodiment, the prediction image correction unit 110 is described as performing filtering processing on the end regions of the inter-frame prediction image using decoded adjacent blocks as an example of the correction processing of the inter-frame prediction image.

[0110] Furthermore, the correction process is not limited to this filtering process, and a weighted average of inter-frame predicted images and intra-frame predicted images can also be used as the correction process.

[0111] In this variation, Figure 1 The image encoding apparatus 1 shown comprises an intra-frame prediction unit 107 and a prediction image correction unit 110, which uses a decoded neighboring block adjacent to the target image block to perform correction processing on the inter-frame prediction image.

[0112] The intra-prediction unit 107 generates an intra-predicted image corresponding to the target image block using decoded neighboring blocks. For example, the intra-prediction unit 107 generates an intra-predicted image from decoded neighboring blocks using a predetermined intra-prediction mode. The prediction image correction unit 110 performs a weighted average process as a correction process on the inter-prediction image input from the inter-prediction unit 108 and the intra-prediction image input from the intra-prediction unit 107. Alternatively, it may be configured such that a prediction mode is selected from multiple intra-prediction mode candidates at the transmitting side to generate the intra-predicted image. In this case, the entropy coding unit 103 performs entropy coding on the intra-prediction mode flag, which indicates which intra-prediction mode was used to generate the intra-predicted image at the transmitting side.

[0113] same, Figure 2The image decoding apparatus 2 shown comprises an intra-prediction unit 204 and a prediction image correction unit 207, which together form a correction unit that performs inter-frame prediction image correction processing using decoded neighboring blocks adjacent to the target image block. The intra-prediction unit 204 generates an intra-prediction image corresponding to the target image block using the decoded neighboring blocks.

[0114] For example, the intra-prediction unit 204 generates an intra-prediction image from decoded adjacent blocks using a predetermined intra-prediction mode. The prediction image correction unit 207 performs a weighted average process as a correction process on the inter-prediction image input from the inter-prediction unit 205 and the intra-prediction image input from the intra-prediction unit 204. Alternatively, the intra-prediction mode for generating the intra-prediction image may be determined based on an intra-prediction mode flag sent from the transmitting side.

[0115] When applying the structure of this modified example, simply replace "filtering processing" in the above embodiment with "weighted averaging processing". Furthermore, it is not only possible to determine whether to perform the weighted averaging processing, but also to adjust the weights in the weighted averaging processing based on the evaluation results of prediction accuracy and continuity.

[0116] <10. Other Implementation Methods>

[0117] In the above embodiments, dual prediction was primarily described as a prediction method using multiple reference images. However, a technique called intra-block copying can also be applied as a prediction method using multiple reference images. In intra-block copying, an intra-frame reference image identical to that of the current frame is used for the prediction of object image blocks in the current frame.

[0118] The image encoding device 1 and the image decoding device 2 can also be provided as programs that enable a computer to perform various processes. Alternatively, the programs can be recorded on a computer-readable medium. If a computer-readable medium is used, the program can be installed on the computer. Here, the computer-readable medium containing the program can also be a non-transitory recording medium. There are no particular limitations on the non-transitory recording medium; for example, it can be a CD-ROM or DVD-ROM. Furthermore, the circuitry for performing the various processes of the image encoding device 1 can be integrated, and the image encoding device 1 can be configured as a semiconductor integrated circuit (chipset, SoC). Similarly, the circuitry for performing the various processes of the image decoding device 2 can be integrated, and the image decoding device 2 can be configured as a semiconductor integrated circuit (chipset, SoC).

[0119] The embodiments have been described in detail above with reference to the accompanying drawings. However, the specific structure is not limited to the above description, and various design modifications can be made without departing from the main idea.

[0120] Furthermore, the entire contents of Japanese Patent Application No. 2018-72451 (filed on April 4, 2018) are incorporated herein by reference.

Claims

1. A predictive image correction device, comprising: The inter-frame prediction unit predicts blocks of the object image obtained by segmenting the image into frame units through inter-frame prediction, and generates inter-frame prediction blocks. The intra-frame prediction unit predicts blocks of the object image through intra-frame prediction and generates intra-frame prediction blocks. The evaluation unit performs an evaluation based on a prediction mode, which controls intra-frame prediction processing of decoded neighboring blocks adjacent to the block of the object image. as well as The correction unit, as a correction process for the inter-frame prediction block, performs a weighted average process on the inter-frame prediction block and the intra-frame prediction block. The correction unit adjusts the weights in the weighted average processing based on the evaluation results.

2. An image encoding apparatus, comprising the predictive image correction apparatus of claim 1.

3. An image decoding device, comprising the predictive image correction device as described in claim 1.

4. A storage medium storing a program that enables a computer to function as the predictive image correction device of claim 1.

Images