Predictive image correction device, image encoding device, image decoding device, and computer program product
By predicting multiple reference images and controlling the filtering process based on similarity evaluation, the problem of the filtering process of the end area of the inter-prediction image in the prior art is solved, and the overall accuracy of the predicted image is improved.
Patent Information
- Application Number
- CN202411356109.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2018-04-04
- Filing Date
- 2019-03-29
- Publication Date
- 2024-12-06
AI Technical Summary
When the existing predicted image correction device exists in the target boundary, even if the prediction accuracy of the inter-frame prediction image is high, filtering processing will still be applied to the problem that the prediction image accuracy will be reduced.
By using a plurality of reference images to predict, the prediction accuracy is evaluated based on the similarity of the predicted image, and the filtering process is controlled based on the prediction accuracy evaluation results, and the end region of the inter-predicted image is filtered only when the prediction accuracy is low.
Effectively prevent the filtering process from decreasing the accuracy of the prediction image, ensuring that the end areas of the inter-prediction image are not filtered when the accuracy is high, and the overall accuracy of the prediction image is improved.
Smart Images

Figure CN119094751A_ABST
Abstract
Description
[0001] Division Statement
[0002] This application is a divisional application of the Chinese patent application with application date of March 29, 2019 and application number 201980028829.3. Technical Field
[0003] The present invention relates to a predictive image correction device, an image encoding device, an image decoding device and a computer program product. Background Art
[0004] In video coding technology, efficient coding (compression) is achieved by performing transformation and prediction in block units obtained by dividing a current image in frame (picture) units. There are two methods of prediction: intra-frame prediction and inter-frame prediction.
[0005] Intra-frame prediction is a method of generating a predicted image by referring to decoded adjacent blocks adjacent to a block that is an encoding target or a decoding target (hereinafter referred to as a "target image block"). Inter-frame prediction is a method of generating a predicted image by referring to a decoded frame different from the current frame to which the target image block belongs.
[0006] Patent document 1 describes a predicted image correction device that predicts a target image block by inter-frame prediction and generates an inter-frame predicted image, predicts a target image block by intra-frame prediction and generates an intra-frame predicted image, and performs weighted averaging of the inter-frame predicted image and the intra-frame predicted image.
[0007] Specifically, the predicted image correction device described in Patent Document 1 evaluates the continuity of the inter-frame predicted image corresponding to the object image block and the decoded adjacent blocks adjacent to the object image block. If the evaluation is discontinuous, the decoded adjacent blocks are used to correct the end area of the inter-frame predicted image through filtering.
[0008] Prior art literature
[0009] Patent Technical Documents Patent Technical Document 1: U.S. Patent Application Publication No. 2013 / 051467 Summary of the invention
[0010] The predicted image correction device described in Patent Document 1 applies filtering processing based on evaluation of continuity between an inter-frame predicted image and decoded adjacent blocks, regardless of prediction accuracy of inter-frame prediction.
[0011] Therefore, when the target boundary exists at the boundary between the inter-frame prediction image and the decoded adjacent block, filtering processing is applied to the end area even when the prediction accuracy of the end area of the inter-frame prediction image is high, so there is a problem that the accuracy of the predicted image may be reduced due to the filtering processing.
[0012] Therefore, an object of the present invention is to provide a predicted image correction device, an image encoding device, an image decoding device, and a program that can appropriately correct a predicted image.
[0013] According to the first feature, the intra-frame prediction device mainly comprises: a prediction unit, which generates a predicted image corresponding to an object image block obtained by segmenting a current image of a frame unit using a plurality of reference images for prediction; a prediction accuracy evaluation unit, which evaluates the prediction accuracy of the predicted image based on the similarity between the plurality of reference images used to generate the predicted image; and a correction unit, which performs correction processing on the predicted image using decoded adjacent blocks adjacent to the object image block, and the correction unit controls the correction processing at least according to the evaluation result of the prediction accuracy evaluation unit.
[0014] In addition, the so-called prediction using multiple reference images is a prediction represented by double prediction in inter-frame prediction, but is not limited to this. For example, the same method can also be applied when generating a predicted image with reference to multiple images, such as the inter-frame BC mode (intra-frame block copy mode) of the image used in the HEVC encoding method.
[0015] An image encoding device according to a second feature is gist of including the predicted image correction device according to the first feature.
[0016] An image decoding device according to a third feature is gist of including the predicted image correction device according to the first feature.
[0017] The program according to the fourth feature is intended to cause a computer to function as the predictive image correction device according to the first feature.
[0018] According to the present invention, a predicted image correction device, an image encoding device, an image decoding device, and a program capable of appropriately correcting a predicted image can be provided. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 It is a diagram showing the structure of an image encoding device according to an embodiment.
[0020] Figure 2 It is a diagram showing the structure of an image decoding device according to an embodiment.
[0021] Figure 3 is a diagram showing an example of inter-frame prediction.
[0022] Figure 4 is a diagram showing an example of a predicted image generated by inter-frame prediction.
[0023] Figure 5: is a diagram showing an example of the structure of the prediction accuracy evaluation unit according to the embodiment.
[0024] Figure 6 It is a diagram showing an example of the structure of a predicted image correction unit according to an embodiment.
[0025] Figure 7 It is a diagram showing an operation example of the continuity evaluation unit according to the embodiment.
[0026] Figure 8 is a diagram showing an example of a predicted image correction operation according to an embodiment.
[0027] Fig. 9 It is a diagram showing the structure of a predicted image correction unit according to a modified example of the embodiment.
[0028] Fig.10 It is a diagram showing an example of the operation of the continuity evaluation unit and the filter processing unit according to the modification example of the embodiment. DETAILED DESCRIPTION
[0029] The image encoding device and the image decoding device according to the embodiment are described with reference to the accompanying drawings. The image encoding device and the image decoding device according to the embodiment encode and decode animation represented by MPEG. In the description of the following drawings, the same or similar parts are marked with the same or similar symbols.
[0030] (1. Structure of Image Coding Device)
[0031] Figure 1 1 is a diagram showing the structure of the image encoding device 1 according to the present embodiment. Figure 1 As shown, the image coding device 1 includes a block division unit 100, a subtraction unit 101, a transformation unit 102a, a quantization unit 102b, an entropy coding unit 103, an inverse quantization unit 104a, an inverse transformation unit 104b, a synthesis unit 105, a memory 106, an intra prediction unit 107, an inter prediction unit 108, a prediction accuracy evaluation unit 109, and a predicted image correction unit (correction unit) 110. In this embodiment, the intra prediction unit 107, the inter prediction unit 108, the prediction accuracy evaluation unit 109, and the predicted image correction unit 110 constitute a predicted image correction device.
[0032] The block division unit 100 divides the input image of the frame (or picture) unit into small block-shaped areas, and outputs the image block to the subtraction unit 101 (and the inter-frame prediction unit 108). The size of the image block is, for example, 32×32 pixels, 16×16 pixels, 8×8 pixels, or 4×4 pixels. The image block is a unit for encoding by the image encoding device 1 and a unit for decoding by the image decoding device 2, and the image block is called an object image block. In addition, the shape of the image block is not limited to a square, and may also be a rectangular shape.
[0033] The subtraction unit 101 calculates a prediction residual, which indicates a difference in pixel units between the target image block input from the block division unit 100 and the prediction image (prediction image block) corresponding to the target image block. Specifically, the subtraction unit 101 calculates the prediction residual by subtracting each pixel value of the prediction image from each pixel value of the encoding target block, and outputs the calculated prediction residual to the transformation unit 102a. In the present embodiment, the prediction image is an image corrected by the prediction image correction unit 110 described later, and is input from the prediction image correction unit 110 to the subtraction unit 101.
[0034] The transform unit 102a and the quantization unit 102b constitute the transform and quantization unit 102 which performs orthogonal transform processing and quantization processing in units of blocks.
[0035] The transform unit 102a performs an orthogonal transform on the prediction residual input from the subtraction unit 101 to calculate a transform coefficient, and outputs the calculated transform coefficient to the quantization unit 102b. The so-called orthogonal transform refers to, for example, discrete cosine transform (DCT: Discrete Cosine Transform), discrete sine transform (DST: Discrete Sine Transform), Karhunen-Loeve transform (KLT: Karhunen-Loeve Transform), etc.
[0036] The quantization unit 102b quantizes the transform coefficients input from the transform unit 102a using the quantization parameter (Qp) and the quantization matrix, and generates quantized transform coefficients. The quantization parameter (Qp) is a parameter commonly applied to each transform coefficient in the block, and is a parameter that determines the coarseness of quantization. The quantization matrix is a matrix having quantization values when each transform coefficient is quantized as elements. The quantization unit 102b outputs the quantization control information, the generated quantized transform coefficient information, etc. to the entropy coding unit 103 and the inverse quantization unit 104a.
[0037] The entropy coding unit 103 performs entropy coding on the quantized transform coefficients input from the quantization unit 102b, performs data compression to generate coded data (bit stream), and outputs the coded data to the outside of the image coding device 1. In entropy coding, Huffman coding, CABAC (Context-based Adaptive Binary Arithmetic Coding; context-based adaptive binary arithmetic coding), etc. can be used. In addition, information about prediction is input from the intra-frame prediction unit 107 and the inter-frame prediction unit 108 to the entropy coding unit 103. The entropy coding unit 103 also performs entropy coding on these information.
[0038] The inverse quantization unit 104a and the inverse transformation unit 104b constitute the inverse quantization and inverse transformation unit 104 that performs inverse quantization processing and inverse orthogonal transformation processing in units of blocks.
[0039] The inverse quantization unit 104a performs an inverse quantization process corresponding to the quantization process performed by the quantization unit 102b. Specifically, the inverse quantization unit 104a performs an inverse quantization process on the quantized transform coefficient input from the quantization unit 102b by using a quantization parameter (Qp) and a quantization matrix, thereby restoring the transform coefficient, and outputs the restored transform coefficient to the inverse transform unit 104b.
[0040] The inverse transform unit 104b performs an inverse orthogonal transform process corresponding to the orthogonal transform process performed by the transform unit 102a. For example, when the transform unit 102a performs a discrete cosine transform, the inverse transform unit 104b performs an inverse discrete cosine transform. The inverse transform unit 104b performs an inverse orthogonal transform on the transform coefficients input from the inverse quantization unit 104a to restore the prediction residual, and outputs the restored prediction residual, i.e., the restored prediction residual, to the synthesis unit 105.
[0041] The synthesizing unit 105 synthesizes the restored prediction residual input from the inverse transform unit 104b with the predicted image input from the predicted image correcting unit 110 in units of pixels. The synthesizing unit 105 reconstructs (decodes) the target image block by adding each pixel value of the restored prediction residual to each pixel value of the predicted image, and outputs the reconstructed target image block, i.e., the reconstructed image block, to the memory 106. In addition, the reconstructed image block is sometimes referred to as a decoded block.
[0042] The memory 106 stores the reconstructed image blocks input from the synthesizing unit 105. The memory 106 stores the reconstructed image blocks in units of frames.
[0043] The intra prediction unit 107 generates an intra prediction image by referring to the decoded adjacent blocks adjacent to the object image block in the reconstructed image block (decoded block) stored in the memory 106. The intra prediction unit 107 selects the best intra prediction mode and uses the selected intra prediction mode for intra prediction. A plurality of intra prediction modes corresponding to a plurality of intra prediction directions are predefined. The so-called intra prediction direction refers to the direction in which the adjacent reference pixels are based on the object pixel when predicting the object pixel in the object image block with reference to the adjacent pixels adjacent to the object image block. That is, according to the intra prediction mode (prediction direction), the adjacent reference pixels that should be used to predict each pixel in the object image block are determined. The intra prediction unit 107 outputs the intra prediction image (and / or the adjacent reference pixels) to the prediction image correction unit 110, and outputs the information of the selected intra prediction mode to the entropy coding unit 103.
[0044] The inter-frame prediction unit 108 uses the reconstructed image (decoded image) of the frame unit stored in the memory 106 as a reference image to perform inter-frame prediction of the prediction target image block. Specifically, the inter-frame prediction unit 108 calculates a motion vector by a method such as block matching, and generates an inter-frame prediction image based on the motion vector. The inter-frame prediction unit 108 selects the best inter-frame prediction method from the inter-frame prediction using multiple reference images (typically bi-prediction) and the inter-frame prediction using one reference image (unidirectional prediction), and performs inter-frame prediction using the selected inter-frame prediction method. In this embodiment, the description is mainly based on the case where the inter-frame prediction unit 108 uses bi-prediction, and the inter-frame prediction unit 108 is equivalent to a prediction unit that performs prediction using multiple reference images. The inter-frame prediction unit 108 outputs the generated inter-frame prediction image to the prediction image correction unit 110, and outputs information related to the selected inter-frame prediction method and motion vector to the entropy coding unit 103. In addition, the inter-frame prediction unit 108 outputs the multiple reference images used for inter-frame prediction to the prediction accuracy evaluation unit 109.
[0045] The prediction accuracy evaluation unit 109 evaluates the prediction accuracy in the end region of the inter-frame prediction image based on the similarity between the multiple reference images used to generate the inter-frame prediction image. Specifically, when the inter-frame prediction unit 108 uses multiple reference images for inter-frame prediction, the similarity between the multiple reference images used for inter-frame prediction is calculated for each image portion composed of one or more pixels, and the prediction accuracy of the inter-frame prediction image is evaluated for each image portion based on the similarity, and information on the evaluation result is output to the prediction image correction unit 110. In this embodiment, an example is mainly described in which the prediction accuracy evaluation unit 109 calculates the similarity between the multiple reference images used for prediction in units of one pixel and evaluates the prediction accuracy of the inter-frame prediction image in units of one pixel. The details of the prediction accuracy evaluation unit 109 will be described later.
[0046] The predicted image correction unit 110 performs correction by weighted averaging the inter-frame predicted image input from the inter-frame prediction unit 108 and the intra-frame predicted image input from the intra-frame prediction unit 107. Specifically, the predicted image correction unit 110 corrects the end region of the inter-frame predicted image by filtering using the decoded adjacent blocks (adjacent reference pixels) corresponding to the intra-frame predicted image. In addition, the predicted image correction unit 110 controls the filtering process at least based on the evaluation result of the prediction accuracy evaluation unit 109. The so-called "control of filtering process" includes control of whether to perform filtering process and control of filtering intensity. In this embodiment, an example in which the predicted image correction unit 110 controls whether to perform filtering process based on at least the evaluation result of the prediction accuracy evaluation unit 109 is mainly described. For example, the predicted image correction unit 110 performs filtering process when the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 is less than the first threshold value, and does not perform filtering process when the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 exceeds the first threshold value. When filtering is performed, the prediction image correction unit 110 outputs the filtered inter-frame prediction image as a prediction image to the subtraction unit 101 and the synthesis unit 105. When filtering is not performed, the prediction image correction unit 110 directly outputs the inter-frame prediction image as a prediction image to the subtraction unit 101 and the synthesis unit 105. In addition, "prediction accuracy" can also be expressed as "prediction inappropriateness". For "prediction inappropriateness", for example, a prediction accuracy below a first threshold value can indicate that the prediction inappropriateness is high. A prediction accuracy exceeding the first threshold value can indicate that the prediction inappropriateness is low. The details of the prediction image correction unit 110 will be described later.
[0047] (2. Structure of Image Decoding Device)
[0048] Figure 2 2 is a diagram showing the structure of the image decoding device 2 according to the present embodiment. Figure 2 As shown, the image decoding device 2 includes an entropy decoding unit 200, an inverse quantization unit 201a, an inverse transformation unit 201b, a synthesis unit 202, a memory 203, an intra prediction unit 204, an inter prediction unit 205, a prediction accuracy evaluation unit 206, and a predicted image correction unit 207. In this embodiment, the intra prediction unit 204, the inter prediction unit 205, the prediction accuracy evaluation unit 206, and the predicted image correction unit 207 constitute a predicted image correction device.
[0049] The entropy decoding unit 200 decodes the coded data generated by the coding device 1, and outputs the quantized transform coefficients to the inverse quantization unit 201a. In addition, the entropy decoding unit 200 decodes the coded data, obtains information related to prediction (intra-frame prediction and inter-frame prediction), and outputs the information related to prediction to the intra-frame prediction unit 204 and the inter-frame prediction unit 205.
[0050] The inverse quantization unit 201a and the inverse transformation unit 201b constitute the inverse quantization and inverse transformation unit 201 that performs inverse quantization processing and inverse orthogonal transformation processing in units of blocks.
[0051] The inverse quantization unit 201a performs an inverse quantization process corresponding to the quantization process performed by the quantization unit 102b of the image encoding device 1. The inverse quantization unit 201a restores the transform coefficient by inverse quantizing the quantized transform coefficient input from the entropy decoding unit 200 using the quantization parameter (Qp) and the quantization matrix, and outputs the restored transform coefficient to the inverse transform unit 201b.
[0052] The inverse transform unit 201b performs an inverse orthogonal transform process corresponding to the orthogonal transform process performed by the transform unit 102a of the image encoding device 1. The inverse transform unit 201b performs an inverse orthogonal transform on the transform coefficient input from the inverse quantization unit 201a to restore the prediction residual, and outputs the restored prediction residual (restored prediction residual) to the synthesis unit 202.
[0053] The synthesis unit 202 reconstructs (decodes) the original target image block by synthesizing the prediction residual input from the inverse transformation unit 201 b and the prediction image input from the prediction image correction unit 207 in units of pixels, and outputs the reconstructed image block to the memory 203 .
[0054] The memory 203 stores the reconstructed image blocks input from the synthesizing unit 202. The memory 203 stores the reconstructed image blocks in frame units. The memory 203 outputs the reconstructed images (decoded images) in frame units to the outside of the image decoding device 2 in the display order.
[0055] The intra prediction unit 204 performs intra prediction based on the intra prediction information (intra prediction mode) input from the entropy decoding unit 200 with reference to the reconstructed image block stored in the memory 203, thereby generating an intra prediction image. Specifically, the intra prediction unit 204 generates the intra prediction image with reference to the adjacent reference pixels determined according to the intra prediction mode in the reconstructed image block (decoded block) stored in the memory 203. The intra prediction unit 204 outputs the intra prediction image (and / or the adjacent reference pixels) to the prediction image correction unit 207.
[0056] The inter prediction unit 205 performs inter prediction of the prediction target image block using the reconstructed image (decoded image) of the frame unit stored in the memory 106 as a reference image. The inter prediction unit 205 performs inter prediction based on the inter prediction information (motion vector information, etc.) input from the entropy decoding unit 200 to generate an inter prediction image, and outputs the inter prediction image to the prediction image correction unit 207. In addition, the inter prediction unit 205 outputs a plurality of reference images used for inter prediction to the prediction accuracy evaluation unit 206.
[0057] The prediction accuracy evaluation unit 206 performs the same operation as the prediction accuracy evaluation unit 109 of the image encoding device 1. When the inter-frame prediction unit 205 uses a plurality of reference images to perform inter-frame prediction, the prediction accuracy evaluation unit 206 calculates the similarity between the plurality of reference images in units of pixels, thereby evaluating the prediction accuracy of the prediction image in units of pixels, and outputs information on the evaluation result to the prediction image correction unit 207.
[0058] The predicted image correction unit 207 performs the same operation as the predicted image correction unit 110 of the image encoding device 1. The predicted image correction unit 207 corrects the end region of the inter-frame predicted image by filtering using the decoded adjacent blocks (adjacent reference pixels) corresponding to the intra-frame predicted image. The predicted image correction unit 207 performs filtering when the prediction accuracy evaluated by the prediction accuracy evaluation unit 206 is less than the first threshold value, and does not perform filtering when the prediction accuracy evaluated by the prediction accuracy evaluation unit 206 exceeds the first threshold value. The predicted image correction unit 207 outputs the filtered inter-frame predicted image as a predicted image to the synthesis unit 202 when filtering is performed, and outputs the inter-frame predicted image as a predicted image to the synthesis unit 202 without filtering. The details of the predicted image correction unit 207 will be described later.
[0059] (3. Inter-frame prediction)
[0060] Figure 3 is a diagram showing an example of inter-frame prediction. Figure 4 : is a diagram showing an example of a predicted image generated by inter-frame prediction. As a simple example of inter-frame prediction, a case of using bi-prediction used in HEVC, in particular, a case of using forward and backward prediction (bidirectional prediction) is described.
[0061] like Figure 3 As shown, the bi-prediction references the frame before and after the target frame (current frame) in time. Figure 3 In the example of , a block in the image of the t-th frame is predicted with reference to the t-1th frame and the t+1th frame. In motion detection, a portion (block) similar to the target image block is detected from the search range set in the system within the reference frames of the t-1th and t+1th frames.
[0062] The detected part is the reference image. The information indicating the relative position of the reference image with respect to the target image block is the arrow shown in the figure, which is called a motion vector. The information of the motion vector is encoded by entropy encoding together with the frame information of the reference image in the image encoding device 1. On the other hand, the image decoding device 2 detects the reference image based on the information of the motion vector generated by the image encoding device 1.
[0063] like Figure 3 and Figure 4 As shown, the reference image 1 and the reference image 2 detected by motion detection are similar partial images that overlap with the position of the object image block in the reference frame, and are therefore images similar to the object image block (encoded object image). Figure 4 In the example of , the object image block contains a pattern of stars and a pattern of a part of a circle. Reference image 1 contains a pattern of stars and a pattern of a whole circle. Reference image 2 contains a pattern of stars but does not contain a pattern of a circle.
[0064] A predicted image is generated from the reference image 1 and the reference image 2. In general, the prediction process generates a predicted image having the characteristics of each reference image by averaging the reference image 1 and the reference image 2, which have different characteristics but are partially similar. However, a more advanced process such as a signal enhancement process using a low-pass filter, a high-pass filter, etc. may be used to generate a predicted image. Here, the reference image 1 includes a circular pattern, and the reference image 2 does not include a circular pattern. Therefore, if the reference images 1 and 2 are averaged to generate a predicted image, the circular pattern in the predicted image has a signal halved compared to the reference image 1.
[0065] The difference between the predicted image obtained from reference image 1 and reference image 2 and the target image block (the image to be encoded) is the prediction residual. Figure 4 In the prediction residual shown in FIG. 1 , a large difference occurs only in the deviated portion of the edge of the star pattern and the deviated portion of the circle pattern (the shaded portion), but for the other portions, a high-precision prediction can be made and the difference becomes small (in Figure 4 No difference is produced in the example above).
[0066] The portion where no difference occurs (the non-edge portion of the star pattern and the background portion) is a portion where the similarity between reference image 1 and reference image 2 is high, and is a portion where a high-precision prediction is performed. On the other hand, the portion where a large difference occurs is a portion unique to each reference image, that is, a portion where the similarity between reference image 1 and reference image 2 is significantly low. Therefore, it can be seen that the prediction accuracy of the portion where the similarity between reference image 1 and reference image 2 is significantly low is low, and a large difference (residual) occurs.
[0067] In this way, the prediction residuals containing a mixture of large differences and no differences are orthogonally transformed. If the transform coefficients are degraded due to quantization, the degraded transform coefficients are propagated to the entire image (block) through inverse quantization and inverse orthogonal transformation. Furthermore, if the prediction residuals (restored prediction residuals) restored by inverse quantization and inverse orthogonal transformation are combined with the predicted image to reconstruct the target image block, the degraded image quality is also propagated to the image block. Figure 4 The non-edge parts and background parts of the star pattern shown are parts predicted with high accuracy.
[0068] (4. Forecast accuracy evaluation department)
[0069] Figure 5 1 is a diagram showing an example of the structure of the prediction accuracy evaluation unit 109 in the image encoding device 1. The prediction accuracy evaluation unit 109 calculates the similarity between multiple reference images used for inter-frame prediction in units of pixels, thereby evaluating the prediction accuracy of the prediction image in units of pixels. Here, an example in which the prediction accuracy evaluation unit 109 evaluates the prediction accuracy for all pixels of the prediction image is described, but the prediction accuracy evaluation unit 109 does not necessarily need to evaluate the prediction accuracy for all pixels of the prediction image. The prediction accuracy evaluation unit 109 may evaluate the prediction accuracy for at least the pixels in the end area of the prediction image.
[0070] like Figure 5 As shown, the prediction accuracy evaluation unit 109 includes a difference calculation unit (subtraction unit) 109a, a normalization unit 109b, and an adjustment unit 109c.
[0071] The difference calculation unit 109a calculates the absolute value of the difference value between the reference image 1 and the reference image 2 in units of pixels, and outputs the calculated absolute value of the difference value to the normalization unit 109b. The absolute value of the difference value is an example of a value indicating similarity. The smaller the absolute value of the difference value, the higher the similarity, and the larger the absolute value of the difference value, the lower the similarity. The difference calculation unit 109a may also calculate the absolute value of the difference value after filtering each reference image. The difference calculation unit 109a may also calculate a statistic such as a square error and use the statistic as the similarity.
[0072] The normalization unit 109b normalizes the difference values of the pixels input from the difference calculation unit 109a using the absolute value of the difference value of the pixel with the largest absolute value of the difference value within the block (i.e., the maximum absolute value of the difference value within the block), and outputs the absolute value of the normalized difference value, i.e., the normalized difference value, to the adjustment unit 109c. In the present embodiment, the normalized difference value is used as a weight for weighting the restored prediction residual to be synthesized with the prediction image in the synthesis unit 105 in pixel units.
[0073] The adjustment unit 109c adjusts the normalized difference value (weight) input from the normalization unit 109b based on the quantization parameter (Qp) that determines the roughness of quantization, and outputs the weight. Since the degradation degree of the restored prediction residual increases as the roughness of quantization increases, the adjustment unit 109c adjusts the normalized difference value (weight) based on the quantization parameter (Qp), thereby weighting the restored prediction residual in consideration of the degradation degree.
[0074] The estimated prediction accuracy Rij of each pixel (ij) output by the prediction accuracy evaluation unit 109 can be expressed by the following equation (1), for example.
[0075] Rij=1-(abs(Xij-Yij) / maxD×Scale(Qp))···(1)
[0076] In formula (1), Xij is the pixel value of pixel ij of reference image 1, Yij is the pixel value of pixel ij of reference image 2, and abs is a function for obtaining an absolute value.
[0077] In addition, in formula (1), maxD is the maximum value of the difference value abs(Xij-Yij) within the block. In order to obtain maxD, it is necessary to obtain the difference value for all pixels in the block, but in order to omit this processing, it can also be replaced by the maximum value of the adjacent blocks that have been processed. For example, when there are values above this value, maxD can be standardized by clipping the used maximum value. Alternatively, maxD can be obtained from the quantization parameter (Qp) using a table that determines the correspondence between the quantization parameter (Qp) and maxD. Alternatively, a fixed value pre-specified in the specification can be used as maxD.
[0078] In formula (1), Scale(Qp) is a coefficient multiplied by the quantization parameter (Qp). Scale(Qp) is designed to be close to 1.0 when Qp is large and close to 0 when Qp is small, and the degree is adjusted by the system. Alternatively, a fixed value pre-specified in the specification may be used as Scale(Qp). In order to simplify the processing, Scale(QP) may be set to a fixed value designed according to the system, such as 1.0.
[0079] The adjustment unit 109c outputs the estimated prediction accuracy Rij. In addition, the Rij may also output a weight adjusted by a sensitivity function designed according to the system. For example, not only may abs(Xii-Yij) / maxD×Scale(Qp)=Rij, Rij=Clip(rij, 1.0, 0.0) be set, but also Rij=Clip(rij+offset, 1.0, 0.0) may be set to add a compensation corresponding to control information such as QP to adjust the sensitivity. In addition, Clip(x, max, min) indicates a process of clipping at max when x exceeds max, and a process of clipping at min when x is less than min.
[0080] The estimated prediction accuracy Rij calculated in this way is a value within the range of 0 to 1.0. Basically, the estimated prediction accuracy Rij is close to 0 when the absolute value of the difference value of the pixel ij between the reference images is large (that is, the prediction accuracy is low), and is close to 1 when the absolute value of the difference value of the pixel ij between the reference images is small (that is, the prediction accuracy is high). The prediction accuracy evaluation unit 109 outputs the map information composed of the estimated prediction accuracy Rij of each pixel ij in the block to the predicted image correction unit 110 in units of blocks.
[0081] In addition, the prediction accuracy evaluation unit 109 may perform evaluation (calculation of estimated prediction accuracy Rij) only when inter-frame prediction using multiple reference images is applied, and may not perform evaluation in other modes, such as unidirectional prediction or intra-frame prediction processing that does not use multiple reference images.
[0082] In addition, although the prediction accuracy evaluation unit 109 in the image encoding device 1 is described, the prediction accuracy evaluation unit 206 in the image decoding device 2 is configured similarly to the prediction accuracy evaluation unit 109 in the image encoding device 1. Specifically, the prediction accuracy evaluation unit 206 in the image decoding device 2 includes a difference calculation unit 206a, a normalization unit 206b, and an adjustment unit 206c.
[0083] (5. Prediction Image Correction Unit)
[0084] Figure 6 1 is a diagram showing an example of the structure of the predicted image correction unit 110 in the image encoding device 1. Figure 6 As shown, the predicted image correction unit 110 includes a continuity evaluation unit 110a, a filter determination unit 110b, and a filter processing unit 110c.
[0085] The continuity evaluation unit 110a evaluates the continuity of the inter-frame prediction image input from the inter-frame prediction unit 108 and the decoded adjacent blocks input from the intra-frame prediction unit 107. The decoded adjacent blocks are decoded blocks adjacent to the target image block. The continuity evaluation unit 110a outputs the evaluation result of the continuity of the inter-frame prediction image and the decoded adjacent blocks to the filter determination unit 110b.
[0086] Figure 7 1 is a diagram showing an example of the operation of the continuity evaluation unit 110a. Figure 7 In the example of , it is assumed that there are decoded blocks on the left and above the target image block to be encoded. The continuity evaluation unit 110a evaluates the continuity of the block boundary between the inter-frame prediction image (target image block) and the decoded block in pixel units in the order from top to bottom (or from bottom to top). The continuity evaluation unit 110a calculates, for example, by the following formula (2): Figure 7 Continuity Cont across the block boundary of pixels p0 and q0.
[0087] Cont=|p2-2×p1+p0|+|q2-2×q1+q0|···(2)
[0088] In addition, the units and evaluation indicators for evaluating continuity are not limited to the units and evaluation indicators shown in formula (2), and can be changed to other units and evaluation indicators as long as they are pre-defined in the system. For example, the unit for evaluating continuity can be the edge unit of the block boundary (based on the continuity evaluation of the left boundary and the continuity evaluation of the upper boundary). In order to simplify the processing, the pixel value difference across the boundary (abs(p0-q0)) can also be used as the continuity evaluation indicator.
[0089] The filtering determination unit 110b determines whether to perform filtering processing based on the prediction accuracy evaluation result input from the prediction accuracy evaluation unit 109 and the continuity evaluation result input from the continuity evaluation unit 110a. The filtering determination unit 110b can make the determination in pixel units of the block boundary between the inter-frame prediction image and the decoded block, or in edge units of the block boundary. The filtering determination unit 110b can also determine to change the strength of the filter (for example, the number of taps or frequency response of the filter) based on the prediction accuracy evaluation result and the continuity evaluation result. In addition, when the prediction accuracy evaluation unit 109 evaluates the prediction accuracy in pixel units, the filtering determination unit 110b can also use the average value of the estimated prediction accuracy R of each pixel in the end area (the object area of the filtering process) of the inter-frame prediction image as the prediction accuracy evaluation result.
[0090] The filtering determination unit 110b determines to perform filtering processing when the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 is less than the first threshold value α and the continuity evaluated by the continuity evaluation unit 110a is less than the second threshold value β. On the other hand, when the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 exceeds the first threshold value α and / or the continuity evaluated by the continuity evaluation unit 110a exceeds the second threshold value β, the filtering determination unit 110b determines not to perform filtering processing. Here, as the threshold α and the threshold β, a system-fixed value can be set, or a variable value can be calculated and set by using a function of a quantization parameter (Qp). In addition, when the inferred prediction accuracy R is not standardized, the threshold α can also be calculated and set as a variable value by using a function of the maximum value of the inferred prediction accuracy within the object image block.
[0091] When the filter determination unit 110b determines that the filter process is to be performed, the filter processing unit 110c performs the filter process on the edge region of the inter-frame prediction image. Figure 7As shown, the filtering processing unit 110c uses the predicted pixels q0, q1, q2 contained in the end area of the inter-frame prediction image (encoded object block) and the adjacent decoded reference pixels p0, p1, p2 input by the intra-frame prediction unit 107, and corrects them using the following formula (3), and outputs the corrected predicted pixels q'0, q'1, q'2.
[0092] [Number 1]
[0093] q′0=Clip3(q0-2×tc, q0+2×tc, (p1+2×p0+2×q0+2×q1+q2+4)>>2)
[0094] q′1=Clip3(q1-2×tc, q1+2×tc, (p0+q0+q1+q2+2)>>2)
[0095] q′2=clip3(q2-2×tc, q2+2×tc, (p0+q0+q1+3×q2+2×q3+4)>>3)…(3)
[0096] Here, clip(x, max, min) indicates that when x exceeds max, clipping is performed with max, and when x is less than min, clipping is performed with min. In addition, tc in equation (3) is an adjustment compensation for the clipping process, which can be calculated and set as a variable value by using a function of a quantization parameter (Qp).
[0097] In addition, although the predicted image correction unit 110 in the image encoding device 1 is described, the predicted image correction unit 207 in the image decoding device 2 is configured similarly to the predicted image correction unit 110 in the image encoding device 1. Specifically, the predicted image correction unit 207 in the image decoding device 2 includes a continuity evaluation unit 207a, a filter determination unit 207b, and a filter processing unit 207c.
[0098] (6. Predictive image correction action)
[0099] Figure 8 1 is a diagram showing an example of a predicted image correction operation. Here, the predicted image correction operation in the image encoding device 1 is described, and the same predicted image correction operation is also performed in the image decoding device 2. Figure 8 The operation flow may be performed in pixel units at the block boundaries between the inter-frame predicted image and the decoded block.
[0100] like Figure 8 As shown, in step S1 , the prediction accuracy evaluation unit 109 evaluates the prediction accuracy in the edge region of the inter prediction image based on the similarity between the plurality of reference images used by the inter prediction unit 108 to generate the inter prediction image.
[0101] In step S2 , the continuity evaluation unit 110 a evaluates the continuity between the inter-frame prediction image output by the inter-frame prediction unit 108 and the decoded adjacent blocks adjacent to the inter-frame prediction image.
[0102] In step S3 , the filter determination unit 110 b compares the prediction accuracy evaluated by the prediction accuracy evaluation unit 109 with the first threshold α, and compares the continuity evaluated by the continuity evaluation unit 110 a with the second threshold β.
[0103] When the prediction accuracy is less than the first threshold α and the continuity is less than the second threshold β (“Yes” in step S3), the filter determination unit 110b determines to perform filtering. In this case, in step S4, the filter processing unit 110c corrects the end area of the inter-frame prediction image by filtering using the decoded adjacent blocks (adjacent reference pixels).
[0104] On the other hand, when the prediction accuracy exceeds the first threshold α and / or the continuity exceeds the second threshold β (“No” in step S3 ), the filter determination unit 110b determines not to perform the filter process. In this case, the filter processing unit 110c does not perform the filter process.
[0105] (7. Summary of implementation methods)
[0106] The image encoding device 1 and the image decoding device 2 according to the present embodiment control the filtering process according to the evaluation result of the prediction accuracy of the inter-frame prediction. As a result, when the prediction accuracy of the end region of the inter-frame prediction image is high, it is possible to control not to apply the filtering process to the end region, thereby preventing the accuracy of the predicted image from being reduced due to the filtering process. In other words, since it is possible to set so that the filtering process can be applied to the end region only when the prediction accuracy of the end region of the inter-frame prediction image is low, the filtering process can be appropriately performed.
[0107] In addition, in the present embodiment, whether to apply filtering is determined by a common operation in the image encoding device 1 and the image decoding device 2. Specifically, the prediction accuracy of inter-frame prediction is evaluated in the image encoding device 1 and the image decoding device 2, and whether to apply filtering is determined based on the evaluation result. As a result, it is not necessary to transmit a flag indicating whether to apply filtering from the image encoding device 1 to the image decoding device 2, so that an increase in the amount of code due to the flag can be prevented.
[0108] (8. Modification 1 of the embodiment)
[0109] In the continuity evaluation and filtering process according to the above-described embodiment, the intra prediction mode (intra prediction direction) applied to the decoded adjacent blocks can be considered, thereby improving the accuracy of the continuity evaluation and filtering process. Fig. 9 is a diagram showing the structure of the predicted image correction unit 110 according to this modification. Fig. 9 As shown, the intra prediction mode applied to the decoded adjacent blocks is input from the intra prediction unit 107 to the continuity evaluation unit 110a and the filter processing unit 110c. The continuity evaluation unit 110a evaluates the continuity according to the intra prediction mode applied to the decoded adjacent blocks. In addition, the filter processing unit 110c performs filter processing according to the intra prediction mode applied to the decoded adjacent blocks.
[0110] Fig.10 1 is a diagram showing an example of the operation of the continuity evaluation unit 110a and the filter processing unit 110c according to this modification. Fig.10 As shown, when evaluating the continuity and filtering the end pixel q0 of the inter-frame prediction image corresponding to the encoding target block, the continuity evaluation unit 110a and the filter processing unit 110c calculate p1, p0, q1, and q2 on the straight line passing through q0 by weighted average according to the intra-frame prediction mode (intra-frame prediction direction) applied to the decoded adjacent block. Specifically, p1 and p0 on the straight line passing through q0 are calculated using the pixel values in the decoded adjacent block, and q1 and q2 on the straight line passing through q0 are calculated using the pixel values in the inter-frame prediction image. The continuity evaluation unit 110a uses the p1, p0, q1, and q2 calculated in this way to evaluate the continuity Cont by the above formula (2). The filter processing unit 110c uses the p1, p0, q1, and q2 calculated in this way to calculate q'0 by the above formula (3).
[0111] (9. Modification 2 of the embodiment)
[0112] In the above-described embodiment, the description is given of an example in which the predicted image correction unit 110 performs the filtering process on the edge region of the inter-frame predicted image using the decoded adjacent blocks as an example of the correction process on the inter-frame predicted image.
[0113] Furthermore, the correction process is not limited to such a filtering process, and a weighted average process of an inter-prediction image and an intra-prediction image may be used as the correction process.
[0114] In this modification, Figure 1 The intra prediction unit 107 and the predicted image correction unit 110 of the illustrated image encoding device 1 constitute a correction unit that performs correction processing on the inter-predicted image using decoded neighboring blocks adjacent to the target image block.
[0115] The intra prediction unit 107 generates an intra prediction image corresponding to the target image block using the decoded adjacent blocks. For example, the intra prediction unit 107 generates the intra prediction image from the decoded adjacent blocks using a predetermined intra prediction mode. The prediction image correction unit 110 performs a weighted average process on the inter prediction image input from the inter prediction unit 108 and the intra prediction image input from the intra prediction unit 107 as a correction process. In addition, it can also be configured so that a prediction mode is selected from a plurality of intra prediction mode candidates on the sending side to generate the intra prediction image. At this time, the entropy coding unit 103 entropy codes the intra prediction mode flag, which indicates which intra prediction mode is used on the sending side to make the intra prediction image.
[0116] same, Figure 2 The intra prediction unit 204 and the predicted image correction unit 207 of the image decoding device 2 shown in the figure constitute a correction unit for correcting the inter-predicted image using the decoded neighboring blocks adjacent to the target image block. The intra prediction unit 204 generates an intra-predicted image corresponding to the target image block using the decoded neighboring blocks.
[0117] For example, the intra prediction unit 204 generates an intra prediction image from the decoded adjacent blocks using a predetermined intra prediction mode. The prediction image correction unit 207 performs a weighted average process as a correction process on the inter prediction image input from the inter prediction unit 205 and the intra prediction image input from the intra prediction unit 204. Alternatively, the intra prediction mode used for generating the intra prediction image may be determined based on the intra prediction mode flag transmitted from the transmitting side.
[0118] In the case of applying the structure of this variation, it is sufficient to replace the "filtering process" in the above-mentioned embodiment with a "weighted average process". In addition, it is possible not only to determine whether to perform the weighted average process, but also to adjust the weights in the weighted average process based on (indicators) such as the evaluation results of the prediction accuracy and the evaluation results of the continuity.
[0119] <10. Other Implementation Methods>
[0120] In the above-mentioned embodiment, as the prediction using multiple reference images, dual prediction is mainly described. However, as the prediction using multiple reference images, a technique called intra-frame block copying can also be applied. In intra-frame block copying, a reference image in the same frame as the current frame is used for prediction of the object image block of the current frame.
[0121] It can also be provided by a program that causes a computer to execute various processes performed by the image encoding device 1 and a program that causes a computer to execute various processes performed by the image decoding device 2. In addition, the program can be recorded in a computer-readable medium. If a computer-readable medium is used, the program can be installed in the computer. Here, the computer-readable medium on which the program is recorded can also be a non-temporary recording medium. The non-temporary recording medium is not particularly limited, and can be, for example, a recording medium such as a CD-ROM or DVD-ROM. In addition, the circuits that execute the various processes performed by the image encoding device 1 can also be integrated, and the image encoding device 1 can be configured as a semiconductor integrated circuit (chip set, SoC). Similarly, the circuits that execute the various processes performed by the image decoding device 2 can also be integrated, and the image decoding device 2 can be configured as a semiconductor integrated circuit (chip set, SoC).
[0122] As mentioned above, although embodiment is demonstrated in detail with reference to drawings, the specific structure is not limited to the above description, Various design deformation|transformation etc. are possible within the range which does not deviate from the summary.
[0123] In addition, the entire contents of Japanese Patent Application No. 2018-72451 (filed on April 4, 2018) are incorporated into this application specification by reference.
Claims
1. A predictive image correction device, comprising: An inter-frame prediction unit generates a block of an inter-frame prediction image corresponding to a block of the target image by partitioning the image in frame units using block predictions of a plurality of reference images; an intra prediction unit for predicting the block of the target image by using adjacent reference pixels of the block of the target image to generate a block of an intra prediction image corresponding to the block of the target image; an evaluation unit that evaluates the accuracy of the inter-frame prediction image based on a value indicating a similarity between blocks of the plurality of reference images used in generating the blocks of the inter-frame prediction image; as well as A correction unit performs weighted averaging of blocks of the inter-frame prediction image and blocks of the intra-frame prediction image as correction processing only when the evaluation result of the accuracy satisfies a threshold condition, and the weight in the weighted averaging is adjusted based on a prediction mode that controls the intra-frame prediction processing related to the adjacent reference pixels.
2. An image encoding device comprising the predicted image correction device according to claim 1.
3. An image decoding device comprising the predicted image correction device according to claim 1.
4. The predictive image correction method comprises the following steps: Generate a block of an inter-frame prediction image corresponding to a block of a target image obtained by dividing an image in frame units using block predictions of a plurality of reference images; generating a block of an intra-frame prediction image corresponding to the block of the object image by predicting the block of the object image using adjacent reference pixels of the block of the object image; evaluating the accuracy of the inter-frame prediction image based on a value indicating a similarity between blocks of the plurality of reference images used in generating the blocks of the inter-frame prediction image; as well as A weighted average of blocks of the inter-frame prediction image and blocks of the intra-frame prediction image is performed as a correction process only when the evaluation result of the accuracy satisfies a threshold condition, and the weights in the weighted average are adjusted based on a prediction mode controlling the intra-frame prediction process related to the adjacent reference pixels.
5. Computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to claim 4 are implemented.
