Prediction device, encoding device, decoding device, and program

Abstract

This prediction device for performing prediction in units of blocks obtained by dividing an image comprises: a generation part for generating, at every prediction target region in the blocks, prediction regions by selecting a prediction process to be applied by threshold determination from among a plurality of prediction processes; a determination part for estimating or specifying, among the prediction regions generated by the generation part in the blocks, a specific region that has applied thereto a prediction process different from the prediction process applied to prediction regions therearound; and a correction part for performing a correction process, in which another region is used, with respect to the specific region estimated or specified by the determination part.

Description

Prediction device, encoding device, decoding device, and program

[0001] The present disclosure relates to a prediction device, an encoding device, a decoding device, and a program.

[0002] In video coding methods such as High Efficiency Video Codec (HEVC) and Versatile Video Coding (VVC), a coding device predicts coding blocks (CUs: Coding Units) obtained by dividing an original image into block units to generate prediction blocks, and then transforms, quantizes, and entropy-codes the prediction residual, which is the difference between the coding block of the original image and the prediction block, before transmitting it.

[0003] The International Video Coding Standardization Working Group (JVET) (ISO / IEC JTC1 SC29 WG5) is currently studying the Enhanced Compression Model (ECM), a next-generation video coding technology. The ECM introduces a color difference signal prediction mode called the Multi Model Linear Model (MMLM) as an intra-prediction mode, which is a prediction mode that takes into account correlation within a frame.

[0004] MMLM is an extension of the cross component linear model (CCLM) used in VVC. CCLM is a mode in which a decoded luminance block is used to predict a corresponding chrominance block, and predictive pixels of the target chrominance block are predicted using a linear model from the decoded luminance block at the corresponding position. Here, a reduced block obtained by downsampling the decoded luminance block is used depending on the chrominance format.

[0005] The linear model is calculated by least mean squares using decoded pixels (chrominance reference pixels) adjacent to the target chrominance block and decoded pixels (luminance reference pixels) adjacent to the corresponding luminance block. In VVC, the luminance and chrominance reference pixels used in linear model calculation are limited to only some pixel positions, thereby reducing the processing load.

[0006] CCLM assumes that there is a certain trend in the distribution of luminance and color difference signals within a block. Therefore, when the distribution of luminance and color difference signals includes multiple trends, such as when an object boundary exists within a block, the approximation accuracy significantly decreases. Therefore, MMLM assumes multiple distributions, clusters the distribution of luminance and color difference signals using, for example, the average value of luminance reference pixels as a threshold, and calculates a linear model for each cluster, thereby preventing a decrease in approximation accuracy. The linear model is an example of a prediction model.

[0007] Specifically, in MMLM, two linear models are calculated using luma reference pixels and chroma reference pixels, and then the linear model to be used for predicting the chroma block is selected depending on whether the pixel value of the luma decoded block at a position corresponding to the target chroma block is greater than a threshold. Each predicted pixel of the chroma block is generated while switching the linear model for each pixel of the luma decoded block at a position corresponding to the target chroma block.

[0008] JVET-D0110 “Enhanced Cross-component Linear Model Intra-prediction”

[0009] A prediction device according to a first aspect is a prediction device that performs predictions on a block-by-block basis obtained by dividing an image, and includes a generation unit that generates a prediction region by selecting a prediction process to be applied from among a plurality of prediction processes using a threshold judgment for each region to be predicted within the block, a determination unit that estimates or identifies a specific region among the prediction regions generated within the block by the generation unit to which a prediction process different from the prediction process applied to surrounding prediction regions has been applied, and a correction unit that performs correction processing on the specific region estimated or identified by the determination unit using other regions.

[0010] A coding device according to a second aspect includes the prediction device according to the first aspect.

[0011] A decoding device according to a third aspect includes the prediction device according to the first aspect.

[0012] A program according to a fourth aspect causes a computer to function as the prediction device according to the first aspect.

[0013] 1 is a diagram illustrating a configuration of an encoding device according to an embodiment. FIG. 2 is a diagram illustrating an example of an intra prediction mode according to an embodiment. FIG. 3 is a diagram illustrating an overview of MMLM according to an embodiment. FIG. 4 is a diagram illustrating an overview of MMLM according to an embodiment. FIG. 5 is a diagram illustrating an overview of MMLM according to an embodiment. FIG. 6 is a diagram illustrating an example of the configuration of an intra prediction unit on the encoding side according to an embodiment. FIG. 7 is a diagram illustrating an example of the operation of an MMLM prediction unit, an isolated pixel determination unit, and an isolated pixel correction unit regarding a first isolated pixel determination operation according to an embodiment. FIG. 8 is a diagram illustrating an example of the operation of an MMLM prediction unit, an isolated pixel determination unit, and an isolated pixel correction unit regarding a second isolated pixel determination operation according to an embodiment. FIG. 9 is a diagram illustrating a configuration of a decoding device according to an embodiment. FIG. 10 is a diagram illustrating a configuration of an intra prediction unit on the decoding side according to an embodiment. FIG. 11 is a diagram illustrating an example of the operation of an intra prediction unit on the decoding side according to an embodiment.

[0014] In MMLM, when a pixel value of a decoded luminance block at a position corresponding to a pixel to be predicted in a chrominance block is near a threshold, a prediction model different from a prediction model applied to a peripheral region (surrounding pixels) of the region to be predicted (pixel) is applied to the region to be predicted (pixel), and a specific region (hereinafter also referred to as an "isolated pixel") to which a prediction model different from that of the peripheral region (pixel) is applied may be generated. Note that, in the following embodiment, an example in which the prediction model is a linear model will be mainly described, but the prediction model is not limited to a linear model and may be a nonlinear model.

[0015] In such a specific region (isolated pixel), the prediction model used for prediction differs from that of the surrounding regions (pixels), causing discontinuity in the predicted pixels, which may result in a decrease in coding performance. This problem is not limited to MMLM, but may also occur in other coding tools that can switch prediction processes on a region (pixel) basis.

[0016] Therefore, the present disclosure provides a prediction device, an encoding device, a decoding device, and a program that suppress degradation of encoding performance caused by discontinuity of predicted pixels, even when prediction processing can be switched on a region (pixel) basis.

[0017] A prediction device according to the present disclosure is a device that performs prediction on a block-by-block basis obtained by dividing an image. The prediction device includes: a generation unit that generates a prediction region by selecting a prediction process to be applied from among a plurality of prediction processes using a threshold value for each region to be predicted within the block; a determination unit that estimates or identifies a specific region within the prediction region generated by the generation unit to which a prediction process different from a prediction process applied to surrounding prediction regions has been applied; and a correction unit that performs correction processing on the specific region estimated or identified by the determination unit using other regions.

[0018] In the embodiments, a "region" is one pixel, and a "region to be predicted" is a pixel to be predicted. However, the "region" may also be a pixel group consisting of two or more consecutive pixels. In the following embodiments, a "pixel to be predicted" may be read as a "pixel group to be predicted."

[0019] Furthermore, in the embodiments, a "specific region to which a prediction process different from that applied to surrounding prediction regions has been applied" is a pixel (isolated pixel) to which a prediction process different from that applied to surrounding prediction pixels has been applied. However, the "specific region to which a prediction process different from that applied to surrounding prediction regions has been applied" may also be a pixel group consisting of two or more consecutive pixels, and the "isolated pixel" in the following embodiments may be read as an "isolated pixel group."

[0020] An encoding device and a decoding device including an intra prediction device according to an embodiment will be described with reference to the drawings. The encoding device and the decoding device respectively encode and decode video (i.e., moving images) represented by MPEG. In the following description of the drawings, identical or similar parts are denoted by identical or similar reference numerals.

[0021] (1) Encoding Device The encoding device according to this embodiment will be described with reference to FIGS.

[0022] (1.1) Configuration of the Encoding Device First, the configuration of the encoding device 1 according to this embodiment will be described. Fig. 1 is a diagram showing the configuration of the encoding device 1 according to this embodiment.

[0023] The encoding device 1 is a device that encodes an input image to generate a bitstream and outputs the bitstream. The encoding device 1 includes a block division unit 100, a subtraction unit 110, a transform / quantization unit 120, an entropy encoding unit 130, an inverse quantization / inverse transform unit 140, a synthesis unit 150, a memory 160, and a prediction unit 170.

[0024] The block division unit 100 divides an original image, which is an input image in units of frames (or pictures) constituting a moving image, into a plurality of image blocks, and outputs the image blocks obtained by division to the subtraction unit 110. The size of the image blocks is, for example, 32 x 32 pixels, 16 x 16 pixels, 8 x 8 pixels, or 4 x 4 pixels. The shape of the image blocks is not limited to squares and may be rectangular (non-square). An image block is a unit for encoding by the encoding device 1 and a unit for decoding by the decoding device. Such an image block is also called a coding block (CU).

[0025] The input image is composed of a luminance signal (Y) and color difference signals (Cb, Cr), and each pixel in the input image is composed of a luminance component (Y) and a color difference component (Cb, Cr). The encoding device 1 supports three color difference formats, for example, 4:4:4, 4:2:2, and 4:2:0. The block division unit 100 performs block division on the luminance signal to output a luminance block, and performs block division on the color difference signal to output a color difference block. The shape of the block division may be the same for the luminance signal and the color difference signal, or the shape of the block division may be independently controllable for the luminance signal and the color difference signal.

[0026] The subtraction unit 110 calculates a prediction residual that represents the difference (error) between the coded block output by the block division unit 100 and a predicted block obtained by predicting the coded block by the prediction unit 170. Specifically, the subtraction unit 110 calculates the prediction residual by subtracting each pixel value of the predicted block from each pixel value of the block, and outputs the calculated prediction residual to the transformation / quantization unit 120.

[0027] The transform / quantization unit 120 performs transform processing and quantization processing on a block-by-block basis. The transform / quantization unit 120 includes a transform unit 121 and a quantization unit 122.

[0028] The transform unit 121 performs a transform process on the prediction residuals output by the subtraction unit 110 to calculate transform coefficients, and outputs the calculated transform coefficients to the quantization unit 122. Examples of transform include a discrete cosine transform (DCT), a discrete sine transform (DST), and a Karhunen-Loeve transform (KLT). The transform process includes a transform skip in which no transform process is performed. The transform skip also includes a transform in which only a horizontal transform process is applied, or a transform in which only a vertical transform process is applied. The transform unit 121 may also perform a secondary transform process in which a further transform process is applied to the transform coefficients obtained by the transform process. The secondary transform process may be applied to only a partial region of the transform coefficients.

[0029] The quantization unit 122 quantizes the transform coefficients output by the transform unit 121 using a quantization parameter and a quantization matrix, and outputs the quantized transform coefficients to the entropy coding unit 130 and the inverse quantization and inverse transform unit 140. The quantization parameter is a parameter that is commonly applied to each transform coefficient in a block and determines the coarseness of quantization. The quantization matrix is ​​a matrix whose elements are quantization values ​​used when quantizing each transform coefficient.

[0030] The entropy coding unit 130 performs entropy coding on the quantized transform coefficients output by the quantization unit 122, compresses the data, generates a bitstream, and outputs the bitstream to the outside of the coding device 1. For entropy coding, Huffman coding, CABAC (Context-based Adaptive Binary Arithmetic Coding), etc. may be used. Note that the entropy coding unit 130 receives input of prediction-related information (flags and indexes) from the prediction unit 170, and also encodes the input information and outputs a bitstream.

[0031] The inverse quantization and inverse transform unit 140 performs inverse quantization processing and inverse transform processing on a block-by-block basis. The inverse quantization and inverse transform unit 140 includes an inverse quantization unit 141 and an inverse transform unit 142.

[0032] The inverse quantization unit 141 performs inverse quantization processing corresponding to the quantization processing performed by the quantization unit 122. Specifically, the inverse quantization unit 141 reconstructs the transform coefficients by inverse quantizing the quantized transform coefficients output by the quantization unit 122 using a quantization parameter and a quantization matrix, and outputs the reconstructed transform coefficients to the inverse transform unit 142.

[0033] The inverse transform unit 142 performs inverse transform processing corresponding to the transform processing performed by the transform unit 121. For example, if the transform unit 121 performs discrete cosine transform, the inverse transform unit 142 performs inverse discrete cosine transform. The inverse transform unit 142 performs inverse transform processing on the transform coefficients output by the inverse quantization unit 141 to reconstruct prediction residuals, and outputs the reconstructed prediction residuals to the synthesis unit 150.

[0034] The synthesis unit 150 synthesizes the reconstructed prediction residual output by the inverse transform unit 142 by adding it pixel by pixel to the predicted block output by the prediction unit 170. The synthesis unit 150 adds each pixel value of the reconstructed prediction residual to each pixel value of the predicted block to decode (reconstruct) the block, and outputs the reconstructed block to the memory 160. Hereinafter, the reconstructed block is also referred to as a decoded block.

[0035] The memory 160 stores the reconstructed blocks output by the synthesis unit 150 and accumulates the reconstructed blocks as decoded images on a frame-by-frame basis. The memory 160 outputs the stored reconstructed blocks or decoded images to the prediction unit 170. Note that a loop filter may be provided between the synthesis unit 150 and the memory 160.

[0036] The prediction unit 170 performs prediction in units of blocks and includes an inter prediction unit 171, an intra prediction unit 172, and a switching unit 173.

[0037] The inter prediction unit 171 uses a decoded image stored in the memory 160 as a reference image to calculate a motion vector by a technique such as block matching, predicts a coding block to generate an inter prediction block, and outputs the generated inter prediction block to the switching unit 173. Here, the inter prediction unit 171 selects an optimal inter prediction method from inter prediction using multiple reference images (typically, bi-prediction) and inter prediction using one reference image (unidirectional prediction), and performs inter prediction using the selected inter prediction method. The inter prediction unit 171 outputs information related to the inter prediction (motion vectors, etc.) to the entropy coding unit 130.

[0038] The intra prediction unit 172 generates an intra prediction block by referring to decoded pixels surrounding a block in the decoded image stored in the memory 160, and outputs the generated intra prediction block to the switching unit 173. Generally, the intra prediction unit 172 selects an intra prediction mode to be applied to a predicted coding block of intra prediction from a plurality of intra prediction modes, and predicts the coding block of intra prediction using the selected intra prediction mode. The intra prediction unit 172 outputs information related to the selected intra prediction mode to the entropy coding unit 130.

[0039] The switching unit 173 switches between the inter-predicted block output by the inter-prediction unit 171 and the intra-predicted block output by the intra-prediction unit 172 , and outputs either the predicted block to the subtraction unit 110 and the synthesis unit 150 .

[0040] 2 is a diagram illustrating an example of intra prediction modes according to this embodiment. The intra prediction unit 172 performs intra prediction on a coding block. In the example shown, the candidate intra prediction modes for a luminance block are planar prediction, DC prediction, and 65 directional predictions, for a total of 67 intra prediction modes.

[0041] Prediction mode mode 0 is planar prediction, prediction mode mode 1 is DC prediction, and prediction modes 2 to 66 are directional prediction (intra angular). In directional prediction, the direction of the arrow indicates the prediction direction (reference direction), the starting point of the arrow indicates the position of the pixel to be predicted, and the end point of the arrow indicates the position of the reference pixel used for predicting this pixel to be predicted (also referred to as "reference pixel position"). A total of 65 modes of directional prediction are prepared, and the selectable prediction direction is determined by the shape (aspect ratio) of the block. Note that in the illustrated example, directional prediction is assumed to be in 65 directions, but the number of directional prediction directions may be more than 65 directions or less than 65 directions.

[0042] As prediction directions parallel to a diagonal line passing through the upper right vertex and the lower left vertex of a block, there are mode 2, which is a prediction mode that references the lower left direction, and mode 66, which is a prediction mode that references the upper right direction, and mode numbers are assigned clockwise from mode 2 to mode 66 at predetermined angles. Mode 34 is a prediction mode that references the upper left direction. Specifically, when the horizontal direction is 0°, the prediction direction of mode 2 is −45°, the prediction direction of mode 18 is 0°, the prediction direction of mode 34 is 45°, the prediction direction of mode 50 is 90°, and the prediction direction of mode 66 is 135°. Note that mode 18 is also referred to as horizontal prediction, and mode 50 is also referred to as vertical prediction.

[0043] Here, the directional predictions below mode 34, i.e., modes 2 to 33, are directional predictions that refer to the left side of the coding block, and the prediction direction is toward the left of the coding block. On the other hand, the directional predictions above mode 34, i.e., modes 35 to 66, are directional predictions that refer to the upper side of the coding block, and the prediction direction is toward the upper side of the coding block.

[0044] On the other hand, the number of candidate intra prediction modes for chrominance blocks is smaller than the number of candidate intra prediction modes for luminance blocks. Specifically, in intra prediction of a chrominance block, the intra prediction unit 172 determines the intra prediction mode used for intra prediction of a luminance block located at a position corresponding to the position of the chrominance block as a first candidate mode, determines a second candidate mode that does not overlap with the first candidate mode, and selects an intra prediction mode to be used for intra prediction of the chrominance block from among these candidate modes. Such a first candidate mode is called DM (Direct Mode or Derived Mode).

[0045] Furthermore, when any of the default modes predetermined as second candidate modes overlaps with the first candidate mode (DM), the intra prediction unit 172 may determine an alternative mode to be used as the second candidate mode in place of the overlapping default mode. Here, the default modes are planar mode (mode 0), vertical mode (mode 50), horizontal mode (mode 18), and DC mode (mode 1). As the alternative mode, a fixed intra prediction mode other than the default mode, for example, mode 66, is used.

[0046] Note that the luminance block located at a position corresponding to the position of a chrominance block refers to the luminance block located at the same position as the chrominance block when the luminance block and the chrominance block have the same block division shapes. However, when the luminance block division shapes and the chrominance block division shapes can be controlled independently, the luminance block located at a position corresponding to the position of the chrominance block refers to the luminance block including coordinates corresponding to a predetermined pixel position within the chrominance block (e.g., the coordinates of the upper left corner of the chrominance block). Here, the coordinates corresponding to the predetermined pixel position within the chrominance block are not necessarily the same because the sizes of luminance and chrominance may differ in a chrominance format such as 4:2:0.

[0047] Furthermore, as an intra prediction mode specific to chrominance blocks, there is inter-color component prediction, in which a chrominance block to be predicted is predicted from a decoded luminance block located at a position corresponding to the position of the chrominance block, using a linear model calculated from luminance and chrominance reference pixels surrounding the chrominance block to be predicted. In this embodiment, MMLM is used as the inter-color component prediction.

[0048] (1.2) Overview of MMLM Next, an overview of MMLM according to this embodiment will be described with reference to Figs. 3 to 6. MMLM is an extended technique of CCLM, which is inter-color component prediction adopted in VVC. The intra prediction unit 172 according to this embodiment performs intra prediction compatible with MMLM.

[0049] As shown in FIG. 3 , CCLM is a mode in which a decoded luma block ( FIG. 3( b)) is used to predict a corresponding chroma block ( FIG. 3( a)). The predicted pixels of the target chroma block are predicted from the decoded luma block at the corresponding position using a linear model ( FIG. 3( c)). Depending on the chroma format, a reduced block obtained by downsampling the decoded luma block may be used. The linear model is calculated by least mean squares using decoded pixels (chroma reference pixels) adjacent to the target chroma block and decoded pixels (luma reference pixels) adjacent to the decoded luma block at the corresponding position.

[0050] CCLM is based on the premise that there is a certain trend in the distribution of luminance and color difference signals within a block. Therefore, when the distribution of luminance and color difference signals contains multiple trends, such as when an object boundary exists within a block, as shown in Figure 4(a), there is a problem in that the approximation accuracy drops significantly.

[0051] Therefore, as shown in Figure 4(b), MMLM assumes multiple distributions, clusters the distributions of luminance and chrominance signals using, for example, the average value of luminance reference pixels as a threshold, and calculates a linear model for each cluster, thereby preventing a decrease in approximation accuracy. Specifically, MMLM calculates two linear models (a first linear model and a second linear model) using luminance reference pixels and chrominance reference pixels, and then selects the linear model to use for predicting the chrominance block depending on whether the pixel value of the luminance decoded block at a position corresponding to the target chrominance block is greater than a threshold. Each predicted pixel of the chrominance block is generated while switching the linear model for each pixel of the luminance decoded block at a position corresponding to the target chrominance block.

[0052] However, as shown in Figure 5, in MMLM, if the pixel value of the decoded luminance block at a position corresponding to the pixel to be predicted in the chrominance block is near a threshold, a linear model different from the linear model applied to the surrounding pixels of the pixel to be predicted in the chrominance block is applied to the pixel to be predicted, which may result in an isolated pixel to which a linear model different from that applied to the surrounding pixels is applied.

[0053] As shown in Figure 6, such isolated pixels cause discontinuities in the color difference predicted pixels (pred_cb) because the linear model used for prediction differs from that of the surrounding pixels, and there is a risk that the coding performance will deteriorate due to the discontinuity.

[0054] In this embodiment, when predicting pixels in a target chrominance block using MMLM, the intra prediction unit 172 estimates, as isolated pixels, pixels whose corresponding decoded luminance pixels have pixel values ​​close to a threshold, and performs correction processing on the estimated isolated pixels. For example, the correction processing is a filtering process using surrounding predicted pixels.

[0055] The determination of whether the pixel value of the corresponding luminance decoded pixel is close to the threshold value may be made based on a fluctuation value predetermined by the system, by determining whether the pixel value of the corresponding luminance decoded pixel is within the range of the threshold value ± the fluctuation value, or may be determined based on features of the distribution of pixel values ​​within the luminance decoded block, such as the average value or variance of the pixel values ​​of the decoded pixels within the block.

[0056] Alternatively, the intra prediction unit 172 may determine whether to perform filtering based on which of the two linear models is applied to pixels near the pixel in question, in addition to determining whether the pixel value is near the threshold value as described above. For example, if it has been determined that the linear model for the pixel in question is to perform prediction processing using the first linear model, and it has been determined that a predetermined number or more of the eight pixels near the pixel are to perform prediction processing using the second linear model, the pixel in question may be identified as an isolated pixel, and it may be determined that filtering is to be applied to the isolated pixel.

[0057] Furthermore, a similar filtering process may be applied to pixels in the vicinity of a pixel determined to be an isolated pixel.

[0058] (1.3) Configuration of Intra Prediction Unit Next, the configuration of the intra prediction unit 172 according to this embodiment will be described with reference to Figures 1 and 7. The intra prediction unit 172 corresponds to a prediction device that performs prediction in units of blocks obtained by dividing an image.

[0059] As shown in FIG. 1, the intra prediction unit 172 includes an MMLM prediction unit 10a, an isolated pixel determination unit 20a, and an isolated pixel correction unit 30a.

[0060] The MMLM prediction unit 10a corresponds to a generation unit that generates predicted pixels by selecting a prediction process to be applied from among multiple prediction processes using threshold value judgment for each pixel to be predicted (also referred to as a "target pixel") in a block to be predicted (also referred to as a "target block"). The isolated pixel determination unit 20a corresponds to a judgment unit that estimates or identifies isolated pixels to which a prediction process different from the prediction process applied to surrounding predicted pixels has been applied, among the predicted pixels generated in the target block by the MMLM prediction unit 10a. The isolated pixel correction unit 30a corresponds to a correction unit that performs correction processing on the isolated pixels estimated or identified by the isolated pixel determination unit 20a using at least one of predicted pixels in the target block and decoded pixels outside the target block.

[0061] In this way, the intra prediction unit 172 estimates or identifies isolated pixels, among the predicted pixels generated within the current block, to which a prediction process different from the prediction process applied to the surrounding predicted pixels has been applied, and performs a correction process on the isolated pixels using at least one of the predicted pixels within the block and the decoded pixels outside the block. This correction process can suppress discontinuity of predicted pixels caused by isolated pixels, thereby suppressing a deterioration in coding performance caused by discontinuity of predicted pixels.

[0062] In this embodiment, the target block is a chrominance block (also referred to as a "target chrominance block"). Each of the multiple prediction processes used by the MMLM prediction unit 10 a is a process for predicting pixels in the target chrominance block using a linear model generated using chrominance reference pixels around the target chrominance block and luminance reference pixels around a predetermined luminance block at a position corresponding to the target chrominance block. Here, the multiple prediction processes each use a different linear model.

[0063] 7 is a diagram showing the configuration of the intra prediction unit 172 according to this embodiment. As shown in FIG. 7, the MMLM prediction unit 10a includes a threshold determination unit 11a, a linear model generation unit 12a, a linear model selection unit 13a, and an inter-color component prediction unit 14a.

[0064] The threshold determination unit 11a determines one or more thresholds to be used for selecting a linear model from the luminance reference pixels. In this embodiment, the threshold determination unit 11a determines one threshold. Specifically, the threshold determination unit 11a determines the threshold to be used for selecting a linear model from the luminance reference pixels adjacent to the luminance block at a position corresponding to the target chrominance block.

[0065] Here, the threshold determination unit 11a may determine the threshold based on, for example, the average value of the luminance reference pixels, or may determine the threshold using another clustering method. Furthermore, the threshold determination unit 11a may determine the threshold by directly using the luminance reference pixels, or may perform subsampling, filtering, or the like to make the luminance reference pixels correspond to the pixel positions of the chrominance reference pixels. For example, when the chrominance format is 4:2:0, the luminance signal has an area twice as large vertically and horizontally as the chrominance signal (see FIG. 3 ). Therefore, as shown in FIG. 3B , the threshold determination unit 11a can align the positions of the luminance reference pixels with the positions of the chrominance reference pixels by performing low-pass filtering on the luminance reference pixels and then subsampling every two pixels vertically and horizontally.

[0066] The linear model generation unit 12a generates a linear model for each cluster determined by the threshold determined by the threshold determination unit 11a. That is, the linear model generation unit 12a generates a plurality of linear models using the threshold determined by the threshold determination unit 11a.

[0067] 4B, the linear model generation unit 12a identifies luminance reference pixels whose pixel values ​​are equal to or less than a threshold (or less than the threshold), and generates a first linear model using chrominance reference pixels at positions corresponding to the identified luminance reference pixels. The linear model generation unit 12a also identifies luminance reference pixels whose pixel values ​​are greater than or equal to the threshold (or greater than or equal to the threshold), and generates a second linear model using chrominance reference pixels at positions corresponding to the identified luminance reference pixels.

[0068] Here, the linear model can be expressed by, for example, Pred_c = a * Rec_y + b, where Prec_c means a predicted pixel of chrominance, and Rec_y means a decoded pixel of luminance at the corresponding position. a and b are coefficients, and the linear model generation unit 12a can set different values ​​for a and b in the first linear model and the second linear model.

[0069] To calculate the coefficients a and b, the linear model generation unit 12a uses a chrominance reference pixel (Ref_c) adjacent to the target chrominance block and a luminance reference pixel (Ref_l) at a position corresponding to the chrominance reference pixel to calculate the coefficients a and b such that arg min_{a,b} [Ref_c - (a * Ref_l + b)], thereby generating a first linear model and a second linear model.

[0070] The linear model selection unit 13a compares, for each chrominance pixel (target pixel) in the target chrominance block, the pixel value of the corresponding luminance pixel in a predetermined luminance block with a threshold value to select a linear model to be used for predicting the chrominance pixel. That is, the linear model selection unit 13a selects a linear model to be used for predicting the target pixel in the target chrominance block based on the pixel value of a decoded pixel in the luminance block at a position corresponding to the target pixel in the target chrominance block.

[0071] Specifically, the linear model selection unit 13 a selects, for each target pixel in the target chrominance block, which linear model to use for generating a predicted pixel from among the linear models generated by the linear model generation unit 12 a. Here, for each target pixel in the target chrominance block, the linear model selection unit 13 a selects a linear model to use for inter-color component prediction depending on whether the pixel value of a decoded luminance pixel at a position corresponding to the target pixel is equal to or less than a threshold value (or less).

[0072] For example, when the pixel value of the decoded luminance pixel corresponding to the target pixel in the target chrominance block is equal to or less than a threshold, the linear model selection unit 13 a determines to use the first linear model for inter-color component prediction of the target pixel in the target chrominance block, whereas when the pixel value of the decoded luminance pixel corresponding to the target pixel in the target chrominance block is greater than or equal to the threshold, the linear model selection unit 13 a determines to use the second linear model for inter-color component prediction of the target pixel in the target chrominance block.

[0073] The linear model selection unit 13a may select a linear model for each pixel using the decoded luminance pixels as they are, or may perform subsampling, filtering, or the like on the decoded luminance pixels to correspond to the position of the target pixel in the target chrominance block and use the resultant linear model for selection. For example, when the chrominance format is 4:2:0, the luminance block has an area twice as large vertically and horizontally as the chrominance block. Therefore, the linear model selection unit 13a can align the position of the decoded luminance pixels with the position of the target pixel in the target chrominance block by performing low-pass filtering on the decoded luminance pixels and then subsampling every two pixels vertically and horizontally.

[0074] The inter-color component prediction unit 14a predicts a target pixel in a target chrominance block by inter-color component prediction using the linear model selected by the linear model selection unit 13a, and generates a predicted pixel for the target pixel. That is, the inter-color component prediction unit 14a performs inter-color component prediction for each target pixel in the target chrominance block based on the linear model selected for each target pixel in the target chrominance block by the linear model selection unit 13a, and generates a predicted pixel for each target pixel in the target chrominance block. Specifically, the inter-color component prediction unit 14a switches the coefficients a and b of the linear model to be applied for each target pixel in the target chrominance block to generate a predicted pixel.

[0075] The isolated pixel determination unit 20a estimates or identifies isolated pixels that have been subjected to a prediction process different from the prediction process applied to surrounding predicted pixels among the predicted pixels generated within the target chrominance block by the inter-color component prediction unit 14a. The operation of the isolated pixel determination unit 20a includes a first isolated pixel determination operation and a second isolated pixel determination operation, which will be described later. These operations will be described in detail later.

[0076] The isolated pixel correction unit 30a performs correction processing on the isolated pixel estimated or identified by the isolated pixel determination unit 20a using predicted pixels in the target block. In this embodiment, the isolated pixel correction unit 30a performs filter processing on the isolated pixel using predicted pixels surrounding the isolated pixel as the correction processing. Details of the operation of the isolated pixel correction unit 30a will be described later. In this embodiment, the isolated pixel correction unit 30a performs filter processing on the isolated pixel among the predicted pixels generated in the target chrominance block by the inter-color component prediction unit 14a, but does not perform filter processing on other predicted pixels. This makes it possible to prevent a situation in which image quality is degraded by unnecessary filter processing.

[0077] (1.4) Operations of the Isolated Pixel Determining Unit and the Isolated Pixel Correcting Unit Next, the operations of the isolated pixel determining unit 20a and the isolated pixel correcting unit 30a according to this embodiment will be described with reference to FIGS.

[0078] (1.4.1) First isolated pixel determination operation The MMLM prediction unit 10a selects a prediction process to be applied to each target pixel in a target chrominance block by comparing the corresponding pixel value (specifically, the pixel value of the luminance-decoded pixel at the corresponding position) with a threshold value.

[0079] In the first isolated pixel determination operation, the isolated pixel determination unit 20a estimates, as isolated pixels, pixels in the target chrominance block whose corresponding pixel values ​​are close to a threshold value. The isolated pixel correction unit 30a controls filtering for each target pixel in the target chrominance block depending on whether the corresponding pixel value is close to the threshold value.

[0080] That is, in the first isolated pixel determination operation, the isolated pixel determination unit 20 a controls whether or not to perform filtering processing for the correction processing for each target pixel in the target chrominance block, depending on the value of the decoded luminance pixel at the corresponding position. As described above, when the values ​​of the decoded luminance pixels are concentrated near the threshold value determined by the threshold determination unit 11 a, there is a high possibility that an isolated pixel will occur for which the applied linear model differs from that of the surrounding pixels.

[0081] Therefore, in the first isolated pixel determination operation, when the value of a decoded luminance pixel is close to the threshold determined by the threshold determination unit 11a, the isolated pixel determination unit 20a and the isolated pixel correction unit 30a estimate a chrominance predicted pixel at a position corresponding to the decoded luminance pixel as an isolated pixel and perform filter processing on the isolated pixel. Performing such filter processing makes it possible to suppress discontinuities that occur when an isolated pixel is subjected to prediction processing using a linear model that differs from that of its surrounding pixels. For example, when the threshold is th and the value of the decoded luminance pixel is rec_L, the pixel may be determined to be an isolated pixel if rec_L > th - K && rec_L < th + K is satisfied, and filter processing is applied.

[0082] Here, K is a constant equal to or greater than 0 that represents a range of determination (when K is 0, control is possible so that filtering is not applied), and may be predetermined by the system or may be determined according to feature quantities such as block size, block shape, and / or mean and variance of pixel values ​​in a decoded luminance block. The feature quantities are not limited to mean and variance, and any index that represents the characteristics of the distribution of pixels in a decoded luminance block can be used. Furthermore, the constant K may be determined according to at least one of the surrounding prediction modes, frame types (I slice, B slice, P slice), and reference structure.

[0083] FIG. 8 is a diagram showing an example of the operations of the MMLM prediction unit 10a, the isolated pixel determination unit 20a, and the isolated pixel correction unit 30a regarding the first isolated pixel determination operation according to this embodiment.

[0084] In steps S101 and S102, when the height of the target chrominance block is CUheight and the width of the target chrominance block is CUwidth, the MMLM prediction unit 10a performs the following loop processing for each target pixel [i, j] in the target chrominance block.

[0085] Specifically, in step S103, the MMLM prediction unit 10 a determines whether the pixel value recLuma[i,j] of a decoded luminance pixel at a position corresponding to the target pixel [i,j] in the target chrominance block is equal to or less than a threshold th. If the pixel value recLuma[i,j] of the decoded luminance pixel is equal to or less than the threshold th, in step S104, the MMLM prediction unit 10 a calculates a predicted chrominance pixel value pred[i,j] from the pixel value recLuma[i,j] of the decoded luminance pixel using a first linear model: pred[i,j] = a0 * recLuma[i,j] + b0.

[0086] On the other hand, if the pixel value recLuma[i,j] of the decoded luminance pixel is greater than the threshold value th (step S105), in step S106, the MMLM prediction unit 10a calculates the predicted chrominance pixel value pred[i,j] from the pixel value recLuma[i,j] of the decoded luminance pixel using the second linear model: pred[i,j] = a1 * recLuma[i,j] + b1.

[0087] Then, in steps S107 and S108, the isolated pixel determination unit 20a and the isolated pixel correction unit 30a perform the following loop processing for each target pixel [i, j] in the target chrominance block, assuming that the height of the target chrominance block is CUheight and the width of the target chrominance block is CUwidth.

[0088] Specifically, in step S109, the isolated pixel determination unit 20a determines whether the pixel value recLuma[i,j] of the decoded luminance pixel is within a predetermined range (±K) based on the threshold value th. In other words, the isolated pixel determination unit 20a determines whether the pixel value recLuma[i,j] of the decoded luminance pixel is near the threshold value th. In other words, the isolated pixel determination unit 20a determines whether the pixel value recLuma[i,j] of the decoded luminance pixel is greater than the threshold value th-K and less than the threshold value th+K.

[0089] When the pixel value recLuma[i,j] of a decoded luminance pixel is close to the threshold th, the corresponding predicted chrominance pixel pred[i,j] can be estimated to be an isolated pixel. In this case, in step S110, the isolated pixel correction unit 30a performs a filter process on the chrominance predicted pixel pred[i,j] using the surrounding predicted pixels (in the illustrated example, pred[i-1,j], pred[i+1,j], pred[i,j-1], and pred[i,j+1]) of the chrominance predicted pixel pred[i,j] as follows: pred[i,j] = (pred[i,j] * c0 + pred[i-1,j] * c1 + pred[i+1,j] * c2 + pred[i,j-1] * c3 + pred[i,j+1] * c4) / (c0 + c1 + c2 + c3 + c4). Here, c0, c1, c2, c3, and c4 are filter coefficients (weighting coefficients), which may be predetermined by the system or variably settable. Such filtering can reduce discontinuities between isolated pixels and their surrounding predicted pixels.

[0090] Although an example has been described in which the surrounding predicted pixels used in the filtering process are four (pred[i-1,j], pred[i+1,j], pred[i,j-1], pred[i,j+1]) from above, below, left, and right, the number of surrounding predicted pixels used in the filtering process is not limited to four. For example, the surrounding predicted pixels used in the filtering process may be eight (pred[i-1,j-1], pred[i-1,j], pred[i-1,j+1], pred[i,j-1], pred[i,j+1], pred[i+1,j-1], pred[i+1,j], pred[i+1,j+1]) by adding diagonal predicted pixels. The same applies to the second isolated pixel determination operation described below.

[0091] (1.4.2) Second Isolated Pixel Determination Operation In the second isolated pixel determination operation, the isolated pixel determination unit 20a stores the prediction process applied to each target pixel in the target chrominance block according to the result of threshold determination. The isolated pixel determination unit 20a then identifies, as isolated pixels, target pixels in the target chrominance block to which the first prediction process has been applied and to which a second prediction process different from the first prediction process has been applied to a predetermined number or more of surrounding pixels. Although the second isolated pixel determination operation requires storage capacity for storing the prediction process applied to each target pixel in the target chrominance block, it is possible to perform isolated pixel determination with higher accuracy than the first isolated pixel determination operation.

[0092] In the second isolated pixel determination operation according to this embodiment, with regard to the correction process for each target pixel in a target chrominance block, the isolated pixel determination unit 20a and the isolated pixel correction unit 30a control whether to perform filtering depending on the linear model applied to the target pixel and the linear model applied to the predicted pixels surrounding the target pixel. For example, if the first linear model has been applied to pixels surrounding a certain target pixel and the second linear model has been applied to the target pixel, the isolated pixel determination unit 20a can identify the target pixel as an isolated pixel. The isolated pixel correction unit 30a can suppress the discontinuity described above by applying filtering to the identified isolated pixel.

[0093] For example, the isolated pixel determination unit 20a stores the type of linear model applied to the target pixel (i, j) as flag[i, j]. Here, if the first linear model is applied, flag[i, j] = 0, and if the second linear model is applied, flag[i, j] = 1. The isolated pixel determination unit 20a calculates the aggregate sum of the linear models applied to the surrounding predicted pixels as follows: sum = flag[i-1, j-1] + flag[i-1, j] + flag[i-1, j+1] + flag[i, j-1] + flag[i, j+1] + flag[i+1, j-1] + flag[i+1, j] + flag[i+1, j+1].

[0094] A larger sum value indicates that the second linear model is applied to a larger proportion of surrounding predicted pixels, whereas a smaller sum value indicates that the first linear model is applied to a larger proportion of surrounding pixels.

[0095] When the first linear model is applied to a target pixel in a target chrominance block (flag[i, j] = 0) and the value of sum is greater than a predetermined value N, the isolated pixel determination unit 20a identifies the target pixel as an isolated pixel, and the isolated pixel correction unit 30a applies filtering. Similarly, when the second linear model is applied to a target pixel in a target chrominance block (flag[i, j] = 1) and the value of sum is less than a predetermined value M, the isolated pixel determination unit 20a identifies the target pixel as an isolated pixel, and the isolated pixel correction unit 30a applies filtering. In all other cases, the target pixel is determined not to be an isolated pixel, and filtering is not applied.

[0096] FIG. 9 is a diagram showing an example of the operations of the MMLM prediction unit 10a, the isolated pixel determination unit 20a, and the isolated pixel correction unit 30a regarding the second isolated pixel determination operation according to this embodiment.

[0097] In steps S201 and S202, when the height of the target chrominance block is CUheight and the width of the target chrominance block is CUwidth, the MMLM prediction unit 10a performs the following loop processing for each target pixel [i, j] in the target chrominance block.

[0098] Specifically, in step S203, the MMLM prediction unit 10a determines whether the pixel value recLuma[i,j] of the decoded luminance pixel at a position corresponding to the target pixel [i,j] in the target chrominance block is equal to or less than a threshold th. If the pixel value recLuma[i,j] of the decoded luminance pixel is equal to or less than the threshold th, in step S204, the MMLM prediction unit 10a calculates a predicted chrominance pixel value pred[i,j] from the pixel value recLuma[i,j] of the decoded luminance pixel using the first linear model: pred[i,j] = a0 * recLuma[i,j] + b0. In this case, in step S205, the isolated pixel determination unit 20a sets and stores a flag flag[i,j] corresponding to the predicted pixel pred[i,j] to 0.

[0099] On the other hand, if the pixel value recLuma[i,j] of the decoded luminance pixel is greater than the threshold value th (step S206), then in step S207, the MMLM prediction unit 10a calculates a predicted chrominance pixel value pred[i,j] from the pixel value recLuma[i,j] of the decoded luminance pixel using the second linear model: pred[i,j] = a1 * recLuma[i,j] + b1. In this case, in step S208, the isolated pixel determination unit 20a sets and stores a flag flag[i,j] corresponding to the predicted pixel pred[i,j] to 1.

[0100] Then, in steps S209 and S210, the isolated pixel determination unit 20a and the isolated pixel correction unit 30a perform the following loop processing for each target pixel [i, j] in the target chrominance block, assuming that the height of the target chrominance block is CUheight and the width of the target chrominance block is CUwidth.

[0101] Specifically, in step S211, the isolated pixel determination unit 20a calculates the aggregate sum of the linear models applied to the surrounding predicted pixels.

[0102] Then, in step S212, the isolated pixel determination unit 20a determines whether a first condition is satisfied, that is, the flag corresponding to the target pixel (predicted pixel) [i, j] is 0 and sum is greater than N, or a second condition is satisfied, that is, the flag corresponding to the target pixel (predicted pixel) [i, j] is 1 and sum is smaller than M. If the first condition or the second condition is satisfied, the isolated pixel determination unit 20a identifies the target pixel (predicted pixel) [i, j] as an isolated pixel.

[0103] Then, in step S213, the isolated pixel correction unit 30a performs a filter process on the predicted pixel pred[i,j] identified as an isolated pixel, using the predicted pixels surrounding the predicted pixel pred[i,j] (in the illustrated example, pred[i-1,j], pred[i+1,j], pred[i,j-1], and pred[i,j+1]), as follows: pred[i,j] = (pred[i,j] * c0 + pred[i-1,j] * c1 + pred[i+1,j] * c2 + pred[i,j-1] * c3 + pred[i,j+1] * c4) / (c0 + c1 + c2 + c3 + c4). Here, c0, c1, c2, c3, and c4 are filter coefficients (weighting coefficients), which may be predetermined by the system or variably settable. Such filtering can reduce discontinuities between isolated pixels and their surrounding predicted pixels.

[0104] (2) Decoding Device Next, the decoding device 2 according to this embodiment will be described with reference to FIGS.

[0105] 10 is a diagram showing the configuration of a decoding device 2 according to this embodiment. The decoding device 2 is a device that derives and outputs a decoded image from an input bitstream. The decoding device 2 includes an entropy decoding unit 200, an inverse quantization and inverse transform unit 210, a synthesis unit 220, a memory 230, and a prediction unit 240.

[0106] The entropy decoding unit 200 decodes the bitstream generated by the encoding device 1 and outputs the quantized transform coefficients to the inverse quantization and inverse transform unit 210. The entropy decoding unit 200 also obtains information related to prediction (intra prediction and inter prediction) and outputs the obtained information to the prediction unit 240. In this embodiment, the entropy decoding unit 200 may obtain a flag indicating that MMLM is applied and output the flag to the prediction unit 240.

[0107] The inverse quantization and inverse transform unit 210 performs inverse quantization and inverse transform processing on a block-by-block basis. The inverse quantization and inverse transform unit 210 includes an inverse quantization unit 211 and an inverse transform unit 212.

[0108] The inverse quantization unit 211 performs inverse quantization processing corresponding to the quantization processing performed by the quantization unit 122 of the encoding device 1. The inverse quantization unit 211 reconstructs the transform coefficients of the coding block by inverse quantizing the quantized transform coefficients output by the entropy decoding unit 200 using the quantization parameter and the quantization matrix, and outputs the reconstructed transform coefficients to the inverse transform unit 212.

[0109] The inverse transform unit 212 performs inverse transform processing corresponding to the transform processing performed by the transform unit 121 of the encoding device 1. The inverse transform unit 212 performs inverse transform processing on the transform coefficients output by the inverse quantization unit 211 to reconstruct prediction residuals, and outputs the reconstructed prediction residuals to the synthesis unit 220. The inverse transform processing includes transform skip, in which the inverse transform processing is not performed. Furthermore, the inverse transform unit 212 may perform inverse secondary transform processing in which a further inverse transform processing is applied to the signal obtained by the inverse transform processing.

[0110] The synthesis unit 220 synthesizes the prediction residual output by the inverse transform unit 212 and the prediction block output by the prediction unit 240 by adding them together pixel by pixel, decodes (reconstructs) the original block, and outputs the reconstructed block to the memory 230.

[0111] The memory 230 stores the reconstructed blocks output by the synthesis unit 220 and accumulates the reconstructed blocks as decoded images on a frame-by-frame basis. The memory 230 outputs the reconstructed blocks or the decoded images to the prediction unit 240. The memory 230 also outputs the decoded images on a frame-by-frame basis to the outside of the decoding device 2. Note that a loop filter may be provided between the synthesis unit 220 and the memory 230.

[0112] The prediction unit 240 performs prediction in units of blocks and includes an inter prediction unit 241, an intra prediction unit 242, and a switching unit 243.

[0113] The inter prediction unit 241 predicts a coded block by inter prediction using a decoded image stored in the memory 230 as a reference image. The inter prediction unit 241 generates an inter prediction block by performing inter prediction in accordance with the motion vector information output by the entropy decoding unit 200, and outputs the generated inter prediction block to the switching unit 243.

[0114] The intra prediction unit 242 generates an intra prediction block by referring to decoded pixels surrounding a block to be predicted (encoded block) among the decoded images stored in the memory 230, and outputs the generated intra prediction block to the switching unit 243. The intra prediction unit 242 according to this embodiment performs intra prediction corresponding to the above-mentioned MMLM.

[0115] The switching unit 243 switches between the inter-predicted block output by the inter-prediction unit 241 and the intra-predicted block output by the intra-prediction unit 242 , and outputs either the predicted block to the synthesis unit 220 .

[0116] 11 is a diagram showing the configuration of the intra prediction unit 242 according to this embodiment. As shown in Fig. 10 and Fig. 11 , the intra prediction unit 242 includes an MMLM prediction unit 10b (a threshold determination unit 11b, a linear model generation unit 12b, a linear model selection unit 13b, and an inter-color component prediction unit 14b), an isolated pixel determination unit 20b, and an isolated pixel correction unit 30b.

[0117] Here, the MMLM prediction unit 10b (threshold determination unit 11b, linear model generation unit 12b, linear model selection unit 13b, and inter-color component prediction unit 14b), the isolated pixel determination unit 20b, and the isolated pixel correction unit 30b perform the same processing as the MMLM prediction unit 10a (threshold determination unit 11a, linear model generation unit 12a, linear model selection unit 13a, and inter-color component prediction unit 14a) on the encoding side, the isolated pixel determination unit 20a, and the isolated pixel correction unit 30a, respectively.

[0118] (3) Operation of Intra Prediction Next, an example of the operation of intra prediction according to this embodiment will be described. Fig. 12 is a diagram showing an example of the operation of the intra prediction unit 242 on the decoding side according to this embodiment. Here, the operation of the intra prediction unit 242 on the decoding side will be described as an example, but the intra prediction unit 172 on the encoding side also performs the same operation as the intra prediction unit 242 on the decoding side. When a flag indicating that MMLM is to be applied to the chroma block to be decoded is signaled from the encoding side, the intra prediction unit 242 performs the following operation.

[0119] In step S1, the MMLM prediction unit 10b selects a prediction process to be applied from among a plurality of prediction processes by threshold value determination for each pixel to be predicted in the target chrominance block, and generates a predicted pixel.

[0120] In step S2, the isolated pixel determination unit 20b estimates, by the above-mentioned first isolated pixel determination operation, or identifies, by the above-mentioned second isolated pixel determination operation, isolated pixels to which a prediction process different from the prediction process applied to the surrounding predicted pixels has been applied, among the predicted pixels generated by the MMLM prediction unit 10b within the target chrominance block.

[0121] In step S3, the isolated pixel correction unit 30b performs a filtering process on the isolated pixel estimated or identified by the isolated pixel determination unit 20b using surrounding predicted pixels in the target chrominance block.

[0122] (4) Summary of the Embodiment Each intra prediction unit 172, 242 according to the present embodiment constitutes a prediction device that performs prediction on a block-by-block basis obtained by dividing an image. Each intra prediction unit 172, 242 includes an MMLM prediction unit 10 that generates predicted pixels by selecting a prediction process to be applied from among multiple prediction processes using a threshold value for each pixel to be predicted in a target chrominance block, an isolated pixel determination unit 20 that estimates or identifies isolated pixels that have been applied with a prediction process different from the prediction processes applied to surrounding predicted pixels among the predicted pixels generated in the target chrominance block by the MMLM prediction unit 10, and an isolated pixel correction unit 30 that performs correction processing (filtering processing) on ​​the isolated pixels estimated or identified by the isolated pixel determination unit 20 using surrounding predicted pixels in the target chrominance block.

[0123] By controlling the filtering process on a pixel-by-pixel basis in this manner, filtering can be performed only in regions where discontinuity occurs due to isolated pixels and prediction accuracy is reduced, thereby suppressing the discontinuity. Furthermore, since filtering is not performed in other regions where prediction accuracy is high, it is possible to suppress image blurring due to filtering. Therefore, according to the intra prediction device of this embodiment, even when prediction processes can be switched on a pixel-by-pixel basis, it is possible to suppress degradation of coding performance due to discontinuity of predicted pixels.

[0124] (5) Controlling whether or not to apply filter processing In the above embodiment, an example was described in which a flag indicating the application of MMLM is signaled from the entropy coding unit 130 of the coding device 1 to the entropy decoding unit 200 of the decoding device 2, i.e., included in the bitstream and transmitted.

[0125] In addition to or instead of such a flag, a flag (also referred to as a "new flag") indicating whether or not the filter processing according to the above-mentioned embodiment is applicable may be signaled from the entropy coding unit 130 of the coding device 1 to the entropy decoding unit 200 of the decoding device 2.

[0126] For example, the new flag may be an enable flag that indicates that the filter process can be applied when it is true ("1") and indicates that the filter process cannot be applied when it is false ("0"), or a disable flag that indicates that the filter process cannot be applied when it is true ("1") and indicates that the filter process can be applied when it is false ("0").

[0127] Such a new flag may be a flag for each sequence, each picture, or each slice. In the case of a sequence, the new flag may be signaled by being included in a Sequence Parameter Set (SPS). In the case of a picture, the new flag may be signaled by being included in a Picture Parameter Set (PPS) or a Picture Header. In the case of a slice, the new flag may be signaled by being included in a Slice Header.

[0128] Alternatively, the encoding side and the decoding side may autonomously determine whether to apply the filtering process according to the above-described embodiment using a common algorithm without signaling a new flag. For example, each of the encoding side and the decoding side may determine whether to apply the filtering process according to the above-described embodiment without a flag, based on features of a picture or a sequence (e.g., features, variance, average, etc. calculated from the distribution of pixel values).

[0129] (6) Modified Example of Chrominance Intra Prediction Processing In the above-described embodiment, an MMLM is described in which a chrominance prediction pixel is generated by switching between two linear models for each target pixel. However, the number of linear models is not limited to two, and three or more linear models may be switchable for each target pixel.

[0130] In this modified example, the isolated pixel correction unit 30 can switch between three or more linear models for each pixel. When switching between three or more linear models, the MMLM prediction unit 10 switches between linear models using two thresholds (threshold 1 and threshold 2). Even with this configuration, the linear model is switched by threshold determination for each pixel. The isolated pixel determination unit 20 and the isolated pixel correction unit 30 can improve prediction accuracy by determining for each pixel whether or not to perform filtering based on threshold 1 and / or threshold 2 and the feature amount (luminance pixel value) used in the threshold determination.

[0131] (7) Modified Example of Filtering In the above embodiment, an example was described in which the first isolated pixel determination operation or the second isolated pixel determination operation was used to control whether or not to perform filtering on each target pixel. However, instead of controlling whether or not to perform filtering, the type of filtering (such as the strength of the filter) may be controlled. In such a modified example, the isolated pixel correction unit 30 applies filtering with different filter lengths or different filter strengths based on the determination result of whether or not the target pixel is an isolated pixel. For example, when the target pixel is an isolated pixel, the isolated pixel correction unit 30 may apply filtering with a longer filter or a stronger filter strength than when the target pixel is not an isolated pixel.

[0132] In the above embodiment, an example has been described in which the isolated pixel correction unit 30 performs filtering using only predicted pixels within the target chrominance block. However, the isolated pixel correction unit 30 may perform filtering using chrominance reference pixels outside the target chrominance block and located near the target chrominance block, instead of or in addition to the predicted pixels within the target chrominance block. By performing filtering using such chrominance reference pixels outside the block, it is expected that prediction accuracy at boundaries of predicted images will be improved.

[0133] (8) Modification Example Using Non-Adjacent Reference Pixels In the above embodiment, an example has been described in which the MMLM prediction unit 10 uses adjacent reference pixels in calculating a linear model. However, the MMLM prediction unit 10 may calculate a linear model using nearby decoded pixels that are not adjacent to the target chrominance block as reference pixels, as in intra prediction using non-adjacent reference pixels (MRL: Multi Reference Line intra prediction) employed in VVC.

[0134] When the isolated pixel correction unit 30 performs filtering using chrominance reference pixels outside the target chrominance block and the MMLM prediction unit 10 is capable of performing the linear model calculation using the non-adjacent reference pixels described above for the target chrominance block, the isolated pixel correction unit 30 may control the filtering based on whether the linear model calculation was performed using non-adjacent reference pixels. For example, when the MMLM prediction unit 10 performs MMLM using non-adjacent reference pixels, the chrominance reference pixels used to generate the linear model are not adjacent to the block, and therefore filtering using the chrominance reference pixels may actually reduce the prediction accuracy near the boundary of the predicted image. Therefore, when the chrominance reference pixels are not adjacent to the target chrominance block, filtering may be performed using only the predicted pixels within the target chrominance block, and the chrominance reference pixels may not be used in the filtering. Furthermore, when a linear model of MMLM is performed using non-adjacent chrominance reference pixels, filtering may be controlled not to be performed.

[0135] Furthermore, when a linear model of an MMLM is generated using non-adjacent chrominance reference pixels for a target chrominance block, filtering may be performed using decoded pixels adjacent to the target chrominance block, rather than the non-adjacent chrominance reference pixels. While it is necessary to store both the non-adjacent chrominance reference pixels for generating the linear model and the adjacent decoded chrominance pixels for filtering in memory, using adjacent decoded chrominance pixels can improve prediction accuracy at boundary portions of the predicted image of the target chrominance block.

[0136] On the other hand, when the reference pixels used to generate the MMLM linear model are non-adjacent reference pixels, the filtering process may be performed using the reference pixels used to generate the linear model. While this may not improve prediction accuracy in some cases, it enables the filtering process without the need to store the chrominance adjacent reference pixels used in the filtering process in memory.

[0137] Furthermore, when the reference pixels used in generating the MMLM linear model are non-adjacent reference pixels, the filtering process may be switched depending on the distance (number of lines) between the target chrominance block and the non-adjacent reference pixels. The presence or absence of the filtering process and the positions (adjacent or non-adjacent) of the reference pixels used in the filtering process may be controlled depending on the distance.

[0138] (9) Other Embodiments The above-described embodiments are intended to prevent a decrease in prediction accuracy due to isolated pixels caused by threshold determination used to select a linear model when applying inter-color component prediction to a chrominance block in which a linear model is selected for each pixel. Therefore, similar control can be applied to prediction processes in which threshold determination is performed that may result in other isolated pixels.

[0139] For example, although the above embodiment describes prediction processing for chrominance blocks, it is also applicable to prediction processing for luminance blocks. Specifically, when isolated pixels are generated by switching between multiple different prediction processing methods for luminance blocks on a pixel-by-pixel basis based on threshold value determination, the prediction accuracy can be improved even in prediction processing for luminance blocks by determining whether or not to perform filtering based on a threshold value and a feature amount for each pixel that is compared with the threshold value.

[0140] Each of the intra prediction units 172, 242 according to this embodiment constitutes a prediction device that performs prediction on a block-by-block basis obtained by dividing an image. Each of the intra prediction units 172, 242 includes an MMLM prediction unit 10 that generates predicted pixels by selecting a prediction process to be applied from among a plurality of prediction processes using a threshold value for each pixel to be predicted in a target luminance block, an isolated pixel determination unit 20 that estimates or identifies isolated pixels to which a prediction process different from the prediction process applied to surrounding predicted pixels has been applied among the predicted pixels generated in the target luminance block by the MMLM prediction unit 10, and an isolated pixel correction unit 30 that performs correction processing (filtering processing) on ​​the isolated pixels estimated or identified by the isolated pixel determination unit 20 using surrounding predicted pixels in the target luminance block.

[0141] Furthermore, although the above-described embodiments have described intra-prediction (inter-color component prediction), similar operations can also be applied to inter-prediction. For example, when the inter-prediction process within a block is selectively switched for each pixel based on a threshold value determined for each block or sequence, and when isolated pixels are generated by switching between multiple different prediction processes for each pixel based on threshold determination, similar to the above-described embodiments, whether or not to perform filtering is determined based on the threshold value and the feature value for each pixel used in the threshold determination. This can improve the accuracy of inter-prediction. For example, the inter-prediction unit 171, 241 selectively applies a first inter-prediction process using a first motion vector and a second inter-prediction process using a second motion vector for each target pixel in the block based on threshold determination.

[0142] Each of the inter prediction units 171, 241 according to this embodiment constitutes a prediction device that performs prediction on a block-by-block basis obtained by dividing an image. Each of the inter prediction units 171, 241 includes a prediction pixel generation unit that selects an inter prediction process to apply from among multiple inter prediction processes using a threshold value for each pixel to be predicted in a current block and generates a prediction pixel, an isolated pixel determination unit 20 that estimates or identifies isolated pixels that have been applied with an inter prediction process different from the inter prediction process applied to surrounding prediction pixels among the prediction pixels generated in the current block by the prediction pixel generation unit, and an isolated pixel correction unit 30 that performs correction processing (filtering processing) on ​​the isolated pixels estimated or identified by the isolated pixel determination unit 20 using surrounding prediction pixels in the current block.

[0143] In the above-described embodiment, the isolated pixel correction unit 30 is provided in the intra prediction unit 172, 242, and an example has been described in which correction processing (filtering) is performed on an isolated pixel in a prediction block using prediction pixels surrounding the isolated pixel in the prediction block. However, to solve the problem of eliminating discontinuities caused by isolated pixels, the filtering processing on the prediction block as described above is not limited to the isolated pixel correction unit 30 being provided in the synthesis unit 150, and filtering may be performed on a block before or after synthesis. For example, filtering may be performed on a reconstructed block (decoded image block) obtained by synthesizing a prediction block and a block of reconstructed prediction residuals (prediction residual block), or filtering may be performed on a block of reconstructed prediction residuals. When filtering processing on a reconstruction block, the isolated pixel correction unit 30 may, for example, perform filtering on a reconstructed pixel corresponding to an isolated pixel in the reconstruction block using reconstructed pixels surrounding the isolated pixel in the reconstruction block (and / or reconstructed pixels outside the reconstruction block). When performing filtering on a prediction residual block, the isolated pixel correction unit 30 may perform filtering on pixel values ​​corresponding to isolated pixels in the prediction residual block using pixel values ​​surrounding the isolated pixels in the prediction residual block (and / or pixel values ​​outside the prediction residual block).

[0144] A program may be provided that causes a computer to execute each process performed by the image processing device (encoding device 1, decoding device 2). The program may be recorded on a computer-readable medium. The computer-readable medium can be used to install the program on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM. Circuits that execute each process performed by the image processing device (encoding device 1, decoding device 2) may be integrated, and the image processing device may be configured as a semiconductor integrated circuit (chip set, SoC).

[0145] The functions performed by the image processing device (encoding device 1, decoding device 2) may be implemented in circuitry or processing circuitry, including general-purpose processors, application-specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), CPUs (Central Processing Units), conventional circuits, and / or combinations thereof, programmed to perform the described functions. A processor includes transistors and other circuits and is considered to be circuitry or processing circuitry. A processor may also be a programmed processor that executes a program stored in memory. In this specification, circuitry, unit, or means refers to hardware that is programmed to perform the described functions or hardware that executes them. The hardware may be any hardware disclosed herein or any hardware known to be programmed or capable of performing the described functions. If the hardware is a processor, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0146] As used in this disclosure, the terms "based on" and "depending on / in response to" do not mean "based only on" or "depending only on," unless expressly stated otherwise. The term "based on" means both "based only on" and "based at least in part on." Similarly, the term "depending on" means both "depending only on" and "depending at least in part on." The terms "include," "comprise," and variations thereof do not mean including only the listed items, but may mean including only the listed items or may include additional items in addition to the listed items. Additionally, the term "or," as used in this disclosure, is not intended to mean an exclusive or. Furthermore, any reference to elements using designations such as "first," "second," etc., as used in this disclosure does not generally limit the quantity or order of those elements. These designations may be used herein as a convenient method of distinguishing between two or more elements. Thus, a reference to a first and a second element does not imply that only two elements may be employed therein or that the first element must precede the second element in some way. In this disclosure, where articles are added by translation, such as a, an, and the in English, these articles shall include the plural unless the context clearly indicates otherwise.

[0147] The above describes the embodiments in detail with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes can be made within the scope that does not deviate from the gist of the invention.

[0148] This application claims priority from Japanese Patent Application No. 2023-109566 (filed July 3, 2023), the entire contents of which are incorporated herein by reference.

[0149] (10) Additional Notes Additional notes will be given regarding the features of the above-described embodiment.

[0150] (Supplementary Note 1) A prediction device (172, 242) that performs prediction on a block-by-block basis obtained by dividing an image, comprising: a generation unit (10) that generates predicted pixels by selecting a prediction process to be applied from among a plurality of prediction processes using a threshold value for each pixel to be predicted within the block; a determination unit (20) that estimates or identifies isolated pixels, among the predicted pixels generated within the block by the generation unit, to which a prediction process different from a prediction process applied to surrounding predicted pixels has been applied; and a correction unit (30) that performs correction processing using other pixels on the isolated pixels estimated or identified by the determination unit.

[0151] (Supplementary Note 2) The prediction device according to Supplementary Note 1, wherein the block is a chrominance block, and each of the plurality of prediction processes is a process for predicting pixels in the chrominance block using a linear model generated using chrominance reference pixels around the chrominance block and luminance reference pixels around a predetermined luminance block at a position corresponding to the chrominance block, and the plurality of prediction processes each use a different linear model.

[0152] (Supplementary Note 3) The prediction device according to Supplementary Note 2, wherein the generation unit includes: a threshold determination unit (11) that determines one or more thresholds used to select the linear model from the luminance reference pixels; a linear model generation unit (12) that generates the linear model for each cluster determined by the one or more thresholds; a linear model selection unit (13) that compares, for each chrominance pixel in the chrominance block, a pixel value of a corresponding luminance pixel in the predetermined luminance block with the one or more thresholds, and selects the linear model to be used to predict the chrominance pixel; and an inter-color component prediction unit (14) that generates a predicted pixel of the chrominance pixel by inter-color component prediction using the selected linear model.

[0153] (Supplementary Note 4) The prediction device according to any one of Supplementary Notes 1 to 3, wherein the generation unit selects a prediction process to be applied to each pixel to be predicted by comparing a corresponding pixel value with a threshold, and the determination unit estimates, among the pixels to be predicted, a pixel whose corresponding pixel value is close to the threshold as the isolated pixel.

[0154] (Supplementary Note 5) The prediction device according to any one of Supplementary Notes 1 to 3, wherein the determination unit stores, for each pixel to be predicted, a prediction process applied in accordance with a result of the threshold determination, and identifies, among the pixels to be predicted, a pixel to which a first prediction process has been applied and to which a second prediction process different from the first prediction process has been applied to a predetermined number or more of surrounding pixels, as the isolated pixel.

[0155] (Supplementary Note 6) The prediction device according to any one of Supplementary Notes 1 to 5, wherein the correction unit performs a filter process on the isolated pixel using predicted pixels surrounding the isolated pixel as the correction process.

[0156] (Supplementary Note 7) A coding device (1) comprising the prediction device according to any one of Supplementary Notes 1 to 6.

[0157] (Supplementary Note 8) A decoding device (2) comprising the prediction device according to any one of Supplementary Notes 1 to 6.

[0158] (Supplementary Note 9) A program that causes a computer to function as the prediction device according to any one of Supplementary Notes 1 to 6.

[0159] 1: Encoding device 2: Decoding device 10: MMLM prediction unit 11: Threshold value determination unit 12: Linear model generation unit 13: Linear model selection unit 14: Inter-color component prediction unit 20: Isolated pixel determination unit 30: Isolated pixel correction unit 100: Block division unit 110: Subtraction unit 120: Transformation and quantization unit 121: Transformation unit 122: Quantization unit 130: Entropy coding unit 140: Inverse quantization and inverse transform unit 141: Inverse quantization unit 142: Inverse transform unit 150: Synthesis unit 160: Memory 170: Prediction unit 171: Inter prediction unit 172: Intra prediction unit 173: Switching unit 200: Entropy decoding unit 210: Inverse quantization and inverse transform unit 211 : Inverse quantization unit 212 : Inverse transformation unit 220 : Combining unit 230 : Memory 240 : Prediction unit 241 : Inter prediction unit 242 : Intra prediction unit 243 : Switching unit

Claims

1. A prediction device that performs prediction on a block-by-block basis obtained by dividing an image, comprising: a generation unit that generates a prediction region by selecting a prediction process to be applied from among a plurality of prediction processes for each region to be predicted within the block through threshold judgment; a judgment unit that estimates or identifies a specific region among the prediction regions generated within the block by the generation unit to which a prediction process different from a prediction process applied to surrounding prediction regions has been applied; and a correction unit that performs correction processing on the specific region estimated or identified by the judgment unit using other regions.

2. The prediction device of claim 1, wherein the block is a chrominance block, and each of the plurality of prediction processes is a process for predicting an area within the chrominance block using a prediction model generated using a chrominance reference area surrounding the chrominance block and a luminance reference area surrounding a specified luminance block at a position corresponding to the chrominance block, and the prediction models for the plurality of prediction processes are different from one another.

3. The prediction device according to claim 2, wherein the generation unit includes: a threshold determination unit that determines one or more thresholds to be used for selecting the prediction model from the luminance reference region; a prediction model generation unit that generates the prediction model for each cluster determined by the one or more thresholds; a prediction model selection unit that compares, for each chrominance region within the chrominance block, pixel values ​​of a corresponding luminance region in the specified luminance block with the one or more thresholds to select the prediction model to be used for predicting the chrominance region; and an inter-color component prediction unit that generates a predicted pixel of the chrominance region by inter-color component prediction using the selected prediction model.

4. A prediction device as described in any one of claims 1 to 3, wherein the generation unit selects a prediction process to be applied to each region to be predicted by comparing corresponding pixel values ​​with a threshold value, and the determination unit estimates, among the region to be predicted, a region whose corresponding pixel values ​​are in the vicinity of the threshold value as the specific region.

5. A prediction device as described in any one of claims 1 to 3, wherein the judgment unit stores a prediction process applied for each region to be predicted in accordance with the result of the threshold judgment, and identifies as the specific region a region to be predicted that has had a first prediction process applied thereto and has had a second prediction process different from the first prediction process applied to a predetermined number or more of surrounding regions.

6. The prediction device according to claim 1, wherein the correction unit performs, as the correction process, a filter process on the specific region using a prediction region surrounding the specific region.

7. An encoding device comprising the prediction device according to claim 1.

8. A decoding device comprising the prediction device according to claim 1.

9. A program for causing a computer to function as the prediction device according to claim 1.