Image decoding apparatus, image decoding method, and program
The image decoding device optimizes coding efficiency by replacing intra prediction methods with matrix prediction, adapting to block characteristics and reducing device size, thus addressing the inefficiencies of separate devices for different intra prediction modes.
Patent Information
- Application Number
- JP2024107808
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-07-03
- Publication Date
- 2026-01-16
AI Technical Summary
Existing image decoding technologies require separate devices for handling different intra prediction methods, leading to increased device size and inefficiencies due to the mixing of intra prediction and matrix prediction, which results in blurred predictions and uneven handling of intra prediction modes with different characteristics.
An image decoding device that replaces part or all of the intra prediction in extended methods like DIMD, OBIC, and TIMD with matrix prediction, adapting the mode and weight settings based on block size, reference pixels, and template costs to optimize processing and reduce device size.
This approach enhances coding efficiency by reducing device size and improving prediction accuracy through adaptive use of matrix prediction, while maintaining high coding performance.
Smart Images
Figure 2026007712000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to an image decoding device, an image decoding method, and a program. [Background technology]
[0002] Non-Patent Document 1 discloses intra prediction, which generates intra predicted values by copying pixels adjacent to a block to be decoded to the block to be decoded.
[0003] Intra prediction is made up of one planar mode, one DC mode, and 65 angle modes, and the angle mode provides a prediction mode that replicates in multiple different directions to improve prediction accuracy.
[0004] Furthermore, the intra prediction values are used to reconstruct the decoded pixels by adding them to the prediction residuals.
[0005] Non-patent documents 2 to 6 disclose methods for extending the intra prediction of Non-patent document 1.
[0006] Non-Patent Documents 1 and 2 disclose decoder-side intra mode derivation (DIMD).
[0007] DIMD generates intra-predicted pixels for a current block by calculating a gradient histogram from a region of decoded pixels in the current frame.
[0008] Specifically, as shown in Fig. 6, DIMD selects N-1 bins from a gradient histogram corresponding to the angle modes of intra prediction in descending order of the number of bins, generates a plurality of predicted pixels corresponding to a total of N modes in combination with predicted pixels in the planar mode of intra prediction or block vectors, applies position dependent intra prediction combination (PDPC), and then takes a weighted average based on the weights of the selected bins as the intra predicted pixel of the block to be decoded. Note that Non-Patent Document 1 adopts N=6.
[0009] Furthermore, Non-Patent Documents 1 and 3 disclose techniques regarding occurrence-based intra coding (OBIC).
[0010] OBIC generates intra-predicted pixels for a block to be coded by calculating a mode histogram of intra-prediction modes that reflects the block size from the decoded pixel area of the frame to be coded, and then applies PDPC.
[0011] Here, OBIC has the same specific procedure as DIMD, except for the histogram.
[0012] In either case, the image decoding device derives multiple intra prediction modes without signaling, and therefore has excellent coding efficiency.
[0013] Non-Patent Documents 1 and 4 disclose Template-based Intra Mode Derivation (TIMD).
[0014] TIMD derives the intra prediction mode of a current block to be coded by applying intra prediction to a decoded pixel region of a current frame to be coded and calculating the cost of each intra prediction mode.
[0015] Specifically, as shown in FIG. 7, TIMD first calculates prediction errors (template costs) in some intra prediction modes when neighboring pixels are used as templates and outer neighboring pixels are used as reference pixels.
[0016] Some intra prediction modes use a most probable mode (MPM) constructed from modes selected in neighboring blocks or the like.
[0017] Next, the TIMD selects the intra prediction mode with the smallest template cost and the intra prediction mode with the second smallest template cost, and also calculates the template costs for the +1 and -1 neighboring modes of each.
[0018] The TIMD again selects the intra prediction mode with the smallest template cost and the intra prediction mode with the second smallest template cost, and determines whether to perform a weighted average based on the difference between the two template costs.
[0019] TIMD determines that if the difference in template cost is small, a weighted average is used.
[0020] If the intra prediction modes with the first and second smallest template costs are both angular modes, the TIMD calculates the template costs of planar mode or DC mode and selects the smallest of the two, resulting in a total of three intra prediction modes. Otherwise, the TIMD selects a total of two intra prediction modes.
[0021] Next, the TIMD generates predicted pixels in each intra prediction mode, applies PDPC to them, and then performs weighted averaging based on the template cost to obtain the intra predicted pixels for the block to be decoded. Otherwise, the TIMD determines not to perform weighted averaging, generates predicted pixels in the intra prediction mode with the smallest template cost, and applies PDPC to obtain the intra predicted pixels for the block to be decoded.
[0022] TIMD allows an image decoding device to derive multiple intra prediction modes without signaling, resulting in excellent coding efficiency. In Non-Patent Document 1, the number of TIMD angle modes is set to 130, which is twice the 65 angle modes of intra prediction.
[0023] Non-Patent Documents 1 and 5 disclose vertical planar and horizontal planar (hereinafter, the two will be collectively referred to as "directional planar").
[0024] In the planar direction, the intra prediction mode of the block to be coded is derived by only horizontal interpolation and only vertical interpolation in the planar direction.
[0025] Specifically, in the case of horizontal planar, a horizontal interpolation value is calculated by taking a weighted average proportional to the x coordinate of a reference pixel at the same y coordinate as the pixel to be decoded and a reference pixel to the upper right.
[0026] On the other hand, in the case of vertical planar, a vertical interpolated value is calculated by taking a weighted average proportional to the y coordinate of a reference pixel at the same x coordinate as the pixel to be decoded and a reference pixel to the lower left.
[0027] Finally, the directional planar interpolation value is subjected to PDPC to obtain an intra-predicted pixel for the block to be decoded. Whether the horizontal planar or vertical planar is used is signaled, and increasing the number of prediction methods has the effect of improving prediction accuracy.
[0028] On the other hand, Non-Patent Documents 1 and 6 hold weights obtained by learning in advance as a matrix, and perform product-sum calculations with neighboring pixels of the block to be decoded to obtain intra-predicted pixels (hereinafter referred to as matrix prediction).
[0029] In Non-Patent Document 6, the Planar mode of intra prediction, DC mode, even-numbered angle modes, even-numbered angle modes of DIMD, and the maximum bin (Primary) mode of OBIC are replaced, and the rest remain as conventional intra prediction, thereby avoiding additional signaling. [Prior art documents] [Non-patent literature]
[0030] [Non-Patent Document 1] M. Coban, et al., “Algorithm description of Enhanced Compression Model 13 (ECM 13),” JVET-AH2025, 2024. [Non-patent document 2] M. Abdoli, et al., "Non-CE3: Decoder-side Intra Mode Derivation (DIMD) with prediction fusion using Planar," JVET-O0449, 2019. [Non-patent document 3] RG Youvalari, et al., "EE2-2.2: Occurrence-based intra coding (OBIC)," JVET-AH0076, 2024. [Non-patent document 4] K. Cao et al., “JVET-W0123: Fusion for template-based intra mode derivation,” JVET-W0123, 2021. [Non-patent document 5] X. Li et al., “EE2-1.15: Horizontal and vertical planar modes,” JVET-AB0127, 2022. [Non-patent document 6] B. Ray, et al., "EE2-2.13: Matrix based intra prediction replacing conventional intra modes," JVET-AH0209, 2024. Summary of the Invention [Problem to be solved by the invention]
[0031] Non-Patent Document 6 has the problem that even if the intra prediction and the extended intra prediction methods (DIMD / OBIC (other than Primary) / TIMD / Directional Planar) are the same mode, intra prediction and matrix prediction are mixed, and therefore separate devices are required, resulting in a large device scale.
[0032] For example, planar intra prediction is a matrix prediction, but the planar mode of the extended method remains the same as before, so two types of devices that support planar mode are required, which creates the problem of doubling the size of the device related to planar mode.
[0033] Furthermore, Non-Patent Document 2 is based on the premise of conventional intra prediction modes, but Non-Patent Document 6 shows that even-numbered angle modes of intra prediction have properties that differ from conventional ones.
[0034] Specifically, even-numbered intra prediction modes are predictions based on product-sum operations with a matrix, and therefore tend to generate relatively blurred intra prediction pixels due to weights that are learned to ensure accurate predictions on average.
[0035] On the other hand, odd-numbered intra prediction modes are conventional intra predictions, and generate intra prediction pixels with relatively preserved edges through directional prediction.
[0036] In this way, since intra prediction modes with different characteristics are handled uniformly, there is room for improvement in coding efficiency. Therefore, the present invention has been made in consideration of the above-mentioned problems, and an object of the present invention is to provide an image decoding device, an image decoding method, and a program with high coding efficiency. [Means for solving the problem]
[0037] A first feature of the present invention is an image decoding device that includes an intra prediction unit that generates a first predicted pixel by replacing part or all of the intra prediction in an extended intra prediction method with another prediction method.
[0038] A second feature of the present invention is summarized as an image decoding method including a step of generating a first predicted pixel by replacing part or all of intra prediction in an extended intra prediction method with another prediction method.
[0039] A third feature of the present invention is a program that causes a computer to function as an image decoding device, wherein the image decoding device includes an intra prediction unit that generates a first predicted pixel by replacing part or all of the intra prediction in an extended intra prediction method with another prediction method. [Effects of the Invention]
[0040] According to the present invention, it is possible to provide an image decoding device, an image decoding method, and a program with high coding efficiency. [Brief explanation of the drawings]
[0041] [Figure 1] FIG. 1 is a diagram showing an example of functional blocks of an image decoding device 200 according to the first embodiment. [Figure 2] FIG. 2 is a diagram illustrating the intra prediction unit 204 of the image decoding device 200 according to the first embodiment. [Figure 3]FIG. 3 is a flowchart illustrating the intra prediction unit 204 of the image decoding device 200 according to the first embodiment. [Figure 4] FIG. 4 is a flowchart showing an example of the operation of the decoding unit 201 of the image decoding device 200 according to the first embodiment. [Figure 5] FIG. 5 is a diagram showing an example of a method for setting a TIMD that reflects the characteristics of an intra prediction mode on a block-by-block basis in the decoding unit 201 of the image decoding device 200 according to the first embodiment. [Figure 6] FIG. 6 is a diagram for explaining the DIMD. [Figure 7] FIG. 7 is a diagram for explaining the TIMD. DETAILED DESCRIPTION OF THE INVENTION
[0042] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, etc., and various variations, including combinations with other existing components, are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.
[0043] First Embodiment The image decoding device 200 according to this embodiment will be described below with reference to FIGS.
[0044] The image decoding device 200 according to the first embodiment of the present invention is adapted to process a variety of image signals (hereinafter referred to as images). For example, the image decoding device 200 according to this embodiment is adapted to process a YUV (YCbCr) image composed of luminance pixels and chrominance pixels, an RGB image composed of RGB pixels, or a monochrome image. Here, the pixels constituting each image have discrete values (pixel values) with a predetermined bit width.
[0045] FIG. 1 is a diagram showing an example of functional blocks of an image decoding device 200 according to this embodiment.
[0046] As shown in FIG. 1, the image decoding device 200 includes a decoding unit 201, an inverse quantization unit 202, an inverse transform unit 203, an intra prediction unit 204, a motion compensation unit 205, an adder 206, and a storage unit 207.
[0047] In the following descriptions of the functions of each part, the word "pixel" may refer to a block (unit) made up of pixels, a tree block which is the maximum size of a block, or a slice, tile, or image (picture) larger than a tree block.
[0048] The decoding unit 201 is configured to decode control information and quantization values from code information coded by the image coding device. For example, the decoding unit 201 is configured to output the control information and the quantization values by performing variable length decoding on the code information.
[0049] Here, the quantized value is sent to the inverse quantization unit 202, and the control information is sent to the intra prediction unit 204 and the motion compensation unit 205. Note that this control information includes information necessary for controlling the inverse quantization unit 202, the intra prediction unit 204, the motion compensation unit 205, etc., and may include header information such as a sequence parameter set, a picture parameter set, a picture header, or a slice header.
[0050] The inverse quantization unit 202 is configured to inverse quantize the quantized values sent from the decoding unit 201 to generate transform coefficients. The transform coefficients are sent to the inverse transform unit 203.
[0051] The inverse transform unit 203 is configured to inversely transform the transform coefficients sent from the inverse quantization unit 202 to generate prediction residuals. The prediction residuals are sent to an adder 207.
[0052] The intra prediction unit 204 is configured to generate a first predicted pixel to be added to the prediction residual in the adder 206, based on the decoded pixel obtained via the adder 206 and the control information decoded by the decoding unit 201. The first predicted pixel is sent to the adder 206.
[0053] The motion compensation unit 205 is configured to generate second predicted pixels to be added to the prediction residual in the adder 206, based on the decoded pixels obtained by referring to the storage unit 207 and the control information decoded by the decoding unit 201. The second predicted pixels are sent to the adder 206.
[0054] The storage unit 207 is configured to cumulatively store the decoded pixels sent from the adder 206. These decoded pixels are referenced by the intra prediction unit 204 and the motion compensation unit 205 via the storage unit 207.
[0055] The adder 206 is configured to obtain decoded pixels by adding the prediction residual sent from the inverse transform unit 203 to the first predicted pixels or the second predicted pixels input from the intra prediction unit 204 or the motion compensation unit 205. The decoded pixels are output as an image and sent to the storage unit 207 and the intra prediction unit 204.
[0056] (Intra prediction unit 204) The function of the intra prediction unit 204 will be described below with reference to FIGS.
[0057] First, the intra prediction unit 204 replaces part or all of the intra prediction used in one or more combinations of DIMD (Decoder-side intra mode derivation), OBIC (Occurrence-based Intra Coding), TIMD (Template-based Intra Mode Derivation), and Planar direction as an extended intra prediction method with Matrix prediction (another prediction method).
[0058] Furthermore, the TIMD to be replaced with matrix prediction may include TIMD used in CIIP (Combined Inter-picture merge and Intra-picture Prediction) and GPM (Geometric Partitioning Mode)-Intra, both of which are disclosed in Non-Patent Document 1.
[0059] Here, the mode of intra prediction substituted for matrix prediction can be changed depending on the block size of the block to be decoded (the block to be coded or the current block).
[0060] For example, in TIMD, for blocks to be decoded that are 16x16 pixels or smaller, the intra prediction angle mode for multiples of 4 is replaced with matrix prediction, and for blocks to be decoded that are larger than 16x16 pixels, the intra prediction angle mode for multiples of 8 is replaced with matrix prediction.
[0061] Conversely, in TIMD, for blocks of 16x16 pixels or smaller, the intra prediction angle mode for multiples of 8 may be replaced with matrix prediction, and for blocks larger than 16x16 pixels, the intra prediction angle mode for multiples of 4 may be replaced with matrix prediction.
[0062] Alternatively, the mode of intra prediction to be substituted for matrix prediction may be changed depending on the width or height of the block to be decoded.
[0063] For example, in TIMD, for blocks to be decoded whose width or height is 16 pixels or less, the intra prediction angle mode of a multiple of 4 is replaced with matrix prediction, and for blocks to be decoded whose width or height is greater than 16 pixels, the intra prediction angle mode of a multiple of 8 is replaced with matrix prediction.
[0064] Conversely, in TIMD, for blocks to be decoded that are 16 pixels or less, the intra prediction angle mode for multiples of 8 may be replaced with matrix prediction, and for blocks to be decoded that are larger than 16 pixels, the intra prediction angle mode for multiples of 4 may be replaced with matrix prediction.
[0065] As described above, by changing the mode of intra prediction to be replaced with matrix prediction depending on the block size or width and height of the block to be decoded, it is possible to reduce the decoding processing time of matrix prediction in TIMD while maintaining the coding performance of matrix prediction.
[0066] Furthermore, in TIMD, whether or not to replace the intra prediction mode with matrix prediction may be controlled depending on whether or not there are reference pixels adjacent to the block to be decoded that are used for template cost calculation.
[0067] For example, if there are reference pixels necessary for template cost calculation, the intra prediction mode may be replaced with matrix prediction.
[0068] Conversely, if there are no reference pixels required for template cost calculation, the intra prediction mode does not need to be replaced with matrix prediction.
[0069] As a modified example, when some of the reference pixels required for template cost calculation do not exist, the reference pixels may be generated by copying (padding) the nearest existing reference pixels.
[0070] Alternatively, in TIMD, whether or not to replace the intra prediction mode with matrix prediction may be controlled depending on whether or not the reference pixels adjacent to the block to be decoded, which are used for template cost calculation, exceed the boundary of the tree block to which the block to be decoded belongs.
[0071] For example, if the reference pixels adjacent to the current block to be decoded and used for template cost calculation do not exceed the boundary of the current tree block to which the current block to be decoded belongs, the intra prediction mode may be replaced with matrix prediction.
[0072] Conversely, if the reference pixels adjacent to the current block to be decoded and used for template cost calculation exceed the boundary of the current tree block to which the current block to be decoded belongs, the intra prediction mode does not need to be replaced with matrix prediction.
[0073] As a modified example, if some of the reference pixels required for template cost calculation exceed the boundary of the tree block to be decoded, those reference pixels may not be used, but may be generated by copying (padding) from reference pixels within the boundary of the tree block to be decoded.
[0074] As another modification, even if the boundary of the tree block to be decoded is crossed, if the line buffer of the tree block to be decoded contains reference pixels required for template cost calculation, the intra prediction mode may be replaced with matrix prediction.
[0075] Here, the intra prediction unit 204 standardizes the mode and weight of the matrix prediction to be replaced.
[0076] For example, by replacing the planar mode in DIMD, OBIC, TIMD, and directional planar with matrix prediction, it is possible to eliminate the planar mode processing device in conventional intra prediction from the image decoding device 200, and it is also possible to standardize the processing device and weights for matrix prediction, thereby achieving the effect of reducing the device size.
[0077] Conversely, by replacing a part of the data, it is possible to obtain an effect of generating a variety of predicted values using different prediction methods.
[0078] Furthermore, when permuted matrix prediction is selected, the intra prediction unit 204 does not apply position dependent intra prediction combination (PDPC).
[0079] In matrix prediction, since position-dependent information is already reflected, by not applying it excessively, it is possible to improve prediction accuracy and reduce processing load.
[0080] Figure 2 shows the procedure for generating predicted pixels using planar matrix prediction in DIMD, generating predicted pixels using the top N-1 modes of the histogram, not applying PDPC to predicted pixels using matrix prediction, and applying PDPC to predicted pixels using non-matrix prediction, and generating a first predicted pixel using a weighted average based on the bins of the histogram.
[0081] The intra prediction unit 204 controls the intra prediction mode and weights used for weighted averaging in DIMD and / or OBIC according to the characteristics of the intra prediction mode.
[0082] First, the intra prediction unit 204 calculates a histogram from neighboring pixels. In the case of a gradient histogram, the intra prediction unit 204 applies a gradient detection filter to each neighboring pixel and counts the intra prediction modes of angles corresponding to the distribution of the neighboring pixels to calculate the gradient histogram.
[0083] In the case of a mode histogram, the intra prediction unit 204 calculates the mode histogram by counting the intra prediction modes of neighboring blocks.
[0084] At this time, the intra prediction unit 204 uses a different counting method for each intra prediction mode in calculating the histogram.
[0085] For example, because the prediction accuracy of even-numbered intra prediction modes is higher on average than the prediction accuracy of odd-numbered intra prediction modes, the intra prediction unit 204 performs counting so as to give priority to even-numbered intra prediction modes.
[0086] Specifically, for even-numbered intra prediction modes, the intra prediction unit 204 calculates a histogram by multiplying the even-numbered intra prediction modes by a preset constant.
[0087] Alternatively, the intra prediction unit 204 calculates the histogram by adding a preset constant.
[0088] In this way, by prioritizing even-numbered intra prediction modes, it is possible to expect an effect of improving prediction accuracy on average.
[0089] Conversely, odd-numbered intra prediction modes preserve edges more than even-numbered intra prediction modes, so the intra prediction unit 204 counts odd-numbered intra prediction modes with priority.
[0090] Specifically, for odd-numbered intra prediction modes, the intra prediction unit 204 calculates the histogram by multiplying by a constant.
[0091] Alternatively, the intra prediction unit 204 calculates the histogram by adding a constant.
[0092] In this way, by prioritizing odd-numbered intra prediction modes, it is possible to expect the effect of improving prediction accuracy in areas with many edges.
[0093] In another embodiment, the intra prediction unit 204 controls the number of intra predictions weighted-averaged in DIMD and / or OBIC according to the distribution of intra prediction modes.
[0094] Even-numbered intra prediction modes have relatively high prediction accuracy, so it is preferable to reduce the number of intra predictions to be weighted averaged to avoid excessive smoothing.
[0095] On the other hand, odd-numbered intra prediction modes preserve edges relatively strongly, so it is preferable to increase the number of intra predictions to be weighted-averaged to generate complex prediction values that cannot be expressed by a single intra prediction.
[0096] Therefore, the intra prediction unit 204 changes the number of intra predictions to be weighted-averaged according to the distribution of intra prediction modes that are included in the top positions in the histogram.
[0097] For example, the intra prediction unit 204 sets N=2 when only even numbers exist, and sets N=6 when only odd numbers exist.
[0098] Otherwise, the intra prediction unit 204 changes N in proportion to the number of odd numbers. Alternatively, the intra prediction unit 204 sets an upper limit on the number of intra prediction modes to be weighted-averaged for a specific intra prediction mode. For example, if the intra prediction unit 204 sets the upper limit to 2, even numbers are set as targets for weighted averaging when selected from the top bins of the histogram until a maximum of two even numbers are selected.
[0099] Figure 3 shows a procedure for generating predicted pixels in TIMD using the top two modes that minimize the template cost in the template, not applying PDPC to predicted pixels using matrix prediction, and applying PDPC to predicted pixels using non-matrix prediction, and generating a first predicted pixel using a weighted average based on the template cost.
[0100] Even when the difference in template costs is greater than the threshold and weighted averaging is not applied, the intra prediction unit 204 generates the first predicted pixel by not applying PDPC to predicted pixels based on matrix prediction, but by applying PDPC to predicted pixels based on non-matrix prediction.
[0101] The intra prediction unit 204 controls the weight of the intra prediction mode used for weighted averaging in TIMD according to the type of intra prediction.
[0102] Matrix prediction has a relatively high prediction accuracy, so it is desirable to increase the weight for weighted averaging to prevent excessive smoothing.
[0103] On the other hand, in intra prediction, edges are preserved relatively strongly, so it is preferable to reduce the weight for weighted averaging to generate complex predicted values that cannot be expressed by a single intra prediction.
[0104] Therefore, when the types of intra predictions for which weighted averaging is performed in TIMD differ, the intra prediction unit 204 changes the weights used for weighted averaging in TIMD.
[0105] For example, the intra prediction unit 204 does not change the weights when only the same type of intra prediction exists.
[0106] On the other hand, when different types of intra predictions are mixed, the intra prediction unit 204 changes the weight of a specific intra prediction. For example, when a matrix prediction weight w0 and an intra prediction weight w1 are mixed, the intra prediction unit 204 changes the weights w0 and w1 using a preset coefficient a as follows:
[0107] w0=a×w0 / (a×w0+w1) w1=w1 / (a×w0+w1) Conversely, for content with many steep edges, it is desirable to reduce the weight of matrix prediction and increase the weight of intra prediction. In this case, the intra prediction unit 204 can also use the above formula. That is, in the above formula, if coefficient a>1, priority is given to matrix prediction, and if coefficient a<1, priority is given to intra prediction.
[0108] In this way, by controlling the weights depending on the type of intra prediction, it is possible to obtain the effect of improving prediction accuracy.
[0109] Increasing the number of intra predictions to be weighted averaged in TIMD can be achieved by setting the denominators w0 and w1 to the sum of the weights of the matrix prediction and intra prediction, respectively. For example, when expanding to three weighted averages, if the third is matrix prediction, the intra prediction unit 204 changes it as follows:
[0110] w0=a×w0 / (a×(w0+w3)+w1) w1=w1 / (a×(w0+w3)+w1) w2=a×w2 / (a×(w0+w3)+w1) In another embodiment, the intra prediction unit 204 controls the template cost evaluated in the TIMD according to the type of intra prediction.
[0111] First, the intra prediction unit 204 applies each intra prediction mode to reference pixels of a nearby template. At this time, for a mode that has been replaced with matrix prediction, the intra prediction unit 204 calculates the template cost using the matrix prediction.
[0112] Although the current block and the template differ in size and shape, the intra prediction unit 204 calculates the template cost by applying the matrix at the relatively matching location.
[0113] In this way, by calculating a more accurate template cost, the prediction accuracy can be improved.
[0114] For example, if the block to be decoded is 16x16 pixels and the template has four lines above and to the left of the block to be decoded, a matrix for 16x16 pixel blocks is applied to the reference pixels outside each template for the 16x4 pixel template above and the 4x16 pixel template to the left, and predicted pixels for each template are calculated.
[0115] Here, in the process of calculating the predicted pixel of each template, the predicted pixel value calculated using the matrix is limited to the pixel area of each template. This reduces the increase in the amount of calculation when applying the matrix to the template. In another embodiment, the intra prediction unit 204 selects the intra prediction mode with the smallest template cost (best mode) and the intra prediction mode with the second smallest template cost (2nd best mode), and checks the adjacent modes within ±1 of each other.
[0116] If the best mode and the adjacent mode are the same type of intra prediction, the intra prediction unit 204 calculates the template cost of the adjacent mode.
[0117] Otherwise (if the best mode and the neighboring mode are different types of intra prediction), the intra predictor 204 does not calculate the template cost of the neighboring mode.
[0118] Similarly, the intra prediction unit 204 determines whether or not to calculate the template cost of the adjacent mode depending on whether the prediction methods of the 2nd best mode and its ±1 modes match.
[0119] Since intra prediction and matrix prediction have different characteristics, the processing load can be reduced by omitting unnecessary template cost calculations for adjacent modes.
[0120] Alternatively, if there is no difference in template cost between the best mode and the second best mode, the intra prediction unit 204 will take a simple average, and therefore emphasizes the change in weight of the weighted average due to the difference in template cost.
[0121] For example, the intra prediction unit 204 raises the template cost to the Nth power and then calculates the weight.
[0122] In this way, by varying the weights of the weighted average, it is possible to avoid the problem of reduced prediction accuracy due to excessive smoothing.
[0123] Alternatively, if there is no difference in template cost between Best mode and 2nd Best mode, the mode with the M (>2)th smallest template cost (M-th Best mode) also has a small difference in template cost from Best mode, so the intra prediction unit 204 generates predicted pixels in M modes, calculates the template cost as a weighted average of any combination, and generates the first predicted pixel using the optimal combination.
[0124] For example, when M=3, the intra prediction unit 204 calculates the template cost using a weighted average of three combinations: Best mode and 2nd Best mode, Best mode and 3rd Best mode, and 2nd Best mode and 3rd Best mode, and generates the first predicted pixel using the combination that minimizes the template cost.
[0125] According to this configuration, the weighted average is evaluated using a template, thereby improving the prediction accuracy.
[0126] Alternatively, if there is no difference in template cost between the Best mode and the 2nd Best mode, the intra prediction unit 204 controls the number of modes to be weighted averaged, since the difference in template cost between the Mth (>2)th smallest template cost mode (M-th Best mode) and the Best mode is also small.
[0127] For example, the intra prediction unit 204 generates the first predicted pixel by multiplying the template cost of the Best mode by a predetermined constant and performing a weighted average of predicted pixels according to any number of modes having template costs smaller than the multiplied template cost. The intra prediction unit 204 can also set an upper limit on the number of modes to be weighted averaged.
[0128] According to this configuration, the effect of improving prediction accuracy can be obtained by taking a weighted average of a large number of modes.
[0129] In another embodiment, the intra predictor 204 limits the mode for calculating the template cost of Matrix prediction.
[0130] Specifically, since matrix prediction takes several times longer to process than conventional intra prediction, the intra prediction unit 204 temporarily substitutes the template cost based on matrix prediction with the template cost based on conventional intra prediction.
[0131] First, even if the mode is Matrix prediction, the intra prediction unit 204 calculates the template cost as if it were a conventional intra prediction mode.
[0132] Next, the intra prediction unit 204 calculates template costs by matrix prediction for the K modes having the smallest template costs that have been replaced by matrix prediction.
[0133] Alternatively, the template cost using matrix prediction may be calculated only when the cost is equal to or less than a constant multiple of the cost in best mode. In other words, when the cost is greater than a constant multiple of the cost in best mode (or equal to or greater than that), the template cost using matrix prediction does not need to be calculated.
[0134] According to this configuration, the number of template cost calculations using matrix prediction can be reduced to K or less, thereby achieving the effect of reducing processing time.
[0135] The number K of templates for calculating template costs by matrix prediction may be adaptively changed.
[0136] For example, the intra prediction unit 204 can change the number K depending on the cost value of the substituted template, the pixel distribution of the template, and the block size.
[0137] In another embodiment, when only angle mode is selected for Best mode and 2nd Best mode (neither Planar mode nor DC mode is selected), the intra prediction unit 204 uses either Planar mode or DC mode as the third mode selection for TIMD without calculating the template cost.
[0138] With this configuration, both planar mode and DC mode are replaced with matrix prediction, so the difference in template cost between the two modes is less likely to appear compared to the modes before replacement, and deciding on one of the modes has the effect of reducing processing time.
[0139] Furthermore, the intra prediction unit 204 may adaptively switch between whether to use the planar mode or the DC mode without calculating the template cost, or whether to calculate the template cost and select either the planar mode or the DC mode.
[0140] For example, the intra prediction unit 204 may perform the above-described switching depending on the template cost values of the Best mode and the 2nd Best mode, the pixel distribution of the template, and the block size.
[0141] Alternatively, even if only angle mode is selected for the Best mode and 2nd Best mode (neither Planar mode nor DC mode is selected), the intra prediction unit 204 does not select a third mode if the Best mode or 2nd Best mode includes Matrix prediction mode.
[0142] According to this configuration, since the prediction accuracy is high due to the matrix prediction, the effect of improving the prediction accuracy can be obtained by not performing excessive weighted averaging.
[0143] Furthermore, with this configuration, the number of modes for calculating template costs can be reduced, which has the effect of reducing processing time.
[0144] Hereinafter, the control information for intra prediction decoded by the decoding unit 201 will be described with reference to Fig. 4. Here, the description will be given taking TIMD as an example, but the same applies to the cases of DIMD, OBIC, and planar direction.
[0145] Here, the coded information input to the image decoding device 200 may include a sequence parameter set (SPS) that compiles control information for each sequence.
[0146] The coded information may also include a picture parameter set (PPS) or a picture header (PH) that summarizes control information for each picture. The coded information may also include a slice header (SH) that summarizes control information for each slice.
[0147] A method for setting intra prediction on a sequence-by-sequence basis will be described below with reference to FIG.
[0148] In step S101, the decoding unit 201 determines whether sps_timd_enabled_flag is 1 in the sequence parameter set.
[0149] Here, sps_timd_enabled_flag is a syntax element that controls whether or not TIMD is applied according to the characteristics of the intra prediction mode.
[0150] When sps_timd_enabled_flag is 1, this indicates that the TIMD according to the characteristics of the intra prediction mode is enabled, and when sps_timd_enabled_flag is 0, this indicates that the TIMD according to the characteristics of the intra prediction mode is disabled.
[0151] In step S102, if sps_timd_enabled_flag is 1, the decoding unit 201 decodes sps_timd_mode.
[0152] sps_timd_mode is a syntax that controls TIMD according to the characteristics of the intra prediction mode.
[0153] In this way, by using sps_timd_mode, the TIMD setting that reflects the characteristics of the intra prediction mode can be changed in accordance with the image characteristics for each sequence, which is expected to maximize the coding efficiency.
[0154] For example, since sequences consisting of CG often have consecutive pixels with the same value, the decoding unit 201 can be set to suppress weighted averaging in TIMD, and for sequences consisting of natural images, the decoding unit 201 can be set to prioritize weighted averaging in TIMD, thereby maximizing coding efficiency.
[0155] When intra prediction is set on a picture-by-picture basis, the decoding unit 201 similarly decodes pps_timd_enabled_flag and pps_timd_mode in the picture parameter set or picture header.
[0156] In this way, by using pps_timd_mode, the TIMD setting that reflects the characteristics of the intra prediction mode can be changed in accordance with the image characteristics on a picture-by-picture basis, which is expected to maximize the coding efficiency.
[0157] For example, since pictures composed of CG often have consecutive pixels with the same value, the decoding unit 201 can be configured to suppress weighted averaging in TIMD, while for pictures composed of natural images, the decoding unit 201 can be configured to prioritize weighted averaging in TIMD, thereby maximizing coding efficiency.
[0158] When intra prediction is set in units of slices, the decoding unit 201 similarly decodes sh_timd_enabled_flag and sh_timd_mode in the slice header.
[0159] In this way, by using sh_timd_mode, the TIMD setting that reflects the characteristics of the intra prediction mode can be changed in units of slices according to the image characteristics, which is expected to maximize the coding efficiency.
[0160] For example, since slices composed of CG often have consecutive pixels with the same value, the decoding unit 201 can be configured to suppress weighted averaging in TIMD, and for slices composed of natural images, the decoding unit 201 can be configured to prioritize weighted averaging in TIMD, thereby maximizing coding efficiency.
[0161] Here, by setting only the upper layer, it is possible to suppress an increase in the amount of code, or by setting also the lower layer and then prioritizing the setting in the lower layer, adaptive control is possible.
[0162] Alternatively, if a TIMD setting method that reflects the characteristics of the intra prediction mode is determined in advance, the decoding of the setting method itself can be omitted.
[0163] In the above example, a method of setting TIMD that reflects the characteristics of the intra prediction mode in units of sequences, pictures, or slices has been described, but it is also possible to directly set it in units of blocks, as described below, without setting these.
[0164] Hereinafter, a method for setting a TIMD that reflects the characteristics of an intra prediction mode in units of blocks will be described with reference to FIG.
[0165] As shown in FIG. 5, in step S201, the decoding unit 201 determines whether any one of sps_timd_enabled_flag, pps_timd_enabled_flag, and sh_timd_enabled_flag is 1 or not.
[0166] If neither is 1, the operation proceeds to step S205, and if any one is 1, the operation proceeds to step S202.
[0167] In step S202, the decoding unit 201 determines whether the current block to be decoded is in the TIMD mode.
[0168] If YES, the operation proceeds to step S203, and if NO, the operation proceeds to step S205.
[0169] In step S203, the decoding unit 201 determines whether reference pixels are available. For example, if the current block is at the end of a frame or a coding tree unit (CTU), the decoding unit 201 determines that reference pixels are unavailable.
[0170] If YES, the operation proceeds to step S204, and if NO, the operation proceeds to step S205.
[0171] In step S204, the decoding unit 201 decodes cu_timd_mode, which is a control signal indicating the setting of the TIMD that reflects the characteristics of the intra prediction mode.
[0172] In step S205, the decoding unit 201 ends the operation without decoding a control signal indicating the setting of the TIMD that reflects the characteristics of the intra prediction mode.
[0173] According to the image decoding device 200 of this embodiment, in DIMD, OBIC, TIMD, and direction Planar, part or all of the intra prediction is replaced with matrix prediction, thereby reducing the size of the image decoding device 200 while adaptively generating the first predicted pixel by reflecting the characteristics of the intra prediction mode, thereby improving coding efficiency.
[0174] Furthermore, according to the image decoding device 200 according to this embodiment, it is possible to improve coding efficiency by applying matrix prediction in TIMD and deriving an optimal intra prediction mode by calculating a template cost according to the matrix prediction.
[0175] The above-described image decoding device 200 may be realized as a program that causes a computer to execute each function (each step). [Industrial Applicability]
[0176] According to this embodiment, for example, it is possible to improve the overall service quality in video communication, which will contribute to the achievement of Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), which is to "Develop resilient infrastructure, promote sustainable industrialization and foster innovation." [Explanation of symbols]
[0177] 200...Image decoding device 201...Decoding section 202...Inverse quantization section 203...Inverse conversion unit 204...Intra prediction unit 205...Motion compensation unit 206...adder 207...Storage section
Claims
1. An image decoding device, An image decoding device comprising an intra prediction unit that generates a first predicted pixel by replacing part or all of intra prediction in an extended intra prediction method with another prediction method.
2. The image decoding device according to claim 1 , wherein the intra prediction unit uses a combination of one or more of DIMD, OBIC, TIMD, and directional planar as the extension method of the intra prediction.
3. The image decoding device according to claim 1 , wherein the intra prediction unit uses matrix prediction as the alternative prediction method to be substituted.
4. The image decoding device according to claim 1 , wherein the intra prediction unit standardizes a prediction mode and a weight for the other prediction method to be substituted in the plurality of combinations of the extension methods of the intra prediction.
5. The image decoding device according to claim 2 , wherein the intra prediction unit replaces part or all of the TIMD intra prediction with matrix prediction.
6. The image decoding device according to claim 1 , wherein the intra prediction unit does not apply PDPC when the replaced prediction method is selected.
7. The image decoding device according to claim 2 , wherein the intra prediction unit controls the intra prediction mode to be evaluated in the TIMD in accordance with the type of intra prediction.
8. The image decoding device according to claim 2 , wherein the intra prediction unit calculates the template cost by the intra prediction for a mode that has been replaced by another type of intra prediction in the TIMD evaluation.
9. The intra prediction unit, in the evaluation of the TIMD, If the predetermined mode and the neighboring mode are the same type of intra prediction, calculate a template cost of the neighboring mode; The image decoding device according to claim 7 , wherein when the predetermined mode and the adjacent mode are not the same type of intra prediction, the template cost of the adjacent mode is not calculated.
10. The image decoding device according to claim 2 , wherein the intra prediction unit controls weights of the intra prediction modes to be weighted-averaged in the TIMD according to the type of intra prediction.
11. The image decoding device according to claim 7 , wherein the intra prediction unit changes the weight used for the weighted average in the TIMD when the type of the intra prediction is different.
12. An image decoding method, comprising: An image decoding method comprising a step of generating a first predicted pixel by replacing part or all of intra prediction in an extended intra prediction method with another prediction method.
13. A program that causes a computer to function as an image decoding device, The image decoding device comprises: A program comprising an intra prediction unit that generates a first predicted pixel by replacing part or all of intra prediction in an extended intra prediction method with another prediction method.