Image decoding device, image decoding method, and program

Abstract

To improve the coding efficiency in a geometric partitioning mode (GPM).SOLUTION: An image decoder 200 is provided with: an intra prediction unit 204 that generates a first prediction pixel based on a decoded pixel and control information; a motion compensation unit 208 that generates a second prediction pixel based on the accumulated decoded pixels and control information; a composition unit 209 that generates a third prediction pixel based on combinations including at least one of the first prediction pixel and the second prediction pixel and the control information; a selection unit 205 that changes the coding order of indices for specifying the shape and the border width of a division boundary in a decoding target block according to the similarity between pixels near accumulated decoded blocks or reference destination blocks; and an adder 206 that adds a prediction residual to any one of a first prediction image, a second prediction image, and the third prediction pixel to obtain a decoded pixel.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image decoding apparatus, an image decoding method, and a program.

Background Art

[0002] In Non-Patent Document 1 and Non-Patent Document 2, a Geometric Partitioning Mode (GPM) is disclosed.

[0003] GPM divides a rectangular block diagonally into two parts and performs motion compensation (inter prediction) on each part. Specifically, the two divided regions are motion-compensated (inter predicted) by merge vectors respectively and synthesized by weighted averaging.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Non-Patent Document 2

Non-Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the techniques disclosed in Non-Patent Document 1 and Non-Patent Document 2, when performing weighted averaging on the boundary portions of the divided small regions, one is selected from five different preset widths.

[0006] In this case, if the boundaries of the sub-regions are clearly defined, it is desirable to select a narrower boundary width; conversely, if the boundaries of the sub-regions are blurred, it is desirable to select a wider boundary width.

[0007] However, the technologies disclosed in Non-Patent Documents 1 and 2 have the problem that there is room for improvement in coding performance because a codeword is given to specify which boundary width to select. Therefore, the present invention has been made in view of the above problems and aims to provide an image decoding device, an image decoding method, and a program that can improve coding efficiency in geometric partitioning mode. [Means for solving the problem]

[0008] The first feature of the present invention is an image decoding device comprising: a decoding unit that decodes control information including codewords that specify the shape and boundary width of the division boundary within the block to be decoded, and quantization values; an inverse quantization unit that dequantizes the quantization values ​​to obtain decoded conversion coefficients; an inverse transformation unit that dequantizes the conversion coefficients to obtain decoded prediction residuals; an intra-prediction unit that generates first prediction pixels based on the decoded pixels and the control information; an accumulation unit that accumulates the decoded pixels; a motion compensation unit that generates second prediction pixels based on the accumulated decoded pixels and the control information; a synthesis unit that generates third prediction pixels based on a combination including at least one of the first prediction pixels and the second prediction pixels and the control information; a selection unit that changes the code order of subscripts that specify the shape and boundary width of the division boundary within the block to be decoded, according to the similarity between neighboring pixels of the accumulated decoded block or reference block; and the first prediction pixels , the second prediction pixels The gist of the invention is that it comprises an adder which adds one of the third predicted pixels and the predicted residual to obtain the decoded pixel.

[0009] A second feature of the present invention is an image decoding device comprising: a decoding unit that decodes control information including codewords that specify the shape and boundary width of the division boundary within the block to be decoded, and quantization values; an inverse quantization unit that dequantizes the quantization values ​​to obtain decoded conversion coefficients; an inverse transformation unit that dequantizes the conversion coefficients to obtain decoded prediction residuals; an intra-prediction unit that generates first prediction pixels based on the decoded pixels and the control information; an accumulation unit that accumulates the decoded pixels; a motion compensation unit that generates second prediction pixels based on the accumulated decoded pixels and the control information; a synthesis unit that generates third prediction pixels based on a combination including at least one of the first prediction pixels and the second prediction pixels and the control information; a selection unit that changes the code order of the control information associated with the division boundary width within the block to be decoded according to the history of the division boundary width within the block to be decoded in the same image; and the first prediction pixels , the second prediction pixels The gist of the invention is that it comprises an adder which adds one of the third predicted pixels and the predicted residual to obtain the decoded pixel.

[0010] A third feature of the present invention is an image decoding method comprising the steps of: decoding control information including codewords that specify the shape and boundary width of the division boundary within the block to be decoded, and a quantization value; inverse quantization of the quantization value to obtain a decoded conversion coefficient; inverse transformation of the conversion coefficient to obtain a decoded prediction residual; generating a first prediction pixel based on the decoded pixel and the control information; accumulating the decoded pixel; generating a second prediction pixel based on the accumulated decoded pixel and the control information; generating a third prediction pixel based on a combination including at least one of the first prediction pixel and the second prediction pixel and the control information; changing the code order of subscripts that specify the shape and boundary width of the division boundary within the block to be decoded, according to the similarity between neighboring pixels of the accumulated decoded block or reference block; and the first prediction pixels , the second prediction pixels The gist of the method is to include the steps of adding one of the third predicted pixels and the predicted residual to obtain the decoded pixel.

[0011] A fourth feature of the present invention is a program that causes a computer to function as an image decoding device, wherein the image decoding device includes a decoding unit that decodes control information including codewords that specify the shape and boundary width of the division boundary within the block to be decoded, and quantization values; an inverse quantization unit that dequantizes the quantization values ​​to obtain decoded conversion coefficients; an inverse transformation unit that decodes the conversion coefficients to obtain decoded prediction residuals; an intra-prediction unit that generates first prediction pixels based on the decoded pixels and the control information; an accumulation unit that accumulates the decoded pixels; a motion compensation unit that generates second prediction pixels based on the accumulated decoded pixels and the control information; a synthesis unit that generates third prediction pixels based on a combination including at least one of the first prediction pixels and the second prediction pixels and the control information; a selection unit that changes the code order of subscripts that specify the shape and boundary width of the division boundary within the block to be decoded, according to the similarity between neighboring pixels of the accumulated decoded block or reference block; and the first prediction pixels , the second prediction pixels The gist of the invention is that it comprises an adder which adds one of the third predicted pixels and the predicted residual to obtain the decoded pixel. [Effects of the Invention]

[0012] According to the present invention, it is possible to provide an image decoding device, an image decoding method, and a program that can improve coding efficiency in geometric partitioning mode. [Brief explanation of the drawing]

[0013] [Figure 1] Figure 1 shows an example of the functional block of an image decoding device 200 according to one embodiment. [Figure 2] Figure 2 shows an example of a case where the block to be decrypted is divided into diagonal sections. [Figure 3] Figure 3 is a diagram illustrating a method for decoding existing partition shapes. [Figure 4]FIG. 4 is a diagram for explaining an example of template matching when intra prediction is applied to small regions A and B. [Figure 5] FIG. 5 is a diagram for explaining an example of the operation of the selection unit 205 of the image decoding apparatus 200 according to an embodiment. [Figure 6] FIG. 6 is a diagram for explaining an example of rearrangement of codewords of boundary widths using template matching or a histogram. [Figure 7] FIG. 7 is a diagram for explaining an example of a case where a division boundary within a decoding target block does not fall within a neighboring pixel region. [Figure 8] FIG. 8 is a diagram for explaining an example of rearrangement of codewords of division boundary widths based on a histogram. [Figure 9] FIG. 9 is a flowchart showing an example of an operation of setting a method for selecting a boundary width in a sequence unit. [Figure 10] FIG. 10 is a flowchart showing an example of an operation of setting a method for selecting a boundary width in a block unit. [Figure 11] FIG. 11 is a diagram showing an example of a functional block of the image decoding apparatus 200 according to an embodiment. [Figure 12] FIG. 12 is a diagram showing an example of a case where a geometric partitioning mode (GPM) is used as small region partitioning. [Figure 13] FIG. 13 is a diagram showing an example of a case where a decoding target block is divided by N = 2 line segments. [Figure 14] FIG. 14 is a diagram showing an example of a case where a decoding target block is divided by N = 3 line segments. [Figure 15] FIG. 15 is a diagram showing examples of dividable shapes of a decoding target block. [[ID=3�]] [Figure 16] FIG. 16 is a diagram showing examples of dividable shapes of a decoding target block. [Figure 17] FIG. 17 is a diagram showing an example of three patterns of weight coefficients assigned to the division boundary of the small region B shown in FIG. 12. [Figure 18]Figure 18 shows an example of a case where a division boundary is formed by two division lines, one vertical and one horizontal, for the block to be decoded, and composite prediction is performed on a pixel-by-pixel basis according to the distance from these two division lines (division line 1 and division line 2). [Figure 19] Figure 19 is a diagram illustrating an example of a blending region relative to dividing line 1 and dividing line 2. [Figure 20] Figure 20 is a flowchart illustrating an example of the operation for setting the selection method for multiple line segment division modes on a sequence-by-sequence basis. [Figure 21] Figure 21 is a flowchart illustrating an example of the operation for setting the selection method for multiple line segment division modes on a block-by-block basis. [Figure 22] Figure 22 shows an example of a case where an angle prediction mode parallel to dividing line 1 is applied. [Figure 23] Figure 23 is a diagram illustrating the intra-prediction mode derivation method according to this embodiment. [Figure 24] Figure 24 is a diagram illustrating the intra-prediction mode derivation method according to this embodiment. [Figure 25] Figure 25 is a diagram illustrating the intra-prediction mode derivation method according to this embodiment. [Figure 26] Figure 26 is a diagram illustrating the intra-prediction mode derivation method according to this embodiment. [Figure 27] Figure 27 shows a method for deriving an intra prediction mode based on adjacent reference blocks for normal intra prediction, as described in Non-Patent Documents 1 and 3, and an example of a method for deriving an intra prediction mode based on adjacent reference blocks for geometric partitioning mode according to this embodiment, which applies the derivation method. [Figure 28] Figure 28 shows an example of the correspondence between cu_div_idx, divDirectionIdx, and divLocationIdx. [Figure 29]Figure 29 is a flowchart illustrating an example of changing the operation to set the selection method for multiple line segment division modes on a sequence-by-sequence basis. [Figure 30] Figure 30 is a flowchart illustrating an example of changing the operation for setting the selection method for division modes, including multiple line segment division modes, on a block-by-block basis. [Modes for carrying out the invention]

[0014] Embodiments of the present invention will be described below with reference to the drawings. Note that the components in the following embodiments can be replaced with existing components as appropriate, and various variations are possible, including combinations with other existing components. Therefore, the description of the following embodiments does not limit the content of the invention as described in the claims.

[0015] <First Embodiment> The image decoding device 200 according to this embodiment will be described below with reference to Figures 1 to 10. Figure 1 is a diagram showing an example of the functional block of the image decoding device 200 according to this embodiment.

[0016] As shown in Figure 1, the image decoding device 200 includes a code input unit 210, a decoding unit 201, an inverse quantization unit 202, an inverse transformation unit 203, an intra prediction unit 204, a sorting unit 205, an adder 206, a storage unit 207, a motion compensation unit 208, a synthesis unit 209, and an image output unit 220.

[0017] The code input unit 210 is configured to acquire code information encoded by the image encoding device.

[0018] The decoding unit 201 is configured to decode control information and quantization values ​​from the code information input from the code input unit 210. For example, the decoding unit 201 is configured to output control information and quantization values ​​by performing variable-length decoding on such code information.

[0019] Here, the quantized values ​​are sent to the inverse quantization unit 202, and the control information is sent to the sorting unit 205, the motion compensation unit 208, the intra prediction unit 204, and the synthesis unit 209. This control information includes information necessary for the control of the motion compensation unit 206, the intra prediction unit 204, and the synthesis unit 207, and may also include header information such as a sequence parameter set, a picture parameter set, a picture header, and a slice header.

[0020] The inverse quantization unit 202 is configured to inversely quantize the quantized values ​​sent from the decoding unit 201 to obtain the decoded conversion coefficients. These conversion coefficients are then sent to the inverse transformation unit 203.

[0021] The inverse transform unit 203 is configured to inversely transform the transformation coefficients sent from the inverse quantization unit 202 to obtain the decoded predicted residual. This predicted residual is then sent to the adder 206.

[0022] The intra-prediction unit 204 is configured to generate a first prediction pixel based on the decoded pixels and control information sent from the decoding unit 201. Here, the decoded pixels are obtained via the adder 206 and stored in the storage unit 207. The first prediction pixel is a prediction pixel to be added with the prediction residual in the adder 206. The first prediction pixel is sent to the adder 206 or the merging unit 209.

[0023] The storage unit 207 is configured to cumulatively store the decoded pixels sent from the adder 206. These decoded pixels are referenced by the motion compensation unit 208 and the sorting unit 205 via the storage unit 207.

[0024] The motion compensation unit 208 is configured to generate a second prediction pixel based on the decoded pixels stored in the storage unit 207 and the control information sent from the decoding unit 201. Here, the second prediction pixel is a prediction pixel to be added with the prediction residual in the adder 206. The second prediction pixel is sent to the adder 206 or the merging unit 209.

[0025] The adder 206 is configured to obtain a decoded pixel by adding the predicted residual sent from the inverse transform unit 203 with one of the input first to third predicted pixels. This decoded pixel is then sent to the image output unit 220, the storage unit 207, and the intra-prediction unit 204.

[0026] The synthesis unit 209 is configured to divide the block to be decoded into multiple shapes (small regions) based on a combination including at least one of multiple first prediction pixels or second prediction pixels sent from the intra prediction unit 204 and the motion compensation unit 208, control information decoded by the decoding unit 201, and boundary information decoded by the selection unit 205 (described later), and synthesize multiple prediction pixels corresponding to each shape to generate a third prediction pixel for addition with the prediction error in the adder 206. The generated third prediction pixel is sent to the adder 206.

[0027] Any method can be used to divide and combine the block to be decoded into multiple shapes (small regions) in the synthesis unit 209, but below, as an example, we will explain the case using the Geometric Partitioning Mode (GPM).

[0028] The sorting unit 205, which is a characteristic configuration of the image decoding device 200 according to this embodiment, will be described below.

[0029] The role of the selection unit 205 is to decode the boundary width used in the subsequent synthesis unit 209 by determining (or limiting) the boundary width based on neighboring pixels of the block to be decoded, thereby decoding the boundary width using a short control information (codeword) called cu_bld_idx.

[0030] Figure 2 shows an example of a case where the block to be decoded is divided into diagonal sections. In the example in Figure 2, the block to be decoded is divided into sub-regions A and B. In each sub-region, prediction pixels are generated using motion compensation or similar methods.

[0031] In this case, to reduce the code size of the split shape, a method of decoding the existing split shape by evaluating the similarity of neighboring pixels can be used (see "EE2-2.4:Template matching based reordering for GPM split modes, JVET-Z0056").

[0032] This method improves encoding efficiency by limiting the candidate division shapes in geometric division mode to those where the error is small, i.e., where the neighboring pixels of the reference block for subregions A and B are similar, as shown in Figure 3, when motion compensation (interpretation) is applied to subregion A or subregion B.

[0033] Furthermore, in this method, when intraprediction is applied to subregion A or subregion B, the candidate division shapes for geometric division mode are compared, as shown in Figure 4, with the error between the decoded (reconstructed) neighboring pixels of the block and the intrapredicted pixels (neighboring intrapredicted pixels) based on the decoded (reconstructed) neighboring pixels that are one or more lines away from the neighboring pixels. By limiting the division shapes to those with small errors, i.e., those in which the neighboring pixels and neighboring intrapredicted pixels are similar, the encoding efficiency is improved.

[0034] The above example shows how different interpretations or intrapredictions are applied to both sub-regions A and B. However, even when interpretations (or intrapredictions) and intrapredictions (or interpretations) are applied to sub-regions A and B respectively, the similarity (error) between the referenced neighbor pixels and the nearby intrapredicted pixels can be compared for the neighboring pixels of the block to be decoded using the same method. This comparison of the errors between the neighbor pixels and the referenced neighbor pixels (or nearby intrapredicted pixels) is generally called template matching.

[0035] In template matching, as shown in Figures 3 and 4, the pixels above, to the left, and the top-left of the block to be decoded are used as neighboring pixels.

[0036] Similarly, since motion information, which serves as control information, indicates the reference point for each sub-region, the reference point also uses the same position as a neighboring pixel.

[0037] The procedure for decoding the division shape of the block to be decoded is as follows:

[0038] Firstly, for all subdivision shapes, the boundary is extended to the corresponding neighboring pixel.

[0039] Secondly, the error of each neighboring pixel is calculated by dividing it by the extended boundary.

[0040] Thirdly, we list only those with small errors and then rearrange the division shapes in order of increasing error.

[0041] Fourth, the final partition shape is decoded using a codeword from among the limited partition shapes.

[0042] However, existing methods do not anticipate the adaptive application of multiple boundary widths to geometric partitioning modes, as disclosed in Non-Patent Document 2, and therefore have the problem of failing to improve coding efficiency.

[0043] To solve this problem, the sorting unit 205 takes the procedure of decoding the boundary width according to the similarity between the neighboring pixels of the block to be decoded (the said neighboring pixels) and the referenced neighboring pixels or nearby intra-predicted pixels.

[0044] Specifically, firstly, the sorting unit 205 decodes the divided shape in the procedure shown in Figure 3 or Figure 4.

[0045] Secondly, as shown in Figure 5, the sorting unit 205 generates neighboring pixels with different boundary widths by applying a weighted average with different boundary widths to the neighboring pixels of the reference target.

[0046] Thirdly, the selection unit 205 compares the similarity between the neighboring pixels of the referenced block to which M (where M is a natural number) different boundary widths are applied, and the neighboring pixels of the block to be decoded, and uses the top N (where N is a natural number, M≧N) of these highly similar pixels as candidate boundary widths.

[0047] Fourth, the selection unit 205 selects a boundary width from the candidate boundary widths by decoding a codeword that uniquely determines the boundary width.

[0048] With this configuration, by assigning shorter codes to candidates with highly similar boundary widths, it is possible to reduce the amount of code required for control information.

[0049] Figure 6 shows a specific example. Figure 6 shows the codewords for five types of control information for geometric partitioning modes disclosed in Non-Patent Document 2. Here, τ is the boundary width in Non-Patent Document 1, and in Non-Patent Document, four additional boundary widths (τ / 2, 2τ, τ / 4, 4τ) are added to τ, which are 1 / 2 times, 2 times, 1 / 4 times, and 4 times. In conventional methods, the boundary width is specified by the code of cu_bld_idx, which is control information (codeword) as shown in Figure 6.

[0050] In contrast, the present invention assigns shorter codes from boundary width candidates with high similarity (in the example in Figure 6, the smaller the boundary width, the higher the similarity) using template matching, that is, by changing (reordering) the code order corresponding to the boundary width candidates, the amount of code in the control information (codeword) cu_bld_idx can be reduced.

[0051] If N=1, the selection unit 205 uniquely decodes the boundary width, which is unnecessary for codeword decoding, and decodes the codeword only if N≠1.

[0052] The selection unit 205 can set an arbitrary width for the boundary width of the boundary when applying a boundary to neighboring pixels during generation.

[0053] The sorting unit 205 may set different boundary widths for each reference target of the small region and generate boundaries with asymmetrical boundary widths.

[0054] Conventionally, in order to suppress the coding amount of the boundary width, only symmetrical boundary widths could be used. However, the function of the selection unit 205 according to this embodiment makes it possible to use a variety of boundary width combinations, resulting in improved prediction accuracy.

[0055] Furthermore, the sorting unit 205 can include the upper right and lower left corners of the block to be decoded and the reference block in the neighboring pixels.

[0056] Furthermore, the sorting unit 205 can utilize multiple lines as neighboring pixels, rather than being limited to just one line. By utilizing many neighboring pixels, the sorting of boundary widths can be made more accurate.

[0057] Furthermore, if the sub-region is a combination of intra-prediction and motion compensation, the selection unit 205 can reuse the results of selecting candidate boundary widths near the sub-region of motion compensation as the boundary width of the sub-region of intra-prediction.

[0058] Furthermore, as shown in Figure 7, if the division boundary within the block to be decoded does not overlap with the regions of neighboring pixels of the decoded block and the reference block, which are pre-set comparison targets, and there are no comparison targets for similarity that would arise from applying different boundary widths to neighboring pixels to be weighted and averaged across the division boundary (i.e., the similarity described above cannot be determined), the selection unit 205 may use a pre-set fixed boundary width or determine the boundary width by decoding an explicit codeword.

[0059] Furthermore, in error comparison using template matching, the only difference between different boundary widths in the same division mode is the weighted average region. That is, regions that are not weighted averaged in the neighboring pixels and the reference neighboring pixels (or nearby intra-predicted pixels) may have the same error. Therefore, to simplify the template matching process, when comparing errors, the comparison region may be limited to only the region with the largest boundary width to be compared.

[0060] Furthermore, for large decryption blocks, errors are less likely to occur between different boundary widths due to this template matching process. Therefore, for large blocks, this template matching process can be skipped. This is expected to simplify the decryption process.

[0061] The following describes an example of how the code length of the codeword control information (codeword) for the boundary width of the splitting mode can be reduced by template matching as described above. Specifically, as shown in Figure 8, the selection unit 205 records the boundary width of the block to be decoded to which the splitting mode is applied as a histogram (history) for each line of the decoding tree block (encoded tree block) from the left edge to the right edge of the picture boundary in the image (picture) containing the block to be decoded, and assigns a shorter code using cu_bld_idx, which is the control information (codeword) that identifies the boundary width of the splitting mode, to the boundary width with a larger histogram.

[0062] The assignment order of codes to the boundary widths of this partitioning mode is changed (reordered) sequentially with each histogram update. Here, the decryption tree block (encoded tree block) is the starting block of the partitioning structure of the decryption target block (encoded block) disclosed in Non-Patent Document 1. The block size of the decryption tree block may be a power of 2, such as 64×64 pixels, 128×128 pixels, or 256×256 pixels. In contrast, the decryption target block may have a pixel size that is the same as or less than the size of the decryption tree block (encoded tree block), but is a power of 2.

[0063] The number of blocks to which the splitting mode is applied, which is added to this histogram, may be initialized when the decoded tree block reaches the leftmost or rightmost picture boundary of the picture.

[0064] Alternatively, the number of blocks to which the split mode is applied may be initialized if the slice boundary is reached before the rightmost picture boundary is reached.

[0065] Furthermore, the order in which the signs are assigned to boundary widths when the histogram for each boundary width is zero (i.e., the number of applied blocks is the initial value) may be the same as in the conventional method. For example, the signs may be set in order from shortest to longest as follows: standard width, 1 / 2 width, 2 width, 1 / 4 width, and 4 width.

[0066] Furthermore, a threshold condition may be set to change (correct) the assignment order of boundary widths during histogram updates. For example, as a threshold for when the standard width boundary is swapped with other boundary widths, if the number of standard widths and other applicable blocks are K and L respectively, the order may be swapped only if K < α*L is satisfied. Here, K and L are natural numbers, and α may be set to, for example, 1.1, 1.2, 1.5, etc.

[0067] Furthermore, this method of shortening control information using a histogram may be applied not only to control information that identifies the boundary width of the division mode, but also to other control information disclosed in Non-Patent Document 1.

[0068] For example, this shortening of control information may be applied to bcw_idx, which is control information (codeword) that identifies a unique weight coefficient from multiple weight coefficient candidates in BCW (Bi-prediction with CU-level Weight) disclosed in Non-Patent Document 1.

[0069] BCW is an effective tool for handling motion fades (uniform changes in the overall brightness of an image). Therefore, the weight coefficients selected may be the same within the same image, the same slice, or the same line of the decoding target tree. Thus, similar to correcting the assignment order of different boundary widths to the code of the control information (codeword) cu_bld_idx based on the histogram, correcting the assignment order of different weight coefficients to the code of the control information (codeword) bcw_idx based on the histogram can be expected to shorten the code length of the control information (codeword) bcw_idx and, consequently, improve coding efficiency.

[0070] Another application example is the application to mmvd_direction_idx and mmvd_distance_idx, which are control information (codewords) that identify candidate directions and magnitudes of discrete motion vector differences in MMVD (Merge with Motion Vector Difference) as disclosed in Non-Patent Document 1.

[0071] MMVD is an effective tool for correcting merged motion vectors in moving images where the motion vector changes uniformly across a wide area of ​​the image. Therefore, the selected motion vector differences may be the same within the same image, the same slice, or the same line of the target tree to be decoded. Thus, similar to correcting the assignment order of different boundary widths to the codes of the control information (codewords) cu_bld_idx and bcw_idx based on the histogram, correcting the assignment order of different weight coefficients to the codes of the control information (codewords) mmvd_direction_idx and mmvd_distance_idx based on the histogram can be expected to shorten the code length of mmvd_direction_idx and mmvd_distance_idx and, consequently, improve coding efficiency.

[0072] The control information used in the boundary width selection method by the selection unit 205 will be described below.

[0073] The coded information input to the image decoding device 200 may include a sequence parameter set (SPS) which summarizes control information (codewords) at the sequence level. Furthermore, such coded information may include a picture parameter set (PPS) or picture header (PH) which summarizes control information at the picture level. In addition, such coded information may include a slice header (SH) which summarizes control information at the slice level.

[0074] Referring to Figure 9, an example of setting the boundary width selection method on a sequence-by-sequence basis is described.

[0075] As shown in Figure 9, in step S101, the decoding unit 201 determines whether sps_div_enabled_flag is 1 in the sequence parameter set.

[0076] Here, sps_div_enabled_flag is a syntax (codeword) that controls whether or not split mode is enabled. If sps_div_enabled_flag is 1, it indicates that split mode is enabled, and if sps_div_enabled_flag is 0, it indicates that split mode is disabled.

[0077] If sps_div_enabled_flag is 1, this operation proceeds to step S102; if sps_div_enabled_flag is 0, this operation terminates.

[0078] In step S102, the decoding unit 201 decodes sps_div_selecting_flag.

[0079] Here, sps_div_selecting_flag is a syntax (codeword) that controls whether or not boundary width candidate selection is performed. If sps_div_selecting_flag is 1, it indicates that boundary width candidate selection is enabled, and if sps_div_selecting_flag is 0, it indicates that boundary width candidate selection is disabled.

[0080] In step S103, the decoding unit 201 decodes sps_div_selecting_mode. Here, sps_div_selecting_mode is a syntax (codeword) that controls the method for selecting candidate boundary widths.

[0081] By using sps_div_selecting_mode, the method for selecting boundary width candidates according to image characteristics can be changed on a sequence-by-sequence basis, which is expected to maximize encoding efficiency.

[0082] For example, for sequences composed of computer graphics, the boundary width is often narrow, so the number of boundary width candidates for each small region can be set to be small. Conversely, for sequences composed of natural images, the number of boundary width candidates for each small region can be set to be large, thereby maximizing encoding efficiency.

[0083] When setting the method for selecting candidate boundary widths on a picture-by-picture basis, the decoding unit 201 similarly decodes pps_div_enabled_flag, pps_div_selecting_flag, and pps_div_selecting_mode in the picture parameter set or picture header.

[0084] By using pps_div_selecting_mode, the method for selecting boundary width candidates according to image characteristics can be changed on a picture-by-picture basis, which is expected to maximize encoding efficiency.

[0085] For example, the number of boundary width candidates for each small region can be set to be small for pictures composed of computer graphics, and large for pictures composed of natural images, thereby maximizing encoding efficiency.

[0086] If the method for selecting candidate boundary widths is set on a slice-by-slice basis, the decoding unit 201 similarly decodes sh_div_enabled_flag, sh_div_selecting_flag, and sh_div_selecting_mode in the slice header.

[0087] By using sh_div_selecting_mode, the method for selecting boundary width candidates according to image characteristics can be changed on a slice-by-slice basis, which is expected to maximize encoding efficiency.

[0088] For example, for slice regions containing CG-composed partial images, the number of candidate boundary widths for each sub-region can be set to be small, while for slice regions containing natural images, the number of candidate boundary widths for each sub-region can be set to be large, thereby maximizing encoding efficiency.

[0089] By setting the encoding only in the upper layers, the increase in the encoding amount can be suppressed, or by setting it in the lower layers as well and prioritizing the settings in the lower layers, adaptive control can be achieved.

[0090] Alternatively, if a method for selecting candidate boundary widths is pre-configured, the decoding of the boundary width selection method itself can be omitted.

[0091] In the examples above, we described how to set the method for selecting candidate boundary widths on a sequence, picture, or slice basis. However, instead of setting these, you can also directly select a pattern for selecting candidate boundary widths on a block basis, as described later. In this case, the degree of freedom in setting boundary width combinations will decrease, but the increase in header information mentioned above can be avoided.

[0092] Referring to Figure 10, an example of the operation of setting a method for selecting candidate boundary widths on a per-block basis for decoding is described.

[0093] As shown in Figure 10, in step S201, the decoding unit 201 determines whether any of sps_gen_enabled_flag, pps_gen_enabled_flag, and sh_gen_enabled_flag is 1.

[0094] If neither is 1, the operation proceeds to step S202; if either is 1, the operation proceeds to step S203.

[0095] In step S202, the decoding unit 201 decides not to apply the selection of candidate boundary widths, and this operation ends.

[0096] In step S203, the decoding unit 201 determines whether the block to be decoded is in split mode or not.

[0097] If the answer is Yes, the process proceeds to step S204; if the answer is No, the process proceeds to step S202.

[0098] In step S204, the decoding unit 201 decodes cu_div_mode, which is a control signal representing the division mode.

[0099] In step S205, the decoding unit 201 further determines whether N=1 or not. If N=1, the operation proceeds to step S206; if N≠1, the operation proceeds to step S207.

[0100] In step S207, the decoding unit 201 decodes cu_bld_idx, which is a control signal that specifies the boundary width from among the candidate boundary widths.

[0101] In the example shown in Figure 10, the decoding unit 201 decodes one cu_bld_idx; however, if two sub-regions are in motion compensation mode, the decoding unit 201 may decode two cu_bld_idx, cu_bld_idx0 and cu_bld_idx1.

[0102] cu_bld_idx is decoded to identify one of the candidate boundary widths selected by the lowest-level div_selecting_mode applied to the block to be decoded.

[0103] According to the image decoding device 200 of this embodiment, decoding can be performed by dividing each unit block into multiple small regions and selecting one or more candidate boundary widths.

[0104] <Second Embodiment> Hereinafter, with reference to Figures 11 to 30, the image decoding device 200 according to the second embodiment of the present invention will be described, focusing on the differences from the image decoding device 200 according to the first embodiment described above.

[0105] Figure 11 is a diagram showing an example of the functional blocks of the image decoding device 200 according to this embodiment. As shown in Figure 11, the image decoding device 200 includes a code input unit 210, a decoding unit 201, an inverse quantization unit 202, an inverse transformation unit 203, an intra prediction unit 204, a synthesis unit 205, an adder 206, a storage unit 207, a motion compensation unit 208, and an image output unit 220.

[0106] The code input unit 210 is configured to acquire code information encoded by the image encoding device.

[0107] The decoding unit 201 is configured to decode control information and quantization values ​​from the code information input from the code input unit 210. For example, the decoding unit 201 is configured to output control information and quantization values ​​by performing variable-length decoding on such code information.

[0108] Here, the quantized values ​​are sent to the inverse quantization unit 202, and the control information is sent to the intra prediction unit 204, the synthesis unit 205, and the motion compensation unit 208. This control information includes information necessary for controlling the intra prediction unit 204, the synthesis unit 205, and the motion compensation unit 208, and may also include header information such as a sequence parameter set, a picture parameter set, a picture header, and a slice header.

[0109] The inverse quantization unit 202 is configured to inversely quantize the quantized values ​​sent from the decoding unit 201 to obtain the decoded conversion coefficients. These conversion coefficients are then sent to the inverse transformation unit 203.

[0110] The inverse transform unit 203 is configured to inversely transform the transformation coefficients sent from the inverse quantization unit 202 to obtain the decoded predicted residual. This predicted residual is then sent to the adder 206.

[0111] The intra-prediction unit 204 is configured to generate a first prediction pixel based on the decoded pixels and control information sent from the decoding unit 201. Here, the decoded pixels are obtained via the adder 206 and stored in the storage unit 207. The first prediction pixel is sent to the adder 206.

[0112] The storage unit 207 is configured to cumulatively store the decoded pixels sent from the adder 206. These decoded pixels are referenced by the motion compensation unit 208 via the storage unit 207.

[0113] The motion compensation unit 208 is configured to generate a second predicted pixel for addition with the predicted residual in the adder 206, based on the decoded pixel obtained by referring to the storage unit 207 and the control information decoded by the decoding unit 201. The generated second predicted pixel is sent to the adder 206 or the merging unit 205.

[0114] The adder 206 is configured to obtain a decoded pixel by adding one of the first to third predicted pixels generated from the decoded pixels, etc., to the predicted residual sent from the inverse transform unit 203. The decoded pixel is then sent to the image output unit 220, the storage unit 207, and the intra-prediction unit 204.

[0115] The synthesis unit 205 is configured to synthesize any combination including at least one of the first and second prediction pixels into small regions divided by multiple line segments, based on the control information decoded by the decoding unit 201, to form a third prediction pixel.

[0116] The synthesis unit 205, which is a characteristic configuration of the image decoding device 200 according to this embodiment, will be described below.

[0117] The role of the synthesis unit 205 is to accurately predict the pixels of the decoding target block by dividing the decoding target block into multiple small regions (small region division) so that the predicted residual can be represented with a small amount of code when the subsequent addition unit 206 calculates the decoded pixels, and then combining the first predicted pixels or second predicted pixels corresponding to each region (synthetic prediction).

[0118] Figure 12 shows an example of a case where geometric partitioning mode (GPM) is used as the sub-region partitioning. In the example in Figure 12, the block to be decoded is divided into sub-region A and sub-region B by a diagonal straight line.

[0119] However, the geometric partitioning mode has a problem in that it cannot handle cases where the boundary between the foreground and background is complex because the partitioning is limited to a single line, and therefore cannot sufficiently improve encoding efficiency.

[0120] (Basic concept of multiple line segment division mode) To solve this problem, the image decoding device 200 according to this embodiment employs a procedure in which the synthesis unit 205 divides the block to be decoded into N (where N is a natural number greater than 1) line segments.

[0121] For example, Figure 13 shows an example of a case where the block to be decoded is divided by N=2 line segments, and Figure 14 shows an example of a case where the block to be decoded is divided by N=3 line segments.

[0122] This method of sub-region partitioning and composite prediction of a decoded block using multiple (N) line segments will be referred to as the "multiple line segment partitioning mode" from now on.

[0123] The synthesis unit 205 may determine the division type of the multiple line segment division mode based on the control information. Details will be described later.

[0124] Increasing the number of line segments N in the multiple line segment division mode improves prediction accuracy, while decreasing the number of line segments N in the multiple line segment division mode reduces the amount of code in the control information that represents the division shape.

[0125] The combining unit 205 may set the number of line segments N in the multiple line segment division mode to a fixed value. For example, as described above, the combining unit 205 may be set to N=2, N=3, N=4, N=5, etc.

[0126] The combining unit 205 may set such fixed values ​​to common values ​​regardless of the length of the short side, the length of the long side, the size (area), or the aspect ratio of the block to be decoded.

[0127] Alternatively, the synthesis unit 205 may set these fixed values ​​to different values ​​based on the length of the short side, the length of the long side, the size (area), or the aspect ratio of the block to be decoded.

[0128] Furthermore, the combining unit 205 may set the number of line segments N in the multiple line segment division mode to be variable. For example, when the combining unit 205 sets the number of line segments N in the multiple line segment division mode to be variable, the number of such line segments N may be determined in proportion to the length of the short side, the length of the long side, and the size (area) of the block to be decoded.

[0129] Furthermore, the combining unit 205 may limit at least one of the positional relationship and angle of the multiple line segments in order to suppress the amount of code in the control information that represents the divided shape.

[0130] For example, the merging unit 205 may limit the first line segment to either the horizontal direction (0 degrees) or the vertical direction (90 degrees) of the block to be decoded. Furthermore, the merging unit 205 may limit the (n+1)th line segment to only the direction perpendicular to the nth line segment (90 degrees).

[0131] However, since a partitioning shape that divides the block to be decoded into four equal parts in a grid pattern can be achieved by the recursive rectangular block partitioning (quadrutree-binary-ternary tree partitioning) disclosed in Non-Patent Literature 1, it is desirable that the synthesis unit 205 be limited so that it cannot select arrangements that can be achieved by such existing block partitioning.

[0132] Applying such limitations, for example, in the case of the number of line segments N=2 in the multiple line segment division mode, is equivalent to selecting the division shape (multiple line segments) of the block to be decoded from among the patterns (32 patterns) of dividing horizontally and vertically at multiple division points as shown in Figure 15.

[0133] Alternatively, applying such limitations would be equivalent to selecting the division shape (multiple line segments) of the block to be decoded from among the patterns shown in Figure 15 and the diagonal division patterns shown in Figure 16, if the first line segment can be in the horizontal (0 degrees), vertical (90 degrees), or diagonal (45 degrees) directions.

[0134] Here, reducing the aforementioned limitations improves prediction accuracy, while increasing the aforementioned limitations reduces the amount of code in the control information used to represent the segmented shape.

[0135] The synthesis unit 205 can be set either fixedly or variably with respect to the above-described limiting method.

[0136] Furthermore, the synthesis unit 205 can set different limitations depending on the size of the block to be decoded (length of the short side, length of the long side, size (area), or aspect ratio, etc.).

[0137] For example, regarding the division points shown in Figure 15, if such division points are placed at regular intervals of a certain number of pixels, the number of division points will be proportional to the length of the edges of the block to be decoded.

[0138] Conversely, the synthesis unit 205 can fix the number of division points, regardless of the size of the block to be decoded, by changing the arrangement of these division points.

[0139] For example, if the compositing unit 205 is arranging a fixed number of K (K is a natural number) division points in the vertical or horizontal direction of the block to be decoded, it may arrange the division points at intervals of L / (K+1) pixels, where L (L is a natural number) is the ratio of the side length of the block to be decoded to the number of pixels.

[0140] Here, L may be a natural number that is a power of 2 greater than or equal to 4, such as 4, 8, 16, 32, 64, or 128, as described in Non-Patent Document 1. Similarly, K may also be a natural number that is a power of 2 greater than or equal to 4, such as 4, 8, 16, 32, 64, or 128.

[0141] Furthermore, the possible values ​​of K may be restricted so that L / (K+1) pixels are natural numbers that are powers of 2 greater than or equal to 4, such as 4, 8, 16, 32, 64, or 128.

[0142] Furthermore, if the aspect ratio of the blocks to be decoded is different, the combining unit 205 may set K different division points for the vertical and horizontal sides (height and width) of the blocks to be decoded.

[0143] Furthermore, the synthesis unit 205 may include the center of the decoded block in Figure 15 as a division point.

[0144] Increasing the number of division points improves prediction accuracy, while decreasing the number of division points reduces the amount of code required to represent the division shape.

[0145] (Method for generating the third prediction pixel in multi-line segment division mode) The method for generating the third predicted pixel by the synthesis unit 205 in the multiple line segment division mode will be described below.

[0146] The synthesis unit 205 is configured to weight the predicted pixels for each of the sub-regions A and B of the decoded block, which have been divided into multiple line segments, according to the distance between the dividing lines (i.e., perform a composite prediction).

[0147] The types of prediction pixels in such small regions A and B may be combinations of different inter-prediction pixels, as in the geometric division mode disclosed in Non-Patent Document 1; combinations of inter-prediction pixels and intra-prediction pixels, as in the geometric division mode intra-prediction disclosed in Non-Patent Document 3; or combinations of different intra-prediction pixels.

[0148] Here, different inter-prediction pixels are generated based on different motion vectors, and different intra-prediction pixels are generated using different intra-prediction modes.

[0149] For the composite prediction of sub-region A and sub-region B by the composite unit 205, a weighted average based on the distance from the dividing line can be used, similar to the one used for composite prediction of geometric partitioning modes disclosed in Non-Patent Document 1 and Non-Patent Document 2.

[0150] The sum of the weight coefficients for multiple prediction pixels is designed to be 1 for each pixel, and the result of combining multiple prediction pixels using these weight coefficients through weighted averaging is used as the prediction pixel by the synthesis unit 205.

[0151] Here, pixels with a weight coefficient of 1 (i.e., the maximum value) are adopted as input prediction pixels, and pixels with a weight coefficient of 0 (i.e., the minimum value) are not used as input prediction pixels. Conceptually, this is equivalent to dividing a unit block into multiple sub-regions, and determining which of the multiple input prediction pixels to apply to which areas in what proportion.

[0152] Figure 17 shows an example of three patterns of weight coefficients to be assigned to the division boundary of subregion B shown in Figure 12. In the example in Figure 17, the horizontal axis represents the distance in pixels from the position of the division boundary (division line), and the vertical axis represents the weight coefficient.

[0153] Specifically, we have prepared three patterns: (1) in which the weight coefficient [0,1] is assigned to the range [a,b] for predicted pixel-level distances a and b from a pre-set division boundary position; (2) in which distances a and b are similarly doubled and the weight coefficient [0,1] is assigned to the range [2a,2b]; and (3) in which distances a and b are similarly halved and the weight coefficient [0,1] is assigned to the range [a / 2,b / 2].

[0154] These are equivalent to providing multiple patterns (variable values) rather than a limited pattern (fixed value) for the width of the subdivision boundary of a small region disclosed in Non-Patent Document 2, i.e., the width τ at which the weight coefficient is other than the minimum or maximum value.

[0155] Here, xc and yc are coordinates within the block to be decoded. That is, the compositing unit 205 may be configured to set multiple weight coefficients according to the inter-pixel distance from the division boundary.

[0156] Alternatively, a=b may be used to set a weight coefficient that is symmetric with respect to the division boundary. That is, the combining unit 205 may be configured to set a weight coefficient that is symmetric with respect to the division boundary as the weight coefficient described above. With such a configuration, b becomes unnecessary, and therefore the amount of sign can be reduced.

[0157] Alternatively, a weight coefficient that is asymmetric with respect to the dividing boundary may be set, where a ≠ b. In other words, the synthesis unit 205 may be configured to set a weight coefficient that is asymmetric with respect to the dividing boundary as the weight coefficient described above. With such a configuration, it is possible to make predictions with high accuracy when there are different degrees of blurring on both sides of the boundary.

[0158] Furthermore, the number of weighting coefficients can be increased beyond just two, a and b, to include multiple line segments. In other words, the synthesis unit 205 may be configured to set weighting coefficients using multiple line segments according to the inter-pixel distance from the division boundary. With such a configuration, high-precision prediction is possible even when blurring occurs non-linearly.

[0159] In this way, by setting multiple weighting coefficients according to the inter-pixel distance from the division boundary, the effect of being able to derive the result uniformly for various block sizes such as 8x8 and 64x16 can be obtained.

[0160] The type, shape, and number of patterns in the synthesis unit 205 can be arbitrarily set.

[0161] For example, in the above example, we explained two and half the distances a and b as multiple patterns, but four times or a quarter of the distances a and b would also be acceptable. Also, in the above formula, the weight coefficients were set to values ​​between 0 and 8, but they can also be set to other values ​​such as 0 to 16 or 0 to 32. In particular, when the inter-pixel distance from the division boundary is two or four times, increasing the maximum value of the weight coefficient can improve the accuracy of the pixel-level weighted average.

[0162] The blending unit 205 may select from pre-prepared combinations for setting multiple weight coefficients (width of the blending region, maximum and minimum values ​​of the weight coefficients) for sub-regions A and B, based on control information sent from the decoding unit 201.

[0163] For example, as disclosed in Non-Patent Document 2, the synthesis unit 205 may use such control information to select the weight coefficients mentioned above from a plurality of pre-prepared patterns (in Non-Patent Document 2, five patterns: 1 / 4 width, 1 / 2 width, 1 width, 2 width, and 4 width).

[0164] Alternatively, as disclosed in Non-Patent Document 2, the blending unit 205 may, using control information, reduce the number of selectable candidates from a plurality of pre-prepared candidate patterns according to the size, short side length, long side length, or aspect ratio of the block to be decoded, and then select the blending width indicated by the control information.

[0165] In this multi-line segment division mode, unlike the composite prediction for geometric division modes disclosed in Non-Patent Documents 1-3, there are multiple division boundaries (division lines) that divide the small region, and therefore, composite prediction is performed for these multiple division lines.

[0166] Figure 18 shows an example of a case where, as shown in Figure 15, a division boundary is formed by two dividing lines, one vertical and one horizontal, with respect to the block to be decoded, and composite prediction is performed on a pixel-by-pixel basis according to the distance from these two dividing lines (dividing line 1 and dividing line 2).

[0167] For the combined prediction of these multiple dividing lines, the synthesis unit 205 may apply a common weighted average.

[0168] Alternatively, the synthesis unit 205 may apply a weighted average consisting of the maximum value of the different weighting coefficients and the distance from the dividing line.

[0169] Furthermore, the synthesis unit 205 may be configured to select weight coefficients from a plurality of weight coefficients according to at least one of the short side length, long side length, aspect ratio, size (number of pixels), or type of division mode of the block to be decoded.

[0170] Alternatively, the synthesis unit 205 may be configured to select weight coefficients from a plurality of weight coefficients depending on the type of intra-prediction mode when intra-prediction is applied to sub-region A or sub-region B.

[0171] Alternatively, the synthesis unit 205 may be configured to select a weighting coefficient from among a plurality of weighting coefficients according to the size (number of pixels) of the sub-region A or sub-region B divided within the block to be decoded by the multiple line segment division mode.

[0172] Alternatively, the synthesis unit 205 may be configured to select a weighting coefficient from among a plurality of weighting coefficients according to the size (number of pixels) of the dividing line.

[0173] Alternatively, the synthesis unit 205 may be configured to select a weighting coefficient from among a plurality of weighting coefficients according to the ratio of the edges of the block to be decoded in the same direction, when the direction of the dividing line is horizontal or perpendicular to the block to be decoded.

[0174] Furthermore, in the multiple line segment division mode, due to the nature of having multiple line segments constituting the division boundary, the region where the weight coefficient in the weighted average is at its maximum or minimum value, that is, the region where multiple different predicted pixels are combined and predicted (blending region), may overlap in the vertical direction of each division line.

[0175] Figure 19 is a diagram illustrating an example of the blending region (the gray area in Figure 19(a)) relative to dividing line 1 and dividing line 2.

[0176] As shown in Figure 19(a), the area where the blending regions for division line 1 and division line 2 overlap (hereinafter referred to as the overlapping blending region) is indicated by a dashed frame.

[0177] Furthermore, Figure 19(b) shows an example of the weight coefficient W_1A applied to the predicted pixels of sub-region A with respect to division line 1, and an example of the weight coefficient W_2A applied to the predicted pixels of sub-region A with respect to division line 2.

[0178] The weight coefficients W_1B and W_2B for each subregion B are obtained by subtracting W_1A and W_2A from the maximum value of the weight coefficient.

[0179] The blending unit 205 may predict and blend the third predicted pixels of the blending region and the overlapping blending region using newly generated weight coefficients by selecting the minimum values ​​of each element W_1A and W_2A of the weight coefficients, as shown in calculation example 1 of Figure 19(c).

[0180] Alternatively, the blending unit 205 may predict and blend the third predicted pixels of the blending region and the overlapping blending region using newly generated weight coefficients, as shown in calculation example 2 of Figure 19(c), by calculating the product of each element W_1A and W_2A of the weight coefficients.

[0181] (Method for determining whether or not the multiple line segment division mode is applicable and the type of division) The control information decoded by the decoding unit 201 will be described below.

[0182] The coded information input to the image decoding device 200 may include a sequence parameter set (SPS) that summarizes control information at the sequence level. Furthermore, such coded information may include a picture parameter set (PPS) or picture header (PH) that summarizes control information at the picture level. In addition, such coded information may include a slice header (SH) that summarizes control information at the slice level.

[0183] Referring to Figure 20, an example of the operation for setting the selection method for multiple line segment division modes on a sequence-by-sequence basis is described.

[0184] As shown in Figure 20, in step S101, the decoding unit 201 determines whether sps_div_enabled_flag is 1 in the sequence parameter set.

[0185] Here, sps_div_enabled_flag is syntax that controls whether or not split mode is enabled. If sps_div_enabled_flag is 1, it indicates that split mode is enabled, and if sps_div_enabled_flag is 0, it indicates that split mode is disabled.

[0186] If sps_div_enabled_flag is 1, this operation proceeds to step S102; if sps_div_enabled_flag is 0, this operation terminates.

[0187] In step S102, the decoding unit 201 decodes sps_div_multi_flag.

[0188] Here, sps_div_multi_flag is syntax that controls whether or not the multiple line segment division mode is enabled. If sps_div_multi_flag is 1, it indicates that the multiple line segment division mode is enabled (N>1), and if sps_div_multi_flag is 0, it indicates that the single line segment division mode is disabled (N=1).

[0189] If sps_div_multi_flag is 1, this operation proceeds to step S103; if sps_div_multi_flag is 0, this operation terminates.

[0190] In step S103, the decoding unit 201 decodes sps_div_multi_mode, where sps_div_multi_mode is the syntax for controlling multiple line segment division modes.

[0191] By using sps_div_multi_mode, it is possible to change the settings of the multiple line segment division mode according to the image characteristics on a sequence-by-sequence basis, which is expected to maximize encoding efficiency.

[0192] For example, for sequences composed of computer graphics, there are many boundaries including horizontal and vertical directions, so the system can be set to limit the division types to those consisting of right angles. Conversely, for sequences composed of natural images, the restrictions on division types can be relaxed, thereby maximizing encoding efficiency.

[0193] When setting the boundary width selection method on a picture-by-picture basis, the decoding unit 201 similarly decodes pps_div_enabled_flag, pps_div_multi_flag, and pps_div_multi_mode in the picture parameter set or picture header.

[0194] By using pps_div_multi_mode, it is possible to change the settings for multiple line segmentation modes according to the image characteristics on a picture-by-picture basis, which is expected to maximize encoding efficiency.

[0195] For example, the system can be configured to limit the division types to right angles for pictures composed of computer graphics, and to relax the restrictions on division types for pictures composed of natural images, thereby maximizing encoding efficiency.

[0196] If the boundary width selection method is set on a slice-by-slice basis, the decoding unit 201 similarly decodes sh_div_enabled_flag, sh_div_multi_flag, and sh_div_multi_mode in the slice header.

[0197] By using sh_div_multi_mode, it is possible to change the settings of the multiple line segment division mode according to the image characteristics on a slice-by-slice basis, which is expected to maximize encoding efficiency.

[0198] For example, the system can be configured to limit the division type to right angles for slice regions containing CG-composed partial images, and to relax the restriction on the division type for slice regions containing natural images, thereby maximizing encoding efficiency.

[0199] By setting the encoding only in the upper layers, the increase in the encoding amount can be suppressed, or by setting it in the lower layers as well and prioritizing the settings in the lower layers, adaptive control can be achieved.

[0200] Alternatively, if a multiple line segment division mode is pre-configured, the decoding of the multiple line segment division mode itself can be omitted.

[0201] In the examples above, we described how to set the multiple line segment division mode on a sequence, picture, or slice basis. However, you can also set the multiple line segment division mode directly on a block basis, as described later, without using these methods.

[0202] In this case, the degree of flexibility in setting the multiple line segment division mode is reduced, but the increase in header information mentioned above can be avoided.

[0203] Referring to Figure 29, an example of changing the operation to set the selection method for multiple line segment division modes on a sequence-by-sequence basis is described.

[0204] As shown in Figure 29, the difference between Figure 29 and Figure 20 is that step S104 is included.

[0205] In step 104, the decoding unit 201 determines whether or not the technique for reordering the division modes associated with the decoded value of cu_div_idx (control information) for identifying the division mode based on template matching (hereinafter referred to as template-based division mode subscript reordering) is effective in the sequence parameter set, by determining whether or not sps_div_template_reordering_enabled_flag (control information), which is controlled on a sequence-by-sequence basis, is 1.

[0206] Here, if sps_div_template_reordering_enabled_flag is 1, it indicates that template-based subscript reordering is enabled, and if sps_div_template_reordering_enabled_flag is 0, it indicates that template-based subscript reordering is disabled.

[0207] Note that sps_div_template_reordering_enabled_flag may be decoded by the decoding unit 201 before step S104, or its value may be estimated without being decoded.

[0208] If sps_div_template_reordering_enabled_flag is not decoded, the decoding unit 201 estimates the value of sps_div_template_reordering_enabled_flag to be 0.

[0209] The decoding unit 201 proceeds to step S102 if sps_div_template_reordering_enabled_flag is 1 (Yes). If sps_div_template_reordering_enabled_flag is 0 (No), this process is terminated.

[0210] As will be explained in more detail later, sorting of partition mode subscripts based on templates has the effect of shortening the code length of the partition mode subscripts. Therefore, by determining that a multi-line partition mode is valid only when sorting of partition mode subscripts based on templates is enabled, the code size of cu_div_idx used to identify the type of multi-line partition mode or the type of partition mode that includes a multi-line partition mode on a per-block basis can be reduced, and as a result, an improvement in coding performance can be expected.

[0211] Referring to Figure 21, an example of the operation for setting the selection method for multiple line segment division modes on a block basis is described.

[0212] As shown in Figure 21, in step S201, the decoding unit 201 determines whether any of sps_div_enabled_flag, pps_div_enabled_flag, and sh_div_enabled_flag is 1.

[0213] If neither is 1, this operation terminates; if either is 1, this operation proceeds to step S202.

[0214] In step S202, the decoding unit 201 determines whether the block to be decoded is in split mode or not.

[0215] If the answer is Yes, the process proceeds to step S203; if the answer is No, the process terminates.

[0216] In step S203, the decoding unit 201 decodes cu_div_idx, which is a control signal representing the division mode.

[0217] cu_div_idx is decoded to identify one of the candidates for multiple line segmentation modes, which are selected by the lowest layer div_multi_mode applied to the block to be decoded.

[0218] The decoding unit 205 decodes the above-mentioned cu_div_idx and identifies the multiple line segment division mode according to the decoded value.

[0219] For example, 32 decoded values ​​of cu_div_idx, as shown in Figure 28, are prepared, corresponding to the 32 candidate division modes for the multiple line segment division modes shown in Figure 15.

[0220] Each decoded value in cu_div_idx corresponds to divDirectionIdx, an internal parameter that indicates one of four division directions (whether the division boundary (the area divided by the division line) is in the upper left, upper right, lower right, or lower left) for identifying the multiple line division mode patterns shown in Figure 15, and divLocationIdx, an internal parameter that determines one of eight division points.

[0221] If this multi-line segment division mode is to be applied to the geometric division mode described in Non-Patent Document 1, it can be achieved by adding cu_div_idx to a table where the 64 decoded values ​​of merge_gpm_partition_idx, which identifies the geometric division mode, are associated.

[0222] merge_gpm_partition_idx is associated with angleIdx, which represents 20 angle patterns, and distanceIdx, which represents 4 distance patterns, to represent 64 patterns of partition lines in geometric partition mode.

[0223] When applying the multiple line segment division mode to the geometric division mode, if divDirectionIdx and divLocationIdx are configured as new values ​​for angleIdx and distanceIdx as internal parameters, the decoding unit 201 can identify the pattern of the multiple line segment division mode in addition to the geometric division mode by decoding merge_gpm_partition_idx.

[0224] Referring to Figure 30, an example of the operation for setting a method for selecting division modes, including multiple line segment division modes, on a block-by-block basis, is described.

[0225] As shown in Figure 30, the difference between Figure 30 and Figure 21 is that steps S204 and S205 are included.

[0226] In step 104, the decoding unit 201 determines whether predetermined conditions are met. If the decoding unit 201 determines that the predetermined conditions are met, it proceeds to step S205; otherwise, it proceeds to step S203.

[0227] Here, the predetermined conditions may include the condition that the block size of the target block is less than or equal to a predetermined block size. The predetermined block size may be specified as a number of pixels that is a power of 2, such as 8x8 pixels, 16x16 pixels, 32x32 pixels, 64x64 pixels, or 128x128 pixels.

[0228] Due to its nature of dividing a target block with multiple line segments, the multiple line segment division mode makes it difficult for the division boundaries to align with the block boundaries within the block in large-sized blocks.

[0229] On the other hand, in small-sized blocks, the division boundaries by multiple line segments are more likely to align with the block boundaries within the block. Therefore, by setting a block size threshold to restrict large-sized blocks as described above, the multiple line segment division mode can be enabled only for small-sized blocks, resulting in improved encoding performance.

[0230] As an example of a change, the threshold determination may be based on the shorter side of the target block instead of the block size of the target block.

[0231] Specifically, the predetermined conditions in step S204 may include the condition that the shorter side of the target block is less than or equal to a predetermined number of pixels. The predetermined number of pixels may be specified as a power of 2, such as 8 pixels, 16 pixels, 32 pixels, 64 pixels, or 128 × 128 pixels.

[0232] This achieves the same effect as the threshold determination based on the block size of the target block described above.

[0233] Conversely, this predetermined condition may include the condition that the block size of the target block is equal to (or greater than) a predetermined block size. The predetermined block size may be specified as a number of pixels that is a power of 2, such as 4x4 pixels, 8x8 pixels, 16x16 pixels, or 32x32 pixels.

[0234] As mentioned above, the multiple line segment division mode, due to its nature of dividing the target block with multiple line segments, tends to align the division boundaries with the block boundaries within the block in small-sized blocks. However, in extremely small block sizes, the distance between multiple line segments or between multiple line segments and the target block boundary becomes short, making it difficult to distinguish it from conventional division modes or conventional coded block division modes that divide the target block with a single line segment.

[0235] Therefore, for extremely small block sizes, as described above, the multiple segment division mode for extremely small target blocks can be disabled by threshold determination based on block size. This reduces the amount of code required for identifying the multiple segment division mode, resulting in improved encoding performance.

[0236] As an example of a modification, the threshold determination may be based on the longer side of the target block instead of the block size of the target block. Specifically, the predetermined conditions in step S204 may include a condition that the longer side of the target block is equal to (or greater than) a predetermined number of pixels. The predetermined number of pixels may be specified as a power of 2, such as 4 pixels, 8 pixels, 16 pixels, or 32 pixels.

[0237] This achieves the same effect as the threshold determination based on the block size of the target block described above.

[0238] The decoding unit 201 determines whether sps_div_template_reordering_enabled_flag (control information), which controls on a sequence-by-sequence basis whether the technique for rearranging the division modes associated with the decoded value of cu_div_idx (control information) for identifying the division mode based on template matching (hereinafter referred to as template-based division mode subscript rearrangement) is enabled, is 1 or not, in the sequence parameter set.

[0239] If sps_div_template_reordering_enabled_flag is 1, it indicates that template-based subscript reordering is enabled, and if sps_div_template_reordering_enabled_flag is 0, it indicates that template-based subscript reordering is disabled.

[0240] If sps_div_template_reordering_enabled_flag is 1 (Yes), the decoding unit 201 proceeds to step S102.

[0241] On the other hand, the decoding unit 201 terminates this process if sps_div_template_reordering_enabled_flag is 0 (No).

[0242] As will be explained in more detail later, sorting of division mode subscripts based on templates has the effect of shortening the code length of the division mode subscripts. Therefore, by determining that a multi-line division mode is valid only when sorting of division mode subscripts based on templates is valid, the code size of cu_div_idx (control information) used to identify the type of multi-line division mode or the type of division mode that includes a multi-line division mode on a per-block basis can be reduced, and as a result, an improvement in coding performance can be expected.

[0243] (Method for deriving motion information for multiple line segment division mode) The following describes how to derive motion information for the multiple line segment division mode.

[0244] The motion information for sub-region A or sub-region B, which is divided in multiple line segment division mode, may be obtained by applying the same motion information derivation method as disclosed in Non-Patent Document 1 for geometric division mode.

[0245] Specifically, the motion compensation unit 208 creates a motion information candidate list (merge candidate list) for small regions A and B, consisting of motion information from neighboring blocks of the block to be decoded, and uses control information (merge index) transmitted from the image encoding device to identify the motion information in the merge candidate list to derive motion information from the merge candidate list.

[0246] If both sub-region A and sub-region B are interpretations, the decoding unit 201 decodes a merge index indicating different motion information candidates so that different motion information is derived from each sub-region.

[0247] Note that, as a method for deriving and registering candidates for motion information in the motion information candidate list, Non-Patent Document 1 discloses a technique called spatial merge. Specifically, the motion information at positions A0, A1, B0, B1, and B2 adjacent to the decoding target block shown in FIG. 27 is registered in the motion information candidate list as a candidate for the motion information of the decoding target block (spatial merge candidate).

[0248] The motion compensation unit 208 may limit the spatial merge candidates that can be registered in this motion information candidate list according to the multi-segment division mode. Specifically, the registration may be limited only to the spatial merges adjacent to each small region divided according to the multi-segment division mode.

[0249] When none of the spatial merge candidates are adjacent to each small region, the registrable spatial merge candidates may be limited only to the nearest spatial merge candidate, or only to N (N is a natural number, N < M) spatial merge candidates at a proximity position for all M (M is a natural number, 5 cases are exemplified above).

[0250] Also, when the small region divided by the multi-segment division mode is located far from all spatial merge candidates, such as at the lower right end of the decoding target block, all candidates may be included as registration targets for this small region.

[0251] Note that in FIG. 27, five cases in total, namely one at the upper left end, two at the right end, and two at the lower left end of the decoding target block, are exemplified as spatial merge candidates. However, the present technology can also be applied to cases where new spatial merge candidate positions are handled with respect to Non-Patent Document 1, such as between B2 and B0 or between B2 and A0.

[0252] (Method for Deriving Intra Prediction Mode for Multi-Segment Division Mode) Hereinafter, a method for deriving an intra prediction mode for the multi-segment division mode will be described.

[0253] The synthesis unit 205 may apply a parallel angle prediction mode (Angular prediction mode) to each dividing line for the intra-prediction mode for small region A or small region B that is divided in the multiple line segment division mode.

[0254] Alternatively, the synthesis unit 205 may apply a vertical angle prediction mode (Angular prediction mode) to each dividing line for the intra-prediction mode for small region A or small region B that is divided in the multiple line segment division mode.

[0255] Figure 22 shows an example of applying an angle prediction mode parallel to dividing line 1.

[0256] Alternatively, the synthesis unit 205 may derive the intra-prediction mode for the sub-region A or sub-region B that is divided in multiple line segment division modes using the derivation technique based on adjacent pixel analysis disclosed in Non-Patent Document 3.

[0257] The following describes three types of derivation techniques.

[0258] [Method 1 for deriving intra-prediction mode based on adjacent reference pixels] The following describes, with reference to Figures 23 to 25, a method for deriving an intra-prediction mode based on adjacent reference pixels for normal intra-prediction according to Non-Patent Document 3, and a method 1 for deriving an intra-prediction mode based on adjacent reference pixels for geometric partitioning mode according to this embodiment, which applies the aforementioned derivation method.

[0259] Figure 23 shows a method for deriving an intra-prediction mode based on adjacent reference pixels for normal intra-prediction according to Non-Patent Document 3, and an example of Method 1 for deriving an intra-prediction mode based on adjacent reference pixels for geometric partitioning mode according to this embodiment, which applies the said derivation method. Hereafter, these derivation methods will be collectively referred to as "DIMD (Decoder-side Intra Mode Derivation)".

[0260] In Non-Patent Document 3, as shown in Figure 23, in such DIMD, a horizontal and vertical Sobel filter with a 3x3 pixel window size is applied to adjacent reference pixels adjacent to the block to be decoded, and a histogram of pixel values ​​for all Angular prediction modes for normal intra-prediction is calculated. Here, the method for calculating the angle and pixel value of adjacent reference pixels to associate with each Angular prediction mode by applying the Sobel filter can be the same as in Non-Patent Document 3, so a detailed explanation is omitted.

[0261] Non-patent document 3 also states that the adjacent reference pixel regions used to calculate the histogram are controlled according to the block size of the block to be decoded, as shown in Figure 23. Specifically, for a 4x4 pixel block, the histogram is calculated using only the 3x3 pixel regions above and to the left of the top leftmost pixel of the block to be decoded.

[0262] Non-patent document 3 describes how intra-prediction pixels are generated using the highest and second-highest pixel values ​​in the calculated histogram, and the intra-prediction mode and Planar mode. Furthermore, the generated intra-prediction pixels are weighted and averaged using predetermined weight values ​​to produce the final intra-prediction pixels.

[0263] In this embodiment, the synthesis unit 205 may apply the DIMD disclosed in Non-Patent Document 3 to the derivation of intra-prediction modes for multiple line segmentation modes only. That is, the synthesis / generation process of intra-prediction pixels using the derived multiple intra-prediction modes is not performed.

[0264] As a result, for the intra prediction regions in the multiple-segment division mode (in the case of the Intra / Intra - multiple-segment division mode, two intra prediction regions), since intra prediction pixels can be generated in one intra prediction mode, while avoiding an increase in the circuit scale required for generating intra prediction pixels in the multiple-segment division mode in a hardware-implemented image decoding device, by analyzing the histogram of adjacent reference pixels of the decoding target block, an intra prediction reflecting textures such as edges suitable for the division shape of the multiple-segment division mode can be applied. Thus, the intra prediction performance is improved, and as a result, an improvement in the coding performance can be expected.

[0265] Note that in this embodiment, similar to Non-Patent Document 3, the decoding unit 201 may be configured to determine whether to derive an intra prediction mode by decoding or estimating a flag indicating whether DIMD can be applied.

[0266] Also, the composition unit 205 according to this embodiment may be configured to register the intra prediction mode derived by such DIMD when the same intra prediction mode is not already included in the intra prediction mode candidate list for the multiple-segment division mode, and not to register the intra prediction mode derived by such DIMD when the same intra prediction mode is already included in the intra prediction mode candidate list for the multiple-segment division mode.

[0267] According to such a configuration, it is possible to avoid the same intra prediction mode being registered repeatedly in the intra prediction mode candidate list.

[0268] Here, when registering a new intra prediction mode for such an intra prediction mode candidate list, comparing the consistency with an existing intra prediction mode, and if the two match, the process of selecting is hereinafter referred to as the "intra prediction mode candidate selection process".

[0269] Furthermore, the synthesis unit 205 according to this embodiment may limit the number of intra prediction modes to be registered in the intra prediction mode candidate list from among the intra prediction modes derived by such DIMD to one. In that case, the synthesis unit 205 derives the Angular prediction mode, which is the highest pixel value (luminance value), from the histogram.

[0270] Furthermore, if the Angular prediction mode (hereinafter referred to as the 1st Angular prediction mode), which has the highest pixel value (luminance value), is pruned during the intra-prediction mode candidate pruning process described above, it may be possible to compare it with existing intra-prediction modes in order from the highest histogram and register those that do not match.

[0271] Alternatively, if the 1st Angular prediction mode is pruned during the intra-prediction mode candidate pruning process described above, the intra-prediction mode derivation process using DIMD may be terminated.

[0272] As an example of modification, the number of intra prediction modes to be registered in the intra prediction mode candidate list from among the intra prediction modes derived by DIMD may be limited to two. In this case, the synthesis unit 205 derives the 1st Angular prediction mode and the 2nd Angular prediction mode, which is the next highest pixel value (luminance value), from the histogram.

[0273] If the 1st Angular prediction mode or the 2nd Angular prediction mode is pruned, as in the case described above, you may compare it with the existing intra prediction mode starting from the one with the next highest histogram and register the one that does not match, or you may simply terminate the intra prediction mode derivation process using DIMD.

[0274] Furthermore, the synthesis unit 205 according to this embodiment may restrict the adjacent reference pixels used for the histogram calculation of DIMD described above to a predetermined area based on the division shape of the multiple line segment division mode (i.e., the angle of the division lines of the multiple line segment division mode).

[0275] Figure 24 shows a table for limiting the area of ​​the template (adjacent reference pixels) referenced based on the division lines of the multiple line division mode, in a template matching technique for multiple line division modes disclosed in Non-Patent Document 3.

[0276] Specifically, A and L in Figure 24 represent the upper and left portions of the block to be decrypted, respectively.

[0277] In this embodiment, by applying a table of adjacent reference pixels defined (restricted) based on the division lines of the multi-segmentation mode disclosed in Non-Patent Document 3 to the calculation of the DIMD histogram, it is possible to derive the Angular prediction using adjacent reference pixels that exist only in the direction of the division lines of the multi-segmentation mode, while avoiding the use of all adjacent reference pixels adjacent to the block to be decoded in the calculation of the DIMD histogram. This reduces the processing load of deriving the intra-prediction mode by DIMD for inter-prediction of the multi-segmentation mode.

[0278] Alternatively, the synthesis unit 205 may derive an intra-prediction mode by DIMD using only the reference pixel regions that span across the left or upper block boundary of the decoded block for small regions A or B, which are divided in multiple line segmentation modes, instead of the adjacent reference pixel table in Figure 24 as described above, as shown in Figure 25.

[0279] Furthermore, in the multiple line segment division mode, as shown in the example in Figure 25(c), if there is no reference pixel region that spans across the left or upper block boundary of the block to be decoded for small region A or small region B, the synthesis unit 205 may derive an intra prediction mode by DIMD using all reference pixels, as shown in Figure 25(c).

[0280] [Method 2 for deriving intra-prediction modes based on adjacent reference pixels] The following describes, with reference to Figures 24 to 26, a method for deriving an intra-prediction mode based on adjacent reference pixels for normal intra-prediction according to Non-Patent Document 2, and a method 2 for deriving an intra-prediction mode based on adjacent reference pixels for geometric partitioning mode according to this embodiment, which applies the aforementioned derivation method.

[0281] Figure 26 shows a method for deriving an intra-prediction mode based on adjacent reference pixels for normal intra-prediction according to Non-Patent Document 2, and an example of a method 2 for deriving an intra-prediction mode based on adjacent reference pixels for geometric partitioning mode according to this embodiment, which applies the aforementioned derivation method. Hereafter, these derivation methods will be collectively referred to as "TIMD (Template-based Intra Mode Derivation)".

[0282] Non-Patent Literature 2 describes how, in TIMD, as shown in Figure 26, the SATD (Sum of Absolute Transformed Difference) is calculated for adjacent reference pixels (hereinafter referred to as templates) of predetermined lines adjacent to the block to be decoded, and for template-oriented intra-prediction pixels (hereinafter referred to as template-oriented intra-prediction pixels) generated using adjacent reference pixels for such templates and predetermined intra-prediction modes. Among the predetermined intra-prediction modes, the intra-prediction mode with the minimum SATD and the next minimum is derived as the intra-prediction mode for TIMD, and intra-prediction pixels are generated.

[0283] Here, the intra-prediction mode used in the calculation of SATD in TIMD described above is an intra-prediction mode that is normally included in the intra-prediction mode candidate list for intra-prediction.

[0284] In TIMD as described in Non-Patent Document 2, if the list of candidate intra-prediction modes for normal intra-prediction does not include vertical prediction mode, horizontal prediction mode, and DC prediction mode, the SATD is calculated with these modes included to derive the intra-prediction mode.

[0285] In this embodiment, by applying the TIMD disclosed in Non-Patent Document 2, the synthesis unit 205 may derive an intra prediction mode. That is, the synthesis unit 205 does not perform the synthesis / generation process of intra prediction pixels using the derived multiple intra prediction modes.

[0286] According to such a configuration, for the intra prediction regions of the multiple line segment division modes (in the case of the Intra / Intra - multiple line segment division mode, two intra prediction regions), an intra prediction pixel can be generated with one intra prediction mode. Therefore, while avoiding an increase in the circuit scale required for generating intra prediction pixels in the multiple line segment division mode in a hardware - implemented image decoding device, by analyzing the histogram of adjacent reference pixels of the decoding target block, an intra prediction reflecting textures such as edges suitable for the division shape of the multiple line segment division mode can be applied, so that the intra prediction performance is improved, and as a result, an improvement in the encoding performance can be expected.

[0287] In this embodiment, similar to Non - Patent Document 3, the decoding unit 201 may be configured to determine whether to derive an intra prediction mode by decoding or estimating a flag for determining whether TIMD can be applied.

[0288] Also, the synthesis unit 205 according to this embodiment may be configured to register the intra prediction mode derived by TIMD when the same intra prediction mode is not already included in the intra prediction mode candidate list for the multiple line segment division mode, and not to register the intra prediction mode derived by TIMD when the same intra prediction mode is already included in the intra prediction mode candidate list for the multiple line segment division mode.

[0289] According to such a configuration, it is possible to avoid the same intra prediction mode being registered repeatedly in the intra prediction mode candidate list.

[0290] Hereinafter, when registering a new intra-prediction mode to the intra-prediction mode candidate list, the process of comparing its consistency with existing intra-prediction modes and pruning it if they match will be referred to as the "intra-prediction mode candidate pruning process."

[0291] Furthermore, the synthesis unit 205 according to this embodiment may limit the number of intra-prediction modes to be registered in the intra-prediction mode candidate list from among the intra-prediction modes derived by TIMD to one. In this case, the synthesis unit 205 derives the intra-prediction mode with the minimum SATD cost (Angular prediction) from the SATD calculation.

[0292] However, in this embodiment, unlike Non-Patent Document 2, the DC prediction mode may be excluded from the calculation of SATD in the TIMD process.

[0293] This is because DC prediction, which generates intra-predicted pixels using all adjacent reference pixels adjacent to the block to be decoded, may generate intra-predicted pixels without properly reflecting textures such as edges corresponding to the division shape of the multi-line segment division mode. Therefore, by excluding DC prediction from the SATD calculation in TIMD processing, it is possible to avoid the derivation of the DC prediction mode in TIMD.

[0294] Furthermore, in the above intra-prediction mode candidate pruning process, if the Angular prediction mode (hereinafter referred to as the 1st Angular prediction mode), which has the lowest SATD cost, is pruned, the existing intra-prediction modes may be compared sequentially, starting with those with the lowest SATD costs, and those that do not match may be registered. Alternatively, if the 1st Angular prediction mode is pruned, the intra-prediction mode derivation process using TIMD may be terminated.

[0295] As an example of a modification, the number of intra prediction modes to be registered in the intra prediction mode candidate list from among the intra prediction modes derived by TIMD may be limited to two. In this case, the synthesis unit 205 derives the 1st Angular prediction mode and the 2nd Angular prediction mode, which is the next best option after the minimum SATD cost, from among the SATD costs.

[0296] If the 1st Angular prediction mode or 2nd Angular prediction mode is pruned, as in the case described above, you may compare it with the existing intra prediction mode, starting with the one with the next lowest SATD cost, and register the one that does not match, or you may terminate the intra prediction mode derivation process using TIMD.

[0297] Furthermore, the synthesis unit 205 according to this embodiment may restrict the adjacent reference pixels used in calculating the TIMD histogram described above to a predetermined region based on the division shape of the multiple line segment division mode (i.e., the angle of the division lines of the multiple line segment division mode).

[0298] In this embodiment, by applying a table of adjacent reference pixel regions defined based on the division lines of the multi-segmentation mode disclosed in Non-Patent Document 3 shown in Figure 24 to the calculation of SATD in TIMD processing, it is possible to derive the Angular prediction using adjacent reference pixels that exist only in the direction of the division lines of the multi-segmentation mode, while avoiding the use of all adjacent reference pixels adjacent to the decoded block in the calculation of SATD in TIMD processing. This reduces the processing load of deriving the intra-prediction mode by TIMD for inter-prediction of the multi-segmentation mode.

[0299] Furthermore, the synthesis unit 205 according to this embodiment may be configured not to process the steps after the calculation of SATD for an intra-prediction mode if the same intra-prediction mode used for calculating SATD in TIMD processing is already registered in the intra-prediction mode candidate list.

[0300] With this configuration, it is possible to avoid the duplicate registration of the same intra-prediction mode after undergoing TIMD processing for intra-prediction modes that have already been registered, thereby reducing the load of TIMD processing on inter-prediction for multiple line segment division modes.

[0301] Alternatively, the synthesis unit 205 may derive an intra-prediction mode by TIMD using only the reference pixel regions that span across the left or upper block boundary of the decoded block for small regions A or B, which are divided in multiple line segmentation modes, rather than the adjacent reference pixel table shown in Figure 24 above, as shown in Figure 25.

[0302] Furthermore, in the multiple line segment division mode, the synthesis unit 205 may, as shown in Figure 25(c), derive an intra prediction mode by TIMD using all reference pixels if there is no reference pixel region that spans across the left or upper block boundary of the block to be decoded for small region A or small region B.

[0303] [Method for deriving intra-predictive modes based on adjacent reference blocks] The following describes, with reference to Figures 24 and 27, a method for deriving an intra-prediction mode based on adjacent reference blocks for normal intra-prediction, as described in Non-Patent Document 1 and Non-Patent Document 3, and a method for deriving an intra-prediction mode based on adjacent reference blocks for geometric partitioning mode according to this embodiment, which applies the same derivation method.

[0304] Figure 27 shows an example of a method for deriving an intra-prediction mode based on adjacent reference blocks for normal intra-prediction, as described in Non-Patent Documents 1 and 3, and an example of a method for deriving an intra-prediction mode based on adjacent reference blocks for geometric partitioning modes according to this embodiment, which applies the aforementioned derivation method. Hereafter, these derivation methods will be collectively referred to as "BIMD (Block-based Intra Mode Derivation)".

[0305] Non-patent document 2 describes how, in BIMD, as shown in Figure 27, the intra-prediction mode of an adjacent reference block at a predetermined position adjacent to the block to be decoded is derived as the intra-prediction mode of BIMD to generate intra-prediction pixels. The intra-prediction mode of the adjacent reference block derived here is the intra-prediction mode of that adjacent reference block if the adjacent reference block is an intra-prediction block. However, if the adjacent reference block is an inter-prediction block, or an inter-prediction block that is also a block to which intra-prediction is applied and which is a multi-line segment division mode applied, the intra-prediction mode stored in 4x4 sub-block pixel units, as described later, is referenced.

[0306] In Non-Patent Documents 1 and 3, the adjacent reference blocks referenced in the above-mentioned BIMD are set to the left (A0), lower left (A1), top (B0), upper right (B1), and upper left (B2) of the block to be decoded, as shown in Figure 27.

[0307] In this embodiment, the synthesis unit 205 may derive intra-prediction modes by applying BIMD disclosed in Non-Patent Documents 1 and 3. That is, the synthesis unit 205 does not perform synthesis / generation processing of intra-prediction pixels using the derived intra-prediction modes.

[0308] With this configuration, for the intra-prediction region of the multiple line segment division mode (two intra-prediction regions in the case of Intra / Intra-multiple line segment division mode), an intra-prediction mode can be selected from a list of intra-prediction mode candidates that may include the intra-prediction mode of the adjacent reference block of the block to be decoded, and intra-prediction pixels can be generated. As a result, intra-prediction that reflects textures such as edges suitable for the division shape of the multiple line segment division mode can be applied, improving intra-prediction performance and consequently improving coding performance.

[0309] Furthermore, the synthesis unit 205 according to this embodiment may be configured to register an intra-prediction mode derived by BIMD if the same intra-prediction mode is not already included in the intra-prediction mode candidate list for the multiple line segment division mode, and not to register an intra-prediction mode derived by BIMD if the same intra-prediction mode is already included in the intra-prediction mode candidate list for the multiple line segment division mode.

[0310] This configuration prevents duplicate registration of the same intra-prediction mode within the intra-prediction mode candidate list.

[0311] Here, when registering a new intra-prediction mode to the intra-prediction mode candidate list, the process of comparing its consistency with existing intra-prediction modes and pruning it if they match will be referred to as the "intra-prediction mode candidate pruning process" from now on.

[0312] Furthermore, unlike Non-Patent Documents 1 and 3, the synthesis unit 205 according to this embodiment may be configured not to register the intra-prediction mode candidate list if the intra-prediction mode derived by BIMD is a DC prediction mode.

[0313] This is because DC prediction, which generates intra-predicted pixels using all adjacent reference pixels adjacent to the block to be decoded, may generate intra-predicted pixels without properly reflecting textures such as edges corresponding to the division shape of the multi-line segment division mode. Therefore, by excluding DC prediction from BIMD's intra-prediction mode, it is possible to avoid the DC prediction mode being used to generate intra-predicted pixels.

[0314] Furthermore, in the synthesis unit 205 according to this embodiment, the order of the up to five adjacent reference blocks shown in Figure 27, which are referenced in the derivation of the intra prediction mode by BIMD, may be configured in the same way as in Non-Patent Documents 1 and 3. Since the reference order is disclosed in Non-Patent Documents 1 and 3, a detailed explanation is omitted in this embodiment.

[0315] As an example of the modification, in this embodiment, by applying the table of adjacent reference pixel regions defined (restricted) based on the division lines of the multi-line division mode disclosed in Non-Patent Document 3 shown in Figure 24 to the adjacent reference block reference in BIMD, it is possible to derive the intra-prediction mode in BIMD by referring only to the intra-prediction mode (Angular prediction) of adjacent reference blocks that exist only in the direction of the division lines of the multi-line division mode, while avoiding the need to refer to the intra-prediction mode of all adjacent reference blocks adjacent to the decoded block. This reduces the load on the BIMD intra-prediction mode derivation process for inter-prediction of the multi-line division mode. [Method for deriving intra-predictive modes using control information] Regarding the different types of intra-prediction modes described above, the synthesis unit 205 may uniquely apply one of the intra-prediction modes described above, or it may select an intra-prediction mode to actually apply from a list of different intra-prediction mode candidates included in the intra-prediction mode candidate list according to the control information.

[0316] The following are examples of configurations for several different intra-prediction modes included in the intra-prediction mode candidate list. Here, we refer to them as the angle prediction mode parallel to the dividing line (Parallel) and the angle prediction mode perpendicular to the dividing line (Perpendicular). Configuration Example 1: DIMD ⇒ Parallel (or Perpendicular) Configuration Example 2: TIMD ⇒ Parallel (or Perpendicular) Configuration Example 3: BIMD ⇒ Parallel (or Perpendicular) Configuration Example 4: DIMD ⇒ TIMD ⇒ Parallel (or Perpendicular) Configuration Example 5: DIMD ⇒ BIMD ⇒ Parallel (or Perpendicular) Configuration Example 6: TIMD ⇒ BIMD ⇒ Parallel (or Perpendicular) Configuration Example 7: DIMD ⇒ TIMD ⇒ BIMD ⇒ Parallel (or Perpendicular) First, configuration examples 1-3 are methods that combine Parallel (or Perpendicular) with DIMD, TIMD, and BIMD, respectively.

[0317] The intra-prediction mode derived using Parallel (or Perpendicular) can derive an intra-prediction mode that reflects textures such as edges based on dividing lines more simply and directly than the intra-prediction modes derived using DIMD, TIMD, and BIMD. However, it is less likely to derive a prediction mode with higher accuracy than DIMD, TIMD, and BIMD, which are based on analysis of adjacent reference pixels, so it is placed in the list after these intra-prediction mode candidates.

[0318] Next, configuration examples 4 and 5 are configuration examples in which DIMD is placed before TIMD or BIMD.

[0319] The reason for placing DIMD before TIMD is that the derivation process of the intra-prediction mode using DIMD is less computationally intensive than the derivation process of the intra-prediction mode using TIMD, which involves relatively heavy computational processing such as the calculation of SATD.

[0320] On the other hand, the reason for placing DIMD before BIMD is that although the DIMD process for deriving the intra-prediction mode, which includes histogram calculation, is not as lightweight as the BIMD process, the intra-prediction mode derived with DIMD may be able to derive an intra-prediction mode that better reflects textures such as edges based on GPM division lines, due to the histogram calculation, compared to the intra-prediction mode derived with BIMD. Therefore, it is thought that this will likely result in a greater improvement in intra-prediction performance.

[0321] In configuration example 6, TIMD is placed before BIMD. The reason for this placement is the same as the reason for placing DIMD before BIMD as described above.

[0322] Configuration Example 7 is a configuration example that combines GIMD, DIMD, TIMD, and BIMD. For the reasons mentioned above, it is expected that deriving the intra-prediction mode in this order will allow for the more efficient deriving of an intra-prediction mode with higher prediction performance.

[0323] (Starting restrictions for each intra-predictive mode derivation method) In this embodiment, the synthesis unit 205 starts each derivation process for intra-prediction modes for the geometric block division mode described above if the number of intra-prediction mode candidates included in the intra-prediction mode candidate list does not reach the maximum size of the intra-prediction mode candidate list, but does not start each derivation process if the number of such candidates reaches the maximum size of the intra-prediction mode candidate list.

[0324] With this configuration, it is possible to avoid executing the derivation process for unnecessary intra-prediction modes, and a reduction in the overall processing load of the synthesis unit 205 can be expected.

[0325] (How to register the intra-prediction mode candidate list after the intra-prediction mode derivation is complete) In this embodiment, the synthesis unit 205 may be configured not to register a predetermined intra-prediction mode if, at the completion of the intra-prediction mode derivation process for the geometric block division mode described above, the number of intra-prediction mode candidates included in the intra-prediction mode candidate list has not reached the maximum size of the intra-prediction mode candidate list, and the same prediction mode is already included in the intra-prediction mode candidate list.

[0326] For example, in Configuration Examples 1 through 7, if the intra-prediction mode derived by DIMD, TIMD, and BIMD is the same as the subsequent Parallel (or Perpendicular) mode, the list size has not reached its maximum value.

[0327] In such a case, the synthesis unit 205 may register an unregistered Perpendicular (or Parallel) mode. Alternatively, the synthesis unit 205 may register a Planar mode. Alternatively, the synthesis unit 205 may register a DC mode. Alternatively, the synthesis unit 205 may register an intra-prediction mode that is near the intra-prediction mode initially registered in the intra-prediction mode candidate list.

[0328] (Method for saving prediction information for multiple line division / multiple line segment division modes) The intra-prediction unit 204 stores the intra-prediction mode applied to sub-region A or sub-region B, which is divided in multiple line segment division mode, in units of sub-blocks of a predetermined size obtained by dividing the block to be decoded.

[0329] Such predetermined size may be, for example, the minimum size of an encoding block, prediction block, or transformation block. Alternatively, such predetermined size may be a fixed size such as 2x2 pixels or 4x4 pixels.

[0330] In this way, by saving the intra-prediction modes applied to each of the sub-regions A or B on a sub-block basis rather than on a decoded block basis, the intra-prediction modes in the blending region can be accurately preserved.

[0331] The intra-prediction unit 204 may save both the intra-prediction modes for sub-region A or sub-region B for a sub-block within the blending region. Alternatively, the intra-prediction unit 204 may save only the corresponding intra-prediction mode depending on whether the sub-block belongs to sub-region A or sub-region B, separated by a dividing line from the center coordinates of the sub-block.

[0332] The motion compensation unit 208 stores motion information (reference image list, reference image index, motion vector) applied to small region A or small region B, which is divided in multiple line segment division mode, in units of subblocks of a predetermined size obtained by dividing the block to be decoded.

[0333] Such predetermined size may be, for example, the minimum size of an encoding block, prediction block, or transformation block. Alternatively, such predetermined size may be a fixed size such as 2x2 pixels or 4x4 pixels.

[0334] In this way, by saving the motion information applied to each of the sub-regions A or B in units of sub-blocks rather than units of the blocks to be decoded, motion information in the blending region can be accurately preserved.

[0335] The motion compensation unit 208 may save motion information for both sub-region A and sub-region B for a sub-block within the blending region. Alternatively, the motion compensation unit 208 may save only the motion information corresponding to which sub-block belongs, separated by a dividing line from the center coordinates of the sub-block.

[0336] Alternatively, if the reference image lists for sub-region A and sub-region B are different (i.e., one refers to a different frame in the future direction relative to the frame in question, and the other refers to a different frame in the past direction relative to the frame in question), the motion compensation unit 208 may generate and store a new motion vector by weighting the respective motion vectors according to the distance between the frame and the reference frame, as disclosed in Non-Patent Literature 1.

[0337] Alternatively, the motion compensation unit 208 may save only the motion vector of sub-region B if the reference image lists for sub-region A and sub-region B are the same (i.e., one refers to a different frame in the future (or past) direction relative to the frame in question, and the other also refers to a different frame in the future (or past) direction relative to the frame in question). Alternatively, in the same case, the motion compensation unit 208 may save only sub-region A.

[0338] (Multiple line division / multiple line segment division method mode subscript code order reordering method) The synthesis unit 205 may rearrange the multiple line segment division modes, which are associated with the decoded value of the control information cu_div_idx for identifying the multiple line segment division modes, by template matching disclosed in Non-Patent Document 3.

[0339] Specifically, if multiple line segment division modes are configured by interpretation, the compositing unit 205 compares the errors (for example, SAD: Sum of Absolute Difference) of the adjacent pixels (templates) of the decoding target block and the reference block for all multiple line segment division modes. When calculating the SAD, the compositing unit 205 weights and averages the division lines by extending them to the adjacent pixels.

[0340] If multiple line segment division modes are configured by intra-prediction, the synthesis unit 205 applies the intra-prediction mode to reference pixels one or more lines ahead of the adjacent reference pixels of the block to be decoded to generate adjacent pixels, and compares the SAD of the generated adjacent pixels with the adjacent pixels of the block to be decoded.

[0341] The synthesis unit 205, by comparing the SADs, rearranges the multiple line division modes associated with the decoded values ​​of cu_div_idx in ascending order of SAD, thereby enabling the use of a multiple line division mode with higher prediction accuracy at a smaller decoded value (code length), resulting in an improvement in coding efficiency.

[0342] Furthermore, the combining unit 205 may not only rearrange the multiple line segment division modes associated with the decoded values ​​of cu_div_idx in ascending order of SAD, but may also exclude them from the list of selectable multiple line segment division modes once a predetermined number is reached in ascending order of SAD.

[0343] For example, the predetermined number could be half the number of candidates for the selectable multiple line segment division modes, or half the total number of candidates obtained by adding the multiple line segment division modes to the geometric division modes.

[0344] This reduces the number of line segment division modes that can be associated with cu_div_idx, and consequently, the number of candidate division modes. This is expected to further shorten the code length of cu_div_idx, resulting in improved coding efficiency.

[0345] According to the image decoding device 200 of this embodiment, decoding is performed by dividing each unit block into small regions composed of multiple line segments, thereby improving encoding efficiency.

[0346] (Example of change) In the embodiments described above, an example is shown in which a region is divided into two sub-regions by multiple line segments. However, the present invention is not limited to this case and can also be applied to cases in which a region is divided into three or more sub-regions by multiple line segments.

[0347] Furthermore, while the above-described embodiment illustrates a case where all subregions are divided to include the edges of the block to be decoded, the present invention is not limited to such a case and can also be applied to a case where at least one subregion is divided so as not to include the edges of the block to be decoded (i.e., a case where at least one subregion is divided so as not to touch the outer perimeter of the block to be decoded).

[0348] The image decoding device 200 described above may be implemented as a program that causes a computer to execute each function (each process). [Industrial applicability]

[0349] Furthermore, according to this embodiment, for example, it is possible to achieve an overall improvement in service quality in video communication, thereby contributing to Goal 9 of the United Nations-led Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote sustainable industrialization and foster innovation." [Explanation of symbols]

[0350] 200…Image decoding device 201...Decoding section 202...Inverse quantization section 203...Inverse Transformation Section 204...Intra Prediction Unit 205... Sorting Department 206... Adder 207...Storage section 208...Motion compensation unit 209...Synthesis part 210... Code input section 220...Image output unit

Claims

1. An image decoding device, A decoding unit that decodes control information including codewords that specify the shape and boundary width of the partitioned boundary within the block to be decoded, and a quantization value, An inverse quantization unit that dequantizes the quantized value and decodes it to obtain a transformation coefficient, An inverse transform unit that converts the aforementioned conversion coefficients inversely to obtain the decoded predicted residual, An intra-prediction unit that generates a first predicted pixel based on the decoded pixel and the control information, A storage unit for storing the decoded pixels, A motion compensation unit that generates a second predicted pixel based on the accumulated decoded pixels and the control information, A synthesis unit that generates a third predicted pixel based on a combination including at least one of the first predicted pixel and the second predicted pixel and the control information, A selection unit that changes the code order of subscripts that specify the shape and width of the division boundary within the block to be decoded, according to the similarity between neighboring pixels of the accumulated decoded block or reference block, An image decoding device characterized by comprising an adder that adds any of the first predicted pixel, the second predicted pixel, and the third predicted pixel with the predicted residual to obtain the decoded pixel.

2. An image decoding device, A decoding unit that decodes control information including codewords that specify the shape and boundary width of the partitioned boundary within the block to be decoded, and a quantization value, An inverse quantization unit that dequantizes the quantized value and decodes it to obtain a transformation coefficient, An inverse transform unit that converts the aforementioned conversion coefficients inversely to obtain the decoded predicted residual, An intra-prediction unit that generates a first predicted pixel based on the decoded pixel and the control information, A storage unit for storing the decoded pixels, A motion compensation unit that generates a second predicted pixel based on the accumulated decoded pixels and the control information, A synthesis unit that generates a third predicted pixel based on a combination including at least one of the first predicted pixel and the second predicted pixel and the control information, A selection unit that changes the code order of the control information associated with the division boundary width within the division boundary width within the block to be decoded, according to the history of the division boundary width within the block to be decoded within the same image, An image decoding device characterized by comprising an adder that adds any of the first predicted pixel, the second predicted pixel, and the third predicted pixel with the predicted residual to obtain the decoded pixel.

3. The image decoding apparatus according to claim 1, characterized in that the sorting unit applies a weighted average of different boundary widths to neighboring pixels of a reference block to generate neighboring pixels with different boundary widths.

4. The image decoding apparatus according to claim 1, characterized in that the selection unit compares the similarity between neighboring pixels of a reference block to which a different boundary width is applied and neighboring pixels of the block to be decoded, and uses the top N pixels with high similarity as candidates for the boundary width.

5. The image decoding apparatus according to claim 4, characterized in that the selection unit selects the boundary width from the candidate boundary widths by decoding a codeword that uniquely determines the boundary width.

6. The image decoding apparatus according to claim 4, characterized in that the selection unit assigns a shorter code from among candidates for boundary widths with high similarity between neighboring pixels of the reference block and neighboring pixels of the block to be decoded.

7. The image decoding apparatus according to claim 5, characterized in that the sorting unit does not decode the codeword when N=1.

8. The image decoding apparatus according to claim 1, characterized in that the sorting unit sets an arbitrary width as the boundary width.

9. The image decoding device according to claim 1, characterized in that the sorting unit sets a different boundary width for each reference destination of a small region divided by the boundary defined by the boundary information.

10. The image decoding apparatus according to claim 1, characterized in that the sorting unit includes the upper right and lower left corners of the reference block and the reference block in the neighboring pixels of the reference block and the reference block, respectively.

11. The image decoding apparatus according to claim 1, characterized in that the sorting unit uses multiple lines as neighboring pixels, rather than being limited to one line.

12. The image decoding apparatus according to claim 1, characterized in that, if there is no comparison target for similarity that arises from applying different boundary widths to neighboring pixels of the reference block and the decoding target block, the selection unit decodes the boundary width with a predetermined fixed width.

13. The image decoding apparatus according to claim 1, characterized in that the selection unit selects the boundary width by decoding an explicit codeword if there is no comparison target for similarity that arises from applying different boundary widths to neighboring pixels of the reference block and the block to be decoded.

14. The image decoding device according to claim 2, characterized in that the selection unit updates the history of the division boundary width each time it detects a block to which the division mode is applied, and modifies the assignment to a shorter code in the control information that identifies the division boundary width for division boundary widths with a larger number of applied blocks in the history.

15. The image decoding apparatus according to claim 2, characterized in that the sorting unit initializes the history of the division boundary width in the decoding tree block at the left edge of the image.

16. An image decoding method, A process of decoding control information including codewords that specify the shape and boundary width of the partitioned boundary within the block to be decoded, and a quantization value, The process of inverse quantization of the quantized value to obtain a decoded transformation coefficient, The process involves inversely transforming the aforementioned transformation coefficient to obtain the decoded predicted residual, A step of generating a first predicted pixel based on the decoded pixel and the control information, The process of accumulating the decoded pixels, A step of generating a second predicted pixel based on the accumulated decoded pixels and the control information, A step of generating a third prediction pixel based on a combination including at least one of the first prediction pixel and the second prediction pixel and the control information, A step of changing the code order of subscripts that specify the shape and width of the division boundary within the block to be decoded, respectively, according to the similarity between neighboring pixels of the accumulated decoded block or reference block, An image decoding method characterized by comprising the step of adding the prediction residual to any one of the first prediction pixel, the second prediction pixel, and the third prediction pixel to obtain the decoded pixel.

17. A program that makes a computer function as an image decoding device, The aforementioned image decoding device is A decoding unit that decodes control information including codewords that specify the shape and boundary width of the partitioned boundary within the block to be decoded, and a quantization value, An inverse quantization unit that dequantizes the quantized value and decodes it to obtain a transformation coefficient, An inverse transform unit that converts the aforementioned conversion coefficients inversely to obtain the decoded predicted residual, An intra-prediction unit that generates a first predicted pixel based on the decoded pixel and the control information, A storage unit for storing the decoded pixels, A motion compensation unit that generates a second predicted pixel based on the accumulated decoded pixels and the control information, A synthesis unit that generates a third predicted pixel based on a combination including at least one of the first predicted pixel and the second predicted pixel and the control information, A selection unit that changes the code order of subscripts that specify the shape and width of the division boundary within the block to be decoded, according to the similarity between neighboring pixels of the accumulated decoded block or reference block, A program characterized by comprising an adder that adds the prediction residual to any one of the first prediction pixel, the second prediction pixel, and the third prediction pixel to obtain the decoded pixel.

Images