Image decoding device, image decoding method, program, and image encoding device
The image decoding device optimizes coding processing by selectively applying non-separable transforms for intra and inter prediction based on sequence-specific control flags and cost calculations, addressing inefficiencies in existing technologies.
Patent Information
- Application Number
- JP2024062457
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-04-09
- Publication Date
- 2025-10-22
AI Technical Summary
Non-separable transforms for intra and inter prediction increase the coding processing amount, particularly in inter pictures or slices where both types of prediction are applicable, leading to inefficiencies.
An image decoding device and method that determines, on a sequence-by-sequence basis, whether to apply non-separable transforms for intra and inter prediction, using control flags and coding cost calculations to optimize processing.
Efficiently reduces the increase in coding processing due to non-separable transforms, maintaining coding performance by selectively applying these transforms only when necessary.
Smart Images
Figure 2025159747000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to an image decoding device, an image decoding method, a program, and an image encoding device. [Background technology]
[0002] Non-Patent Documents 1 and 2 disclose a transform that converts residual pixel values of the current decoded block into the frequency domain (generating transform coefficients).
[0003] Furthermore, Non-Patent Documents 1 to 3 disclose non-separable transforms for intra prediction or inter prediction. [Prior art documents] [Non-patent literature]
[0004] [Non-Patent Document 1] ITU-T H.266 VVC [Non-patent document 2] M. Coban, et al., “Algorithm description of Enhanced Compression Model 10 (ECM 10)”, JVET-AE2025, 2023 [Non-patent document 3] F. Wang, et al., “Non-EE2: On LFNST / NSPT for inter coding”, JVET-AF0082, October 2023 Summary of the Invention [Problem to be solved by the invention]
[0005] However, the non-separable transforms disclosed in Non-Patent Documents 1 to 3 increase the amount of coding processing. In particular, for inter pictures or inter slices to which both intra prediction and inter prediction are applicable, both non-separable transforms for intra prediction and non-separable transforms for inter prediction can be applied, which causes a problem that the increase in coding processing amount due to the application of non-separable transform to intra pictures or intra slices to which only intra prediction is applicable is relatively large. Therefore, the present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an image decoding device, an image decoding method, a program, and an image coding device that can efficiently suppress (reduce) the increase in coding processing amount due to non-separable transforms for intra prediction and non-separable transforms for inter prediction under coding conditions including inter pictures or inter slices to which both intra prediction and inter prediction are applicable. [Means for solving the problem]
[0006] A first feature of the present invention is summarized as an image decoding device comprising: an intra prediction unit that applies intra prediction to a current block; an inter prediction unit that applies inter prediction to the current block; an inverse transform / inverse quantization unit that applies inverse transform and inverse quantization to the current block; and a decoding unit that decodes coded data including control information, wherein the decoding unit determines whether or not each of the non-separable transform for intra prediction and the non-separable transform for inter prediction is applicable to the current block, based on the control information that specifies whether or not each of the non-separable transform for intra prediction and the non-separable transform for inter prediction is applicable on a sequence-by-sequence basis.
[0007] A second feature of the present invention is an image decoding method comprising step A of applying intra prediction to a current block, step B of applying inter prediction to the current block, step C of applying inverse transform and inverse quantization to the current block, and step D of decoding encoded data including control information, wherein in step D, it is determined whether or not a non-separable transform for intra prediction and a non-separable transform for inter prediction can be applied to the current block, based on control information that specifies whether or not each of the non-separable transform for intra prediction and the non-separable transform for inter prediction can be applied on a sequence-by-sequence basis.
[0008] A third feature of the present invention is summarized as a program causing a computer to function as an image decoding device, the image decoding device including an intra prediction unit that applies intra prediction to a current block, an inter prediction unit that applies inter prediction to the current block, an inverse transform / inverse quantization unit that applies inverse transform and inverse quantization to the current block, and a decoding unit that decodes encoded data including control information, wherein the decoding unit determines whether or not each of the non-separable transform for intra prediction and the non-separable transform for inter prediction is applicable to the current block, based on the control information that specifies whether or not each of the non-separable transform for intra prediction and the non-separable transform for inter prediction is applicable on a sequence-by-sequence basis.
[0009] A fourth aspect of the present invention is an image coding device including an intra prediction unit that applies intra prediction to a current block, an inter prediction unit that applies inter prediction to the current block, a transform and inverse quantization unit that applies transform and quantization to the current block, and a coding unit that encodes coefficient level values and control information to generate coded data, wherein the coding unit compares a coding cost of the current block to which inter prediction has been applied with a coding cost of the current block to which intra prediction has been applied but to which non-separable transform for intra prediction has not been applied, to determine whether or not non-separable transform for intra prediction can be applied to the current block. [Effects of the Invention]
[0010] According to the present invention, it is possible to provide an image decoding device, an image decoding method, a program, and an image encoding device that can efficiently suppress (reduce) the increase in the amount of encoding processing due to non-separable transforms for intra prediction and non-separable transforms for inter prediction under encoding conditions having inter pictures or inter slices to which both intra prediction and inter prediction can be applied. [Brief explanation of the drawings]
[0011] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an image processing system 1 according to an embodiment. [Figure 2] FIG. 2 is a diagram showing an example of functional blocks of the image encoding device 100 according to an embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of functional blocks of an image decoding device 200 according to an embodiment. [Figure 4] FIG. 4 is a flowchart showing calculation of a coding cost by the coding device 100 according to an embodiment and determination of whether or not it is necessary to calculate the coding cost of non-separable transform for intra prediction. [Figure 5] FIG. 5 is a diagram illustrating an example of a decoding operation based on the value of sps_nst_enabled_flag. [Figure 6] FIG. 6 is a diagram illustrating an example of a decoding operation based on the value of sps_intra_nst_enabled_flag. DETAILED DESCRIPTION OF THE INVENTION
[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, etc., and various variations, including combinations with other existing components, are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.
[0013] First Embodiment An image processing system 10 according to a first embodiment of the present invention will be described below with reference to Figures 1 to 4. Figure 1 is a diagram showing the image processing system 10 according to this embodiment.
[0014] As shown in FIG. 1, an image processing system 10 according to this embodiment includes an image encoding device 100 and an image decoding device 200.
[0015] The image coding device 100 is configured to generate coded data by coding an input image signal (picture). The image decoding device 200 is configured to generate an output image signal by decoding the coded data.
[0016] The encoded data may be transmitted from the image encoding device 100 to the image decoding device 200 via a transmission path. The encoded data may be stored in a storage medium and then provided from the image encoding device 100 to the image decoding device 200.
[0017] (Block type and block division tree structure) The image signals (frames) handled by the image encoding device 100 and the image decoding device 200 are made up of RGB signals or Y, Cb, and Cr signals (luminance signal and color difference signal components).
[0018] The minimum division unit of such an image signal (frame) and the basic unit of encoding are called a coding tree unit (CTU) and a coding unit (CU).
[0019] Such a CTU is divided into variable-sized CUs by recursive quad-tree block division, binary-tree block division, or ternary-tree block division.
[0020] The processing content of these block division tree structures can be the well-known processing described in Non-Patent Document 1, so a detailed description will be omitted.
[0021] Furthermore, Non-Patent Document 1 discloses processing details in which a block having a luminance signal component (hereinafter, luminance block) and a block having a chrominance signal component (hereinafter, chrominance block) of a frame to be coded (hereinafter, target frame), a CTU to be coded (hereinafter, target CTU), or a CU to be coded (hereinafter, target block or target CU) are made to have the same block partitioning tree structure (hereinafter, single tree) or different block partitioning tree structures (hereinafter, dual tree).
[0022] (Image encoding device 100) The image encoding device 100 according to this embodiment will be described below with reference to Fig. 2. Fig. 2 is a diagram showing an example of functional blocks of the image encoding device 100 according to this embodiment.
[0023] As shown in FIG. 2, the image encoding device 100 includes an inter prediction unit 111, an intra prediction unit 112, a subtractor 121, an adder 122, a transform / quantization unit 131, an inverse transform / inverse quantization unit 132, an encoding unit 140, an in-loop filter processing unit 150, and a frame buffer 160.
[0024] The inter prediction unit 111 is configured to generate a prediction signal by inter prediction (inter-frame prediction).
[0025] Specifically, the inter prediction unit 111 is configured to identify a reference block included in the reference frame by comparing the target frame with a reference frame stored in the frame buffer 160, and to determine a motion vector (mv) for the identified reference block.
[0026] The inter prediction unit 111 is configured to generate, for each current block, a prediction signal included in the current block based on the reference block and the motion vector. The inter prediction unit 111 is configured to output the prediction signal to the subtractor 121 and the adder 122. Here, the reference frame is a frame different from the current frame.
[0027] The intra prediction unit 112 is configured to generate a prediction signal by intra prediction (prediction within a frame).
[0028] Specifically, the intra prediction unit 112 is configured to identify a reference block included in a target frame, and generate a prediction signal for each target block based on the identified reference block. The intra prediction unit 112 is also configured to output the prediction signal to the subtractor 121 and the adder 122.
[0029] Here, the reference block is a block that is referenced for the current block, for example, a block that is adjacent to the current block.
[0030] Furthermore, the processing unit for the inter prediction 111 and intra prediction 112 may be the same size as the current block, or may be a prediction block (PU: Prediction Unit) of a size obtained by subdividing the current block.
[0031] The subtractor 121 is configured to subtract the prediction signal from the input image signal and output the prediction residual signal to the transform / quantization unit 131. Here, the subtractor 121 is configured to generate a prediction residual signal that is the difference between the prediction signal generated by intra prediction or inter prediction and the input image signal.
[0032] The adder 122 is configured to add the prediction signal to the prediction residual signal output from the inverse transform / inverse quantization unit 132 to generate a pre-filter decoded signal, and to output the pre-filter decoded signal to the intra prediction unit 112 and the in-loop filter processing unit 150.
[0033] Here, the unfiltered decoded signal forms a reference block used by the intra prediction unit 112.
[0034] The transform / quantization unit 131 is configured to perform a transform process on the prediction residual signal and to obtain coefficient level values. Furthermore, the transform / quantization unit 131 may be configured to quantize the coefficient level values.
[0035] Here, the transform process is a process of transforming a prediction residual signal into a frequency component signal. For such a transform process, a basis pattern (transform matrix) corresponding to a discrete cosine transform (hereinafter referred to as DCT) or a basis pattern (transform matrix) corresponding to a discrete sine transform (hereinafter referred to as DST) may be used.
[0036] Furthermore, it is also possible to use the low frequency non-separable transform or the non-separable primary transform disclosed in Non-Patent Documents 1 to 3. Details will be described later.
[0037] Furthermore, the processing unit of the transform / inverse quantization unit 131 may be the same size as the target block, or may be a transform block (TU) having a size obtained by subdividing the target block.
[0038] The inverse transform and inverse quantization unit 132 is configured to perform inverse transform processing on the coefficient level values output from the transform and quantization unit 131. Here, the inverse transform and inverse quantization unit 132 may be configured to perform inverse quantization on the coefficient level values prior to the inverse transform processing.
[0039] Here, the inverse transform processing and inverse quantization are performed in the reverse order to the transform processing and quantization performed by the transform / quantization unit 131 .
[0040] The encoding unit 140 is configured to encode the coefficient level values output from the transform / quantization unit 131 and output encoded data.
[0041] Here, for example, the coding is entropy coding that assigns codes of different lengths based on the probability of occurrence of coefficient level values.
[0042] The encoding unit 140 is also configured to encode control data used in the decoding process in addition to the coefficient level values.
[0043] In addition, such control information (syntax) includes information necessary for controlling intra prediction, inter prediction, inverse transform / inverse quantization units, etc., and may also include header information such as a sequence parameter set (SPS), a picture parameter set (PPS), a picture header (PH), and a slice header (SH).
[0044] The in-loop filtering unit 150 is configured to perform filtering on the unfiltered decoded signal output from the adder 122 and to output the filtered decoded signal to the frame buffer 160 .
[0045] Here, for example, the filtering process is deblocking filtering process that reduces distortion occurring at the boundary portion of the current block (or PU, TU).
[0046] The frame buffer 160 is configured to store reference frames used by the inter prediction unit 111.
[0047] Here, the filtered decoded signal forms a reference frame used in the inter prediction unit 111.
[0048] (Image decoding device 200) The image decoding device 200 according to this embodiment will be described below with reference to Fig. 3. Fig. 3 is a diagram showing an example of functional blocks of the image decoding device 200 according to this embodiment.
[0049] As shown in FIG. 3, the image decoding device 200 includes a decoding unit 210, an inverse transform / inverse quantization unit 220, an adder 230, an inter prediction unit 241, an intra prediction unit 242, an in-loop filtering unit 250, and a frame buffer 260.
[0050] The decoding unit 210 is configured to decode the coded data generated by the image coding device 100, and to decode the coefficient level values.
[0051] Here, the decoding is, for example, entropy decoding, which is the reverse procedure of the entropy encoding performed by the encoding unit 140.
[0052] The decoding unit 210 may also be configured to obtain the control data by decoding the encoded data.
[0053] In addition, such control information (syntax) includes information necessary for controlling intra prediction, inter prediction, inverse transform / inverse quantization units, etc., and may also include header information such as a sequence parameter set (SPS), a picture parameter set (PPS), a picture header (PH), and a slice header (SH).
[0054] The inverse transform / inverse quantization unit 220 is configured to perform inverse transform processing on the coefficient level values output from the decoding unit 210. Here, the inverse transform / inverse quantization unit 220 may be configured to perform inverse quantization on the coefficient level values prior to the inverse transform processing.
[0055] Here, the inverse transform processing and inverse quantization are performed in the reverse order to the transform processing and quantization performed by the transform / quantization unit 131 .
[0056] The adder 230 is configured to add the prediction signal to the prediction residual signal output from the inverse transform / inverse quantization unit 220 to generate a pre-filtered decoded signal, and output the pre-filtered decoded signal to the intra prediction unit 242 and the in-loop filter processing unit 250.
[0057] Here, the unfiltered decoded signal forms a reference block used by the intra prediction unit 242.
[0058] Like the inter prediction unit 111, the inter prediction unit 241 is configured to generate a prediction signal by inter prediction (inter-frame prediction).
[0059] Specifically, the inter prediction unit 241 is configured to generate a prediction signal based on a motion vector decoded from encoded data and a reference signal included in a reference frame. The inter prediction unit 241 is configured to output the prediction signal to the adder 230.
[0060] Like the intra prediction unit 112, the intra prediction unit 242 is configured to generate a prediction signal by intra prediction (intra-frame prediction).
[0061] Specifically, the intra prediction unit 242 is configured to identify a reference block included in the target frame, and generate a prediction signal for each prediction block based on the identified reference block. The intra prediction unit 242 is configured to output the prediction signal to the adder 230.
[0062] Similar to the in-loop filter processing unit 150, the in-loop filter processing unit 250 is configured to perform filtering on the unfiltered decoded signal output from the adder 230 and to output the filtered decoded signal to the frame buffer 260.
[0063] Here, for example, the filtering process is deblocking filtering process that reduces distortion occurring at the boundary portion of the current block (or PU, TU).
[0064] Similar to the frame buffer 160, the frame buffer 260 is configured to store reference frames used by the inter prediction unit 241.
[0065] Here, the filtered decoded signal forms a reference frame used by the inter prediction unit 241.
[0066] (low frequency non-separable transform) Low Frequency Non-Separable Transform (LFNST) is a technique disclosed in Non-Patent Documents 1-3.
[0067] LFNST is a technique that performs a non-separable transform on correlations that remain after a first transform (hereinafter referred to as a primary transform) process of a current block to which intra prediction has been applied.
[0068] The spatial redundancy of the prediction residual signal depends on the prediction mode. In particular, in intra-prediction, when directional prediction (angular prediction) is used, it is known that a directional pattern of the prediction residual signal (directional prediction residual signal) is observed.
[0069] For example, the transform coefficients generated after the primary transform process of the directional prediction residual signal have a patterned bias in the coefficient distribution in the low frequency region, i.e., spatial redundancy remains between the transform coefficients.
[0070] Therefore, LFNST concentrates the transform coefficients after the primary transform in the low frequency region, reducing the redundancy between the transform coefficients and improving the compression efficiency.
[0071] The transform coefficients to be transformed are arranged in a one-dimensional array by a transform set specified according to the intra-prediction mode number. This one-dimensional array is called a transform set.
[0072] Next, the transform coefficients to be transformed, arranged in a linear edge array, are transformed using a transform base specified according to lfnst_idx, which is control information that specifies the intra prediction mode, the current block size (or transform block size), whether LFNST is applied, and the type of transform base if it is applied, to obtain re-transformed coefficients in a one-dimensional array.
[0073] The retransformed coefficients in the one-dimensional array are rearranged in the two-dimensional array based on the scan order of the diagonal scan. Note that the coefficients outside the area where the retransformed coefficients are rearranged are zeroed.
[0074] Here, the transformation base of LFNST is defined by a transformation set and a transformation kernel, but in Non-Patent Document 2, the number of patterns of the transformation set and transformation kernel that make up the transformation base is expanded.
[0075] Specifically, the transformation set is increased from 4 to 35, the transformation kernels are increased from 2 to 3, and larger kernels are introduced for larger blocks.
[0076] (Non-separable linear transformation) Non-Patent Document 2 discloses a non-separable primary transform (NSPT) that is not included in Non-Patent Document 1.
[0077] In Non-Patent Document 2, NSPT is applied to small blocks. Specifically, it is introduced to replace a combination of separable DCT-II and LFNST transforms for current blocks of 4x4, 4x8 / 8x4, 4x16 / 16x4, 8x8, 8x16 / 16x8, 4x32 / 32x4, and 8x32 / 32x8. The method of generating the transform bases is the same as that of LFNST.
[0078] (LFNST / NSPT for inter-prediction) Non-Patent Document 3 discloses LFNST / NSPT (hereinafter, non-separable transform for inter prediction) for a current block to which inter prediction is applied.
[0079] The prediction residual signal of the current block to which inter prediction is applied contains a directional residual signal, similar to that of an intra prediction block, and is applied to eliminate remaining correlation in the transform coefficients, similar to the LFNST / NSPT (hereinafter referred to as non-separable transform for intra prediction) for the block to which intra prediction is applied described above.
[0080] In non-patent document 3, a gradient histogram of coded or decoded neighboring pixel values adjacent to the current block is calculated, and a direction equivalent to the intra prediction mode is calculated to derive a transform base (i.e., a transform set and a transform kernel) for a non-separable transform for inter prediction.
[0081] Here, as a technique not disclosed in Non-Patent Document 3, in order to derive a transform base, a transform base for a non-separable transform for inter prediction may be derived using the intra prediction mode used by the block referenced by the motion vector of the current block to which inter prediction is applied (hereinafter referred to as the reference block).
[0082] Here, the intra prediction mode used by the reference block may be stored in the frame buffer.
[0083] In addition, if there are multiple reference blocks for the current block, the presence or absence of an intra prediction mode and the prediction mode number of the reference blocks can be scanned in the order of the center of the current block, the four corners (top left, top right, bottom left, bottom right), and once an intra prediction mode is found, that intra prediction mode can be used to derive a transform base for the non-separable transform for inter prediction of the current block.
[0084] Here, the non-separable transforms disclosed in Non-Patent Documents 1 to 3 increase the amount of coding processing.
[0085] In particular, in interpictures or interslices, where both intraprediction and interprediction can be applied, both non-separable transforms for intraprediction and non-separable transforms for interprediction can be applied, so the increase in the amount of coding processing due to the application of non-separable transforms is relatively large compared to intrapictures or intraslices, where only intraprediction can be applied.
[0086] For example, under low-delay encoding conditions such as Low-delay B and Low-delay P, all pictures except the first decoded picture (encoded picture) are interpictures.
[0087] Therefore, when both non-separable transforms for intra prediction and non-separable transforms for inter prediction can be applied on a sequence-by-sequence basis, the amount of coding processing required to determine whether or not to apply non-separable transforms for intra prediction and non-separable transforms for inter prediction on a picture-by-picture or slice-by-slice basis increases.
[0088] Therefore, in the present invention, a technique for efficiently suppressing (reducing) the increase in the amount of coding processing due to non-separable transformation for intra prediction and non-separable transformation for inter prediction will be described below in a coding condition having an inter picture or inter slice to which both intra prediction and inter prediction can be applied.
[0089] (Technology to reduce the amount of coding processing performed by a single coding device) A technique for reducing the amount of encoding processing performed by the encoding device 100 alone according to this embodiment will be described with reference to FIG.
[0090] 4 is a flowchart showing calculation of the coding cost of the coding device 100 according to this embodiment and determination of whether or not it is necessary to calculate the coding cost of non-separable transform for intra prediction. Specific processing is as follows.
[0091] As shown in FIG. 4, in step S401, the encoding device 100 calculates the encoding cost J of the current block to which inter prediction is applied. Inter Then, the process proceeds to step 402.
[0092] Here, the coding cost J is generally calculated by the following equation:
[0093] J=D+λR D is a prediction error (for example, a prediction residual signal), λ is a Lagrange constant, and R is the amount of code.
[0094] In step S402, the encoding device 100 calculates the encoding cost J of the current block to which intra prediction has been applied and to which intra prediction non-separable transform has not been applied. IntraThen, the process proceeds to step 403.
[0095] In step S403, the encoding device 100 calculates J by the following inequality: Inter and J Intra Evaluate the magnitude relationship with
[0096] Min(J Intra )>αMin(J Inter ) Here, Min() in the inequality is the calculated J Inter and J Intra is a function that calculates the minimum value of each of the above, and α is a constant greater than 1 (α>1).
[0097] Here, as a modified example, α may be a constant (α>1) having two or more different values based on the block size of the current block. Specifically, α may be set to a larger value as the block size becomes smaller, and a smaller value as the block size becomes larger.
[0098] In step S403, if the above inequality is satisfied, the encoding apparatus 100 ends this process, and if the above inequality is not satisfied, the encoding apparatus 100 proceeds to step S404.
[0099] In step S404, the encoding device 100 calculates the encoding cost J of the current block to which intra prediction and intra-prediction non-separable transformation are applied. Intra-NST Calculate.
[0100] As described above, in cases where the inequality in step S403 is satisfied, i.e., cases where intra prediction is less likely to be applied than inter prediction, by not calculating the coding cost of the current block to which intra prediction is applied and to which a non-separable transform for intra prediction is applied (by early termination of the non-separable transform for intra prediction), it is possible to reduce the calculation time required to calculate the coding cost of the current block to which intra prediction is applied and to which an intra-prediction non-separable transform is applied.
[0101] In coding conditions having an inter-picture or inter-slice where both intra-prediction and inter-prediction can be applied, the early termination of the non-separable transform for intra-prediction described above can efficiently suppress (reduce) the increase in the amount of coding processing required while maintaining as much as possible the improvement in coding performance achieved by the non-separable transform for intra-prediction and the non-separable transform for inter-prediction.
[0102] As a modification, J in step S402 in FIG. Intra If ∑ i = 1 ...
[0103] That is, the encoding device 100 does not need to calculate the encoding cost of the current block to which intra prediction is applied and to which non-separable transform for intra prediction is applied.
[0104] In cases where intra prediction is terminated early before encoding, for example, the absolute error SAD (sum of absolute difference) or SATD (sum of absolute transformed difference) that can be calculated after the inter prediction and intra prediction are completed can be compared, and a determination can be made whether the SAD or SATD of the intra prediction is sufficiently larger than the SAD or SATD of the inter prediction.
[0105] (Technology to reduce the amount of coding processing involved in both the encoding device and the decoding device) A technique for reducing the amount of encoding processing performed by both the encoding device 100 and the decoding device 200 will be described.
[0106] As an example, the decoding device 200 (or the decoding unit 210 or the inverse transform / inverse quantization unit 220) may be configured to decode a flag (sps_intra_nst_enabled_flag) that controls whether or not a non-separable transform for intra prediction can be applied on a sequence-by-sequence basis, and a flag that controls whether or not a non-separable transform for inter prediction (sps_inter_nst_enabled_flag) can be applied on a sequence-by-sequence basis.
[0107] Here, sps_intra_nst_enabled_flag is a flag that specifies whether or not non-separable transform for intra prediction is applicable for each sequence.
[0108] When the flag is 1, the decoding device 200 may determine that non-separable transform for intra prediction is applicable on a sequence-by-sequence basis, and when the flag is 0, may determine that non-separable transform for intra prediction is not applicable on a sequence-by-sequence basis.
[0109] If such a flag does not exist (if the data is not decoded), the decoding device 200 may estimate the value of the flag to be 0.
[0110] Similarly, sps_inter_nst_enabled_flag is a flag that specifies whether or not non-separable transform for inter prediction is applicable on a sequence-by-sequence basis.
[0111] When the flag is 1, the decoding device 200 may determine that non-separable transform for inter prediction is applicable on a sequence-by-sequence basis, and when the flag is 0, may determine that non-separable transform for intra prediction is not applicable on a sequence-by-sequence basis.
[0112] If such a flag does not exist (if the data is not decoded), the decoding device 200 may estimate the value of the flag to be 0.
[0113] By using these flags, it is possible to efficiently suppress (reduce) an increase in the amount of coding processing due to non-separable transform for intra prediction and non-separable transform for inter prediction under coding conditions including inter pictures or inter slices.
[0114] For example, in an encoding condition having an inter-picture or inter-slice where both intra-prediction and inter-prediction can be applied, by always decoding sps_intra_nst_enabled_flag as 0 while always decoding sps_inter_nst_enabled_flag as 1, only non-separable transforms for inter-prediction can be explicitly applied in the inter-picture or inter-slice, thereby suppressing (reducing) the increase in the amount of encoding processing for non-separable transforms for intra-prediction.
[0115] As a modification, the decoding device 200 may decode a flag for each type of non-separable transform.
[0116] For example, when there is a low-frequency non-separable transform and a non-separable primary transform, the decoding device 200 may decode sps_intra_lfnst_enabled_flag, sps_intra_nspt_enabled_flag, sps_inter_lfnst_enabled_flag, and sps_inter_nspt_enabled_flag as two flags for intra prediction and two flags for inter prediction, respectively.
[0117] Here, sps_intra_lfnst_enabled_flag is a flag that specifies whether or not low frequency non-separable transform for intra prediction is applicable on a sequence-by-sequence basis.
[0118] If the flag is 1, the decoding device 200 may determine that low-frequency non-separable transform for intra prediction is applicable on a sequence-by-sequence basis, and if the flag is 0, may determine that low-frequency non-separable transform for intra prediction is not applicable on a sequence-by-sequence basis.
[0119] If such a flag does not exist (if the data is not decoded), the decoding device 200 may estimate the value of the flag to be 0.
[0120] Furthermore, sps_intra_nspt_enabled_flag is a flag that specifies whether or not non-separable linear transform for intra prediction is applicable for each sequence.
[0121] If the flag is 1, the decoding device 200 may determine that a non-separable linear transform for intra prediction is applicable on a sequence-by-sequence basis, and if the flag is 0, may determine that a low-frequency non-separable transform for intra prediction is not applicable on a sequence-by-sequence basis.
[0122] If such a flag does not exist (if the data is not decoded), the decoding device 200 may estimate the value of the flag to be 0.
[0123] Furthermore, sps_inter_lfnst_enabled_flag is a flag that specifies whether or not low frequency non-separable transform for inter prediction is applicable on a sequence-by-sequence basis.
[0124] If the flag is 1, the decoding device 200 may determine that low-frequency non-separable transform for inter prediction is applicable on a sequence-by-sequence basis, and if the flag is 0, may determine that low-frequency non-separable transform for inter prediction is not applicable on a sequence-by-sequence basis.
[0125] If such a flag does not exist (if the data is not decoded), the decoding device 200 may estimate the value of the flag to be 0.
[0126] Furthermore, sps_inter_nspt_enabled_flag is a flag that specifies whether or not non-separable linear transform for inter prediction is applicable for each sequence.
[0127] If the flag is 1, the decoding device 200 may determine that a non-separable linear transform for inter prediction is applicable on a sequence-by-sequence basis, and if the flag is 0, may determine that a low-frequency non-separable transform for inter prediction is not applicable on a sequence-by-sequence basis.
[0128] If such a flag does not exist (if the data is not decoded), the decoding device 200 may estimate the value of the flag to be 0.
[0129] As another modification, the decoding device 200 may decode a flag that controls whether or not the above-described non-separable transform, low-frequency non-separable transform, or non-separable primary transform is applicable, not in units of sequences, but in units of pictures or slices. Alternatively, the decoding device 200 may decode a similar flag hierarchically, such as in units of sequences, pictures, or slices.
[0130] As another modification, the decoding device 200 may decode sps_nst_enabled_flag before decode sps_intra_nst_enabled_flag and sps_inter_nst_enabled_flag, and control whether or not to decode sps_intra_nst_enabled_flag and sps_inter_nst_enabled_flag based on the value of sps_nst_enabled_flag.
[0131] Here, sps_nst_enabled_flag is a flag that specifies whether or not a non-separable transform is applicable on a sequence-by-sequence basis.
[0132] If the flag is 1, the decoding device 200 may determine that a non-separable transform is applicable on a sequence-by-sequence basis, and if the flag is 0, may determine that a non-separable transform is not applicable on a sequence-by-sequence basis.
[0133] If such a flag does not exist (if the data is not decoded), the decoding device 200 may estimate the value of the flag to be 0.
[0134] The amount of coding can be reduced by controlling whether or not to decode sps_intra_nst_enabled_flag and sps_inter_nst_enabled_flag based on the value of sps_nst_enabled_flag.
[0135] For example, as shown in Figure 5, if the value of sps_nst_enabled_flag is 1, sps_intra_nst_enabled_flag and sps_inter_nst_enabled_flag are decoded, and if the value of sps_nst_enabled_flag is 0, sps_intra_nst_enabled_flag and sps_inter_nst_enabled_flag can be estimated to be 0 without being decoded.
[0136] As another modification, the decoding device 200 may decode sps_intra_nst_enabled_flag before decode sps_intra_lfnst_enabled_flag and sps_intra_nspt_enabled_flag, and control whether to decode sps_intra_lfnst_enabled_flag and sps_intra_nspt_enabled_flag based on the value of sps_intra_nst_enabled_flag.
[0137] The amount of coding can be reduced by controlling whether or not to decode sps_intra_lfnst_enabled_flag and sps_intra_nspt_enabled_flag based on the value of sps_intra_nst_enabled_flag.
[0138] Furthermore, the decoding device 200 can reduce the amount of code by controlling whether or not to decode sps_inter_lfnst_enabled_flag and sps_inter_nspt_enabled_flag based on the value of sps_inter_nst_enabled_flag.
[0139] For example, as shown in Figure 6, if the value of sps_intra_nst_enabled_flag is 1, sps_intra_lfnst_enabled_flag and sps_intra_nspt_enabled_flag are decoded, and if the value of sps_intra_nst_enabled_flag is 0, sps_intra_lfnst_enabled_flag and sps_intra_nspt_enabled_flag can be estimated to be 0 without being decoded.
[0140] Similarly, for example, as shown in Figure 6, if the value of sps_inter_nst_enabled_flag is 1, sps_inter_lfnst_enabled_flag and sps_inter_nspt_enabled_flag are decoded, and if the value of sps_inter_nst_enabled_flag is 0, sps_inter_lfnst_enabled_flag and sps_inter_nspt_enabled_flag can be estimated to be 0 without being decoded.
[0141] Here, when it is determined that NSPT for inter prediction is not applicable on a sequence-by-sequence basis and LFNST for inter prediction is applicable on a sequence-by-sequence basis, the inverse transform / inverse quantization unit 220 may control whether to apply LFNST based on the current block size.
[0142] For example, the inverse transform / inverse quantization unit 220 may determine that NSPT is inapplicable to the current block of the block size to which NSPT is to be applied.
[0143] Alternatively, the inverse transform / inverse quantization unit 220 may determine that LFNST is applicable to the current block of the block size to which NSPT is to be applied.
[0144] As a result, when the encoding device 100 is set to not apply NSPT on a sequence-by-sequence basis and to apply LFNST, LFNST can be applied to the current block of the block size to which NSPT is to be applied, thereby reducing the amount of NSPT encoding processing and compensating for the possibility of reduced encoding performance due to disabling NSPT on a sequence-by-sequence basis by applying LFNST. In this example, control of whether to apply NSPT and LFNST is described without distinguishing between NSPT and LFNST for intra prediction and inter prediction, but the two may also be controlled separately.
[0145] As another example, the decoding device 200 (or the decoding unit 210 or the inverse transform / inverse quantization unit 220) may control whether or not to apply a non-separable transform for inter prediction depending on the distance (difference in POC: Picture Order Count) between the current frame to which the current block belongs and the reference frame to which the reference block referenced for inter prediction belongs.
[0146] For example, decoding device 200 may determine that non-separable conversion for inter prediction is applicable if the POC difference is greater than a first predetermined value (or the POC difference is equal to or greater than the first predetermined value), and may determine that non-separable conversion for inter prediction is not applicable if the POC difference is less than the first predetermined value (or the POC difference is less than the first predetermined value).
[0147] Here, the first predetermined value may be, for example, the length of a Group Of Pictures (GOP), which is a unit of inter-frame reference structure, i.e., the same value as the GOP length. Alternatively, the first predetermined value may be 1 / 2 the GOP length. Alternatively, the first predetermined value may be 1 / 4 the GOP length. Alternatively, the first predetermined value may be a fixed value. For example, the fixed value may be any of 2, 4, 8, or 16.
[0148] In general, when the difference in POC is large, the prediction residual signal tends to be large, and when the difference in POC is small, the prediction residual signal tends to be small.
[0149] Therefore, by controlling whether or not to apply non-separable transforms for inter prediction depending on the mPOC difference as described above, it is possible to suppress the increase in the amount of coding processing due to non-separable transforms for inter prediction while maintaining the coding performance of non-separable transforms for inter prediction as much as possible.
[0150] As another example, the decoding device 200 (or the decoding unit 210 or the inverse transform / inverse quantization unit 220) may control whether or not to apply at least one or both of a non-separable transform for intra prediction and a non-separable transform for inter prediction, depending on the quantization parameter (QP) of the current block.
[0151] For example, decoding device 200 may determine that at least one or both of non-separable transform for intra prediction and non-separable transform for inter prediction are applicable when the QP of the current block is smaller than a second predetermined value (or the QP of the current block is equal to or smaller than the second predetermined value), and may determine that at least one or both of non-separable transform for intra prediction and non-separable transform for inter prediction are not applicable when the QP of the current block is equal to or larger than the second predetermined value (or the QP of the current block is larger than the second predetermined value).
[0152] Here, the second predetermined value may be the same value as the QP of the current picture (current frame) or current slice to be decoded or coded, which includes the current block.
[0153] Alternatively, the second predetermined value may be the same value as the QP (base QP) to which the QP offset set in each current picture to be decoded or coded by the image coding device 100 is added.
[0154] In general, when QP is large, non-zero transform coefficients other than those near the DC component (low frequency component) are less likely to remain, and when QP is small, non-zero transform coefficients other than those near the DC component (low frequency component) are more likely to remain.
[0155] Therefore, by controlling whether or not at least one or both of the non-separable transform for intra prediction and the non-separable transform for inter prediction can be applied depending on the QP, it is possible to suppress the increase in the amount of coding processing due to the non-separable transform while maintaining the coding performance of the non-separable transform as much as possible.
[0156] As another example, the decoding device 200 (or the decoding unit 210 or the inverse transform / inverse quantization unit 220) may control whether or not to apply non-separable transform for inter prediction depending on the quantization parameter (QP) of the reference block.
[0157] For example, decoding device 200 may determine that non-separable transform for inter prediction is applicable when the QP of the reference block is smaller than a second predetermined value, and may determine that non-separable transform for inter prediction is not applicable when the QP of the current block is larger than the second predetermined value. Generally, when the QP is large, non-zero transform coefficients other than those near the DC component (low frequency component) are less likely to remain, and when the QP is small, non-zero transform coefficients other than those near the DC component (low frequency component) are more likely to remain.
[0158] Therefore, by controlling whether or not to apply non-separable transform for inter prediction according to QP, it is possible to suppress an increase in the amount of coding processing due to non-separable transform for inter prediction while maintaining the coding performance of non-separable transform for inter prediction as much as possible.
[0159] The image encoding device 100 and the image decoding device 200 described above may be realized as a program that causes a computer to execute each function (each step).
[0160] In each of the above-described embodiments, the present invention has been described as being applied to the image encoding device 100 and the image decoding device 200, but the present invention is not limited to this and can be similarly applied to image encoding systems and image decoding systems that have the functions of the image encoding device 100 and the image decoding device 200. [Explanation of symbols]
[0161] 10...Image processing system 100...Image encoding device 111, 241...Inter prediction section 112, 242...Intra prediction section 121...Subtractor 122, 230...adder 131...Transformation and quantization unit 132, 220...Inverse transform and inverse quantization units 140...encoding section 150, 250...In-loop filter processing section 160, 260...frame buffer 200...Image decoding device 210...Decoding unit
Claims
1. An image decoding device, an intra prediction unit that applies intra prediction to a current block; an inter prediction unit that applies inter prediction to the current block; an inverse transform and inverse quantization unit that applies inverse transform and inverse quantization to the current block; a decoding unit that decodes the coded data including the control information, The decoding unit determines whether or not a non-separable transform for intra prediction and a non-separable transform for inter prediction can be applied to the current block based on control information that specifies whether or not each of the non-separable transforms for intra prediction and the non-separable transforms for inter prediction can be applied on a sequence-by-sequence basis.
2. The decoding unit If the value of the control information is 1, the non-separable transform for intra prediction is determined to be applicable on a sequence-by-sequence basis; If the value of the control information is 0, determining that the non-separable transform for intra prediction is not applicable on a sequence-by-sequence basis; 2. The image decoding device according to claim 1, wherein if the control information is not decoded, the value of the control information is estimated to be 0.
3. The decoding unit If the value of the control information is 1, the non-separable transform for inter prediction is determined to be applicable on a sequence-by-sequence basis; If the value of the control information is 0, determining that the non-separable transform for inter prediction cannot be applied on a sequence-by-sequence basis; 2. The image decoding device according to claim 1, wherein if the control information is not decoded, the value of the control information is estimated to be 0.
4. An image decoding method, comprising: a step A of applying intra prediction to a current block; A step B of applying inter prediction to the current block; a step C of applying an inverse transform and an inverse quantization to the current block; and a step D of decoding the coded data including the control information, An image decoding method characterized in that in step D, it is determined whether or not a non-separable transform for intra prediction and a non-separable transform for inter prediction can be applied to the current block based on control information that specifies whether or not each of the non-separable transforms for intra prediction and the non-separable transforms for inter prediction can be applied on a sequence-by-sequence basis.
5. A program that causes a computer to function as an image decoding device, The image decoding device comprises: an intra prediction unit that applies intra prediction to a current block; an inter prediction unit that applies inter prediction to the current block; an inverse transform and inverse quantization unit that applies inverse transform and inverse quantization to the current block; a decoding unit that decodes the coded data including the control information, The decoding unit determines whether or not a non-separable transform for intra prediction and a non-separable transform for inter prediction can be applied to the current block based on control information that specifies whether or not each of the non-separable transforms for intra prediction and the non-separable transform for inter prediction can be applied on a sequence-by-sequence basis.
6. An image encoding device, an intra prediction unit that applies intra prediction to a current block; an inter prediction unit that applies inter prediction to the current block; a transform and inverse quantization unit that applies transform and quantization to the current block; a coding unit that encodes the coefficient level values and the control information to generate coded data, The encoding unit compares the encoding cost of a current block to which inter prediction has been applied with the encoding cost of a current block to which intra prediction has been applied but to which a non-separable transform for intra prediction has not been applied, and determines whether or not a non-separable transform for intra prediction can be applied to the current block.
7. 7. The image encoding device of claim 6, wherein the encoding unit does not calculate the encoding cost for a case in which the non-separable transform for intra prediction is applied to the current block when the encoding cost of the current block to which the intra prediction is applied and to which the non-separable transform for intra prediction is not applied is larger than a constant multiple of the encoding cost of the current block to which the inter prediction is applied.
8. 7. The image encoding device according to claim 6, wherein the encoding unit calculates the encoding cost for a case in which the non-separable transform for intra prediction is applied to the current block when the encoding cost of the current block to which the intra prediction is applied and to which the non-separable transform for intra prediction is not applied is smaller or equal to a constant multiple of the encoding cost of the current block to which the inter prediction is applied.
9. 9. The image encoding device of claim 7, wherein the predetermined constant for multiplying the encoding cost of the current block to which inter prediction is applied is greater than 1.
10. 10. The image encoding device according to claim 9, wherein the predetermined constant for multiplying the encoding cost of the current block to which inter prediction is applied by a constant is set to two or more different values depending on the block size of the current block.