Image processing device and coefficient data generation method

Abstract

The present disclosure pertains to an image processing device, a bit stream generation method, a coefficient data generation method, and a quantization coefficient generation method that make it possible to suppress reduction in encoding efficiency. The present disclosure, in encoding of an image, generates a transform skip flag that is flag information indicating, for each component, whether to skip transform processing for transforming a residual between the image and a prediction image of the image to coefficient data; encodes the generated transform skip flag; generates encoded data of the transform skip flag; and generates a bit stream including the generated encoded data of the transform skip flag. The present disclosure can be applied to, for example, an image processing device, an image encoding device, an image decoding device, a transmission device, a reception device, a transmission / reception device, an information processing device, an imaging device, a reproduction device, a bit stream generation method, a coefficient data generation method, a quantization coefficient generation method, or the like.

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus and method for generating coefficient data and, in particular, to an image processing apparatus capable of suppressing a reduction in coding efficiency and method for generating coefficient data relates to.

Background Art

[0002] Conventionally, in image coding, a method has been proposed in which conversion processing for converting the difference between an image and its predicted image into coefficient data is skipped (omitted) for coding (for example, Non-Patent Document 1).

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the case of the method described in this Non-Patent Document 1, this conversion skip mode is applied only to the luminance component (luminance conversion skip). That is, this conversion skip mode could not be applied to the color difference component. Therefore, there was a risk of a reduction in coding efficiency.

[0005] The present disclosure has been made in view of such a situation and is intended to suppress a reduction in coding efficiency. [Means for solving the problem]

[0010] An image processing apparatus, one aspect of this technology, includes a flag decoding unit that decodes encoded data of a conversion skip flag corresponding to a component identifier that identifies brightness and color difference; a mode control unit that controls, based on the decoded conversion skip flag corresponding to the component identifier, whether to set the decoding mode of the encoded data of the coefficient data corresponding to the component identifier to a TS residual decoding mode, which is a mode for skipping the inverse transformation process that converts the coefficient data into residuals between the image and the predicted image, or to a non-TS residual decoding mode, which is a mode for not skipping the inverse transformation process; and a mode control unit that applies the set decoding mode and processes the coefficient data corresponding to the component identifier according to the syntax structure corresponding to the decoding mode. Encoded data It comprises a coefficient data decoding unit that decodes the data. If the inverse transformation process is skipped, the coefficient data decoding unit shares context variables between decoding the encoded data of the luminance component and decoding the encoded data of the chrominance component of the coefficient data. It is an image processing device. One aspect of this technology is a coefficient data generation method which includes: decoding encoded data of a conversion skip flag corresponding to a component identifier that identifies luminance and chrominance; controlling, based on the decoded conversion skip flag corresponding to the component identifier, whether the decoding mode of the encoded data of the coefficient data corresponding to the component identifier is a TS residual decoding mode, which is a mode for skipping the inverse transformation process that converts the coefficient data into residuals between an image and a predicted image, or a non-TS residual decoding mode, which is a mode for not skipping the inverse transformation process; and applying the set decoding mode, decoding the encoded data of the coefficient data corresponding to the component identifier according to the syntax structure corresponding to the decoding mode, and, if the inverse transformation process is skipped, sharing a context variable between the decoding of the encoded data of the luminance component and the decoding of the encoded data of the chrominance component of the coefficient data.

[0011] Another aspect of this technology is an image processing apparatus comprising: a flag decoding unit that decodes encoded data of a conversion skip flag corresponding to a component identifier that identifies brightness and color difference; a mode control unit that controls, based on the decoded conversion skip flag corresponding to the component identifier, whether to set the decoding mode of the encoded data of the coefficient data corresponding to the component identifier to a TS residual decoding mode, which is a mode for skipping the inverse transformation process that converts the coefficient data into residuals between an image and a predicted image, or to a non-TS residual decoding mode, which is a mode for not skipping the inverse transformation process; and a coefficient data decoding unit that applies the set decoding mode and decodes the encoded data of the coefficient data corresponding to the component identifier according to the syntax structure corresponding to the decoding mode, wherein the coefficient data decoding unit applies a decoding method of a sine code corresponding to the decoded conversion skip flag corresponding to the component identifier. This technology other The coefficient data generation method for the side involves decoding the encoded data of a conversion skip flag corresponding to a component identifier that identifies luminance and chrominance; controlling the decoding mode of the encoded data of the coefficient data corresponding to the component identifier based on the decoded conversion skip flag corresponding to the component identifier, to either a TS residual decoding mode, which is a mode for skipping the inverse transformation process that converts the coefficient data into residuals between the image and the predicted image, or a non-TS residual decoding mode, which is a mode for not skipping the inverse transformation process; and applying the set decoding mode and, according to the syntax structure corresponding to the decoding mode, Applying a method for decoding a sign code corresponding to the conversion skip flag that corresponds to the decoded component identifier, The coefficient data corresponding to the component identifier Encoded data This is a method for generating coefficient data, which includes decoding the data.

[0016] In one aspect of this technology, an image processing apparatus and coefficient data generation method, the encoded data of a conversion skip flag corresponding to a component identifier that identifies luminance and chrominance is decoded, and based on the decoded conversion skip flag corresponding to the component identifier, the decoding mode of the encoded data of the coefficient data corresponding to that component identifier is determined. of The system controls whether to use TS residual decoding mode, which is for cases where the inverse transform process of converting coefficient data into residuals between the image and the predicted image is skipped, or non-TS residual decoding mode, which is for cases where the inverse transform process is not skipped. The set decoding mode is then applied, and the encoded data of the coefficient data corresponding to the component identifier is decoded according to the syntax structure corresponding to that decoding mode. If the inverse transformation process is skipped, the context variables are shared between decoding the encoded data of the luminance component and decoding the encoded data of the chrominance component of the coefficient data. The action will be carried out. In the image processing apparatus and coefficient data generation method of other aspects of this technology, the following is performed: the encoded data of a conversion skip flag corresponding to a component identifier that identifies luminance and chrominance is decoded; based on the conversion skip flag corresponding to the decoded component identifier, the decoding mode of the encoded data of the coefficient data corresponding to that component identifier is controlled to be either a TS residual decoding mode, which is a mode for skipping the inverse transformation process that converts the coefficient data into residuals between the image and the predicted image, or a non-TS residual decoding mode, which is a mode for not skipping the inverse transformation process; the set decoding mode is applied, and according to the syntax structure corresponding to the decoding mode, a decoding method of the sine code corresponding to the conversion skip flag corresponding to the decoded component identifier is applied, and the encoded data of the coefficient data corresponding to that component identifier is decoded. [Brief explanation of the drawing]

[0018] [Figure 1] This diagram illustrates the extension of conversion skipping. [Figure 2] This is a block diagram showing a typical configuration of an image coding device. [Figure 3] This is a block diagram showing the main configuration examples of the coding section. [Figure 4] This is a flowchart illustrating an example of the image encoding process. [Figure 5] This is a flowchart illustrating an example of the encoding process. [Figure 6] This block diagram shows a typical configuration of an image decoding device. [Figure 7] This is a block diagram showing the main configuration examples of the decoding unit. [Figure 8] This is a flowchart illustrating an example of the image decoding process. [Figure 9] This flowchart shows an example of the decryption process. [Figure 10] This figure shows an example of TU syntax. [Figure 11]It is a diagram showing an example of the syntax of conversion mode information. [Figure 12] It is a diagram showing an example of the syntax of the TS residual coding mode. [Figure 13] It is a diagram showing an example of the semantics of conversion mode information. [Figure 14] It is a diagram showing another example of the syntax of conversion mode information. [Figure 15] It is a diagram showing an example of the syntax of the sequence parameter set. [Figure 16] It is a diagram showing an example of the syntax of the TU. [Figure 17] It is a diagram showing an example of the semantics of the sequence parameter set. [Figure 18] It is a diagram showing an example of the syntax of the sequence parameter set. [Figure 19] It is a diagram showing an example of the syntax of the TU. [Figure 20] It is a diagram showing an example of the semantics of the sequence parameter set. [Figure 21] It is a block diagram showing a main configuration example of the quantization unit. [Figure 22] It is a flowchart showing an example of the quantization process flow. [Figure 23] It is a block diagram showing a main configuration example of the inverse quantization unit. [Figure 24] It is a flowchart showing an example of the inverse quantization process flow. [Figure 25] It is a diagram showing an example of the syntax of the quantization parameter. [Figure 26] It is a diagram showing an example of the syntax of the context variable. [Figure 27] It is a diagram showing an example of the syntax of the context variable. [Figure 28] [[ID=5l]]It is a diagram for explaining the switching of the sign coding mode. [Figure 29] It is a block diagram showing a main configuration example of the computer.

Embodiments for Carrying Out the Invention

[0019] The following describes the forms for implementing this disclosure (hereinafter referred to as embodiments). The explanation will be given in the following order. 1. Supporting literature for technical content and terminology 2. Skip conversion 3. First Embodiment (Extension of Conversion Skip) 4. Second Embodiment (Correction of Quantization Parameters) 5. Third Embodiment (Sharing of Context Variables) 6. Fourth Embodiment (Sine Code Encoding / Decoding Mode Control) 7. Addendum

[0020] <1. Literature and other resources supporting the technical content and terminology> The scope disclosed in this technology includes not only the contents described in the examples, but also the contents described in the following non-patent documents that were publicly known at the time of filing.

[0021] Non-patent document 1: (mentioned above) Non-patent document 2: Takeshi Tsukuba, Masaru Ikeda, Yoichi Yagasaki, Teruhiko Suzuki, "CE8: Chroma Transform Skip (CE8-3.2)", JVET-O0081-v2, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 15th Meeting: Gothenburg, SE, 3-12 July 2019 Non-Patent Document 3: Tung Nguyen, Benjamin Bross, Heiko Schwarz, Detlev Marpe, Thomas Wiegand, "Non-CE8: Minimum Allowed QP for Transform Skip Mode", JVET-O0405-v1, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 15th Meeting: Gothenburg, SE, 3-12 July 2019 Non-Patent Document 4: Jianle Chen, Yan Ye, Seung Hwan Kim, "Algorithm description for Versatile Video Coding and Test Model 6 (VTM 6)", JVET-O2002-v2, Joint Video Experts Team (JVET), of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 15th Meeting: Gothenburg, SE, 3-12 July 2019 Non-Patent Document 5: Takeshi Tsukuba, Masaru Ikeda, Yoichi Yagasaki, Teruhiko Suzuki, "CE8-2.1: Transform Skip for Chroma with limiting maximum number of context-coded bin in TS residual coding", JVET-P0058-v1, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 16th Meeting: Geneva, CH, 1-11 October 2019 Non-patent document 6: Gordon Clare, Felix Henry, Takeshi Tsukuba, Masaru Ikeda, Yoich Yagasaki, Teruhiko Suzuki, "CE8-4.1: BDPCM and Transform skip for Chroma", JVET-P0059-v1, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 16th Meeting: Geneva, CH, 1-11 October 2019 Non-patent document 7: TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (International Telecommunication Union), "Advanced video coding for generic audiovisual services", H.264, 04 / 2017 Non-patent document 8: TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU (International Telecommunication Union), "High efficiency video coding", H.265, 12 / 2016

[0022] In other words, the content described in the aforementioned non-patent literature also serves as a basis for determining the support requirement. For example, even if the Quad-Tree Block Structure and QTBT (Quad Tree Plus Binary Tree) Block Structure described in the aforementioned non-patent literature are not directly described in the examples, they are considered to be within the scope of the disclosure of this technology and satisfy the support requirement of the claims. Similarly, technical terms such as Parsing, Syntax, and Semantics are also considered to be within the scope of the disclosure of this technology and satisfy the support requirement of the claims, even if they are not directly described in the examples.

[0023] Furthermore, in this specification, unless otherwise specified, the term "block" (not a block indicating a processing unit) used in the description as a sub-region or processing unit of an image (picture) refers to any sub-region within a picture, and its size, shape, and characteristics are not limited. For example, "block" includes any sub-region (processing unit) such as TB (Transform Block), TU (Transform Unit), PB (Prediction Block), PU (Prediction Unit), SCU (Smallest Coding Unit), CU (Coding Unit), LCU (Largest Coding Unit), CTB (Coding Tree Block), CTU (Coding Tree Unit), transformation block, sub-block, macroblock, tile, or slice, as described in the aforementioned non-patent literature.

[0024] Furthermore, when specifying the size of such blocks, it is possible to specify the block size not only directly but also indirectly. For example, the block size may be specified using identification information that identifies the size. Alternatively, the block size may be specified by a ratio or difference with the size of a reference block (e.g., LCU or SCU). For example, when transmitting information that specifies the block size as a syntax element, the information that specifies the size indirectly as described above may be used. By doing so, the amount of information can be reduced, and encoding efficiency may be improved. In addition, this specification of block size also includes specifying a range of block sizes (e.g., specifying a range of acceptable block sizes).

[0025] Furthermore, in this specification, encoding includes not only the entire process of converting an image into a bitstream, but also some of the processes. For example, it includes not only processes that encompass prediction processing, orthogonal transformations, quantization, arithmetic coding, etc., but also processes that collectively refer to quantization and arithmetic coding, processes that encompass prediction processing, quantization, and arithmetic coding, etc. Similarly, decoding includes not only the entire process of converting a bitstream into an image, but also some of the processes. For example, it includes not only processes that encompass inverse arithmetic decoding, inverse quantization, inverse orthogonal transformation, prediction processing, etc., but also processes that encompass inverse arithmetic decoding and inverse quantization, processes that encompass inverse arithmetic decoding, inverse quantization, and prediction processing, etc.

[0026] <2. Skip conversion> <Skip conversion of luminance component> Conventionally, in the encoding of still and moving images, the images input to the encoding device are generally color images having luminance components and color components (which may include color difference components). In such image encoding, for example, as described in Non-Patent Document 1, a method has been considered in which the conversion process of converting the residual between the image and its predicted image into coefficient data for the luminance component is skipped (omitted).

[0027] However, this conversion skip mode could not be applied to color components. Therefore, in screen content where conversion skipping was enabled, there was a risk that the encoding efficiency of color components would be reduced.

[0028] <Expansion of conversion skip flag> Therefore, the conversion skip function is extended to allow conversion skip settings for each component (luminance component and color component), enabling conversion skipping not only for luminance components but also for color components. For example, in the image encoding / decoding method described in the non-patent document mentioned above, the skip setting can be indicated by the conversion skip flag (transform_skip_flag). Components can be indicated by component identifiers (cIdx). Therefore, for example, as in Method 1 in the top row of the table in Figure 1, the value of this conversion skip flag is set to correspond to the value of the component identifier.

[0029] For example, an image processing device may include a flag generation unit that generates conversion skip flags, which are flag information indicating whether or not to skip the conversion process for each component in image encoding, which converts the residual between the image and its predicted image into coefficient data; a flag encoding unit that encodes the conversion skip flags generated by the flag generation unit and generates encoded data for the conversion skip flags; and a bitstream generation unit that generates a bitstream containing the encoded data for the conversion skip flags generated by the flag encoding unit.

[0030] Furthermore, for example, when generating a bitstream, a conversion skip flag is generated for each component, which is flag information indicating whether or not to skip the conversion process in image encoding where the residual between the image and its predicted image is converted into coefficient data. This generated conversion skip flag is then encoded to generate encoded data for the conversion skip flag, and finally, a bitstream containing this encoded data for the conversion skip flag is generated.

[0031] This allows the decoding side to provide settings for conversion skipping on a component-by-component basis. Therefore, when conversion skipping is applied to the color component, the decoding side can correctly decode the bitstream. Consequently, the reduction in encoding efficiency can be suppressed compared to when conversion skipping is only applicable to the luminance component.

[0032] Furthermore, the decryption side may be made able to obtain this conversion skip flag.

[0033] For example, an image processing device may be equipped with a flag decoding unit that decodes encoded data of a conversion skip flag corresponding to a component identifier and obtains the conversion skip flag corresponding to that component identifier.

[0034] Alternatively, for example, when generating coefficient data, the encoded data of the conversion skip flag corresponding to the component identifier may be decoded to obtain the conversion skip flag corresponding to that component identifier.

[0035] By doing so, during decoding, the component-specific conversion skip settings indicated by the conversion skip flag can be applied. Therefore, the bitstream of color components to which conversion skipping has been applied can be correctly decoded. Consequently, the reduction in encoding efficiency can be suppressed compared to the case where conversion skipping can only be applied to the luminance component.

[0036] Furthermore, during image encoding, the conversion process for color components may be controlled based on the conversion skip settings for each component using the conversion skip flags described above. In other words, conversion skipping may be applied to the conversion process for color components. By doing so, the reduction in encoding efficiency can be suppressed compared to the case where conversion skipping is only applicable to the luminance component.

[0037] Similarly, when decoding encoded image data, the inverse transformation process of converting the coefficient data of the color components into residuals between the image and the predicted image may be controlled based on the conversion skip settings for each component using the conversion skip flags described above. In other words, conversion skipping may be applied to the inverse transformation process of the color components. By doing so, the bitstream of the color components to which conversion skipping has been applied can be correctly decoded. Therefore, the reduction in encoding efficiency can be suppressed compared to the case where conversion skipping is only applicable to the luminance component.

[0038] As mentioned above, by setting conversion skips for each component, conversion skips can be set for color components independently of luminance components. Therefore, conversion skips for color components can be set depending on whether or not conversion skips are enabled for color components. By doing so, the reduction in encoding efficiency can be suppressed compared to applying the conversion skip setting for luminance components to color components.

[0039] <Controlling the encoding mode> Furthermore, in the method described in Non-Patent Document 1, the encoding mode (decoding mode) of the luminance component coefficient data was controlled depending on whether or not a conversion skip was performed. For example, when a conversion skip was performed, the luminance component coefficient data was encoded using a TS residual encoding mode optimized for the conversion-skipped coefficient data (the encoded data of the luminance component coefficient data was decoded using a TS residual decoding mode optimized for the conversion-skipped coefficient data). In contrast, when a conversion skip was not performed, the luminance component coefficient data was encoded using a non-TS residual encoding mode optimized for the conversion-processed coefficient data (the encoded data of the luminance component coefficient data was decoded using a non-TS residual decoding mode optimized for the conversion-processed coefficient data).

[0040] However, since conversion skipping was not applied to the color difference component, this type of encoding mode (decoding mode) control was not performed. Therefore, when conversion skipping was applied to the color component as described above, there was a risk that the encoding efficiency would be reduced compared to when this type of encoding mode (decoding mode) control was performed.

[0041] Therefore, when applying conversion skipping to color components as in Method 1 described above, control of the encoding mode (decoding mode) may also be applied. For example, as in Method 1-1 in the second row from the top of the table in Figure 1, when applying conversion skipping to each value of the component identifier, the TS residual encoding mode (TS residual decoding mode) may be applied as the encoding mode (decoding mode), and when not applying conversion skipping (no conversion skipping), the non-TS residual encoding mode (non-TS residual decoding mode) may be applied as the encoding mode (decoding mode).

[0042] For example, based on a conversion skip flag corresponding to a component identifier, the encoding mode of the coefficient data corresponding to that component identifier may be controlled to either be the TS residual encoding mode, which is the mode for when the conversion process is skipped, or the non-TS residual encoding mode, which is the mode for when the conversion process is not skipped. The coefficient data corresponding to the component identifier may then be encoded using the encoding mode set in this manner, and encoded data of the coefficient data may be generated.

[0043] Alternatively, for example, based on a conversion skip flag corresponding to a component identifier, the decoding mode of the encoded data of the coefficient data corresponding to that component identifier may be controlled to either a TS residual decoding mode, which is used when the inverse transformation process of converting the coefficient data into residuals between the image and the predicted image is skipped, or a non-TS residual decoding mode, which is used when the inverse transformation process is not skipped. The encoded data of the coefficient data corresponding to the component identifier may then be decoded using the decoding mode set in this manner, and the coefficient data corresponding to that component identifier may be generated.

[0044] By doing so, it is possible to apply an encoding mode (decoding mode) according to the conversion skip setting, which helps to suppress the reduction in encoding efficiency compared to when encoding (decoding) is performed using a single encoding mode (decoding mode).

[0045] <Use conversion skip residual coding flag> A flag indicating the selection between TS residual coding mode (TS residual decoding mode) and non-TS residual coding mode (non-TS residual decoding mode) may be applied. For example, as shown in method 1-1-1 in the third row from the top of the table in Figure 1, a residual coding mode selection flag (also called a conversion skip residual coding use flag) indicating the coding mode selection setting may be applied.

[0046] For example, during encoding, a conversion skip residual coding usage flag, which is flag information indicating whether to apply TS residual coding mode or non-TS residual coding mode, may be generated. This conversion skip residual coding usage flag may then be encoded, and a bitstream containing the encoded data of this conversion skip residual coding usage flag may be generated.

[0047] Furthermore, during decoding, the encoded data of the conversion skip residual coding usage flag may be decoded to generate a conversion skip residual coding usage flag corresponding to the component identifier, and based on the conversion skip residual coding usage flag corresponding to the component identifier, the decoding mode of the encoded data of the coefficient data corresponding to that component identifier may be controlled to be either TS residual decoding mode or non-TS residual decoding mode.

[0048] For example, such a conversion skip residual coding usage flag may be set at a high level, such as in a sequence parameter. By doing so, even encoders and decoders that do not support conversion skipping can correctly perform coding and decoding based on this flag. Therefore, a reduction in coding efficiency can be suppressed. In other words, the implementation of the TS residual coding mode (TS residual decoding mode) in encoders and decoders can be omitted, and an increase in circuit size can be suppressed.

[0049] <Conversion skip residual coding usage specific mode flag> In a particular mode, a flag may be applied to indicate whether to apply the TS residual coding mode (TS residual decoding mode) or the non-TS residual coding mode (non-TS residual decoding mode). For example, as shown in method 1-1-2 in the fourth row from the top of the table in Figure 1, a flag that enables the selection of the coding mode in a particular mode (also called the conversion skip residual coding use specific mode flag) may be applied.

[0050] For example, during encoding, a conversion skip residual coding usage specific mode flag may be generated, which is flag information indicating whether to apply TS residual coding mode or non-TS residual coding mode in a particular mode. This conversion skip residual coding usage specific mode flag may then be encoded, and a bitstream containing the encoded data of this conversion skip residual coding usage specific mode flag may be generated.

[0051] Furthermore, during decoding, the encoded data of the conversion skip residual coding usage specific mode flag may be decoded, a conversion skip residual coding usage specific mode flag corresponding to the component identifier may be generated, and based on the conversion skip residual coding usage specific mode flag corresponding to the component identifier, the decoding mode of the encoded data of the coefficient data corresponding to the component identifier may be controlled to be either TS residual decoding mode or non-TS residual decoding mode.

[0052] For example, such a conversion skip residual coding usage specific mode flag may be set at a high level, such as in a sequence parameter. This makes it possible to switch between TS residual decoding mode and non-TS residual decoding mode in a specific mode.

[0053] Methods 1, 1-1, 1-1-1, and 1-1-2 described above will be explained later in the first embodiment.

[0054] <Correction of quantization parameters> Non-patent document 3 states that in a luminance conversion block to which luminance conversion skipping is applied, the PSNR (Peak Signal-to-Noise Ratio) may decrease when the quantization parameter QP < 4. As a countermeasure, a method is proposed to clip the quantization parameter QP applied to the luminance conversion block in the case of luminance conversion skipping to QP = 4 or QP = QpPrimeTsMin (minimum TSQP).

[0055] Similarly, applying conversion skipping to the color component may also reduce the PSNR, but conventionally, conversion skipping has not been applied to the color component, and therefore, this point has not been taken into consideration.

[0056] Therefore, for example, as shown in Method 1-2 in the fifth row from the top of the table in Figure 1, the quantization parameters may be corrected for each value of the component identifier in the case of conversion skipping. In other words, the quantization parameters may be corrected when conversion skipping is applied. Furthermore, such control may be performed for each component.

[0057] For example, in an image processing device, when a conversion skip flag corresponding to a component identifier indicates a conversion skip in image encoding, the device may include a quantization parameter correction unit that corrects the quantization parameters applied to the target conversion block corresponding to the component identifier, and a quantization unit that quantizes the target conversion block corresponding to the component identifier using the quantization parameters corrected by the quantization parameter correction unit.

[0058] For example, in a quantization coefficient generation method, if a conversion skip flag corresponding to a component identifier indicates a conversion skip in image encoding, the quantization parameters applied to the target conversion block corresponding to that component identifier may be corrected, and the target conversion block corresponding to that component identifier may be quantized using the corrected quantization parameters to generate the quantization coefficients corresponding to that component identifier.

[0059] For example, an image processing device may include a quantization parameter correction unit that corrects the quantization parameters applied to the processing target conversion block corresponding to a component identifier when a conversion skip flag corresponding to a component identifier indicates a conversion skip, and a dequantization unit that performs dequantization of the processing target conversion block corresponding to the component identifier using the quantization parameters corrected by the quantization parameter correction unit.

[0060] For example, in a coefficient data generation method, if a conversion skip flag corresponding to a component identifier indicates a conversion skip, the quantization parameters applied to the target conversion block corresponding to that component identifier may be corrected, and the target conversion block corresponding to that component identifier may be dequantized using the corrected quantization parameters to generate coefficient data corresponding to that component identifier.

[0061] By doing so, the reduction in PSNR can be suppressed in the encoder, decoder, or both.

[0062] For example, in encoding, if the conversion skip flag corresponding to a component identifier indicates a conversion skip, the larger of the minimum quantization parameter for that conversion skip and the quantization parameter corresponding to that component identifier may be set as the quantization parameter applied to the conversion block to be processed corresponding to that component identifier. If the conversion skip flag corresponding to that component identifier indicates a non-conversion skip where no conversion skip is performed, the quantization parameter corresponding to that component identifier may be set as the quantization parameter applied to the conversion block to be processed corresponding to that component identifier.

[0063] Furthermore, for example, in decoding, if the conversion skip flag corresponding to a component identifier indicates a conversion skip, the larger of the minimum quantization parameter for that conversion skip and the quantization parameter corresponding to that component identifier may be set as the quantization parameter applied to the target conversion block corresponding to that component identifier. If the conversion skip flag corresponding to that component identifier indicates a non-conversion skip where no conversion skip is performed, the quantization parameter corresponding to that component identifier may be set as the quantization parameter applied to the target conversion block corresponding to that component identifier.

[0064] By doing so, the reduction in PSNR can be suppressed in the encoder, decoder, or both.

[0065] For example, in decoding, if the encoded data of the conversion skip flag corresponding to the component identifier is decoded and the conversion skip flag corresponding to that component identifier indicates a conversion skip, the quantization parameters applied to the target conversion block corresponding to that component identifier may be corrected.

[0066] In this way, the correction of quantization parameters can be controlled based on the transformation skip flag corresponding to the encoded component identifier.

[0067] Methods 1 and 2 described above will be explained later in the second embodiment.

[0068] <Sharing context variables> Furthermore, as mentioned above, if a conversion skip is introduced for the color component, and the conversion block to which the conversion skip is applied for that color component is encoded using TS residual coding mode, then if independent context variables are used for the luminance component and the color component, it becomes necessary to add a new context variable for the color component. This increases the memory capacity required to hold the context variable, which could potentially increase hardware costs.

[0069] Therefore, in the case of conversion skipping, for example, as shown in method 1-3 in the sixth row from the top of the table in Figure 1, the context variable corresponding to each binIdx in the bin sequence of each syntax may be shared between the luminance component and the color component.

[0070] For example, in encoding, if the conversion process is skipped, the context variable may be shared between the encoding of the luminance component and the encoding of the chrominance component of the coefficient data.

[0071] Furthermore, for example, if the inverse transformation process is skipped during decoding, the context variable may be shared between the decoding of the encoded data for the luminance component of the coefficient data and the decoding of the encoded data for the chrominance component.

[0072] This approach helps to suppress the increase in memory size required to hold context variables, thereby reducing the increase in hardware costs.

[0073] Methods 1-3 described above will be explained later in the third embodiment.

[0074] <Sine code encoding and decoding mode control> Alternatively, as shown in method 1-4 in the seventh row from the top of the table in Figure 1, the encoding and decoding method of the sine code may be switched according to the conversion skip flag corresponding to the component identifier.

[0075] For example, as shown in method 1-4-1 in the eighth row from the top of the table in Figure 1, if conversion skipping is not performed, bypass coding and decoding may be applied to the coding and decoding of sine codes.

[0076] Alternatively, as shown in method 1-4-2 in the ninth row from the top of the table in Figure 1, in the case of conversion skipping, if the number of remaining context coding bins is greater than or equal to a threshold, context coding / decoding may be applied to the coding / decoding of the sine code; otherwise, bypass coding / decoding may be applied to the coding / decoding of the sine code.

[0077] By doing so, the reduction in encoding efficiency can be suppressed.

[0078] Methods 1-4, 1-4-1, and 1-4-2 described above will be explained later in the fourth embodiment.

[0079] <3. First Embodiment> <3-1. Image encoding device> The technology described above can be applied to any device, system, etc. For example, this technology can be applied to an image encoding device that encodes image data.

[0080] Figure 2 is a block diagram showing an example of the configuration of an image encoding device, which is one embodiment of an image processing device to which this technology is applied. The image encoding device 100 shown in Figure 2 is a device for encoding image data of moving images. For example, the image encoding device 100 implements the technology described in at least one of the above-mentioned non-patent documents and encodes image data of moving images in a manner compliant with the standards described in any of those documents.

[0081] Note that Figure 2 shows the main components such as the processing unit and data flow, and does not necessarily represent everything. In other words, there may be processing units in the image encoding device 100 that are not shown as blocks in Figure 2, and there may be processes and data flows that are not shown as arrows or other symbols in Figure 2. The same applies to other diagrams that explain the processing units within the image encoding device 100.

[0082] As shown in Figure 2, the image coding device 100 includes a control unit 101, a sorting buffer 111, an arithmetic unit 112, an orthogonal transform unit 113, a quantization unit 114, an encoding unit 115, and a storage buffer 116. The image coding device 100 also includes an inverse quantization unit 117, an inverse orthogonal transform unit 118, an arithmetic unit 119, an in-loop filter unit 120, a frame memory 121, a prediction unit 122, and a rate control unit 123.

[0083] <Department Head> The control unit 101 divides the video data held by the sorting buffer 111 into processing unit blocks (CU, PU, ​​conversion block, etc.) based on an external or pre-specified block size for the processing unit. The control unit 101 also determines the encoding parameters to be supplied to each block (header information Hinfo, prediction mode information Pinfo, conversion information Tinfo, filter information Finfo, etc.) based, for example, on RDO (Rate-Distortion Optimization). For example, the control unit 101 can set a conversion skip flag, etc.

[0084] Details of these encoding parameters will be described later. Once the control unit 101 determines the above encoding parameters, it supplies them to each block. Specifically, it does so as follows.

[0085] Header information Hinfo is supplied to each block. Prediction mode information Pinfo is supplied to the encoding unit 115 and the prediction unit 122. Transformation information Tinfo is supplied to the encoding unit 115, the orthogonal transform unit 113, the quantization unit 114, the inverse quantization unit 117, and the inverse orthogonal transform unit 118. Filter information Finfo is supplied to the in-loop filter unit 120.

[0086] <sort buffer> The image encoding device 100 receives each field (input image) of the moving image data in the order of playback (display order). The sorting buffer 111 acquires and holds (stores) each input image in the order of playback (display order). Based on the control of the control unit 101, the sorting buffer 111 sorts the input images in the order of encoding (decoding order) or divides them into processing units. The sorting buffer 111 supplies each processed input image to the arithmetic unit 112.

[0087] <Arithmetic section> The calculation unit 112 subtracts the predicted image P supplied by the prediction unit 122 from the image corresponding to the processing unit block supplied from the sorting buffer 111 to derive the predicted residual D, and supplies it to the orthogonal transformation unit 113.

[0088] <Orthogonal Transformation Unit> The orthogonal transformation unit 113 receives the predicted residual supplied from the calculation unit 112 and the transformation information Tinfo supplied from the control unit 101 as inputs, and performs an orthogonal transformation on the predicted residual based on the transformation information Tinfo to derive the transformation coefficient Coeff. The orthogonal transformation unit 113 supplies the obtained transformation coefficient to the quantization unit 114.

[0089] <Quantization section> The quantization unit 114 receives the conversion coefficients supplied from the orthogonal transformation unit 113 and the conversion information Tinfo supplied from the control unit 101 as input, and scales (quantizes) the conversion coefficients based on the conversion information Tinfo. The rate of this quantization is controlled by the rate control unit 123. The quantization unit 114 supplies the quantized conversion coefficients (also called the quantization conversion coefficient level) level obtained by this quantization to the encoding unit 115 and the inverse quantization unit 117.

[0090] <Encoding section> The encoding unit 115 receives the quantization conversion coefficient levels supplied from the quantization unit 114, various encoding parameters supplied from the control unit 101 (header information Hinfo, prediction mode information Pinfo, conversion information Tinfo, filter information Finfo, etc.), filter information such as filter coefficients supplied from the in-loop filter unit 120, and information on the optimal prediction mode supplied from the prediction unit 122 as input.

[0091] The encoding unit 115 performs entropy encoding (reversible encoding), such as CABAC (Context-based Adaptive Binary Arithmetic Code) or CAVLC (Context-based Adaptive Variable Length Code), on the quantization conversion coefficient level to generate a bit sequence (encoded data).

[0092] Furthermore, the encoding unit 115 derives residual information Rinfo from the quantization conversion coefficient level, encodes the residual information Rinfo, and generates a bit sequence.

[0093] Furthermore, the encoding unit 115 includes information about the filter supplied from the in-loop filter unit 120 in the filter information Finfo, and information about the optimal prediction mode supplied from the prediction unit 122 in the prediction mode information Pinfo. Then, the encoding unit 115 encodes the various encoding parameters mentioned above (header information Hinfo, prediction mode information Pinfo, transformation information Tinfo, filter information Finfo, etc.) and generates a bit sequence.

[0094] Furthermore, the encoding unit 115 multiplexes the bit sequences of the various information generated as described above to generate encoded data. The encoding unit 115 supplies this encoded data to the storage buffer 116.

[0095] <Storage buffer> The storage buffer 116 temporarily holds the encoded data obtained by the encoding unit 115. At a predetermined timing, the storage buffer 116 outputs the stored encoded data to the outside of the image encoding device 100, for example, as a bitstream. For example, this encoded data is transmitted to the decoding side via any recording medium, any transmission medium, any information processing device, etc. In other words, the storage buffer 116 also functions as a transmission unit that transmits encoded data (bitstream).

[0096] <Inverse quantization part> The inverse quantization unit 117 performs processing related to inverse quantization. For example, the inverse quantization unit 117 takes the quantization conversion coefficient level level supplied from the quantization unit 114 and the conversion information Tinfo supplied from the control unit 101 as input, and scales (inverse quantizes) the value of the quantization conversion coefficient level based on the conversion information Tinfo. This inverse quantization is the reverse process of the quantization performed in the quantization unit 114. The inverse quantization unit 117 supplies the conversion coefficient Coeff_IQ obtained by such inverse quantization to the inverse orthogonal transformation unit 118. Since the inverse quantization unit 117 is the same as the inverse quantization unit on the decoding side (described later), the explanation given for the decoding side (described later) can be applied to the inverse quantization unit 117.

[0097] <Inverse orthogonal transformation section> The inverse orthogonal transform unit 118 performs processing related to the inverse orthogonal transform. For example, the inverse orthogonal transform unit 118 takes the transformation coefficients supplied from the inverse quantization unit 117 and the transformation information Tinfo supplied from the control unit 101 as input, and performs an inverse orthogonal transform on the transformation coefficients based on the transformation information Tinfo to derive the predicted residual D'. This inverse orthogonal transform is the inverse process of the orthogonal transform performed in the orthogonal transform unit 113. The inverse orthogonal transform unit 118 supplies the predicted residual obtained by this inverse orthogonal transform to the calculation unit 119. Since the inverse orthogonal transform unit 118 is the same as the inverse orthogonal transform unit on the decoding side (described later), the explanation given for the decoding side (described later) can be applied to the inverse orthogonal transform unit 118.

[0098] <Arithmetic section> The calculation unit 119 receives the predicted residual D' supplied from the inverse orthogonal transform unit 118 and the predicted image P supplied from the prediction unit 122 as input. The calculation unit 119 adds the predicted residual and the predicted image corresponding to the predicted residual to derive a locally decoded image. The calculation unit 119 supplies the derived locally decoded image to the in-loop filter unit 120 and the frame memory 121.

[0099] <In-loop filter section> The in-loop filter unit 120 performs processing related to in-loop filtering. For example, the in-loop filter unit 120 receives the locally decoded image supplied from the calculation unit 119, the filter information Finfo supplied from the control unit 101, and the input image (original image) supplied from the sorting buffer 111 as input. The information input to the in-loop filter unit 120 is arbitrary, and other information may also be input. For example, if necessary, information such as prediction mode, motion information, code amount target value, quantization parameter QP, picture type, and block (CU, CTU, etc.) may be input to the in-loop filter unit 120.

[0100] The in-loop filter unit 120 performs appropriate filtering on the locally decoded image based on its filter information Finfo. The in-loop filter unit 120 also uses the input image (original image) and other input information in its filtering process as needed.

[0101] For example, the in-loop filter unit 120 can apply four in-loop filters in this order, as described in Non-Patent Document 1: a bilateral filter, a deblocking filter (DBF), an adaptive offset filter (SAO), and an adaptive loop filter (ALF). Note that the choice of which filters to apply and in what order is arbitrary and can be selected as appropriate.

[0102] Of course, the filtering process performed by the in-loop filter unit 120 is arbitrary and not limited to the examples described above. For example, the in-loop filter unit 120 may apply a Wiener filter or the like.

[0103] The in-loop filter unit 120 supplies the filtered locally decoded image to the frame memory 121. When transmitting information about the filter, such as filter coefficients, to the decoding side, the in-loop filter unit 120 supplies this filter information to the encoding unit 115.

[0104] <Frame Memory> The frame memory 121 performs processing related to the storage of image data. For example, the frame memory 121 takes locally decoded images supplied from the arithmetic unit 119 and filtered locally decoded images supplied from the in-loop filter unit 120 as input and stores (stores) them. The frame memory 121 also uses these locally decoded images to reconstruct decoded images for each picture unit and stores them (stores them in a buffer within the frame memory 121). The frame memory 121 supplies the decoded images (or a part thereof) to the prediction unit 122 upon request from the prediction unit 122.

[0105] <Prediction Section> The prediction unit 122 performs processing related to the generation of a predicted image. For example, the prediction unit 122 takes prediction mode information Pinfo supplied from the control unit 101, the input image (original image) supplied from the sorting buffer 111, and the decoded image (or a part thereof) read from the frame memory 121 as input. The prediction unit 122 uses the prediction mode information Pinfo and the input image (original image) to perform prediction processing such as inter-prediction or intra-prediction, makes predictions by referencing the decoded image as a reference image, performs motion compensation processing based on the prediction results, and generates a predicted image. The prediction unit 122 supplies the generated predicted image to the calculation unit 112 and the calculation unit 119. The prediction unit 122 also supplies information about the prediction mode selected by the above processing, i.e., the optimal prediction mode, to the encoding unit 115 as needed.

[0106] <Rate Control Section> The rate control unit 123 performs processing related to rate control. For example, the rate control unit 123 controls the rate of the quantization operation of the quantization unit 114 based on the amount of encoded data stored in the storage buffer 116, so as to prevent overflow or underflow.

[0107] <Encoding section> Figure 3 is a block diagram showing an example of the main configuration of the encoding unit 115 in Figure 2. As shown in Figure 3, the encoding unit 115 includes a conversion mode information encoding unit 150, a control unit 151, a selection unit 152, a TS residual encoding unit 153, a non-TS residual encoding unit 154, and a selection unit 155.

[0108] The transformation mode information encoding unit 150 performs processing related to the encoding of transformation mode information (transform_mode). This transformation mode information is information related to the mode of the transformation process performed by the orthogonal transformation unit 113. For example, the transformation mode information may include a transformation skip flag (transform_skip_flag[xTbY][yTbY][cIdx]) and an identifier related to the primary transformation (mts_idx[xTbY][yTbY][cIdx]).

[0109] For example, the conversion mode information encoding unit 150 can acquire conversion mode information supplied from the control unit 101. Furthermore, the conversion mode information encoding unit 150 can encode the acquired conversion mode information and generate encoded conversion mode information data. In addition, the conversion mode information encoding unit 150 can supply the generated encoded conversion mode information data to the storage buffer 116 (i.e., provide it to the decoding side).

[0110] The control unit 151 performs processing related to the control of the encoding mode. For example, the control unit 151 may obtain transformation mode information ((transform_skip_flag[xTbY][yTbY][cIdx]) and mts_idx[xTbY][yTbY][cIdx], etc.) and component identifiers (cIdx) supplied from the control unit 101. Furthermore, the control unit 151 can switch between TS residual encoding mode and non-TS residual encoding mode as the encoding mode for coefficient data (quantization coefficients) by controlling the selection of the selection unit 152 and selection unit 155 based on the transformation skip flag corresponding to the component identifier. For example, if transformation skipping is applied, the control unit 151 connects the selection unit 152 and selection unit 155 to the TS residual encoding unit 153. Alternatively, if transformation skipping is not applied (if transformation processing is performed), the control unit 151 connects the selection unit 152 and selection unit 155 to the non-TS residual encoding unit 154.

[0111] The selection unit 152 performs processing related to the selection of the destination for the coefficient data (quantization coefficients). For example, the selection unit 152 may acquire the quantization coefficients supplied from the quantization unit 114. The selection unit 152 may also supply the acquired quantization coefficients to the TS residual coding unit 153 or the non-TS residual coding unit 154 (whichever is specified by the control unit 151) according to the control unit 151. For example, if a conversion skip is applied, the selection unit 152 supplies the quantization coefficients to the TS residual coding unit 153. Alternatively, if a conversion skip is not applied (i.e., a conversion process is performed), the selection unit 152 supplies the quantization coefficients to the non-TS residual coding unit 154.

[0112] The TS residual coding unit 153 performs processing related to the TS residual coding mode. For example, the TS residual coding unit 153 can acquire quantization coefficients supplied from the selection unit 152. The TS residual coding unit 153 can also encode the acquired quantization coefficients in the TS residual coding mode. This TS residual coding mode is an encoding mode for cases where the conversion process is skipped. For example, the encoding in the TS residual coding mode is optimized for coefficient data where the conversion process is skipped, such as when scanning from the DC component to the high-frequency component of the coefficient data, or when the position information of the last coefficient is not transmitted to the decoding side. More specific methods for the TS residual coding mode are described in Non-Patent Literature 4, etc. The TS residual coding unit 153 can thus encode the quantization coefficients and generate encoded data of the quantization coefficients. The TS residual coding unit 153 can thus supply the generated encoded data to the selection unit 155.

[0113] The non-TS residual coding unit 154 performs processing related to the non-TS residual coding mode. For example, the non-TS residual coding unit 154 can acquire quantization coefficients supplied from the selection unit 152. The non-TS residual coding unit 154 can also encode the acquired quantization coefficients in the non-TS residual coding mode. This non-TS residual coding mode is an encoding mode for when transformation processing is performed. For example, the encoding in the non-TS residual coding mode is optimized for transformed coefficient data, such as scanning from the last coefficient of the coefficient data toward the DC component, or transmitting positional information of the last coefficient to the decoding side. More specific methods for the non-TS residual coding mode are described in Non-Patent Document 4, etc. The non-TS residual coding unit 154 can thus encode the quantization coefficients and generate encoded data of the quantization coefficients. The non-TS residual coding unit 154 can thus supply the generated encoded data to the selection unit 155.

[0114] The selection unit 155 performs processing related to the selection of the source of the encoded data. For example, the selection unit 155 may acquire encoded data supplied from the TS residual encoding unit 153 or the non-TS residual encoding unit 154 (whichever is specified by the control unit 151). For example, if a conversion skip is applied, the selection unit 155 acquires encoded data supplied from the TS residual encoding unit 153. Alternatively, for example, if a conversion skip is not applied (if a conversion process is performed), the selection unit 155 acquires encoded data supplied from the non-TS residual encoding unit 154. The selection unit 155 may then supply the acquired encoded data to the storage buffer 116 (i.e., provide it to the decoding side).

[0115] <Skip conversion for each component> The image encoding device 100 can encode a color image that has luminance and color components. Furthermore, the image encoding device 100 can encode not only the luminance component but also the color component by skipping (omitting) the conversion process that converts the residual between the image and its predicted image into coefficient data.

[0116] For example, the control unit 101 can set a transform_skip_flag for each component, indicating whether or not to apply a transform skip. A component can be identified by a component identifier (cIdx). In other words, the control unit 101 can set the value of the transform_skip_flag to correspond to the value of the component identifier.

[0117] In other words, the control unit 101 can generate a transformation skip flag (transform_skip_flag[xTbY][yTbY][cIdx]) that indicates whether or not to skip the transformation process that converts the residual between the image to be encoded and its predicted image into coefficient data for each component (cIdx). Furthermore, the transformation mode information encoding unit 150 of the encoding unit 115 can encode the transformation skip flag generated by the control unit 101 and generate encoded data for the transformation skip flag. In addition, the storage buffer 116 can generate a bitstream containing the encoded data for the transformation skip flag generated by the transformation mode information encoding unit 150 and output it to the outside of the image encoding device 100.

[0118] In this way, the control unit 101 can, for example, set a conversion skip flag for the color component and apply conversion skipping to the color component. Furthermore, the image encoding device 100 can provide the generated conversion skip flag (i.e., the conversion skip setting for each component) to the decoding side. Therefore, when conversion skipping is applied to the color component, the decoding side can correctly decode the bitstream. Consequently, the reduction in encoding efficiency can be suppressed compared to the case where conversion skipping is only applicable to the luminance component.

[0119] Furthermore, the image encoding device 100 may control the color component conversion process based on the conversion skip settings for each component using the conversion skip flags described above. For example, the orthogonal transformation unit 113 may control the conversion skip for each component based on the conversion skip flags generated by the control unit 101. By doing so, the orthogonal transformation unit 113 can apply conversion skips to, for example, the color component conversion process.

[0120] Furthermore, as described above, the control unit 101 can set conversion skips for each component, thereby enabling conversion skips for color components independently of luminance components. Therefore, the control unit 101 can set conversion skips for color components depending on whether conversion skips are effective for color components. As a result, the orthogonal transformation unit 113 can perform conversion skips for color components independently of luminance components. By doing so, the reduction in encoding efficiency can be suppressed compared to applying the conversion skip setting for luminance components to color components.

[0121] <Controlling the encoding mode> Furthermore, the encoding unit 115 may control the encoding mode of the coefficient data corresponding to a component identifier based on a conversion skip flag corresponding to that component identifier. For example, the control unit 151, selection unit 152, and selection unit 155 of the encoding unit 115 may control whether to use TS residual encoding mode or non-TS residual encoding mode based on the conversion skip flag. For example, the TS residual encoding unit 153 and the non-TS residual encoding unit 154 may encode the coefficient data corresponding to the component identifier using the encoding mode set in this way, and generate encoded data of that coefficient data.

[0122] In other words, multiple coding modes with different characteristics are provided as candidates, and the coding unit 115 selects and applies a coding mode from among these candidates based on a conversion skip flag corresponding to a component identifier. That is, the coding unit 115 codes the coefficient data using the selected coding mode. By doing this, the coding unit 115 can apply a coding mode with characteristics that are more suitable for the conversion skip setting among the coding modes with different characteristics, thus suppressing the reduction in coding efficiency compared to when coding is performed using a single coding mode.

[0123] <Image encoding process flow> Next, we will explain the flow of each process performed by the image encoding device 100 as described above. First, we will explain an example of the image encoding process flow with reference to the flowchart in Figure 4.

[0124] When the image encoding process is started, in step S101, the sorting buffer 111 is controlled by the control unit 101 to sort the order of the frames of the input video data from display order to encoding order.

[0125] In step S102, the control unit 101 sets processing units (performs block division) for the input image held by the sorting buffer 111.

[0126] In step S103, the control unit 101 determines (sets) the encoding parameters for the input image held by the sorting buffer 111.

[0127] In step S104, the control unit 101 generates transformation mode information (transform_mode) for the transformation block corresponding to the component identifier (cIdx).

[0128] In step S105, the prediction unit 122 performs prediction processing and generates predicted images, etc., for the optimal prediction mode. For example, in this prediction processing, the prediction unit 122 performs intra-prediction to generate predicted images, etc., for the optimal intra-prediction mode, performs inter-prediction to generate predicted images, etc., for the optimal inter-prediction mode, and selects the optimal prediction mode from among them based on cost function values, etc.

[0129] In step S106, the calculation unit 112 calculates the difference between the input image and the predicted image of the optimal mode selected by the prediction process in step S105. In other words, the calculation unit 112 generates a predicted residual D between the input image and the predicted image. The predicted residual D obtained in this way has a reduced data size compared to the original image data. Therefore, the amount of data can be compressed compared to encoding the image as is.

[0130] In step S107, the orthogonal transformation unit 113 performs an orthogonal transformation on the predicted residual D generated by the processing in step S106, according to the transformation mode information generated in step S104, and derives the transformation coefficient Coeff.

[0131] In step S108, the quantization unit 114 quantizes the conversion coefficient Coeff obtained in step S107 by using the quantization parameters calculated by the control unit 101, and derives the quantization conversion coefficient level level.

[0132] In step S109, the inverse quantization unit 117 inversely quantizes the quantization conversion coefficient level generated by the processing in step S108 using characteristics corresponding to the quantization characteristics of step S108, and derives the conversion coefficient Coeff_IQ.

[0133] In step S110, the inverse orthogonal transform unit 118 performs an inverse orthogonal transform on the transformation coefficient Coeff_IQ obtained by the processing in step S109, in accordance with the transformation mode information generated in step S104, in a manner corresponding to the orthogonal transform process in step S107, and derives the predicted residual D'. Since this inverse orthogonal transform process is the same as the inverse orthogonal transform process performed on the decoding side (described later), the explanation given for the decoding side (described later) can be applied to the inverse orthogonal transform process in step S110.

[0134] In step S111, the calculation unit 119 generates a locally decoded decoded image by adding the predicted image obtained by the prediction process in step S105 to the predicted residual D' derived by the processing in step S110.

[0135] In step S112, the in-loop filter unit 120 performs in-loop filtering on the locally decoded decoded image derived from the processing in step S111.

[0136] In step S113, the frame memory 121 stores the locally decoded decoded images derived from the processing in step S111, and the locally decoded decoded images that were filtered in step S112.

[0137] In step S114, the encoding unit 115 encodes the quantization conversion coefficient level obtained in step S108 and the conversion mode information generated in step S104. For example, the encoding unit 115 encodes the quantization conversion coefficient level, which is information about the image, using arithmetic encoding or the like to generate encoded data. At this time, the encoding unit 115 also encodes various encoding parameters (header information Hinfo, prediction mode information Pinfo, and conversion information Tinfo). Furthermore, the encoding unit 115 derives residual information RInfo from the quantization conversion coefficient level and encodes the residual information RInfo.

[0138] In step S115, the storage buffer 116 stores the encoded data obtained in this way and outputs it to the outside of the image encoding device 100 as, for example, a bitstream. This bitstream is transmitted to the decoding side, for example, via a transmission line or recording medium.

[0139] In step S116, the rate control unit 123 performs rate control as necessary.

[0140] When the process in step S116 is completed, the image encoding process is finished.

[0141] <Encoding process flow> Next, referring to the flowchart in Figure 5, we will explain an example of the encoding process performed in step S114 of Figure 4.

[0142] When the encoding process is started, the transformation mode information encoding unit 150 of the encoding unit 115 encodes the transformation mode information (transform_mode) of the transformation block corresponding to the component identifier cIdx in step S151.

[0143] In step S152, the control unit 151 derives Condition1 using the following equation (1). That is, the control unit 151 generates Condition1 using the transform_skip_flag[cIdx] corresponding to the component. Note that the transform_skip_flag[cIdx] may also include the coordinates (xTbY, yTbY) of the block to be processed (for example, trnform_skip_flag[xTbY][yTbY][cIdx]), but for simplicity, this is omitted here. It will also be omitted as appropriate below.

[0144] TIFF0007863977000001.tif18132

[0145] In step S153, the control unit 151 determines whether Condition1 is true or not. If Condition1 is true, the transform skip flag (transform_skip_flag[cIdx]) corresponding to the component is true (IS_SKIP). Therefore, the selection unit 152 and the selection unit 155 connect to the TS residual coding unit 153 according to the control unit 151. The process then proceeds to step S154.

[0146] In step S154, the TS residual coding unit 153 encodes the quantization coefficients using the TS residual coding mode to generate coded data. Once the coded data is generated, the coding process ends.

[0147] Furthermore, in step S153, if Condition1 is false, the transform_skip_flag[cIdx] corresponding to the component is false. Therefore, the selection units 152 and 155 connect to the non-TS residual coding unit 154 according to the control of their control unit 151. This causes the process to proceed to step S155.

[0148] In step S155, the non-TS residual coding unit 154 encodes the quantization coefficients using the non-TS residual coding mode to generate coded data. Once the coded data is generated, the coding process is completed.

[0149] As described above, by performing each process, the image encoding device 100 can apply the TS residual encoding mode to the color components as well, thereby suppressing a reduction in encoding efficiency.

[0150] In the above explanation, it was described that whether or not a transformation is skipped is notified by the transformation skip flag (transform_skip_flag), but this is not limited to this, and it may also be notified as one mode of the identifier mts_idx related to the primary transformation. The identifier mts_idx is an identifier that indicates the horizontal and vertical transformation types of the primary transformation. In that case, the control unit 151 can derive Condition1 as shown in the following equation (2).

[0151] TIFF0007863977000002.tif21118

[0152] <3-2. Image Decoding Device> Figure 6 is a block diagram showing an example of the configuration of an image decoding device, which is one embodiment of an image processing device to which this technology is applied. The image decoding device 200 shown in Figure 6 is a device that decodes encoded data in which the predicted residual between an image and its predicted image is encoded, such as AVC or HEVC. For example, the image decoding device 200 can implement the technology described in the above-mentioned non-patent literature and decode encoded data in which the image data of a moving image is encoded in a manner compliant with the standards described in any of those documents. For example, the image decoding device 200 can decode encoded data (bitstream) generated by the above-mentioned image encoding device 100.

[0153] Note that Figure 6 shows the main components such as the processing unit and data flow, and does not necessarily represent everything. In other words, there may be processing units in the image decoding device 200 that are not shown as blocks in Figure 6, and there may be processes and data flows that are not shown as arrows or other symbols in Figure 6. The same applies to other figures that explain the processing units within the image decoding device 200.

[0154] In Figure 6, the image decoding device 200 includes a storage buffer 211, a decoding unit 212, an inverse quantization unit 213, an inverse orthogonal transform unit 214, an arithmetic unit 215, an in-loop filter unit 216, a sorting buffer 217, a frame memory 218, and a prediction unit 219. The prediction unit 219 includes an intra-prediction unit and an inter-prediction unit (not shown).

[0155] <Storage buffer> The storage buffer 211 acquires and stores (stores) the bitstream input to the image decoding device 200. At a predetermined timing, or when predetermined conditions are met, the storage buffer 211 extracts the encoded data contained in the stored bitstream and supplies it to the decoding unit 212.

[0156] <Decryption section> The decoding unit 212 performs processing related to image decoding. For example, the decoding unit 212 takes encoded data supplied from the storage buffer 211 as input, and, in accordance with the definition of the syntax table, entropy decodes (reversible decoding) the syntax value of each syntax element from the bit sequence to derive the parameters.

[0157] The parameters derived from the syntax elements and their syntax values ​​include, for example, header information Hinfo, prediction mode information Pinfo, transformation information Tinfo, residual information Rinfo, and filter information Finfo. In other words, the decoding unit 212 parses (analyzes and obtains) this information from the bitstream. This information is described below.

[0158] <Header Information Hinfo> The header information Hinfo includes, for example, header information such as VPS (Video Parameter Set), SPS (Sequence Parameter Set), PPS (Picture Parameter Set), PH (Picture Header), and SH (Slice Header). The header information Hinfo includes information that defines, for example, the image size (width PicWidth, height PicHeight), bit depth (luminance bitDepthY, chrominance bitDepthC), chrominance array type ChromaArrayType, maximum CU size MaxCUSize / minimum CUSize, maximum depth MaxQTDepth / minimum depth MinQTDepth for quad-tree partitioning, maximum depth MaxBTDepth / minimum depth MinBTDepth for binary-tree partitioning, maximum value MaxTSSize (also called maximum conversion skip block size), and on / off flags (also called enable flags) for each encoding tool.

[0159] For example, the on / off flags for encoding tools included in the header information Hinfo include the on / off flags related to the following transformation and quantization processes. Note that the on / off flag for an encoding tool can also be interpreted as a flag indicating whether or not the syntax related to that encoding tool exists in the encoded data. Furthermore, a value of 1 (true) for the on / off flag indicates that the encoding tool is usable, and a value of 0 (false) indicates that the encoding tool is unavailable. Note that the interpretation of the flag values ​​may be reversed.

[0160] Cross-Component Prediction (CCP_enabled_flag): This flag indicates whether cross-component prediction (CCP, also known as CC prediction) is enabled or disabled. For example, a flag value of "1" (true) indicates that it is enabled, while a flag value of "0" (false) indicates that it is disabled.

[0161] This CCP is also known as inter-component linear prediction (CCLM or CCLMP).

[0162] <Predictive Mode Information Pinfo> The prediction mode information Pinfo includes, for example, information such as the size information of the PB (prediction block) to be processed (PBSize, prediction block size), intra prediction mode information IPinfo, and motion prediction information MVinfo.

[0163] The intra-prediction mode information IPinfo includes, for example, prev_intra_luma_pred_flag, mpm_idx, rem_intra_pred_mode in the JCTVC-W1005, 7.3.8.5 Coding Unit syntax, and the luminance intra-prediction mode IntraPredModeY derived from that syntax.

[0164] Furthermore, the intra-prediction mode information IPinfo includes, for example, the inter-component prediction flag (ccp_flag(cclmp_flag)), the multi-class linear prediction mode flag (mclm_flag), the color difference sample location type identifier (chroma_sample_loc_type_idx), the color difference MPM identifier (chroma_mpm_idx), and the luminance intra-prediction mode (IntraPredModeC) derived from these syntaxes.

[0165] The inter-component prediction flag (ccp_flag(cclmp_flag)) is a flag indicating whether or not to apply inter-component linear prediction. For example, ccp_flag==1 indicates that inter-component prediction will be applied, and ccp_flag==0 indicates that inter-component prediction will not be applied.

[0166] The multi-class linear prediction mode flag (mclm_flag) is information about the linear prediction mode (linear prediction mode information). More specifically, the multi-class linear prediction mode flag (mclm_flag) is flag information that indicates whether or not to use multi-class linear prediction mode. For example, if it is "0", it indicates that it is a one-class mode (single-class mode) (e.g., CCLMP), and if it is "1", it indicates that it is a two-class mode (multi-class mode) (e.g., MCLMP).

[0167] The chroma_sample_loc_type_idx is an identifier that identifies the type of pixel position of a chroma difference component (also known as the chroma_sample_loc_type).

[0168] This color difference sample location type identifier (chroma_sample_loc_type_idx) is transmitted as information about the pixel location of the color difference component (chroma_sample_loc_info()), which is stored within it.

[0169] The chroma_mpm_idx identifier is an identifier that indicates which prediction mode candidate from the intraPredModeCandListC chroma_mpm_idx is designated as the intraPredMode prediction mode.

[0170] The motion prediction information MVinfo includes information such as merge_idx, merge_flag, inter_pred_idc, ref_idx_LX, mvp_lX_flag, X={0,1}, and mvd (see, for example, JCTVC-W1005, 7.3.8.6 Prediction Unit Syntax).

[0171] Of course, the information included in the predictive mode information Pinfo is arbitrary, and other information may be included as well.

[0172] <Conversion Information Tinfo> The conversion information Tinfo may include, for example, the following information. Of course, the information included in the conversion information Tinfo is arbitrary, and it may include information other than these.

[0173] The width (TBWSize) and height (TBHSize) of the transformation block to be processed (or the base-2 logarithms of each TBWSize and TBHSize, log2TBWSize and log2TBHSize). Transformation skip flag (ts_flag): A flag indicating whether or not to skip the (inverse) primary and (inverse) secondary transformations. Scan identifier (scanIdx) Quantization parameter (qp) Quantization matrix (scaling_matrix (e.g., JCTVC-W1005, 7.3.4 Scaling list data syntax))

[0174] <Residual Information Rinfo> The residual information Rinfo (see, for example, 7.3.8.11 Residual Coding syntax in JCTVC-W1005) includes syntax such as the following:

[0175] cbf(coded_block_flag): residual data presence / absence flag last_sig_coeff_x_pos: Last non-zero coefficient X coordinate last_sig_coeff_y_pos: Last non-zero coefficient Y coordinate coded_sub_block_flag: Subblock non-zero coefficient flag sig_coeff_flag: Flag indicating presence or absence of non-zero coefficients gr1_flag: A flag indicating whether the level of non-zero coefficients is greater than 1 (also called the GR1 flag). gr2_flag: A flag indicating whether the level of non-zero coefficients is greater than 2 (also called the GR2 flag). sign_flag: The sign indicating the positive or negative sign of non-zero coefficients (also called the sine sign). coeff_abs_level_remaining: Residual level of non-zero coefficients (also called non-zero coefficient residual level) etc.

[0176] Of course, the information included in the residual information Rinfo is arbitrary, and it is also possible to include information other than this.

[0177] <Filter Information Finfo> The filter information Finfo includes, for example, control information related to each of the following filter processes:

[0178] Control information regarding deblocking filters (DBFs) Control information regarding Pixel Adaptive Offset (SAO) Control information for Adaptive Loop Filters (ALF) Other control information related to linear and nonlinear filters

[0179] More specifically, this includes information such as the picture to which each filter applies, the area within the picture, filter On / Off control information for each CU, and filter On / Off control information related to slice and tile boundaries. Of course, the information included in the filter information Finfo is arbitrary, and other information may also be included.

[0180] Returning to the description of the decoding unit 212, the decoding unit 212 refers to the residual information Rinfo and derives the quantization transformation coefficient level level for each coefficient position within each transformation block. The decoding unit 212 supplies this quantization transformation coefficient level to the inverse quantization unit 213.

[0181] Furthermore, the decoding unit 212 supplies the parsed header information Hinfo, prediction mode information Pinfo, conversion information Tinfo, and filter information Finfo to each block. Specifically, this is done as follows:

[0182] Header information Hinfo is supplied to the inverse quantization unit 213, the inverse orthogonal transform unit 214, the prediction unit 219, and the in-loop filter unit 216. Prediction mode information Pinfo is supplied to the inverse quantization unit 213 and the prediction unit 219. Transformation information Tinfo is supplied to the inverse quantization unit 213 and the inverse orthogonal transform unit 214. Filter information Finfo is supplied to the in-loop filter unit 216.

[0183] Of course, the above example is just one example and is not limited to this example. For example, each encoding parameter may be supplied to any processing unit. Also, other information may be supplied to any processing unit.

[0184] <Inverse quantization part> The inverse quantization unit 213 performs processing related to inverse quantization. For example, the inverse quantization unit 213 takes the conversion information Tinfo and the quantization conversion coefficient level level supplied from the decoding unit 212 as input, scales (inverse quantizes) the value of the quantization conversion coefficient level based on the conversion information Tinfo, and derives the conversion coefficient Coeff_IQ after inverse quantization.

[0185] This inverse quantization is performed as the reverse process of quantization by the quantization unit 114. Furthermore, this inverse quantization is the same process as the inverse quantization performed by the inverse quantization unit 117. In other words, the inverse quantization unit 117 performs the same process (inverse quantization) as the inverse quantization unit 213.

[0186] The inverse quantization unit 213 supplies the derived conversion coefficient Coeff_IQ to the inverse orthogonal transformation unit 214.

[0187] <Inverse orthogonal transformation section> The inverse orthogonal transform unit 214 performs processing related to the inverse orthogonal transform. For example, the inverse orthogonal transform unit 214 takes the transformation coefficients Coeff_IQ supplied from the inverse quantization unit 213 and the transformation information Tinfo supplied from the decoding unit 212 as input, and performs an inverse orthogonal transform process (inverse transform process) on the transformation coefficients based on the transformation information Tinfo to derive the predicted residual D'.

[0188] This inverse orthogonal transformation is performed as the reverse of the orthogonal transformation performed by the orthogonal transformation unit 113. Furthermore, this inverse orthogonal transformation is the same process as the inverse orthogonal transformation performed by the inverse orthogonal transformation unit 118. In other words, the inverse orthogonal transformation unit 118 performs the same process (inverse orthogonal transformation) as the inverse orthogonal transformation unit 214.

[0189] The inverse orthogonal transform unit 214 supplies the derived predicted residual D' to the calculation unit 215.

[0190] <Arithmetic section> The calculation unit 215 performs processing related to the addition of information about the image. For example, the calculation unit 215 takes the predicted residual supplied from the inverse orthogonal transform unit 214 and the predicted image supplied from the prediction unit 219 as input. The calculation unit 215 adds the predicted residual and the predicted image (predicted signal) corresponding to that predicted residual to derive a locally decoded image.

[0191] The calculation unit 215 supplies the derived local decoded image to the in-loop filter unit 216 and the frame memory 218.

[0192] <In-loop filter section> The in-loop filter unit 216 performs processing related to in-loop filtering. For example, the in-loop filter unit 216 receives the locally decoded image supplied from the calculation unit 215 and the filter information Finfo supplied from the decoding unit 212 as input. Note that the information input to the in-loop filter unit 216 is arbitrary, and information other than this may be input.

[0193] The in-loop filter unit 216 performs appropriate filtering on the locally decoded image based on its filter information Finfo.

[0194] For example, the in-loop filter section 216 applies four in-loop filters in this order: a bilateral filter, a deblocking filter (DBF), an adaptive offset filter (SAO), and an adaptive loop filter (ALF). Note that the order in which these filters are applied is arbitrary and can be selected as appropriate.

[0195] The in-loop filter unit 216 performs filtering corresponding to the filtering performed by the encoding side (for example, the in-loop filter unit 120 of the image encoding device 100). Of course, the filtering performed by the in-loop filter unit 216 is arbitrary and not limited to the example described above. For example, the in-loop filter unit 216 may apply a Wiener filter or the like.

[0196] The in-loop filter unit 216 supplies the filtered locally decoded images to the sorting buffer 217 and the frame memory 218.

[0197] <sort buffer> The sorting buffer 217 receives locally decoded images supplied from the in-loop filter unit 216 as input and holds (stores) them. The sorting buffer 217 uses these locally decoded images to reconstruct decoded images for each picture unit and holds (stores in the buffer). The sorting buffer 217 sorts the obtained decoded images from the decoding order to the playback order. The sorting buffer 217 outputs the sorted group of decoded images as moving image data to the outside of the image decoding device 200.

[0198] <Frame Memory> The frame memory 218 performs processing related to the storage of image data. For example, the frame memory 218 takes locally decoded images supplied by the arithmetic unit 215 as input, reconstructs the decoded images for each picture unit, and stores them in a buffer within the frame memory 218.

[0199] Furthermore, the frame memory 218 receives locally decoded images that have been filtered by the in-loop filter unit 216 as input, reconstructs the decoded image for each picture unit, and stores it in a buffer within the frame memory 218. The frame memory 218 then supplies the decoded image (or a part thereof) stored in it as a reference image to the prediction unit 219 as appropriate.

[0200] The frame memory 218 may also store header information Hinfo, prediction mode information Pinfo, conversion information Tinfo, filter information Finfo, etc., related to the generation of the decoded image.

[0201] <Encoding section> Figure 7 is a block diagram showing an example of the main configuration of the decoding unit 212 in Figure 6. As shown in Figure 7, the decoding unit 212 includes a conversion mode information decoding unit 250, a control unit 251, a selection unit 252, a TS residual decoding unit 253, a non-TS residual decoding unit 254, and a selection unit 255.

[0202] The transformation mode information decoding unit 250 performs processing related to the decoding of the encoded data of the transformation mode information (transform_mode). This transformation mode information is information related to the mode of the inverse transformation process performed by the inverse orthogonal transformation unit 214. For example, the transformation mode information may include a transformation skip flag (transform_skip_flag[xTbY][yTbY][cIdx]) and an identifier related to the primary transformation (mts_idx[xTbY][yTbY][cIdx]).

[0203] For example, the conversion mode information decoding unit 250 can acquire encoded data supplied from the storage buffer 211. The conversion mode information decoding unit 250 can also decode the acquired encoded data and generate conversion mode information and component identifiers (cIdx). Furthermore, the conversion mode information encoding unit 150 can supply the generated conversion mode information, etc., to the control unit 251.

[0204] The control unit 251 performs processing related to the control of the decoding mode. For example, the control unit 251 may obtain transformation mode information ((transform_skip_flag[xTbY][yTbY][cIdx]) and mts_idx[xTbY][yTbY][cIdx], etc.) and component identifiers (cIdx) supplied from the transformation mode information decoding unit 250. Furthermore, the control unit 251 may switch between TS residual decoding mode and non-TS residual decoding mode as the decoding mode for the encoded data of the coefficient data by controlling the selection of the selection unit 252 and selection unit 255 based on the transformation skip flag corresponding to the component identifier. For example, if transformation skipping is applied, the control unit 251 connects the selection unit 252 and selection unit 255 to the TS residual decoding unit 253. Alternatively, if transformation skipping is not applied (if transformation processing is performed), the control unit 251 connects the selection unit 252 and selection unit 255 to the non-TS residual decoding unit 254.

[0205] The selection unit 252 performs processing related to the selection of the destination for the encoded data of the coefficient data (quantization coefficients). For example, the selection unit 252 may acquire encoded data supplied from the storage buffer 211. The selection unit 252 may also, in accordance with the control unit 251, supply the acquired encoded data to the TS residual decoding unit 253 or the non-TS residual decoding unit 254 (whichever is specified by the control unit 251). For example, if a conversion skip is applied, the selection unit 252 supplies the encoded data to the TS residual decoding unit 253. Alternatively, for example, if a conversion skip is not applied (if a conversion process is performed), the selection unit 252 supplies the encoded data to the non-TS residual decoding unit 254.

[0206] The TS residual decoding unit 253 performs processing related to the TS residual decoding mode. For example, the TS residual decoding unit 253 can acquire encoded data supplied from the selection unit 252. The TS residual decoding unit 253 can also decode the acquired encoded data in the TS residual decoding mode. This TS residual decoding mode is a decoding mode for cases where the conversion process is skipped. For example, the decoding in the TS residual decoding mode corresponds to the encoding in the TS residual coding mode and is optimized for encoded data of coefficient data where the conversion process has been skipped. More specific methods for the TS residual decoding mode are described in Non-Patent Literature 4, etc. The TS residual decoding unit 253 can decode the encoded data in this way and generate quantization coefficients. The TS residual decoding unit 253 can supply the quantization coefficients thus generated to the selection unit 255.

[0207] The non-TS residual decoding unit 254 performs processing related to the non-TS residual decoding mode. For example, the non-TS residual decoding unit 254 can acquire encoded data supplied from the selection unit 252. The non-TS residual decoding unit 254 can also decode the acquired encoded data in the non-TS residual decoding mode. This non-TS residual decoding mode is a decoding mode for when transformation processing is performed. For example, the decoding in the non-TS residual decoding mode corresponds to the encoding in the non-TS residual coding mode and is optimized for encoded data of coefficient data that has undergone transformation processing. More specific methods for the non-TS residual decoding mode are described in Non-Patent Document 4, etc. The non-TS residual decoding unit 254 can decode the encoded data in this way and generate quantization coefficients. The non-TS residual decoding unit 254 can supply the quantization coefficients generated in this way to the selection unit 255.

[0208] The selection unit 255 performs processing related to the selection of the source of the quantization coefficients. For example, the selection unit 255 may obtain quantization coefficients supplied from the TS residual decoding unit 253 or the non-TS residual decoding unit 254 (whichever is specified by the control unit 251). For example, if a conversion skip is applied, the selection unit 255 obtains quantization coefficients supplied from the TS residual decoding unit 253. Alternatively, for example, if a conversion skip is not applied (i.e., a conversion process is performed), the selection unit 255 obtains encoded data supplied from the non-TS residual decoding unit 254. The selection unit 255 may supply the quantization coefficients thus obtained to the inverse quantization unit 213.

[0209] <Skip conversion for each component> The image decoding device 200 can decode encoded data of a color image that has luminance and color components. Furthermore, the image decoding device 200 can skip (omit) the inverse transform process that converts the coefficient data generated by decoding the encoded data into the residuals between the image and its predicted image, not only for the luminance component but also for the color component.

[0210] For example, the conversion mode information decoding unit 250 can decode the encoded data of the conversion skip flag corresponding to the component identifier and obtain the conversion skip flag corresponding to that component identifier.

[0211] By doing so, during decoding, the component-specific conversion skip settings indicated by the conversion skip flag can be applied. Therefore, the bitstream of color components to which conversion skipping has been applied can be correctly decoded. Consequently, the reduction in encoding efficiency can be suppressed compared to the case where conversion skipping can only be applied to the luminance component.

[0212] Furthermore, when decoding the encoded image data, the inverse conversion process of the color components may be controlled based on the conversion skip settings for each component using the conversion skip flags described above. In other words, conversion skipping may be applied to the inverse conversion process of the color components. By doing so, the bitstream of the color components to which conversion skipping has been applied can be correctly decoded. Therefore, the reduction in encoding efficiency can be suppressed compared to the case where conversion skipping can only be applied to the luminance component.

[0213] As mentioned above, by setting conversion skips for each component, conversion skips can be set for color components independently of luminance components. Therefore, conversion skips for color components can be set depending on whether or not conversion skips are enabled for color components. By doing so, the reduction in encoding efficiency can be suppressed compared to applying the conversion skip setting for luminance components to color components.

[0214] <Controlling the encoding mode> Furthermore, when applying conversion skipping to color components as in Method 1 described above, such control over the decoding mode may also be applied. For example, when applying conversion skipping to each value of a component identifier, the TS residual decoding mode may be applied as the decoding mode, and in the case of non-conversion skipping where conversion skipping is not applied, the non-TS residual decoding mode may be applied as the decoding mode.

[0215] For example, based on a conversion skip flag corresponding to a component identifier, the decoding mode of the encoded data of the coefficient data corresponding to that component identifier may be controlled to either a TS residual decoding mode, which is used when the inverse transformation process of converting the coefficient data into residuals between the image and the predicted image is skipped, or a non-TS residual decoding mode, which is used when the inverse transformation process is not skipped. The encoded data of the coefficient data corresponding to the component identifier may then be decoded using the decoding mode set in this manner, and the coefficient data corresponding to that component identifier may be generated.

[0216] By doing so, it is possible to apply a decoding mode according to the conversion skip setting, which helps to suppress the reduction in encoding efficiency compared to when decoding is performed with a single decoding mode.

[0217] <Image Decoding Process Flow> Next, we will describe the flow of each process performed by the image decoding device 200 with the above configuration. First, we will explain an example of the image decoding process flow with reference to the flowchart in Figure 8.

[0218] When the image decoding process starts, the storage buffer 211 acquires and holds (stores) the bitstream supplied from outside the image decoding device 200 in step S401.

[0219] In step S202, the decoding unit 212 extracts and decodes the encoded data from the bitstream to obtain the quantization conversion coefficient level. The decoding unit 212 also parses (analyzes and obtains) various encoding parameters from the bitstream through this decoding.

[0220] In step S203, the inverse quantization unit 213 performs inverse quantization, which is the reverse process of quantization performed on the encoding side, on the quantization conversion coefficient level level obtained in the processing of step S202, to obtain the conversion coefficient Coeff_IQ.

[0221] In step S204, the inverse orthogonal transform unit 214, in accordance with the control in step S203, performs an inverse orthogonal transform process on the transformation coefficient Coeff_IQ obtained in step S203, which is the inverse process of the orthogonal transform process performed on the encoding side, to obtain the predicted residual D'.

[0222] In step S205, the prediction unit 219 performs prediction processing based on the information parsed in step S202 using a prediction method specified by the encoding side, and generates a predicted image P by referring to a reference image stored in the frame memory 218, etc.

[0223] In step S206, the calculation unit 215 adds the predicted residual D' obtained in step S204 and the predicted image P obtained in step S205 to obtain the local decoded image R local Derive the following.

[0224] In step S207, the in-loop filter unit 216 processes the local decoded image R obtained by the processing in step S206. local In-loop filtering is performed on this.

[0225] In step S208, the sorting buffer 217 is used to sort the filtered local decoded image R obtained by the processing in step S207. local The decoded image R is derived using this method, and the order of the decoded image R group is rearranged from the decoding order to the playback order. The decoded image R group rearranged in playback order is output to the outside of the image decoding device 200 as a moving image.

[0226] Furthermore, in step S209, the frame memory 218 stores the local decoded image R obtained by the processing in step S206. local , and the filtered local decoded image R obtained by the processing in step S207 local Remember at least one of them.

[0227] Once the process in step S209 is completed, the image decoding process is finished.

[0228] <Decryption process flow> Next, with reference to the flowchart in Figure 9, an example of the decoding process performed in step S202 of Figure 8 will be explained.

[0229] When the decryption process is started, the transformation mode information decoding unit 250 of the decoding unit 212 decrypts the transformation mode information (transform_mode) of the transformation block corresponding to the component identifier cIdx in step S251.

[0230] In step S252, the control unit 251 derives Condition1 using the above-described equation (1). In other words, the control unit 251 generates Condition1 using the transform_skip_flag[cIdx] corresponding to the component.

[0231] In step S253, the control unit 251 determines whether Condition1 is true or not. If Condition1 is true, the transform skip flag (transform_skip_flag[cIdx]) corresponding to the component is true (IS_SKIP). Therefore, the selection unit 252 and the selection unit 255 connect to the TS residual decoding unit 253 according to the control unit 251. The process then proceeds to step S254.

[0232] In step S254, the TS residual decoding unit 253 decodes the encoded data using the TS residual decoding mode to generate quantization coefficients. Once the quantization coefficients are generated, the decoding process ends.

[0233] Furthermore, in step S253, if Condition1 is false, the transform_skip_flag[cIdx] corresponding to the component is false. Therefore, the selection units 252 and 255 connect to the non-TS residual decoding unit 254 according to the control of their control unit 251. This causes the process to proceed to step S255.

[0234] In step S255, the non-TS residual decoding unit 254 encodes the encoded data using the non-TS residual decoding mode to generate quantization coefficients. Once the quantization coefficients are generated, the decoding process ends.

[0235] As described above, each process is performed, allowing the image decoding device 200 to apply the TS residual decoding mode to the color components as well, thereby suppressing a reduction in encoding efficiency.

[0236] In the above explanation, it was described that whether or not to skip a transformation is indicated by the transformation skip flag (transform_skip_flag), but this is not limited to this, and it may also be indicated as one mode of the identifier mts_idx related to the primary transformation. The identifier mts_idx is an identifier that indicates the horizontal and vertical transformation types of the primary transformation. In that case, the control unit 251 can derive Condition1 as shown in equation (2) above.

[0237] <3-3. Syntax and Semantics> An example of the TU (transform-unit) syntax in this case is shown in Figure 10. In the example in Figure 10, transformation mode information corresponding to each component (transform_mode(... ,0), transform_mode(... ,0), transform_mode(... ,2)) is derived, and whether or not to apply transformation skipping (residual_coding(... ,cIdx) or residual_ts_coding(... ,cIdx)) is determined for each component (tu_cbf_luma, tu_cbf_cb, tu_cbf_cr).

[0238] An example of the syntax for this transformation mode information is shown in Figure 11. In the example in Figure 11, the transformation mode information for each component (transform_mode(... ,cIdx)) is set to include a transformation skip flag for each component (transform_skip_flag[x0][y0][cIdx]) and an identifier for the transformation type of the primary transformation for each component (mts_idx[x0][y0][cIdx]). In this case, the conditions for each component to generate this information are the same. The content of these conditions is arbitrary. For example, the transformation skip enable flag (sps_transform_skip_enabled_flag) set in the sequence parameter set (SPS) can be used as such a condition.

[0239] Figure 12 shows an example of the syntax for transformation skip (residual_ts_coding(... ,cIdx)). As shown in Figure 12, transformation skip settings are made for each component (cIdx). Figure 13A shows an example of the semantics of the transformation skip flag (transform_skip_flag[x0][y0][cIdx]) in the transformation mode information. Furthermore, Figure 13B shows an example of the semantics of the transform type identifier for the primary transformation (mts_idx[x0][y0][cIdx]). In this way, information about transformation skip is set for each component. Therefore, as described above, transformation skip can be applied to color components, and the reduction in coding efficiency can be suppressed.

[0240] Furthermore, in the syntax of the transformation mode information, the conditions for each component to generate the transformation skip flag (transform_skip_flag[x0][y0][cIdx]) and the identifier of the transformation type of the primary transformation for each component (mts_idx[x0][y0][cIdx]) may be different from each other, as shown in the example in Figure 14. By providing conditions for each component in this way, the redundancy of the judgment conditions can be suppressed, and the increase in processing load can be suppressed.

[0241] <3-4. Flag for using conversion skip residual coding> Additionally, a conversion skip residual coding usage flag may be applied to indicate the selection between TS residual coding mode (TS residual decoding mode) and non-TS residual coding mode (non-TS residual decoding mode).

[0242] An example of the syntax for a sequence parameter set (SPS) is shown in Figure 15. For example, the control unit 101 may set the conversion skip residual coding use flag sps_ts_residual_coding_use_flag in this sequence parameter set.

[0243] The sps_ts_residual_coding_use_flag is a conversion skip residual coding use flag notified at the sequence parameter set level. A value of "1" indicates that the TS residual coding mode will be applied when a conversion is skipped. A value of "0" indicates that the non-TS residual coding mode will be applied when a conversion is skipped.

[0244] Figure 16 shows an example of the TU syntax in that case. In the example in Figure 16, this transform skip residual coding use flag is used as a condition for determining whether or not to apply transform skip. The semantics of sps_ts_residual_coding_use_flag are shown in Figure 17. Also, Condition1 in this case can be derived, for example, using equation (3) below.

[0245] TIFF0007863977000003.tif22131

[0246] Furthermore, if the identifier of the primary transformation type is used, it is derived as shown in (4) below.

[0247] TIFF0007863977000004.tif24133

[0248] By using such high-level flags, even encoders and decoders that do not support conversion skipping can perform encoding and decoding correctly based on these flags. Therefore, it is possible to suppress a reduction in encoding efficiency. In other words, in the encoder and decoder, the implementation of the TS residual encoding mode (TS residual decoding mode) can be omitted, and an increase in circuit scale can be suppressed.

[0249] Note that the data unit for setting this conversion skip residual encoding use flag is arbitrary and may be other than the sequence parameter set. For example, this conversion skip residual encoding use flag may be notified at the CU level, slice level, picture level, etc. The finer the granularity of the data unit, the higher the degree of freedom in mode switching and the greater the room for improving encoding efficiency.

[0250] <3-5. Conversion Skip Residual Encoding Use Specific Mode Flag> Also, in a specific mode, a flag indicating whether to apply the TS residual encoding mode (TS residual decoding mode) or the non-TS residual encoding mode (non-TS residual decoding mode) may be applied.

[0251] An example of the syntax of the sequence parameter set (SPS) is shown in FIG. 18. For example, the control unit 101 may set the conversion skip residual encoding use specific mode flag sps_ts_residual_coding_use_for_bdpcm_flag in this sequence parameter set.

[0252] The `sps_ts_residual_coding_use_for_bdpcm_flag` is a flag notified at the sequence parameter set level and indicates that the selection of the coding mode is enabled in BDPCM (Block-based Differential Pulse Code Modulation). A value of "1" for this flag indicates that the TS residual coding mode will be applied in the case of BDPCM. A value of "0" for this flag indicates that the non-TS residual coding mode will be applied in the case of BDPCM.

[0253] Figure 19 shows an example of the TU syntax in that case. In the example in Figure 19, this conversion skip residual coding use specific mode flag is used as a condition for determining whether or not to apply conversion skip. The semantics of sps_ts_residual_coding_use_for_bdpcm_flag are shown in Figure 20. Also, Condition1 in this case can be derived, for example, using the following equation (5).

[0254] TIFF0007863977000005.tif17141

[0255] Furthermore, if the identifier of the primary transformation type is used, it is derived as shown in (6) below.

[0256] TIFF0007863977000006.tif20141

[0257] By using such high-level flags, it becomes possible to switch between TS residual coding mode and non-TS residual coding mode in BDPCM.

[0258] The data unit to which this conversion skip residual coding usage specific mode flag is set is arbitrary and does not have to be a sequence parameter set. For example, this conversion skip residual coding usage specific mode flag may be notified at the CU level, slice level, picture level, etc. The finer the granularity of the data unit, the greater the flexibility in switching modes and the greater the room for improving coding efficiency.

[0259] <4. Second Embodiment> <4-1. Correction of quantization parameters in quantization> For each value of a component identifier, the quantization parameters may be corrected in the case of a conversion skip. In other words, the quantization parameters may be corrected when a conversion skip is applied. Furthermore, such control may be made possible on a component-by-component basis.

[0260] <Quantization section> In that case as well, the image coding device 100 is the same as in the example in Figure 2. Figure 21 is a block diagram showing an example of the main configuration of the quantization unit 114 in that case. As shown in Figure 21, the quantization unit 114 in this case has a QP correction unit 311 and a quantization processing unit 312.

[0261] The QP correction unit 311 performs processing related to the correction of quantization parameters. For example, the QP correction unit 311 can derive a quantization parameter Qp to be applied to the target transformation block corresponding to the component identifier cIdx by referring to the transformation skip flag corresponding to the component identifier cIdx, the joint chromatic difference coding mode information TuResMode, the CU level QP (Qp') corresponding to the component identifier cIdx, and the minimum QP for transformation skips (QpPrimeTsMin). The types of these quantization parameters include Qp'y, Qp'cb, Qp'cr, and Qp'cbcr. In addition, the minimum quantization parameter QpPrimeTsMin for transformation skips is notified in the parameter set.

[0262] The joint chrominance coding mode is a mode in which only one of the chrominance components (Cb, Cr) is transmitted, while the other is derived from the first component and not transmitted. For example, only the residual of Cr is coded and transmitted, while the residual of Cb is derived from the residual of Cr and not transmitted. The joint chrominance coding mode information TuResMode is information related to such a joint chrominance coding mode.

[0263] The QP correction unit 311 supplies the corrected quantization parameter (corrected QP(qP)) to the quantization processing unit 312.

[0264] The quantization processing unit 312 quantizes the coefficient data (conversion coefficients) using the quantization parameters (corrected QP (qP)) supplied from the QP correction unit 311, and generates quantization coefficients. The quantization processing unit 312 supplies the generated quantization coefficients to the encoding unit 115 and the inverse quantization unit 117.

[0265] <Quantization Process Flow> An example of the quantization process flow in step S108 of Figure 4 will be explained with reference to the flowchart in Figure 22. When the quantization process starts, the QP correction unit 311 of the quantization unit 114 derives Condition2 in step S301 using the following equation (7). In other words, the QP correction unit 311 generates Condition2 using the transform_skip_flag[cIdx] corresponding to the component.

[0266] TIFF0007863977000007.tif17130

[0267] Furthermore, if the identifier of the primary transformation type is used, it is derived as shown in (8) below.

[0268] TIFF0007863977000008.tif20113

[0269] In step S302, the QP correction unit 311 determines whether this Condition2 is true. If Condition2 is true, that is, when the transform skip flag (transform_skip_flag[cIdx]) corresponding to the component is true (IS_SKIP), the process proceeds to step S303.

[0270] In step S303, the QP correction unit 311 corrects the quantization parameter QP. In this case, the QP correction unit 311, for example, sets the larger value between the minimum quantization parameter for transform skip (QpPrimeTsMin) and the quantization parameter QP'x at the CU level as the quantization parameter (corrected QP (qP)) to be applied to the processing target transform block corresponding to the component identifier. When the processing of step S303 ends, the process proceeds to step S305.

[0271] Also, if it is determined that Condition2 is false, the process proceeds to step S304. In this case, the QP correction unit 311, for example, sets the quantization parameter QP'x at the CU level as the quantization parameter (corrected QP (qP)) to be applied to the processing target transform block corresponding to the component identifier. When the processing of step S304 ends, the process proceeds to step S305.

[0272] In step S305, the quantization processing unit 312 quantizes the transform coefficients using the quantization parameter updated in step S303 or step S304. When the processing of step S305 ends, the quantization process ends and the process returns to FIG. 4.

[0273] By doing so, it is possible to suppress the reduction of PSNR.

[0274] <4-2. Correction of Quantization Parameter in Inverse Quantization> Similarly, in inverse quantization, the quantization parameters may be corrected for each value of the component identifier in the case of a conversion skip. In other words, the quantization parameters may be corrected when a conversion skip is applied. Furthermore, such control may be performed on a component-by-component basis.

[0275] <Inverse quantization part> In that case as well, the image decoding device 200 is the same as in the example in Figure 6. Figure 23 is a block diagram showing the main configuration example of the inverse quantization unit 213 in that case. As shown in Figure 23, the inverse quantization unit 213 in this case has a QP correction unit 411 and an inverse quantization processing unit 412.

[0276] The QP correction unit 411 performs processing related to the correction of quantization parameters. For example, the QP correction unit 411 can derive a quantization parameter Qp to be applied to the target transformation block corresponding to the component identifier cIdx by referring to the transformation skip flag corresponding to the component identifier cIdx, the joint chromatic difference coding mode information TuResMode, the CU level QP (Qp') corresponding to the component identifier cIdx, and the minimum QP for transformation skips (QpPrimeTsMin). The types of this quantization parameter include Qp'y, Qp'cb, Qp'cr, and Qp'cbcr. In addition, the minimum quantization parameter QpPrimeTsMin for transformation skips is notified in the parameter set.

[0277] The QP correction unit 411 supplies the corrected quantization parameter (corrected QP(qP)) to the inverse quantization processing unit 412.

[0278] The inverse quantization processing unit 412 performs inverse quantization of the quantization coefficients using the quantization parameters (corrected QP(qP)) supplied from the QP correction unit 411, and generates coefficient data (conversion coefficients). The inverse quantization processing unit 412 supplies the generated coefficient data to the inverse orthogonal transformation unit 214.

[0279] <Flow of inverse quantization process> An example of the inverse quantization process flow in step S203 of Figure 8 will be explained with reference to the flowchart in Figure 24. When the inverse quantization process is started, the QP correction unit 411 of the inverse quantization unit 213 derives Condition2 in step S401 using the above-mentioned equation (7). That is, the QP correction unit 411 generates Condition2 using a transform_skip_flag[cIdx] corresponding to the component. This transform_skip_flag is obtained by decoding it with the transform mode information decoding unit 250, as described in the first embodiment.

[0280] In step S402, the QP correction unit 411 determines whether Condition2 is true or not. If Condition2 is true, that is, if the transformation skip flag (transform_skip_flag[cIdx]) corresponding to the component is true (IS_SKIP), the process proceeds to step S403.

[0281] In step S403, the QP correction unit 411 corrects the quantization parameter QP. In this case, the QP correction unit 411 sets the larger of the minimum quantization parameter for conversion skips (QpPrimeTsMin) and the CU level quantization parameter QP'x as the quantization parameter (corrected QP(qP)) to be applied to the conversion block to be processed corresponding to the component identifier. When the processing in step S403 is completed, the process proceeds to step S405.

[0282] Furthermore, if Condition2 is determined to be false, the process proceeds to step S404. In this case, the QP correction unit 411 sets, for example, the CU level quantization parameter QP'x as the quantization parameter (corrected QP(qP)) to be applied to the processing target transformation block corresponding to the component identifier. When the processing in step S404 is completed, the process proceeds to step S405.

[0283] In step S405, the inverse quantization processing unit 412 inversely quantizes the quantization coefficients using the quantization parameters updated in step S403 or step S404. When the processing in step S405 is completed, the inverse quantization process is finished, and the process returns to Figure 8.

[0284] By doing so, the reduction in PSNR can be suppressed.

[0285] <4-3. Syntax> An example of the syntax for the quantization parameters in this case is shown in Figure 25. As mentioned above, the quantization parameter qP is corrected for each component using the transform_skip_flag.

[0286] <5. Third Embodiment> <Sharing context variables> When applying conversion skipping as described above, the context variable corresponding to each binIdx in the bin sequence of each syntax may be shared between the luminance component and the color component.

[0287] For example, in encoding, if the conversion process is skipped, the context variables may be shared between the encoding of the luminance component and the encoding of the chrominance component of the coefficient data.

[0288] Furthermore, for example, if the inverse transform process is skipped during decoding, the context variables may be shared between the decoding of the encoded data for the luminance component of the coefficient data and the decoding of the encoded data for the chrominance component.

[0289] An example of the syntax for context variables in that case is shown in Figure 26. As in the example in Figure 26, when applying conversion skipping, the context variables may be derived in a common way for both the luminance and color components.

[0290] Another example of the syntax for context variables is shown in Figure 27. As in the example in Figure 27, when applying conversion skipping, the context variables may be derived in a way that is independent of the luminance and color components.

[0291] As described above, by sharing the context variables corresponding to each binIdx in the bin sequence of each syntax between the luminance component and the color component, the increase in memory size required to hold the context variables can be suppressed. This, in turn, can suppress the increase in hardware costs.

[0292] <6. Fourth Embodiment> <Sine code encoding and decoding mode control> Alternatively, the encoding and decoding method for sine codes may be switched depending on the conversion skip flag corresponding to the component identifier. Figure 28 shows an example of the syntax in that case.

[0293] As shown in the table in Figure 28, in the column for binIdx = 0, for example, if conversion skipping is not performed, bypass coding / decoding may be applied to the coding / decoding of the sine code. Alternatively, for example, in the case of conversion skipping, if the number of remaining context coding bins is greater than or equal to a threshold, context coding / decoding may be applied to the coding / decoding of the sine code; otherwise, bypass coding / decoding may be applied to the coding / decoding of the sine code ((MaxCcbs > 0) ? (0...5) : bypass).

[0294] By doing so, the reduction in encoding efficiency can be suppressed.

[0295] <7. Addendum> <Computer> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on a computer. Here, "computer" includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0296] Figure 29 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above by a program.

[0297] In the computer 900 shown in Figure 29, the CPU (Central Processing Unit) 901, ROM (Read Only Memory) 902, and RAM (Random Access Memory) 903 are interconnected via a bus 904.

[0298] An input / output interface 910 is also connected to the bus 904. The input / output interface 910 is connected to an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915.

[0299] The input unit 911 consists of, for example, a keyboard, mouse, microphone, touch panel, and input terminals. The output unit 912 consists of, for example, a display, speaker, and output terminals. The storage unit 913 consists of, for example, a hard disk, RAM disk, and non-volatile memory. The communication unit 914 consists of, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0300] In a computer configured as described above, the CPU 901 loads, for example, a program stored in the memory unit 913 into the RAM 903 via the input / output interface 910 and the bus 904, and executes it, thereby performing the series of processes described above. The RAM 903 also stores data necessary for the CPU 901 to perform various processes as appropriate.

[0301] The program to be executed by the computer can be recorded and applied, for example, on removable media 921 such as a package medium. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by inserting the removable media 921 into the drive 915.

[0302] Furthermore, this program can also be provided via wired or wireless transmission media such as a local area network, the internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0303] Additionally, this program can be pre-installed on ROM902 or memory unit913.

[0304] <Units of Information and Processing> The data units to which the various types of information described above are set, and the data units targeted by the various processes, are arbitrary and not limited to the examples given above. For example, this information and these processes may be set for each TU (Transform Unit), TB (Transform Block), PU (Prediction Unit), PB (Prediction Block), CU (Coding Unit), LCU (Largest Coding Unit), subblock, block, tile, slice, picture, sequence, or component, or they may target the data of those data units. Of course, these data units can be set for each piece of information or process, and it is not necessary for all information and processes to have the same data unit. The storage location of this information is arbitrary and may be stored in the header or parameter set of the data units mentioned above. It may also be stored in multiple locations.

[0305] <Control Information> In each of the embodiments described above, control information relating to the technology described may be transmitted from the encoding side to the decoding side. For example, control information (e.g., enabled_flag) that controls whether or not to allow (or prohibit) the application of the technology described above may be transmitted. Alternatively, control information (e.g., present_flag) indicating the target (or target to which the technology described above is applied or not applied) may be transmitted. For example, control information specifying the block size (upper or lower limit, or both), frame, component, or layer to which the technology is applied (or to which its application is permitted or prohibited) may be transmitted.

[0306] <Applicability of this technology> This technology can be applied to any image encoding and decoding scheme. In other words, as long as it does not contradict the technology described above, the specifications for various processes related to image encoding and decoding, such as transformation (inverse transformation), quantization (dequantization), encoding (decoding), and prediction, are arbitrary and not limited to the examples given above. Furthermore, some of these processes may be omitted as long as they do not contradict the technology described above.

[0307] Furthermore, this technology can be applied to a multi-view image encoding and decoding system that performs encoding and decoding of multi-view images containing images from multiple viewpoints (views). In that case, this technology should be applied to the encoding and decoding of each viewpoint (view).

[0308] Furthermore, this technology can be applied to a hierarchical image coding (scalable coding) and decoding system that performs coding and decoding of hierarchical images that are layered (hierarchically structured) to have scalability functionality for predetermined parameters. In that case, this technology should be applied to the coding and decoding of each layer.

[0309] Furthermore, although the image coding device 100 and image decoding device 200 were described above as examples of applications of this technology, this technology can be applied to any configuration.

[0310] For example, this technology can be applied to various electronic devices such as transmitters and receivers (e.g., television sets and mobile phones) used in satellite broadcasting, cable TV and other wired broadcasting, internet distribution, and cellular communication distribution to terminals, or devices (e.g., hard disk recorders and cameras) that record images on media such as optical discs, magnetic discs, and flash memory, or play back images from these storage media.

[0311] Furthermore, this technology can also be implemented as part of a device, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set with additional functions added to a unit (e.g., a video set).

[0312] Furthermore, this technology can also be applied to network systems composed of multiple devices. For example, this technology may be implemented as cloud computing, where multiple devices share and collaborate on processing via a network. For example, this technology may be implemented in a cloud service that provides image (video) related services to any terminal such as computers, AV (Audio Visual) equipment, portable information processing terminals, and IoT (Internet of Things) devices.

[0313] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules within a single enclosure, are both considered systems.

[0314] <Fields and applications where this technology can be applied> Systems, devices, and processing units incorporating this technology can be used in any field, such as transportation, healthcare, security, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. Furthermore, their applications are entirely arbitrary.

[0315] For example, this technology can be applied to systems and devices used to provide entertainment content. Furthermore, for example, this technology can be applied to systems and devices used for traffic management, such as traffic condition monitoring and automated driving control. In addition, for example, this technology can be applied to systems and devices used for security. Furthermore, for example, this technology can be applied to systems and devices used for automatic control of machinery, etc. Furthermore, for example, this technology can be applied to systems and devices used for agriculture and livestock farming. Furthermore, for example, this technology can be applied to systems and devices that monitor natural conditions such as volcanoes, forests, and oceans, as well as wildlife. Furthermore, for example, this technology can be applied to systems and devices used for sports.

[0316] <Other> In this specification, "flag" refers to information used to identify multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the values ​​that this "flag" can take are, for example, two values, 1 / 0, or three or more values. In other words, the number of bits that constitute this "flag" is arbitrary, and can be one bit or multiple bits. Furthermore, identification information (including flags) can be included not only in the form of the identification information itself in the bitstream, but also in the form of differential information of the identification information relative to a certain reference information in the bitstream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also differential information relative to the reference information.

[0317] Furthermore, various types of information (metadata, etc.) related to encoded data (bitstream) may be transmitted or recorded in any form, as long as they are associated with the encoded data. Here, the term "associate" means, for example, making it possible to use (link) one piece of data when processing the other. In other words, associated data may be combined into a single piece of data, or they may be individual pieces of data. For example, information associated with encoded data (image) may be transmitted on a different transmission path than the encoded data (image). Also, for example, information associated with encoded data (image) may be recorded on a different recording medium (or a different recording area on the same recording medium) than the encoded data (image). Note that this "association" may not apply to the entire data, but only to a part of it. For example, an image and the information corresponding to that image may be associated with each other in any unit, such as multiple frames, a single frame, or a part within a frame.

[0318] In this specification, terms such as "combine," "multiplex," "add," "integrate," "include," "store," "insert," "insert," and "place" mean combining multiple things into one, such as combining encoded data and metadata into a single data, and represent one method of "associating" as described above.

[0319] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0320] For example, the configuration described as a single device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as a single device (or processing unit). Furthermore, it is also possible to add configurations other than those described above to the configuration of each device (or each processing unit). In addition, if the overall system configuration and operation are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0321] Furthermore, for example, the program described above may be executed on any device. In that case, the device should have the necessary functions (such as functional blocks) and be able to obtain the necessary information.

[0322] Furthermore, for example, each step of a flowchart may be executed by one device, or it may be divided among multiple devices. Additionally, if a single step includes multiple processes, these processes may be executed by one device, or they may be divided among multiple devices. In other words, multiple processes included in a single step can be executed as multiple steps. Conversely, processes described as multiple steps can be combined and executed as a single step.

[0323] Furthermore, for example, a program executed by a computer may be structured so that the steps of the program are executed chronologically in the order described herein, or they may be executed in parallel or individually at necessary times, such as when a call is made. In other words, the steps may be executed in an order different from the order described above, as long as no inconsistencies arise. Moreover, the steps of this program may be executed in parallel with the processing of other programs, or in combination with the processing of other programs.

[0324] Furthermore, for example, multiple technologies relating to this technology can be implemented independently, as long as they do not create a contradiction. Of course, any multiple technologies can also be implemented in combination. For example, some or all of the technologies described in one embodiment can be implemented in combination with some or all of the technologies described in another embodiment. Also, some or all of the above-mentioned technologies can be implemented in combination with other technologies not mentioned above.

[0325] Furthermore, this technology can also be configured as follows. (1) A flag generation unit that generates a conversion skip flag, which is flag information that indicates whether or not to skip the conversion process that converts the residual between the image and the predicted image of the image into coefficient data in the image encoding process, A flag encoding unit encodes the conversion skip flag generated by the flag generation unit and generates encoded data of the conversion skip flag, A bitstream generation unit generates a bitstream that includes encoded data of the conversion skip flag generated by the flag encoding unit. An image processing device equipped with the following features. (2) A mode control unit that controls, based on the conversion skip flag corresponding to the component identifier generated by the flag generation unit, whether to set the encoding mode of the coefficient data corresponding to the component identifier to a TS residual encoding mode, which is the mode for when the conversion process is skipped, or to a non-TS residual encoding mode, which is the mode for when the conversion process is not skipped. A coefficient data encoding unit encodes coefficient data corresponding to the component identifier and generates encoded data of the coefficient data according to the encoding mode set by the mode control unit. Furthermore, The bitstream generation unit generates a bitstream that includes encoded data of the conversion skip flag generated by the flag encoding unit and encoded data of the coefficient data generated by the coefficient data encoding unit. (1) The image processing apparatus described above. (3) If the conversion process is skipped, the coefficient data encoding unit shares a context variable between encoding the luminance component and encoding the chrominance component of the coefficient data. (2) The image processing apparatus described above. (4) The coefficient data encoding unit applies a sine code encoding method corresponding to the conversion skip flag generated by the flag generation unit. The image processing apparatus described in (2) or (3). (5) The flag generation unit further generates a conversion skip residual coding use flag, which is flag information indicating whether to apply the TS residual coding mode, which is a mode for when the conversion process is skipped, or the non-TS residual coding mode, which is a mode for when the conversion process is not skipped. The flag encoding unit further encodes the conversion skip residual coding usage flag generated by the flag generation unit, and generates encoded data of the conversion skip residual coding usage flag. The bitstream generation unit generates the bitstream which further includes encoded data of the conversion skip residual coding use flag generated by the flag coding unit. (1) The image processing apparatus described in any of (4). (6) The flag generation unit further generates a conversion skip residual coding usage specific mode flag, which is flag information indicating whether to apply a TS residual coding mode, which is a mode for skipping the conversion process, or a non-TS residual coding mode, which is a mode for not skipping the conversion process, in a specific mode. The flag encoding unit further encodes the conversion skip residual encoding usage specific mode flag generated by the flag generation unit, and generates encoded data of the conversion skip residual encoding usage specific mode flag. The bitstream generation unit generates the bitstream which further includes encoded data of the conversion skip residual encoding usage specific mode flag generated by the flag encoding unit. (1) The image processing apparatus described in any of (5). (7) In the image encoding process, a conversion skip flag is generated, which is flag information that indicates for each component whether or not to skip the conversion process that converts the residual between the image and the predicted image of the image into coefficient data. Encode the generated conversion skip flag and generate encoded data for the conversion skip flag. Generate a bitstream containing the encoded data of the generated conversion skip flag. Bitstream generation method.

[0326] (8) In the encoding of an image, if a conversion skip flag corresponding to a component identifier indicates a conversion skip that skips the conversion process that converts the residual between the image and the predicted image of the image into a conversion coefficient, a quantization parameter correction unit corrects the quantization parameters applied to the processing target conversion block corresponding to the component identifier, A quantization unit that quantizes the processing target transformation block corresponding to the component identifier using the quantization parameter corrected by the quantization parameter correction unit. An image processing device equipped with the following features. (9) The quantization parameter correction unit is: If the conversion skip flag corresponding to the component identifier indicates a conversion skip, the larger of the minimum quantization parameter of the conversion skip and the quantization parameter corresponding to the component identifier is set as the quantization parameter to be applied to the conversion block to be processed corresponding to the component identifier. If the conversion skip flag corresponding to the component identifier indicates a non-conversion skip where no conversion skip is performed, the quantization parameter corresponding to the component identifier is set as the quantization parameter to be applied to the processing target conversion block corresponding to the component identifier. (8) The image processing apparatus described above. (10) In image encoding, if a conversion skip flag corresponding to a component identifier indicates a conversion skip that skips the conversion process that converts the residual between the image and the predicted image of the image into a conversion coefficient, the quantization parameters applied to the target conversion block corresponding to the component identifier are corrected. Using the corrected quantization parameters, the processing target transformation block corresponding to the component identifier is quantized, and a quantization coefficient corresponding to the component identifier is generated. A method for generating quantization coefficients.

[0327] (11) A flag decoding unit that decodes encoded data of a conversion skip flag corresponding to a component identifier and obtains the conversion skip flag corresponding to the component identifier, A mode control unit controls, based on the conversion skip flag corresponding to the component identifier obtained by the flag decoding unit, whether to set the decoding mode of the encoded data of the coefficient data corresponding to the component identifier to a TS residual decoding mode, which is a mode for skipping the inverse transform process that converts the coefficient data into residuals between the image and the predicted image, or to a non-TS residual decoding mode, which is a mode for not skipping the inverse transform process. A coefficient data decoding unit decodes the encoded data of the coefficient data corresponding to the component identifier and generates the coefficient data corresponding to the component identifier, based on the decoding mode set by the mode control unit. An image processing device equipped with the following features. (12) If the inverse transformation process is skipped, the coefficient data decoding unit shares a context variable between decoding the encoded data of the luminance component of the coefficient data and decoding the encoded data of the chrominance component. (11) The image processing apparatus described above. (13) The coefficient data decoding unit applies a sine code decoding method corresponding to the conversion skip flag that corresponds to the component identifier obtained by the flag decoding unit. The image processing apparatus described in (11) or (12). (14) The flag decoding unit further includes a flag decoding unit that decodes the encoded data of a conversion skip residual coding usage flag, which is flag information indicating whether to apply the TS residual coding mode or the non-TS residual coding mode when the inverse conversion process is skipped, and obtains the conversion skip residual coding usage flag corresponding to the component identifier, The mode control unit further controls whether the decoding mode of the encoded data of the coefficient data corresponding to the component identifier is the TS residual decoding mode or the non-TS residual decoding mode, based on the conversion skip residual coding usage flag corresponding to the component identifier generated by the flag decoding unit. The image processing apparatus described in any of (11) to (13). (15) The flag decoding unit further decodes the encoded data of the conversion skip residual coding usage specific mode flag, which is flag information indicating whether to apply the TS residual decoding mode or the non-TS residual decoding mode in a particular mode, to obtain the conversion skip residual coding usage specific mode flag corresponding to the component identifier. The mode control unit further controls whether the decoding mode of the encoded data of the coefficient data corresponding to the component identifier is the TS residual decoding mode or the non-TS residual decoding mode, based on the conversion skip residual coding usage specific mode flag corresponding to the component identifier generated by the flag decoding unit. The image processing apparatus described in any of (11) to (14). (16) Decode the encoded data of the conversion skip flag corresponding to the component identifier to obtain the conversion skip flag corresponding to the component identifier, Based on the conversion skip flag corresponding to the obtained component identifier, the decoding mode of the encoded data of the coefficient data corresponding to the component identifier is controlled to either be a TS residual decoding mode, which is a mode for skipping the inverse transform process that converts the coefficient data into residuals between the image and the predicted image, or a non-TS residual decoding mode, which is a mode for not skipping the inverse transform process. According to the configured decoding mode, the encoded data of the coefficient data corresponding to the component identifier is decoded, and the coefficient data corresponding to the component identifier is generated. Method for generating coefficient data.

[0328] (17) If the conversion skip flag corresponding to the component identifier indicates a conversion skip that skips the inverse conversion process that converts coefficient data into residuals between the image and the predicted image, the quantization parameter correction unit corrects the quantization parameters applied to the processing target conversion block corresponding to the component identifier, An inverse quantization unit performs inverse quantization of the processing target transformation block corresponding to the component identifier using the quantization parameter corrected by the quantization parameter correction unit. An image processing device equipped with the following features. (18) The quantization parameter correction unit is: If the conversion skip flag corresponding to the component identifier indicates a conversion skip, the larger of the minimum quantization parameter for the conversion skip and the quantization parameter corresponding to the component identifier is set as the quantization parameter to be applied to the conversion block to be processed corresponding to the component identifier. If the conversion skip flag corresponding to the component identifier indicates a non-conversion skip where no conversion skip is performed, the quantization parameter corresponding to the component identifier is set as the quantization parameter to be applied to the processing target conversion block corresponding to the component identifier. (17) The image processing apparatus described above. (19) The flag decoding unit further comprises decoding the encoded data of the conversion skip flag corresponding to the component identifier and obtaining the conversion skip flag corresponding to the component identifier, The quantization parameter correction unit corrects the quantization parameters applied to the processing target conversion block corresponding to the component identifier when the conversion skip flag corresponding to the component identifier obtained by the flag decoding unit indicates a conversion skip. The image processing apparatus described in (17) or (18). (20) If the conversion skip flag corresponding to the component identifier indicates a conversion skip that skips the inverse conversion process that converts the coefficient data into residuals between the image and the predicted image, the quantization parameters applied to the target conversion block corresponding to the component identifier are corrected. Using the corrected quantization parameters, the processing target transformation block corresponding to the component identifier is dequantized to generate the coefficient data corresponding to the component identifier. Method for generating coefficient data. [Explanation of symbols]

[0329] 100 Image coding unit, 101 Control unit, 114 Quantization unit, 115 Coding unit, 150 Conversion mode information coding unit, 151 Control unit, 152 Selection unit, 153 TS residual coding unit, 154 Non-TS residual coding unit, 155 Selection unit, 200 Image decoding unit, 212 Decoding unit, 213 Inverse quantization unit, 250 Conversion mode information decoding unit, 251 Control unit, 252 Selection unit, 253 TS residual decoding unit, 254 Non-TS residual decoding unit, 255 Selection unit, 311 QP correction unit, 312 Quantization processing unit, 411 QP correction unit, 412 Inverse quantization processing unit

Claims

1. A flag decoding unit that decodes the encoded data of a conversion skip flag corresponding to a component identifier that identifies brightness and color difference, A mode control unit controls, based on the conversion skip flag corresponding to the decoded component identifier, whether to set the decoding mode of the encoded data of the coefficient data corresponding to the component identifier to a TS residual decoding mode, which is a mode for skipping the inverse transformation process that converts the coefficient data into residuals between the image and the predicted image, or to a non-TS residual decoding mode, which is a mode for not skipping the inverse transformation process. A coefficient data decoding unit that applies the set decoding mode and decodes the encoded data of the coefficient data corresponding to the component identifier according to the syntax structure corresponding to the decoding mode. Equipped with, If the inverse transformation process is skipped, the coefficient data decoding unit shares a context variable between decoding the encoded data of the luminance component and decoding the encoded data of the chrominance component of the coefficient data. Image processing device.

2. The flag decoding unit further decodes the encoded data of the conversion skip residual coding usage flag, which is flag information indicating whether to apply the TS residual decoding mode or the non-TS residual decoding mode. The mode control further controls whether the decoding mode is the TS residual decoding mode or the non-TS residual decoding mode based on the decoded conversion skip residual coding use flag. The image processing apparatus according to claim 1.

3. The flag decoding unit further decodes the encoded data of the conversion skip residual coding usage specific mode flag, which is flag information indicating whether to apply the TS residual decoding mode or the non-TS residual decoding mode in a particular mode. The mode control further controls whether the decoding mode is the TS residual decoding mode or the non-TS residual decoding mode based on the decoded conversion skip residual coding usage specific mode flag. The image processing apparatus according to claim 1.

4. Decoding the encoded data of the conversion skip flag corresponding to the component identifier that identifies luminance and color difference, Based on the conversion skip flag corresponding to the decoded component identifier, the decoding mode of the encoded data of the coefficient data corresponding to the component identifier is controlled to be either a TS residual decoding mode, which is a mode for skipping the inverse transform process that converts the coefficient data into residuals between the image and the predicted image, or a non-TS residual decoding mode, which is a mode for not skipping the inverse transform process. When applying the set decoding mode, decoding the encoded data of the coefficient data corresponding to the component identifier according to the syntax structure corresponding to the decoding mode, and skipping the inverse transformation process, the context variables are shared between the decoding of the encoded data of the luminance component and the decoding of the encoded data of the chrominance component of the coefficient data. A method for generating coefficient data that includes this data.

5. A flag decoding unit that decodes the encoded data of a conversion skip flag corresponding to a component identifier that identifies brightness and color difference, A mode control unit controls, based on the conversion skip flag corresponding to the decoded component identifier, whether to set the decoding mode of the encoded data of the coefficient data corresponding to the component identifier to a TS residual decoding mode, which is a mode for skipping the inverse transformation process that converts the coefficient data into residuals between the image and the predicted image, or to a non-TS residual decoding mode, which is a mode for not skipping the inverse transformation process. A coefficient data decoding unit that applies the set decoding mode and decodes the encoded data of the coefficient data corresponding to the component identifier according to the syntax structure corresponding to the decoding mode. Equipped with, The coefficient data decoding unit applies a sine code decoding method corresponding to the conversion skip flag that corresponds to the decoded component identifier. Image processing device.

6. If the inverse transformation process is not skipped, the coefficient data decoding unit applies bypass decoding as the decoding method. The image processing apparatus according to claim 5.

7. The flag decoding unit further decodes the encoded data of the conversion skip residual coding usage flag, which is flag information indicating whether to apply the TS residual decoding mode or the non-TS residual decoding mode. The mode control further controls whether the decoding mode is the TS residual decoding mode or the non-TS residual decoding mode based on the decoded conversion skip residual coding use flag. The image processing apparatus according to claim 5.

8. The flag decoding unit further decodes the encoded data of the conversion skip residual coding usage specific mode flag, which is flag information indicating whether to apply the TS residual decoding mode or the non-TS residual decoding mode in a particular mode. The mode control further controls whether the decoding mode is the TS residual decoding mode or the non-TS residual decoding mode based on the decoded conversion skip residual coding usage specific mode flag. The image processing apparatus according to claim 5.

9. Decoding the encoded data of the conversion skip flag corresponding to the component identifier that identifies luminance and color difference, Based on the conversion skip flag corresponding to the decoded component identifier, the decoding mode of the encoded data of the coefficient data corresponding to the component identifier is controlled to be either a TS residual decoding mode, which is a mode for skipping the inverse transform process that converts the coefficient data into residuals between the image and the predicted image, or a non-TS residual decoding mode, which is a mode for not skipping the inverse transform process. The configured decoding mode is applied, and a decoding method for the sign code corresponding to the conversion skip flag corresponding to the decoded component identifier is applied according to the syntax structure corresponding to the decoding mode, thereby decoding the encoded data of the coefficient data corresponding to the component identifier. A method for generating coefficient data that includes this data.

Images