Image processing device, bit stream generation method, and quantization coefficient generation method

Abstract

To enable a reduction in encoding efficiency to be suppressed.SOLUTION: In encoding of an image: a conversion skip flag being flag information showing whether to skip conversion processing for converting a residual error between the image and a prediction image of the image into coefficient data is generated; the generated conversion skip flag is encoded; encoded data of the conversion skip flag is generated; and a bit stream including the encoded data of the generated conversion skip flag is generated. This disclosure can be applied, for example, to an image processing device, an image encoding device, an image decoding device, a transmission device, a reception device, a transmission / reception device, an information processing device, an imaging device, a reproduction device, a bit stream generation method, a coefficient data generation method, a quantization coefficient generation method, or the like.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, a bitstream generation method, and a quantization coefficient generation method, and more particularly, to an image processing device, a bitstream generation method, and a quantization coefficient generation method that are capable of suppressing a decrease in encoding efficiency. [Background technology]

[0002] 2. Description of the Related Art Conventionally, in image coding, a method has been proposed in which coding is performed by skipping (omitting) a conversion process for converting a residual between an image and its predicted image into coefficient data (for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Benjamin Bross, Jianle Chen, Shan Liu, "Versatile Video Coding (Draft 5)", JVET-N1001-v10, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 14th Meeting: Geneva, CH, 19-27 Mar. 2019 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the case of the method described in Non-Patent Document 1, this transform skip mode is applied only to the luminance component (luminance transform skip). In other words, this transform skip mode cannot be applied to the chrominance component. Therefore, there is a risk of reducing the coding efficiency.

[0005] The present disclosure has been made in consideration of such circumstances, and makes it possible to suppress a decrease in coding efficiency. [Means for solving the problem]

[0006] An image processing device according to one aspect of the present technology is an image processing device including: a flag generation unit that generates a transform skip flag, which is flag information indicating, for each component, whether or not to skip a transform process for converting a residual between an image and a predicted image of the image into coefficient data when encoding the image; a flag encoding unit that encodes the transform skip flag generated by the flag generation unit and generates encoded data of the transform skip flag; and a bit stream generation unit that generates a bit stream including the encoded data of the transform skip flag generated by the flag encoding unit.

[0007] A bitstream generation method according to one aspect of the present technology is a bitstream generation method that generates a transform skip flag, which is flag information indicating, for each component, whether or not to skip a transform process in which a residual between an image and a predicted image of the image is converted into coefficient data, in encoding an image, encodes the generated transform skip flag, generates encoded data of the transform skip flag, and generates a bitstream including the encoded data of the generated transform skip flag.

[0008] Another aspect of the image processing device of the present technology is an image processing device that, when encoding an image, a transform skip flag corresponding to a component identifier indicates a transform skip that skips a transform process that converts a residual between the image and a predicted image of the image into a transform coefficient, includes a quantization parameter correction unit that corrects a quantization parameter to be applied to a transform block to be processed that corresponds to the component identifier, and a quantization unit that quantizes the transform block to be processed that corresponds to the component identifier using the quantization parameter corrected by the quantization parameter correction unit.

[0009] Another aspect of the quantization coefficient generation method of the present technology is a quantization coefficient generation method for encoding an image, when a transform skip flag corresponding to a component identifier indicates a transform skip that skips a transform process that converts a residual between the image and a predicted image of the image into a transform coefficient, correcting a quantization parameter to be applied to a transform block to be processed that corresponds to the component identifier, quantizing the transform block to be processed that corresponds to the component identifier using the corrected quantization parameter, and generating a quantization coefficient corresponding to the component identifier.

[0010] In an image processing device and a bitstream generation method according to one aspect of the present technology, in encoding an image, a transform skip flag is generated, which is flag information indicating for each component whether or not to skip a transform process for converting a residual between the image and a predicted image of the image into coefficient data, the generated transform skip flag is encoded, encoded data of the transform skip flag is generated, and a bitstream including the encoded data of the generated transform skip flag is generated.

[0011] In another aspect of the image processing device and quantization coefficient generating method of the present technology, in encoding an image, if a transform skip flag corresponding to a component identifier indicates a transform skip that skips a transform process that converts a residual between the image and a predicted image of that image into a transform coefficient, a quantization parameter to be applied to a transform block to be processed that corresponds to the component identifier is corrected, and the corrected quantization parameter is used to quantize the transform block to be processed that corresponds to the component identifier, and a quantization coefficient corresponding to the component identifier is generated. [Brief description of the drawings]

[0012] [Figure 1] FIG. 13 is a diagram illustrating an extension of a conversion skip. [Diagram 2] FIG. 1 is a block diagram showing an example of the main configuration of an image encoding device. [Diagram 3] 2 is a block diagram showing an example of the main configuration of an encoding unit. FIG. [Figure 4] 13 is a flowchart showing an example of the flow of an image encoding process. [Diagram 5] 11 is a flowchart illustrating an example of the flow of an encoding process. [Figure 6] FIG. 2 is a block diagram showing an example of the main configuration of an image decoding device. [Figure 7] 2 is a block diagram showing an example of the main configuration of a decoding unit. FIG. [Figure 8] 13 is a flowchart showing an example of the flow of an image decoding process. [Figure 9] 13 is a flowchart showing an example of the flow of a decoding process. [Figure 10] FIG. 13 is a diagram illustrating an example of the syntax of a TU. [Figure 11] FIG. 13 is a diagram illustrating an example of the syntax of conversion mode information. [Figure 12] A figure showing an example of the syntax of a TS residual coding mode. [Figure 13] A diagram showing an example of the semantics of conversion mode information. [Figure 14] FIG. 13 is a diagram showing another example of the syntax of the conversion mode information. [Figure 15] A figure showing an example of the syntax of a sequence parameter set. [Figure 16] FIG. 13 is a diagram illustrating an example of the syntax of a TU. [Figure 17] A diagram illustrating an example of the semantics of a sequence parameter set. [Figure 18] A figure showing an example of the syntax of a sequence parameter set. [Figure 19] FIG. 13 is a diagram illustrating an example of the syntax of a TU. [Figure 20] A diagram illustrating an example of the semantics of a sequence parameter set. [Figure 21] 4 is a block diagram showing an example of the main configuration of a quantization unit. FIG. [Figure 22] 13 is a flowchart showing an example of the flow of a quantization process. [Figure 23]13 is a block diagram showing an example of the main configuration of an inverse quantization unit. [Figure 24] 13 is a flowchart showing an example of the flow of an inverse quantization process. [Diagram 25] FIG. 13 is a diagram illustrating an example of the syntax of a quantization parameter. [Figure 26] FIG. 13 is a diagram illustrating an example of the syntax of a context variable. [Figure 27] FIG. 13 is a diagram illustrating an example of the syntax of a context variable. [Figure 28] FIG. 11 is a diagram illustrating switching of sign encoding modes. [Figure 29] FIG. 1 is a block diagram showing an example of a main configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described in the following order. 1. Literature supporting technical content and technical terms 2. Skip conversion 3. First embodiment (extension of conversion skip) 4. Second embodiment (quantization parameter correction) 5. Third embodiment (sharing of context variables) 6. Fourth embodiment (sign code encoding / decoding mode control) 7. Notes

[0014] <1. Literature that supports technical content and technical terms> The scope of the disclosure of the present technology includes not only the contents described in the examples but also the contents described in the following non-patent documents that were publicly known at the time of filing the application.

[0015] Non-patent document 1: (mentioned above) Non-Patent Document 2: Takeshi Tsukuba, Masaru Ikeda, Yoichi Yagasaki, Teruhiko Suzuki, "CE8: Chroma Transform Skip (CE8-3.2)", JVET-O0081-v2, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 15th Meeting: Gothenburg, SE, 3-12 July 2019 Non-Patent Document 3: Tung Nguyen, Benjamin Bross, Heiko Schwarz, Detlev Marpe, Thomas Wiegand, "Non-CE8: Minimum Allowed QP for Transform Skip Mode", JVET-O0405-v1, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 15th Meeting: Gothenburg, SE, 3-12 July 2019 Non-Patent Document 4: Jianle Chen, Yan Ye, Seung Hwan Kim, "Algorithm description for Versatile Video Coding and Test Model 6 (VTM 6)", JVET-O2002-v2, Joint Video Experts Team (JVET), of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 15th Meeting: Gothenburg, SE, 3-12 July 2019 Non-Patent Document 5: Takeshi Tsukuba, Masaru Ikeda, Yoichi Yagasaki, Teruhiko Suzuki, "CE8-2.1: Transform Skip for Chroma with limiting maximum number of context-coded bin in TS residual coding", JVET-P0058-v1, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 16th Meeting: Geneva, CH, 1-11 October 2019 Non-Patent Document 6: Gordon Clare, Felix Henry, Takeshi Tsukuba, Masaru Ikeda, Yoich Yagasaki, Teruhiko Suzuki, "CE8-4.1: BDPCM and Transform skip for Chroma", JVET-P0059-v1, Joint Video Experts Team (JVET) of ITU-T SG 16 WP 3 and ISO / IEC JTC 1 / SC 29 / WG 11 16th Meeting: Geneva, CH, 1-11 October 2019 Non-Patent Document 7: TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU(International Telecommunication Union), "Advanced video coding for generic audiovisual services", H.264, 04 / 2017 Non-Patent Document 8: TELECOMMUNICATION STANDARDIZATION SECTOR OF ITU(International Telecommunication Union), "High efficiency video coding", H.265, 12 / 2016

[0016] In other words, the contents described in the above non-patent documents are also the basis for determining the support requirements. For example, even if the Quad-Tree Block Structure and QTBT (Quad Tree Plus Binary Tree) Block Structure described in the above non-patent documents are not directly described in the examples, they are considered to be within the disclosure scope of the present technology and to satisfy the support requirements of the claims. Similarly, for example, technical terms such as parsing, syntax, and semantics are also considered to be within the disclosure scope of the present technology and to satisfy the support requirements of the claims, even if they are not directly described in the examples.

[0017] In addition, in this specification, a "block" (not a block indicating a processing unit) used in the description as a partial region or processing unit of an image (picture) indicates any partial region in a picture, and its size, shape, characteristics, etc. are not limited, unless otherwise specified. For example, a "block" includes any partial region (processing unit) such as a TB (Transform Block), TU (Transform Unit), PB (Prediction Block), PU (Prediction Unit), SCU (Smallest Coding Unit), CU (Coding Unit), LCU (Largest Coding Unit), CTB (Coding Tree Block), CTU (Coding Tree Unit), transform block, sub-block, macroblock, tile, or slice described in the above-mentioned non-patent document.

[0018] In addition, when specifying the size of such a block, the block size may be specified indirectly in addition to directly. For example, the block size may be specified using identification information for identifying the size. For example, the block size may be specified by a ratio or difference with respect to the size of a reference block (for example, LCU or SCU). For example, when transmitting information specifying a block size as a syntax element or the like, the information may be used to indirectly specify the size as described above. In this way, the amount of information can be reduced, and coding efficiency may be improved. The specification of the block size also includes specification of a range of block sizes (for example, specification of a range of allowable block sizes, etc.).

[0019] In this specification, encoding includes not only the whole process of converting an image into a bit stream, but also a part of the process. For example, it includes not only a process including prediction processing, orthogonal transformation, quantization, arithmetic coding, etc., but also a process collectively referring to quantization and arithmetic coding, a process including prediction processing, quantization, and arithmetic coding, etc. Similarly, decoding includes not only the whole process of converting a bit stream into an image, but also a part of the process. For example, it includes not only a process including inverse arithmetic decoding, inverse quantization, inverse orthogonal transformation, prediction processing, etc., but also a process including inverse arithmetic decoding and inverse quantization, a process including inverse arithmetic decoding, inverse quantization, and prediction processing, etc.

[0020] <2. Skip conversion> <Skip conversion of luminance component> In the past, in still image and video coding, the image input to the coding device is generally a color image having a luminance component and a chrominance component (which may include a chrominance component). In such image coding, for example, as described in Non-Patent Document 1, a coding method has been devised in which the conversion process for converting the residual between an image and its predicted image into coefficient data is skipped (omitted) for the luminance component.

[0021] However, this transform skip mode could not be applied to color components, so there was a risk of the coding efficiency of color components decreasing in screen content, etc., where the transform skip is effective.

[0022] <Extended conversion skip flag> Therefore, the transform skip is extended so that the transform skip can be set for each component (for each luminance component or color component), and the transform skip can be performed not only for the luminance component but also for the color components. For example, in the case of the image encoding / decoding method described in the above-mentioned non-patent document, the skip setting can be indicated by a transform skip flag (transform_skip_flag). Also, a component can be indicated by a component identifier (cIdx). Therefore, for example, as in method 1 in the top row of the table in FIG. 1, the value of this transform skip flag is set in correspondence with the value of the component identifier.

[0023] For example, an image processing device is provided with a flag generation unit that generates a transform skip flag, which is flag information indicating for each component whether or not to skip a transform process for converting a residual between an image and a predicted image of that image into coefficient data when encoding the image, a flag encoding unit that encodes the transform skip flag generated by the flag generation unit and generates encoded data of the transform skip flag, and a bit stream generation unit that generates a bit stream including the encoded data of the transform skip flag generated by the flag encoding unit.

[0024] Also, for example, when generating a bitstream, a transform skip flag is generated, which is flag information indicating for each component whether or not to skip the transform process for converting the residual between an image and a predicted image of that image into coefficient data when encoding an image, the generated transform skip flag is encoded, encoded data of the transform skip flag is generated, and a bitstream including the encoded data of the generated transform skip flag is generated.

[0025] In this way, the setting of the transform skip for each component can be provided to the decoding side. Therefore, when the transform skip is applied to the color component, the decoding side can correctly decode the bit stream. Therefore, the reduction in coding efficiency can be suppressed compared to the case where the transform skip is applicable only to the luminance component.

[0026] Then, the conversion skip flag may be obtained on the decoding side.

[0027] For example, an image processing device may include a flag decoding unit that decodes encoded data of a transform skip flag corresponding to a component identifier and obtains a transform skip flag corresponding to the component identifier.

[0028] Also, for example, when generating coefficient data, encoded data of a transform skip flag corresponding to a component identifier may be decoded to obtain the transform skip flag corresponding to that component identifier.

[0029] In this way, the transform skip setting for each component indicated by the transform skip flag can be applied during decoding. Therefore, the bit stream of the color component to which the transform skip is applied can be correctly decoded. Therefore, the reduction in coding efficiency can be suppressed compared to the case where the transform skip is applicable only to the luminance component.

[0030] Also, when encoding an image, the conversion process of a color component may be controlled based on the setting of the conversion skip for each component by the above-mentioned conversion skip flag. In other words, the conversion skip may be applied to the conversion process of a color component. In this way, it is possible to suppress a decrease in the encoding efficiency compared to the case where the conversion skip is applicable only to the luminance component.

[0031] Similarly, when decoding the encoded data of an image, an inverse transform process for transforming coefficient data of a color component into a residual between an image and a predicted image may be controlled based on the transform skip setting for each component by the transform skip flag described above. That is, a transform skip may be applied to the inverse transform process of the color component. In this way, the bit stream of the color component to which the transform skip is applied can be correctly decoded. Therefore, it is possible to suppress a decrease in encoding efficiency compared to a case where the transform skip is applicable only to the luminance component.

[0032] As described above, by setting the transform skip for each component, the transform skip can be set for the color components independently of the luminance component. Therefore, the transform skip for the color components can be set depending on whether the transform skip is valid for the color components. In this way, it is possible to suppress a decrease in coding efficiency compared to the case where the transform skip setting for the luminance component is applied to the color components.

[0033] <Encoding mode control> Furthermore, in the case of the method described in Non-Patent Document 1, the encoding mode (decoding mode) of the coefficient data of the luminance component was controlled depending on the presence or absence of a transform skip. For example, when a transform skip was performed, the coefficient data of the luminance component was encoded in a TS residual encoding mode optimized for the coefficient data that was transformed skipped (the encoded data of the coefficient data of the luminance component was decoded in a TS residual decoding mode optimized for the coefficient data that was transformed skipped). In contrast, when a transform skip was not performed, the coefficient data of the luminance component was encoded in a non-TS residual encoding mode optimized for the coefficient data that was transformed (the encoded data of the coefficient data of the luminance component was decoded in a non-TS residual decoding mode optimized for the coefficient data that was transformed).

[0034] However, since the transform skip was not applied to the chrominance components, the coding mode (decoding mode) was not controlled in this way. Therefore, when the transform skip is applied to the chrominance components as described above, the coding efficiency may be reduced compared to when the coding mode (decoding mode) is controlled in this way.

[0035] Therefore, such control of the encoding mode (decoding mode) may also be applied when applying transform skip to color components as in the above-mentioned method 1. For example, as in method 1-1 in the second row from the top of the table in Fig. 1, for each value of the component identifier, when transform skip is applied, the TS residual encoding mode (TS residual decoding mode) may be applied as the encoding mode (decoding mode), and when transform skip is not applied (in the case of non-transform skip), the non-TS residual encoding mode (non-TS residual decoding mode) may be applied as the encoding mode (decoding mode).

[0036] For example, based on a transform skip flag corresponding to a component identifier, the encoding mode of the coefficient data corresponding to that component identifier may be controlled to be a TS residual encoding mode, which is a mode for when the transform process is skipped, or a non-TS residual encoding mode, which is a mode for when the transform process is not skipped, and the coefficient data corresponding to the component identifier may be encoded using the encoding mode thus set, thereby generating encoded data of the coefficient data.

[0037] Also, for example, based on a transform skip flag corresponding to a component identifier, the decoding mode of the encoded data of coefficient data corresponding to that component identifier may be controlled to be a TS residual decoding mode, which is a mode for skipping the inverse transform process of converting coefficient data into a residual between an image and a predicted image, or a non-TS residual decoding mode, which is a mode for not skipping the inverse transform process, and the encoded data of the coefficient data corresponding to the component identifier may be decoded using the decoding mode thus set, to generate coefficient data corresponding to that component identifier.

[0038] By doing this, it is possible to apply an encoding mode (decoding mode) according to the setting of the transform skip, thereby making it possible to suppress reduction in encoding efficiency more than when encoding (decoding) is performed using a single encoding mode (decoding mode).

[0039] <Transform skip residual coding use flag> A flag indicating selection between a TS residual coding mode (TS residual decoding mode) and a non-TS residual coding mode (non-TS residual decoding mode) may be applied. For example, as in method 1-1-1 in the third row from the top of the table in Fig. 1, a residual coding mode selection flag (also called a transform skip residual coding use flag) indicating a coding mode selection setting may be applied.

[0040] For example, when encoding, a transform skip residual encoding usage flag may be generated, which is flag information indicating whether to apply a TS residual encoding mode or a non-TS residual encoding mode, and the transform skip residual encoding usage flag may be encoded to generate a bitstream including encoded data of the transform skip residual encoding usage flag.

[0041] Also, at the time of decoding, the encoded data of the transform skip residual coding usage flag may be decoded, a transform skip residual coding usage flag corresponding to the component identifier may be generated, and based on the transform skip residual coding usage flag corresponding to the component identifier, the decoding mode of the encoded data of the coefficient data corresponding to the component identifier may be controlled to be the TS residual decoding mode or the non-TS residual decoding mode.

[0042] For example, such a transform skip residual coding use flag may be set at a high level such as a sequence parameter. In this way, even an encoder or decoder that does not support transform skip can correctly perform coding and decoding based on this flag. Therefore, it is possible to suppress a decrease in coding efficiency. In other words, it is possible to omit the implementation of the TS residual coding mode (TS residual decoding mode) in the encoder or decoder, and it is possible to suppress an increase in circuit scale.

[0043] <Conversion skip residual coding use specific mode flag> A flag may be applied to indicate whether to apply a TS residual coding mode (TS residual decoding mode) or a non-TS residual coding mode (non-TS residual decoding mode) in a specific mode. For example, a flag (also referred to as a transform skip residual coding use specific mode flag) that enables selection of a coding mode in a specific mode may be applied, as in method 1-1-2 in the fourth row from the top of the table in Fig. 1.

[0044] For example, when encoding, a transform skip residual coding use specific mode flag may be generated, which is flag information indicating whether to apply a TS residual coding mode or a non-TS residual coding mode in a specific mode, and the transform skip residual coding use specific mode flag may be encoded to generate a bitstream including encoded data of the transform skip residual coding use specific mode flag.

[0045] In addition, at the time of decoding, the encoded data of the transform skip residual coding use specific mode flag may be decoded, a transform skip residual coding use specific mode flag corresponding to the component identifier may be generated, and based on the transform skip residual coding use specific mode flag corresponding to the component identifier, the decoding mode of the encoded data of the coefficient data corresponding to the component identifier may be controlled to be the TS residual decoding mode or the non-TS residual decoding mode.

[0046] For example, such a transform skip residual coding use specific mode flag may be set at a high level of a sequence parameter, etc. In this way, in a specific mode, it becomes possible to switch between a TS residual decoding mode and a non-TS residual decoding mode.

[0047] The above methods 1, 1-1, 1-1-1, and 1-1-2 will be described later in the first embodiment.

[0048] <Quantization parameter correction> In non-patent document 3, when a quantization parameter QP<4 is applied to a luminance transform block to which a luminance transform skip is applied, there is a risk of a decrease in PSNR (Peak Signal-to-Noise Ratio). As a countermeasure, a method is proposed in which the quantization parameter QP applied to the luminance transform block in the case of a luminance transform skip is clipped to QP=4 or QP=QpPrimeTsMin (minimum TSQP).

[0049] Similarly, when transform skip is applied to color components, there is a risk of a decrease in PSNR. However, in the past, transform skip has not been applied to color components, and naturally, this point has not been taken into consideration.

[0050] Therefore, for example, as in method 1-2 in the fifth row from the top of the table in Fig. 1, for each value of the component identifier, in the case of transform skip, the quantization parameter may be corrected. In other words, when transform skip is applied, the quantization parameter may be corrected. And, such control may be performed for each component.

[0051] For example, in an image processing device, when encoding an image, if a transform skip flag corresponding to a component identifier indicates a transform skip, the image processing device may be provided with a quantization parameter correction unit that corrects a quantization parameter to be applied to a transform block to be processed that corresponds to the component identifier, and a quantization unit that quantizes the transform block to be processed that corresponds to the component identifier using the quantization parameter corrected by the quantization parameter correction unit.

[0052] For example, in a quantization coefficient generation method, when encoding an image, if a transform skip flag corresponding to a component identifier indicates a transform skip, a quantization parameter to be applied to a transform block to be processed corresponding to the component identifier may be corrected, and the corrected quantization parameter may be used to quantize the transform block to be processed corresponding to the component identifier, thereby generating a quantization coefficient corresponding to the component identifier.

[0053] For example, in an image processing device, when a transform skip flag corresponding to a component identifier indicates a transform skip, the image processing device may be provided with a quantization parameter correction unit that corrects a quantization parameter to be applied to a transform block to be processed that corresponds to the component identifier, and an inverse quantization unit that uses the quantization parameter corrected by the quantization parameter correction unit to perform inverse quantization of the transform block to be processed that corresponds to the component identifier.

[0054] For example, in a coefficient data generation method, when a transform skip flag corresponding to a component identifier indicates a transform skip, a quantization parameter to be applied to a transform block to be processed corresponding to the component identifier may be corrected, and the corrected quantization parameter may be used to perform inverse quantization of the transform block to be processed corresponding to the component identifier, thereby generating coefficient data corresponding to the component identifier.

[0055] In this way, it is possible to suppress a decrease in PSNR in the encoder or the decoder, or in both.

[0056] For example, in encoding, if a transform skip flag corresponding to a component identifier indicates a transform skip, the larger of the minimum quantization parameter for that transform skip and the quantization parameter corresponding to that component identifier may be set as the quantization parameter to be applied to the transform block to be processed corresponding to that component identifier, and if the transform skip flag corresponding to that component identifier indicates a non-transform skip that does not perform a transform skip, the quantization parameter corresponding to that component identifier may be set as the quantization parameter to be applied to the transform block to be processed corresponding to that component identifier.

[0057] Also, for example, in decoding, if a transform skip flag corresponding to a component identifier indicates a transform skip, the larger of the minimum quantization parameter for that transform skip and the quantization parameter corresponding to that component identifier may be set as the quantization parameter to be applied to the transform block to be processed corresponding to that component identifier, and if the transform skip flag corresponding to that component identifier indicates a non-transform skip that does not perform a transform skip, the quantization parameter corresponding to that component identifier may be set as the quantization parameter to be applied to the transform block to be processed corresponding to that component identifier.

[0058] In this way, it is possible to suppress a decrease in PSNR in the encoder or the decoder, or in both.

[0059] For example, in decoding, encoded data of a transform skip flag corresponding to a component identifier is decoded, and if the transform skip flag corresponding to that component identifier indicates a transform skip, the quantization parameter to be applied to the transform block to be processed corresponding to that component identifier may be corrected.

[0060] In this way, it is possible to control the correction of the quantization parameter based on the transform skip flag corresponding to the coded component identifier.

[0061] The above method 1-2 will be described later in the second embodiment.

[0062] <Sharing context variables> Furthermore, when introducing a transform skip to a color component as described above and encoding a transform block of the color component to which the transform skip is applied using a TS residual encoding mode, if independent context variables are used for the luminance component and the chrominance component, it is necessary to add a new context variable for the color component, which increases the memory capacity for storing the context variables and may result in increased hardware costs.

[0063] Therefore, for example, in the case of transform skip, as in method 1-3 in the sixth row from the top of the table in Figure 1, the context variables corresponding to each binIdx in the bin string of each syntax may be shared between the luminance component and the chrominance component.

[0064] For example, in the case of skipping the transform process in the encoding, the context variables may be shared between the encoding of the luminance component of the coefficient data and the encoding of the chrominance component.

[0065] Furthermore, for example, when the inverse transform process is skipped in decoding, the context variables may be shared between the decoding of the coded data of the luminance component of the coefficient data and the decoding of the coded data of the chrominance component.

[0066] In this way, it is possible to suppress an increase in memory size for holding context variables, and therefore an increase in hardware costs.

[0067] The above methods 1 to 3 will be described later in the third embodiment.

[0068] <Sign code encoding / decoding mode control> Also, for example, as in method 1-4 in the seventh row from the top of the table in FIG. 1, the encoding / decoding method of the sign code may be switched depending on the transform skip flag corresponding to the component identifier.

[0069] For example, as in method 1-4-1 in the eighth row from the top of the table in FIG. 1, in the case of not using transform skip, bypass encoding / decoding may be applied to encoding / decoding of the sign code.

[0070] Also, for example, as in method 1-4-2 in the ninth row from the top of the table in Figure 1, in the case of transform skip, if the number of remaining context coding bins is equal to or greater than a threshold, context coding / decoding may be applied to the coding / decoding of the sign code, and otherwise bypass coding / decoding may be applied to the coding / decoding of the sign code.

[0071] In this way, it is possible to suppress a decrease in coding efficiency.

[0072] The above methods 1-4, 1-4-1, and 1-4-2 will be described later in the fourth embodiment.

[0073] <3. First embodiment> <3-1. Image encoding device> The present technology described above can be applied to any apparatus, device, system, etc. For example, the present technology described above can be applied to an image encoding device that encodes image data.

[0074] Fig. 2 is a block diagram showing an example of the configuration of an image encoding device, which is one aspect of an image processing device to which the present technology is applied. The image encoding device 100 shown in Fig. 2 is a device that encodes image data of a moving image. For example, the image encoding device 100 implements a technology described in at least one of the non-patent documents mentioned above, and encodes image data of a moving image in a manner conforming to a standard described in any of those documents.

[0075] Note that Fig. 2 shows the main processing units, data flows, etc., and is not limited to all that is shown in Fig. 2. In other words, in the image encoding device 100, there may be processing units that are not shown as blocks in Fig. 2, and there may be processing and data flows that are not shown as arrows, etc. in Fig. 2. This is the same in other figures that explain processing units, etc. in the image encoding device 100.

[0076] 2, the image coding device 100 includes a control unit 101, a rearrangement buffer 111, a calculation unit 112, an orthogonal transformation unit 113, a quantization unit 114, an encoding unit 115, and an accumulation buffer 116. The image coding device 100 also includes an inverse quantization unit 117, an inverse orthogonal transformation unit 118, a calculation unit 119, an in-loop filter unit 120, a frame memory 121, a prediction unit 122, and a rate control unit 123.

[0077] <Control Unit> The control unit 101 divides the video data held by the rearrangement buffer 111 into blocks (CU, PU, ​​transform block, etc.) of processing units based on an external or pre-specified block size of the processing units. The control unit 101 also determines the coding parameters (header information Hinfo, prediction mode information Pinfo, transform information Tinfo, filter information Finfo, etc.) to be supplied to each block based on, for example, RDO (Rate-Distortion Optimization). For example, the control unit 101 can set a transform skip flag, etc.

[0078] These coding parameters will be described in detail later. After determining the coding parameters as described above, the control unit 101 supplies them to each block. More specifically, the process is as follows.

[0079] The header information Hinfo is supplied to each block. The prediction mode information Pinfo is supplied to the encoding unit 115 and the prediction unit 122. The transformation information Tinfo is supplied to the encoding unit 115, the orthogonal transformation unit 113, the quantization unit 114, the inverse quantization unit 117, and the inverse orthogonal transformation unit 118. The filter information Finfo is supplied to the in-loop filter unit 120.

[0080] <Sorting Buffer> Each field (input image) of video data is input to the image coding device 100 in its playback order (display order). The reordering buffer 111 acquires and holds (stores) each input image in its playback order (display order). Under the control of the control unit 101, the reordering buffer 111 reorders the input images in coding order (decoding order) and divides them into blocks, which are processing units. The reordering buffer 111 supplies each processed input image to the calculation unit 112.

[0081] <Arithmetic section> The calculation unit 112 subtracts the predicted image P supplied from the prediction unit 122 from the image corresponding to the block of processing units supplied from the sorting buffer 111 to derive a prediction residual D, and supplies it to the orthogonal transformation unit 113.

[0082] <Orthogonal transformation section> The orthogonal transform unit 113 receives the prediction residual supplied from the calculation unit 112 and the transformation information Tinfo supplied from the control unit 101, and performs an orthogonal transform on the prediction residual based on the transformation information Tinfo to derive a transformation coefficient Coeff. The orthogonal transform unit 113 supplies the obtained transformation coefficient to the quantization unit 114.

[0083] <Quantization part> The quantization unit 114 receives the transform coefficients supplied from the orthogonal transform unit 113 and the transform information Tinfo supplied from the control unit 101, and scales (quantizes) the transform coefficients based on the transform information Tinfo. The quantization rate is controlled by the rate control unit 123. The quantization unit 114 supplies the quantized transform coefficients (also referred to as quantized transform coefficient levels) level obtained by such quantization to the encoding unit 115 and the inverse quantization unit 117.

[0084] <Encoding part> The encoding unit 115 receives as input the quantization transformation coefficient levels supplied from the quantization unit 114, various encoding parameters (header information Hinfo, prediction mode information Pinfo, transformation information Tinfo, filter information Finfo, etc.) supplied from the control unit 101, information regarding filters such as filter coefficients supplied from the in-loop filter unit 120, and information regarding the optimal prediction mode supplied from the prediction unit 122.

[0085] The encoding unit 115 performs entropy encoding (lossless encoding) such as CABAC (Context-based Adaptive Binary Arithmetic Code) or CAVLC (Context-based Adaptive Variable Length Code) on the quantized transform coefficient levels to generate a bit string (encoded data).

[0086] Furthermore, the encoding unit 115 derives residual information Rinfo from the quantized transform coefficient levels, encodes the residual information Rinfo, and generates a bit string.

[0087] Furthermore, the encoding unit 115 includes information on the filter supplied from the in-loop filter unit 120 in the filter information Finfo, and includes information on the optimal prediction mode supplied from the prediction unit 122 in the prediction mode information Pinfo. Then, the encoding unit 115 encodes the various encoding parameters (header information Hinfo, prediction mode information Pinfo, transformation information Tinfo, filter information Finfo, etc.) described above to generate a bit string.

[0088] The encoding unit 115 multiplexes the bit strings of the various pieces of information generated as described above to generate encoded data, and supplies the encoded data to the accumulation buffer 116.

[0089] <Accumulation buffer> The accumulation buffer 116 temporarily holds the encoded data obtained in the encoding unit 115. The accumulation buffer 116 outputs the held encoded data, for example as a bit stream, to the outside of the image encoding device 100 at a predetermined timing. For example, this encoded data is transmitted to the decoding side via an arbitrary recording medium, an arbitrary transmission medium, an arbitrary information processing device, etc. In other words, the accumulation buffer 116 is also a transmission unit that transmits the encoded data (bit stream).

[0090] <Inverse quantization part> The inverse quantization unit 117 performs processing related to inverse quantization. For example, the inverse quantization unit 117 receives the quantized transform coefficient level level supplied from the quantization unit 114 and the transformation information Tinfo supplied from the control unit 101 as input, and scales (inverse quantizes) the value of the quantized transform coefficient level based on the transformation information Tinfo. Note that this inverse quantization is an inverse process of the quantization performed in the quantization unit 114. The inverse quantization unit 117 supplies the transformation coefficient Coeff_IQ obtained by such inverse quantization to the inverse orthogonal transformation unit 118. Note that the inverse quantization unit 117 is the same as the inverse quantization unit (described later) on the decoding side, and therefore the description (described later) on the decoding side can be applied to the inverse quantization unit 117.

[0091] <Inverse orthogonal transform section> The inverse orthogonal transform unit 118 performs processing related to inverse orthogonal transform. For example, the inverse orthogonal transform unit 118 receives the transform coefficients supplied from the inverse quantization unit 117 and the transform information Tinfo supplied from the control unit 101, and performs inverse orthogonal transform on the transform coefficients based on the transform information Tinfo to derive a prediction residual D'. Note that this inverse orthogonal transform is an inverse process of the orthogonal transform performed in the orthogonal transform unit 113. The inverse orthogonal transform unit 118 supplies the prediction residual obtained by such inverse orthogonal transform to the calculation unit 119. Note that the inverse orthogonal transform unit 118 is the same as the inverse orthogonal transform unit (described later) on the decoding side, and therefore the description of the decoding side (described later) can be applied to the inverse orthogonal transform unit 118.

[0092] <Arithmetic section> The calculation unit 119 receives as input the prediction residual D' supplied from the inverse orthogonal transform unit 118 and the predicted image P supplied from the prediction unit 122. The calculation unit 119 adds the prediction residual to a predicted image corresponding to the prediction residual to derive a locally decoded image. The calculation unit 119 supplies the derived locally decoded image to the in-loop filter unit 120 and the frame memory 121.

[0093] <In-loop filter section> The in-loop filter unit 120 performs processing related to in-loop filter processing. For example, the in-loop filter unit 120 receives as input a locally decoded image supplied from the calculation unit 119, filter information Finfo supplied from the control unit 101, and an input image (original image) supplied from the rearrangement buffer 111. Note that any information is input to the in-loop filter unit 120, and information other than the above information may be input. For example, a prediction mode, motion information, a code amount target value, a quantization parameter QP, a picture type, information on a block (CU, CTU, etc.), etc. may be input to the in-loop filter unit 120 as necessary.

[0094] The in-loop filter unit 120 performs appropriate filtering on the locally decoded image based on the filter information Finfo. The in-loop filter unit 120 also uses the input image (original image) and other input information for the filtering, as necessary.

[0095] For example, the in-loop filter unit 120 can apply four in-loop filters, a bilateral filter, a deblocking filter (DBF (DeBlocking Filter)), an adaptive offset filter (SAO (Sample Adaptive Offset)), and an adaptive loop filter (ALF (Adaptive Loop Filter)), in this order, as described in Non-Patent Document 1. Note that which filters are applied and in what order they are applied are arbitrary and can be appropriately selected.

[0096] Of course, the filtering process performed by the in-loop filter unit 120 is arbitrary and is not limited to the above example. For example, the in-loop filter unit 120 may apply a Wiener filter or the like.

[0097] The in-loop filter unit 120 supplies the filtered locally decoded image to the frame memory 121. When transmitting information about the filter, such as a filter coefficient, to the decoding side, the in-loop filter unit 120 supplies the information about the filter to the encoding unit 115.

[0098] <Frame memory> The frame memory 121 performs processing related to storage of image-related data. For example, the frame memory 121 receives as input a locally decoded image supplied from the calculation unit 119 or a locally decoded image subjected to a filter process supplied from the in-loop filter unit 120, and holds (stores) the input. The frame memory 121 also uses the locally decoded image to reconstruct a decoded image for each picture, and holds the reconstructed image (stores the reconstructed image in a buffer in the frame memory 121). The frame memory 121 supplies the decoded image (or a part thereof) to the prediction unit 122 in response to a request from the prediction unit 122.

[0099] <Prediction Department> The prediction unit 122 performs processing related to the generation of a predicted image. For example, the prediction unit 122 receives as input the prediction mode information Pinfo supplied from the control unit 101, the input image (original image) supplied from the rearrangement buffer 111, and the decoded image (or a part thereof) read from the frame memory 121. The prediction unit 122 performs prediction processing such as inter prediction or intra prediction using the prediction mode information Pinfo and the input image (original image), performs prediction by referring to the decoded image as a reference image, performs motion compensation processing based on the prediction result, and generates a predicted image. The prediction unit 122 supplies the generated predicted image to the calculation unit 112 and the calculation unit 119. The prediction unit 122 also supplies information on the prediction mode selected by the above processing, that is, the optimal prediction mode, to the encoding unit 115 as necessary.

[0100] <Rate control section> The rate control unit 123 performs processing related to rate control. For example, the rate control unit 123 controls the rate of the quantization operation of the quantization unit 114 based on the code amount of the encoded data accumulated in the accumulation buffer 116 so as to prevent overflow or underflow from occurring.

[0101] <Encoding part> Fig. 3 is a block diagram showing an example of the main configuration of the encoding unit 115 in Fig. 2. As shown in Fig. 3, the encoding unit 115 has a conversion mode information encoding unit 150, a control unit 151, a selection unit 152, a TS residual encoding unit 153, a non-TS residual encoding unit 154, and a selection unit 155.

[0102] The transform mode information encoding unit 150 performs processing related to encoding of transform mode information (transform_mode). This transform mode information is information related to the mode of transform processing by the orthogonal transform unit 113. For example, the transform mode information may include a transform skip flag (transform_skip_flag[xTbY][yTbY][cIdx]), an identifier related to primary transform (mts_idx[xTbY][yTbY][cIdx]), etc.

[0103] For example, the transform mode information encoding unit 150 may acquire transform mode information supplied from the control unit 101. The transform mode information encoding unit 150 may also encode the acquired transform mode information to generate encoded data of the transform mode information. The transform mode information encoding unit 150 may also supply the generated encoded data of the transform mode information to the accumulation buffer 116 (i.e., provide it to the decoding side).

[0104] The control unit 151 performs processing related to the control of the coding mode. For example, the control unit 151 may acquire transform mode information (transform_skip_flag[xTbY][yTbY][cIdx], mts_idx[xTbY][yTbY][cIdx], etc.) and a component identifier (cIdx) supplied from the control unit 101. Furthermore, the control unit 151 may switch between a TS residual coding mode and a non-TS residual coding mode as a coding mode for coefficient data (quantization coefficients) by controlling the selection of the selection unit 152 and the selection unit 155 based on the transform skip flag corresponding to the component identifier. For example, when the transform skip is applied, the control unit 151 connects the selection unit 152 and the selection unit 155 to the TS residual coding unit 153. Furthermore, for example, when the transform skip is not applied (when the transform process is performed), the control unit 151 connects the selection unit 152 and the selection unit 155 to the non-TS residual coding unit 154.

[0105] The selection unit 152 performs processing related to selection of a supply destination of coefficient data (quantization coefficients). For example, the selection unit 152 may acquire the quantization coefficients supplied from the quantization unit 114. Furthermore, the selection unit 152 may supply the acquired quantization coefficients to the TS residual encoding unit 153 or the non-TS residual encoding unit 154 (the one designated by the control unit 151) in accordance with the control of the control unit 151. For example, when a transform skip is applied, the selection unit 152 supplies the quantization coefficients to the TS residual encoding unit 153. Furthermore, for example, when a transform skip is not applied (when a transform process is performed), the selection unit 152 supplies the quantization coefficients to the non-TS residual encoding unit 154.

[0106] The TS residual coding unit 153 performs processing related to the TS residual coding mode. For example, the TS residual coding unit 153 may acquire the quantization coefficients supplied from the selection unit 152. Furthermore, the TS residual coding unit 153 may code the acquired quantization coefficients in the TS residual coding mode. This TS residual coding mode is a coding mode for skipping the conversion process. For example, coding in the TS residual coding mode is optimized for coefficient data for which the conversion process has been skipped, such as by scanning from the DC component of the coefficient data toward the high frequency component, or by not transmitting the position information of the last coefficient to the decoding side. A more specific method of the TS residual coding mode is described in Non-Patent Document 4 and the like. The TS residual coding unit 153 may code the quantization coefficients in this way and generate coded data of the quantized coefficients. The TS residual coding unit 153 may supply the coded data generated in this way to the selection unit 155.

[0107] The non-TS residual coding unit 154 performs processing related to the non-TS residual coding mode. For example, the non-TS residual coding unit 154 may acquire the quantized coefficients supplied from the selection unit 152. Furthermore, the non-TS residual coding unit 154 may code the acquired quantized coefficients in the non-TS residual coding mode. This non-TS residual coding mode is a coding mode for the case where a conversion process is performed. For example, coding in the non-TS residual coding mode is optimized for coefficient data that has undergone a conversion process, such as by scanning from the last coefficient of the coefficient data toward the DC component, or by transmitting position information of the last coefficient to the decoding side. A more specific method of the non-TS residual coding mode is described in Non-Patent Document 4 and the like. The non-TS residual coding unit 154 may code the quantized coefficients in this way and generate coded data of the quantized coefficients. The non-TS residual coding unit 154 may supply the coded data generated in this way to the selection unit 155.

[0108] The selection unit 155 performs processing related to selection of a supply source of encoded data. For example, the selection unit 155 may acquire encoded data supplied from the TS residual encoding unit 153 or the non-TS residual encoding unit 154 (the one specified by the control unit 151). For example, when a transform skip is applied, the selection unit 155 acquires encoded data supplied from the TS residual encoding unit 153. Also, for example, when a transform skip is not applied (when a transform process is performed), the selection unit 155 acquires encoded data supplied from the non-TS residual encoding unit 154. The selection unit 155 may supply the encoded data acquired in this manner to the accumulation buffer 116 (i.e., may provide the encoded data to the decoding side).

[0109] <Skip conversion of each component> The image coding device 100 can code a color image having a luminance component and a chrominance component. The image coding device 100 can code not only the luminance component but also the chrominance component by skipping (omitting) the conversion process that converts the residual between an image and its predicted image into coefficient data.

[0110] For example, the control unit 101 can set a transform skip flag (transform_skip_flag) indicating whether or not to apply a transform skip for each component. The component can be indicated by a component identifier (cIdx). That is, the control unit 101 can set the value of the transform skip flag in correspondence with the value of the component identifier.

[0111] That is, the control unit 101 can generate a transform skip flag (transform_skip_flag[xTbY][yTbY][cIdx]) indicating whether or not to skip the transform process for transforming the residual between the image to be coded and its predicted image into coefficient data for each component (cIdx). Furthermore, the transform mode information coding unit 150 of the coding unit 115 can code the transform skip flag generated by the control unit 101 and generate coded data of the transform skip flag. Furthermore, the accumulation buffer 116 can generate a bit stream including the coded data of the transform skip flag generated by the transform mode information coding unit 150 and output the bit stream to the outside of the image coding device 100.

[0112] In this way, the control unit 101 can set a transform skip flag for a color component, for example, and can also apply a transform skip to the color component. Furthermore, the image encoding device 100 can provide the generated transform skip flag (i.e., a transform skip setting for each component) to the decoding side. Therefore, when a transform skip is applied to a color component, the decoding side can correctly decode the bit stream. Therefore, it is possible to suppress a decrease in encoding efficiency compared to a case where a transform skip is applicable only to a luminance component.

[0113] Furthermore, in the image encoding device 100, the transform processing of the color components may be controlled based on the setting of the transform skip for each component by the transform skip flag described above. For example, the orthogonal transform unit 113 may control the transform skip for each component based on the transform skip flag generated by the control unit 101. In this way, the orthogonal transform unit 113 can apply the transform skip to the transform processing of the color components, for example.

[0114] As described above, the control unit 101 sets the transform skip for each component, so that the transform skip can be set for the color components independently of the luminance component. Therefore, the control unit 101 can set the transform skip for the color components depending on whether the transform skip is valid for the color components. This allows the orthogonal transform unit 113 to perform the transform skip for the color components independently of the luminance component. In this way, it is possible to suppress a decrease in the coding efficiency compared to the case where the transform skip setting for the luminance component is applied to the color components.

[0115] <Encoding mode control> Furthermore, the encoding unit 115 may control the encoding mode of the coefficient data corresponding to the component identifier based on a transform skip flag corresponding to the component identifier. For example, the control unit 151, the selection unit 152, and the selection unit 155 of the encoding unit 115 may control whether to use the TS residual encoding mode or the non-TS residual encoding mode based on the transform skip flag. For example, the TS residual encoding unit 153 and the non-TS residual encoding unit 154 may encode the coefficient data corresponding to the component identifier in the encoding mode set in this way, and generate encoded data of the coefficient data.

[0116] That is, a plurality of encoding modes with different characteristics are prepared as candidates, and the encoding unit 115 selects and applies an encoding mode from the plurality of candidates based on a transform skip flag corresponding to a component identifier. That is, the encoding unit 115 encodes coefficient data in the selected encoding mode. In this manner, the encoding unit 115 can apply an encoding mode with characteristics more suitable for the setting of the transform skip among the encoding modes with different characteristics, and thus it is possible to suppress a decrease in encoding efficiency compared to the case of encoding using a single encoding mode.

[0117] <Image encoding process flow> Next, a description will be given of the flow of each process executed by the above-described image encoding device 100. First, an example of the flow of image encoding process will be described with reference to the flowchart in FIG.

[0118] When the image encoding process starts, in step S101, the reordering buffer 111 is controlled by the control unit 101 to reorder the frames of the input video data from the display order to the encoding order.

[0119] In step S102, the control unit 101 sets a processing unit for the input image held by the sorting buffer 111 (performs block division).

[0120] In step S103, the control unit 101 determines (sets) coding parameters for the input image held by the reordering buffer 111.

[0121] In step S104, the control unit 101 generates transform mode information (transform_mode) of the transform block corresponding to the component identifier (cIdx).

[0122] In step S105, the prediction unit 122 performs a prediction process to generate a predicted image, etc., of an optimal prediction mode. For example, in this prediction process, the prediction unit 122 performs intra prediction to generate a predicted image, etc., of an optimal intra prediction mode, performs inter prediction to generate a predicted image, etc., of an optimal inter prediction mode, and selects an optimal prediction mode from among them based on a cost function value, etc.

[0123] In step S106, the calculation unit 112 calculates the difference between the input image and the prediction image of the optimal mode selected by the prediction process in step S105. That is, the calculation unit 112 generates a prediction residual D between the input image and the prediction image. The prediction residual D thus obtained has a reduced data amount compared to the original image data. Therefore, the data amount can be compressed compared to the case where the image is encoded as it is.

[0124] In step S107, the orthogonal transform unit 113 performs an orthogonal transform process on the prediction residual D generated in the process of step S106, in accordance with the transform mode information generated in step S104, and derives the transform coefficient Coeff.

[0125] In step S108, the quantization unit 114 quantizes the transform coefficient Coeff obtained by the process of step S107 by using the quantization parameter calculated by the control unit 101, etc., to derive the quantized transform coefficient level level.

[0126] In step S109, the inverse quantization unit 117 inverse quantizes the quantized transform coefficient level generated by the process of step S108 with characteristics corresponding to the quantization characteristics of step S108, to derive the transform coefficient Coeff_IQ.

[0127] In step S110, the inverse orthogonal transform unit 118 performs inverse orthogonal transform on the transform coefficient Coeff_IQ obtained by the process of step S109 in accordance with the transform mode information generated in step S104, using a method corresponding to the orthogonal transform process of step S107, to derive a prediction residual D'. Note that this inverse orthogonal transform process is similar to the inverse orthogonal transform process (described later) performed on the decoding side, and therefore the explanation (described later) on the decoding side can be applied to the inverse orthogonal transform process of step S110.

[0128] In step S111, the calculation unit 119 generates a locally decoded image by adding the prediction image obtained by the prediction process in step S105 to the prediction residual D' derived in the process in step S110.

[0129] In step S112, the in-loop filter unit 120 performs an in-loop filter process on the locally decoded image derived in the process of step S111.

[0130] In step S113, the frame memory 121 stores the locally decoded image derived by the process in step S111 and the locally decoded image filtered in step S112.

[0131] In step S114, the encoding unit 115 encodes the quantized transform coefficient level LEVEL obtained by the process of step S108 and the transform mode information generated in step S104. For example, the encoding unit 115 encodes the quantized transform coefficient level LEVEL, which is information related to an image, by arithmetic encoding or the like to generate encoded data. At this time, the encoding unit 115 also encodes various encoding parameters (header information Hinfo, prediction mode information Pinfo, transform information Tinfo). Furthermore, the encoding unit 115 derives residual information RInfo from the quantized transform coefficient level LEVEL and encodes the residual information RInfo.

[0132] In step S115, the accumulation buffer 116 accumulates the encoded data thus obtained and outputs it, for example, as a bit stream, to the outside of the image encoding device 100. This bit stream is transmitted to the decoding side, for example, via a transmission path or a recording medium.

[0133] In step S116, the rate control unit 123 performs rate control as necessary.

[0134] When the process of step S116 ends, the image encoding process ends.

[0135] <Encoding process flow> Next, an example of the flow of the encoding process executed in step S114 in FIG. 4 will be described with reference to the flowchart in FIG.

[0136] When the encoding process starts, in step S151, the transform mode information encoding unit 150 of the encoding unit 115 encodes the transform mode information (transform_mode) of the transform block corresponding to the component identifier cIdx.

[0137] In step S152, the control unit 151 derives Condition1 by the following formula (1). That is, the control unit 151 generates Condition1 by using a transform skip flag (transform_skip_flag[cIdx]) corresponding to a component. Note that the transform skip flag (trsnform_skip_flag[cIdx]) may also include coordinates (xTbY, yTbY) of the block to be processed (for example, trsnform_skip_flag[xTbY][yTbY][cIdx]), but this is omitted here for simplicity. It will also be omitted below as appropriate.

[0138] TIFF2025026585000002.tif18132

[0139] In step S153, the control unit 151 determines whether or not Condition1 is true. If Condition1 is true, the transform skip flag (transform_skip_flag[cIdx]) corresponding to the component is true (IS_SKIP). Therefore, the selection unit 152 and the selection unit 155 are connected to the TS residual encoding unit 153 under the control of the control unit 151. As a result, the process proceeds to step S154.

[0140] In step S154, the TS residual encoding unit 153 encodes the quantized coefficients in the TS residual encoding mode to generate encoded data. When the encoded data is generated, the encoding process ends.

[0141] Also, in step S153, if Condition1 is false, the transform skip flag (transform_skip_flag[cIdx]) corresponding to the component is false. Therefore, the selection unit 152 and the selection unit 155 connect to the non-TS residual encoding unit 154 under the control of the control unit 151. As a result, the process proceeds to step S155.

[0142] In step S155, the non-TS residual encoding unit 154 encodes the quantized coefficients in the non-TS residual encoding mode to generate encoded data. When the encoded data is generated, the encoding process ends.

[0143] By performing each process as described above, the image coding device 100 can apply the TS residual coding mode to color components as well, and can suppress a decrease in coding efficiency.

[0144] In the above, it has been described that whether or not a transform skip is performed is notified by the transform skip flag (transform_skip_flag), but this is not limiting and may be notified as one mode of the identifier mts_idx related to the primary transform. The identifier mts_idx is an identifier indicating the horizontal and vertical transform types of the primary transform. In this case, the control unit 151 can derive Condition1 as shown in the following formula (2).

[0145] TIFF2025026585000003.tif21118

[0146] <3-2. Image Decoding Device> FIG. 6 is a block diagram showing an example of the configuration of an image decoding device, which is one aspect of an image processing device to which the present technology is applied. The image decoding device 200 shown in FIG. 6 is a device that decodes encoded data in which a prediction residual between an image and its predicted image is encoded, such as AVC or HEVC. For example, the image decoding device 200 may implement a technology described in the above-mentioned non-patent documents and decode encoded data in which image data of a moving image is encoded in a method that complies with a standard described in any of those documents. For example, the image decoding device 200 may decode encoded data (bit stream) generated by the above-mentioned image encoding device 100.

[0147] Note that Fig. 6 shows the main processing units, data flows, etc., and is not limited to all that are shown in Fig. 6. In other words, in the image decoding device 200, there may be processing units not shown as blocks in Fig. 6, and there may be processing and data flows not shown as arrows, etc. in Fig. 6. This is the same in other figures explaining processing units, etc. in the image decoding device 200.

[0148] 6, the image decoding device 200 includes an accumulation buffer 211, a decoding unit 212, an inverse quantization unit 213, an inverse orthogonal transformation unit 214, a calculation unit 215, an in-loop filter unit 216, a rearrangement buffer 217, a frame memory 218, and a prediction unit 219. Note that the prediction unit 219 includes an intra prediction unit and an inter prediction unit, not shown.

[0149] <Accumulation buffer> The accumulation buffer 211 acquires and holds (stores) the bitstream input to the image decoding device 200. At a predetermined timing, or when a predetermined condition is met, the accumulation buffer 211 extracts coded data included in the accumulated bitstream and supplies the coded data to the decoding unit 212.

[0150] <Decryption section> The decoding unit 212 performs processing related to image decoding. For example, the decoding unit 212 receives the encoded data supplied from the accumulation buffer 211 as input, and performs entropy decoding (lossless decoding) on ​​the syntax values ​​of the syntax elements from the bit string in accordance with the definition of the syntax table, to derive parameters.

[0151] The parameters derived from the syntax elements and the syntax values ​​of the syntax elements include, for example, header information Hinfo, prediction mode information Pinfo, transformation information Tinfo, residual information Rinfo, filter information Finfo, and other information. That is, the decoding unit 212 parses (analyzes and obtains) these pieces of information from the bitstream. These pieces of information will be described below.

[0152] <Header information Hinfo> The header information Hinfo includes, for example, header information such as a video parameter set (VPS), a sequence parameter set (SPS), a picture parameter set (PPS), a picture header (PH), and a slice header (SH). The header information Hinfo includes, for example, information that specifies an image size (horizontal width PicWidth, vertical width PicHeight), bit depth (luminance bitDepthY, chrominance bitDepthC), a chrominance array type ChromaArrayType, a maximum value MaxCUSize / minimum value MinCUSize of CU size, a maximum depth MaxQTDepth / minimum depth MinQTDepth of quad-tree partitioning, a maximum depth MaxBTDepth / minimum depth MinBTDepth of binary-tree partitioning, a maximum value MaxTSSize of transform skip blocks (also called maximum transform skip block size), an on / off flag (also called valid flag) of each encoding tool, and the like.

[0153] For example, the on / off flag of the encoding tool included in the header information Hinfo includes on / off flags related to the conversion and quantization processes shown below. The on / off flag of the encoding tool can also be interpreted as a flag indicating whether or not syntax related to the encoding tool is present in the encoded data. When the value of the on / off flag is 1 (true), it indicates that the encoding tool is usable, and when the value of the on / off flag is 0 (false), it indicates that the encoding tool is not usable. The interpretation of the flag value may be reversed.

[0154] Inter-component prediction enabled flag (ccp_enabled_flag): Flag information indicating whether inter-component prediction (CCP (Cross-Component Prediction), also called CC prediction) is available. For example, when this flag information is "1" (true), it indicates that it is available, and when it is "0" (false), it indicates that it is not available.

[0155] This CCP is also called component-to-component linear prediction (CCLM or CCLMP).

[0156] <Prediction mode information Pinfo> The prediction mode information Pinfo includes, for example, size information PBSize (prediction block size) of the processing target PB (prediction block), intra prediction mode information IPinfo, motion prediction information MVinfo, and other information.

[0157] The intra prediction mode information IPinfo includes, for example, prev_intra_luma_pred_flag, mpm_idx, and rem_intra_pred_mode in JCTVC-W1005, 7.3.8.5 Coding Unit syntax, and a luma intra prediction mode IntraPredModeY derived from the syntax.

[0158] In addition, the intra-prediction mode information IPinfo includes, for example, an inter-component prediction flag (ccp_flag (cclmp_flag)), a multi-class linear prediction mode flag (mclm_flag), a chrominance sample position type identifier (chroma_sample_loc_type_idx), a chrominance MPM identifier (chroma_mpm_idx), and a luma intra-prediction mode (IntraPredModeC) derived from these syntaxes.

[0159] The inter-component prediction flag (ccp_flag (cclmp_flag)) is flag information indicating whether or not inter-component linear prediction is applied. For example, when ccp_flag==1, it indicates that inter-component prediction is applied, and when ccp_flag==0, it indicates that inter-component prediction is not applied.

[0160] The multi-class linear prediction mode flag (mclm_flag) is information on the linear prediction mode (linear prediction mode information). More specifically, the multi-class linear prediction mode flag (mclm_flag) is flag information indicating whether or not to use the multi-class linear prediction mode. For example, "0" indicates a one-class mode (single class mode) (e.g., CCLMP), and "1" indicates a two-class mode (multi-class mode) (e.g., MCLMP).

[0161] The chrominance sample location type identifier (chroma_sample_loc_type_idx) is an identifier that identifies the type of pixel location of the chrominance component (also referred to as the chrominance sample location type).

[0162] Note that this chrominance sample location type identifier (chroma_sample_loc_type_idx) is transmitted (stored in) information about the pixel location of the chrominance component (chroma_sample_loc_info()).

[0163] The chroma MPM identifier (chroma_mpm_idx) is an identifier indicating which prediction mode candidate in the chroma intra prediction mode candidate list (intraPredModeCandListC) is to be specified as the chroma intra prediction mode.

[0164] The motion prediction information MVinfo includes information such as merge_idx, merge_flag, inter_pred_idc, ref_idx_LX, mvp_lX_flag, X={0,1}, mvd, etc. (see, for example, JCTVC-W1005, 7.3.8.6 Prediction Unit Syntax).

[0165] Of course, the prediction mode information Pinfo may include any information, and may include information other than the above information.

[0166] <Conversion information Tinfo> The conversion information Tinfo includes, for example, the following information: Of course, the information included in the conversion information Tinfo is arbitrary, and information other than the above information may be included.

[0167] The width size TBWSize and height TBHSize of the transform block to be processed (or the logarithmic values ​​log2TBWSize and log2TBHSize of TBWSize and TBHSize in base 2) Transform skip flag (ts_flag): A flag indicating whether to skip the (inverse) primary transform and the (inverse) secondary transform. Scan Identifier (scanIdx) Quantization parameter (qp) Quantization matrix (scaling_matrix (e.g., JCTVC-W1005, 7.3.4 Scaling list data syntax))

[0168] <Residual information Rinfo> The residual information Rinfo (see, for example, 7.3.8.11 Residual Coding syntax in JCTVC-W1005) includes, for example, the following syntax:

[0169] cbf(coded_block_flag): Residual data presence flag last_sig_coeff_x_pos: last non-zero coefficient X coordinate last_sig_coeff_y_pos: Y coordinate of the last non-zero coefficient coded_sub_block_flag: Sub-block non-zero coefficient presence / absence flag sig_coeff_flag: Non-zero coefficient presence / absence flag gr1_flag: Flag indicating whether the level of the non-zero coefficient is greater than 1 (also called the GR1 flag) gr2_flag: Flag indicating whether the level of the non-zero coefficient is greater than 2 (also called the GR2 flag) sign_flag: Sign indicating the positive or negative of the non-zero coefficient (also called the sign sign) coeff_abs_level_remaining: The remaining level of non-zero coefficients (also called the non-zero coefficient residual level) etc.

[0170] Of course, the information included in the residual information Rinfo is arbitrary, and information other than the above information may be included.

[0171] <Filter information Finfo> The filter information Finfo includes, for example, control information related to each of the following filter processes:

[0172] Control information for the deblocking filter (DBF) Control information for pixel adaptive offset (SAO) Control information for the adaptive loop filter (ALF) Other control information for linear and nonlinear filters

[0173] More specifically, for example, information specifying a picture to which each filter is applied, an area within a picture, filter On / Off control information for each CU, filter On / Off control information related to slice and tile boundaries, etc. Of course, any information is included in the filter information Finfo, and information other than these pieces of information may be included.

[0174] Returning to the description of the decoding unit 212, the decoding unit 212 derives the quantized transform coefficient level level of each coefficient position in each transform block by referring to the residual information Rinfo. The decoding unit 212 supplies the quantized transform coefficient level to the inverse quantization unit 213.

[0175] Furthermore, the decoding unit 212 supplies the parsed header information Hinfo, prediction mode information Pinfo, transformation information Tinfo, and filter information Finfo to each block.

[0176] The header information Hinfo is supplied to the inverse quantization unit 213, the inverse orthogonal transformation unit 214, the prediction unit 219, and the in-loop filter unit 216. The prediction mode information Pinfo is supplied to the inverse quantization unit 213 and the prediction unit 219. The transformation information Tinfo is supplied to the inverse quantization unit 213 and the inverse orthogonal transformation unit 214. The filter information Finfo is supplied to the in-loop filter unit 216.

[0177] Of course, the above example is merely an example, and the present invention is not limited to this example. For example, each encoding parameter may be supplied to an arbitrary processing unit. In addition, other information may be supplied to an arbitrary processing unit.

[0178] <Inverse quantization part> The inverse quantization unit 213 performs processing related to inverse quantization. For example, the inverse quantization unit 213 receives the transformation information Tinfo and the quantized transformation coefficient level level supplied from the decoding unit 212 as input, scales (inverse quantizes) the value of the quantized transformation coefficient level based on the transformation information Tinfo, and derives the transformation coefficient Coeff_IQ after inverse quantization.

[0179] Note that this inverse quantization is performed as an inverse process of the quantization performed by the quantization unit 114. Also, this inverse quantization is the same process as the inverse quantization performed by the inverse quantization unit 117. That is, the inverse quantization unit 117 performs the same process (inverse quantization) as the inverse quantization unit 213.

[0180] The inverse quantization unit 213 supplies the derived transform coefficient Coeff_IQ to the inverse orthogonal transformation unit 214 .

[0181] <Inverse orthogonal transform section> The inverse orthogonal transform unit 214 performs processing related to inverse orthogonal transform. For example, the inverse orthogonal transform unit 214 receives the transform coefficient Coeff_IQ supplied from the inverse quantization unit 213 and the transform information Tinfo supplied from the decoding unit 212 as input, and performs inverse orthogonal transform processing (inverse transform processing) on ​​the transform coefficient based on the transform information Tinfo to derive a prediction residual D'.

[0182] This inverse orthogonal transform is performed as an inverse process of the orthogonal transform performed by the orthogonal transform unit 113. This inverse orthogonal transform is the same process as the inverse orthogonal transform performed by the inverse orthogonal transform unit 118. That is, the inverse orthogonal transform unit 118 performs the same process (inverse orthogonal transform) as the inverse orthogonal transform unit 214.

[0183] The inverse orthogonal transform unit 214 supplies the derived prediction residual D′ to the calculation unit 215.

[0184] <Arithmetic section> The calculation unit 215 performs processing related to addition of information related to an image. For example, the calculation unit 215 receives as input a prediction residual supplied from the inverse orthogonal transform unit 214 and a predicted image supplied from the prediction unit 219. The calculation unit 215 adds the prediction residual and a predicted image (prediction signal) corresponding to the prediction residual, and derives a locally decoded image.

[0185] The calculation unit 215 supplies the derived locally decoded image to the in-loop filter unit 216 and the frame memory 218.

[0186] <In-loop filter section> The in-loop filter unit 216 performs processing related to in-loop filter processing. For example, the in-loop filter unit 216 receives as input a locally decoded image supplied from the calculation unit 215 and filter information Finfo supplied from the decoding unit 212. Note that any information is input to the in-loop filter unit 216, and information other than the above information may also be input.

[0187] The in-loop filter unit 216 performs appropriate filtering on the locally decoded image based on the filter information Finfo.

[0188] For example, the in-loop filter unit 216 applies four in-loop filters, a bilateral filter, a deblocking filter (DBF (DeBlocking Filter)), an adaptive offset filter (SAO (Sample Adaptive Offset)), and an adaptive loop filter (ALF (Adaptive Loop Filter)), in this order. Note that which filters are applied and in what order they are applied are arbitrary and can be appropriately selected.

[0189] The in-loop filter unit 216 performs a filter process corresponding to the filter process performed by the encoding side (for example, the in-loop filter unit 120 of the image encoding device 100). Of course, the filter process performed by the in-loop filter unit 216 is arbitrary and is not limited to the above example. For example, the in-loop filter unit 216 may apply a Wiener filter or the like.

[0190] The in-loop filter unit 216 supplies the filtered locally decoded image to a reordering buffer 217 and a frame memory 218 .

[0191] <Sorting Buffer> The reordering buffer 217 receives the locally decoded images supplied from the in-loop filter unit 216 and holds (stores) them. The reordering buffer 217 uses the locally decoded images to reconstruct decoded images for each picture and holds them (stores them in the buffer). The reordering buffer 217 reorders the obtained decoded images from the decoding order to the playback order. The reordering buffer 217 outputs the reordered decoded images to the outside of the image decoding device 200 as video data.

[0192] <Frame memory> The frame memory 218 performs processing related to storage of image-related data. For example, the frame memory 218 receives the locally decoded image supplied from the calculation unit 215, reconstructs the decoded image for each picture, and stores the reconstructed image in a buffer within the frame memory 218.

[0193] Furthermore, the frame memory 218 receives as input the locally decoded image that has been in-loop filtered and is supplied from the in-loop filter unit 216, reconstructs a decoded image for each picture, and stores the reconstructed image in a buffer within the frame memory 218. The frame memory 218 appropriately supplies the stored decoded image (or a part thereof) to the prediction unit 219 as a reference image.

[0194] The frame memory 218 may store header information Hinfo, prediction mode information Pinfo, transformation information Tinfo, filter information Finfo, and the like, which are related to the generation of a decoded image.

[0195] <Encoding part> Fig. 7 is a block diagram showing an example of the main configuration of the decoding unit 212 in Fig. 6. As shown in Fig. 7, the decoding unit 212 has a transform mode information decoding unit 250, a control unit 251, a selection unit 252, a TS residual decoding unit 253, a non-TS residual decoding unit 254, and a selection unit 255.

[0196] The transform mode information decoding unit 250 performs processing related to decoding of encoded data of transform mode information (transform_mode). This transform mode information is information related to the mode of inverse transform processing by the inverse orthogonal transform unit 214. For example, the transform mode information may include a transform skip flag (transform_skip_flag[xTbY][yTbY][cIdx]), an identifier related to primary transform (mts_idx[xTbY][yTbY][cIdx]), etc.

[0197] For example, the transform mode information decoding unit 250 may obtain encoded data supplied from the accumulation buffer 211. The transform mode information decoding unit 250 may also decode the obtained encoded data and generate transform mode information and a component identifier (cIdx). Furthermore, the transform mode information encoding unit 150 may supply the generated transform mode information, etc. to the control unit 251.

[0198] The control unit 251 performs processing related to control of the decoding mode. For example, the control unit 251 may acquire transform mode information (transform_skip_flag[xTbY][yTbY][cIdx], mts_idx[xTbY][yTbY][cIdx], etc.) and a component identifier (cIdx) supplied from the transform mode information decoding unit 250. Furthermore, the control unit 251 may switch between a TS residual decoding mode and a non-TS residual decoding mode as a decoding mode of the encoded data of the coefficient data by controlling the selection of the selection unit 252 and the selection unit 255 based on the transform skip flag corresponding to the component identifier. For example, when the transform skip is applied, the control unit 251 connects the selection unit 252 and the selection unit 255 to the TS residual decoding unit 253. Furthermore, for example, when the transform skip is not applied (when the transform process is performed), the control unit 251 connects the selection unit 252 and the selection unit 255 to the non-TS residual decoding unit 254.

[0199] The selection unit 252 performs processing related to selection of a supply destination of the coded data of coefficient data (quantized coefficients). For example, the selection unit 252 may acquire coded data supplied from the accumulation buffer 211. Furthermore, the selection unit 252 may supply the acquired coded data to the TS residual decoding unit 253 or the non-TS residual decoding unit 254 (the one designated by the control unit 251) in accordance with the control of the control unit 251. For example, when a transform skip is applied, the selection unit 252 supplies the coded data to the TS residual decoding unit 253. Furthermore, for example, when a transform skip is not applied (when a transform process is performed), the selection unit 252 supplies the coded data to the non-TS residual decoding unit 254.

[0200] The TS residual decoding unit 253 performs processing related to the TS residual decoding mode. For example, the TS residual decoding unit 253 may obtain the encoded data supplied from the selection unit 252. Furthermore, the TS residual decoding unit 253 may decode the obtained encoded data in the TS residual decoding mode. This TS residual decoding mode is a decoding mode for skipping the conversion process. For example, the decoding in the TS residual decoding mode corresponds to the encoding in the TS residual encoding mode, and is optimized for the encoded data of the coefficient data in which the conversion process has been skipped. A more specific method of the TS residual decoding mode is described in Non-Patent Document 4 and the like. The TS residual decoding unit 253 may decode the encoded data in this way and generate the quantization coefficients. The TS residual decoding unit 253 may supply the quantization coefficients generated in this way to the selection unit 255.

[0201] The non-TS residual decoding unit 254 performs processing related to the non-TS residual decoding mode. For example, the non-TS residual decoding unit 254 may obtain the encoded data supplied from the selection unit 252. Furthermore, the non-TS residual decoding unit 254 may decode the obtained encoded data in the non-TS residual decoding mode. This non-TS residual decoding mode is a decoding mode for the case where a transform process is performed. For example, the decoding in the non-TS residual decoding mode corresponds to the encoding in the non-TS residual coding mode, and is optimized for the encoded data of the coefficient data that has been subjected to the transform process. A more specific method of the non-TS residual decoding mode is described in Non-Patent Document 4 and the like. The non-TS residual decoding unit 254 may decode the encoded data in this way and generate a quantization coefficient. The non-TS residual decoding unit 254 may supply the quantization coefficient generated in this way to the selection unit 255.

[0202] The selection unit 255 performs processing related to selection of a supply source of the quantization coefficient. For example, the selection unit 255 may acquire the quantization coefficient supplied from the TS residual decoding unit 253 or the non-TS residual decoding unit 254 (the one specified by the control unit 251). For example, when a transform skip is applied, the selection unit 255 acquires the quantization coefficient supplied from the TS residual decoding unit 253. Also, for example, when a transform skip is not applied (when a transform process is performed), the selection unit 255 acquires the coded data supplied from the non-TS residual decoding unit 254. The selection unit 255 may supply the quantization coefficient thus acquired to the inverse quantization unit 213.

[0203] <Skip conversion of each component> The image decoding device 200 can decode coded data of a color image having a luminance component and a chrominance component. The image decoding device 200 can skip (omit) an inverse transform process for converting coefficient data generated by decoding coded data not only for the luminance component but also for the chrominance component into a residual between an image and its predicted image.

[0204] For example, the transform mode information decoding unit 250 can decode the encoded data of the transform skip flag corresponding to the component identifier, and obtain the transform skip flag corresponding to that component identifier.

[0205] In this way, the transform skip setting for each component indicated by the transform skip flag can be applied during decoding. Therefore, the bit stream of the color component to which the transform skip is applied can be correctly decoded. Therefore, the reduction in coding efficiency can be suppressed compared to the case where the transform skip is applicable only to the luminance component.

[0206] Furthermore, when decoding the encoded data of an image, the inverse transform process of a color component may be controlled based on the setting of the transform skip for each component by the transform skip flag described above. That is, the transform skip may be applied to the inverse transform process of a color component. In this way, the bit stream of the color component to which the transform skip is applied can be correctly decoded. Therefore, it is possible to suppress a decrease in encoding efficiency compared to the case where the transform skip is applicable only to the luminance component.

[0207] As described above, by setting the transform skip for each component, the transform skip can be set for the color components independently of the luminance component. Therefore, the transform skip for the color components can be set depending on whether the transform skip is valid for the color components. In this way, it is possible to suppress a decrease in coding efficiency compared to the case where the transform skip setting for the luminance component is applied to the color components.

[0208] <Encoding mode control> In addition, such control of the decoding mode may also be applied when applying transform skip to color components as in the above-mentioned method 1. For example, when applying transform skip to each value of the component identifier, the TS residual decoding mode may be applied as the decoding mode, and in the case of non-transform skip where transform skip is not applied, the non-TS residual decoding mode may be applied as the decoding mode.

[0209] For example, based on a transform skip flag corresponding to a component identifier, the decoding mode of the encoded data of coefficient data corresponding to that component identifier may be controlled to be a TS residual decoding mode, which is a mode for skipping the inverse transform process of converting coefficient data into a residual between an image and a predicted image, or a non-TS residual decoding mode, which is a mode for not skipping the inverse transform process, and the encoded data of the coefficient data corresponding to the component identifier may be decoded using the decoding mode thus set to generate coefficient data corresponding to that component identifier.

[0210] In this way, a decoding mode according to the setting of the transform skip can be applied, so that a decrease in coding efficiency can be suppressed more than when decoding is performed in a single decoding mode.

[0211] <Flow of image decoding process> Next, a description will be given of the flow of each process executed by the image decoding device 200 configured as above. First, an example of the flow of an image decoding process will be described with reference to the flowchart in FIG.

[0212] When the image decoding process starts, in step S401, the accumulation buffer 211 acquires and holds (accumulates) a bitstream supplied from outside the image decoding device 200.

[0213] In step S202, the decoding unit 212 extracts and decodes the coded data from the bit stream to obtain the quantized transform coefficient level. Furthermore, the decoding unit 212 parses (analyzes and obtains) various coding parameters from the bit stream through this decoding.

[0214] In step S203, the inverse quantization unit 213 performs inverse quantization, which is the inverse process of the quantization performed on the encoding side, on the quantized transform coefficient level obtained by the process in step S202 to obtain a transform coefficient Coeff_IQ.

[0215] In step S204, the inverse orthogonal transform unit 214, under the control of step S203, performs inverse orthogonal transform processing, which is the inverse processing of the orthogonal transform processing performed on the encoding side, on the transform coefficient Coeff_IQ obtained in step S203, to obtain a prediction residual D'.

[0216] In step S205, the prediction unit 219 performs a prediction process using a prediction method specified by the encoding side based on the information parsed in step S202, and generates a predicted image P by, for example, referring to a reference image stored in the frame memory 218.

[0217] In step S206, the calculation unit 215 adds the prediction residual D′ obtained in step S204 and the predicted image P obtained in step S205 to obtain a locally decoded image R local Derive.

[0218] In step S207, the in-loop filter unit 216 filters the locally decoded image R obtained by the process in step S206. local In-loop filtering is performed on

[0219] In step S208, the sorting buffer 217 sorts the filtered locally decoded image R obtained by the process in step S207. local The decoded images R are derived using the above formula, and the order of the decoded images R is rearranged from the decoding order to the playback order. The decoded images R rearranged in the playback order are output to the outside of the image decoding device 200 as a moving image.

[0220] In step S209, the frame memory 218 stores the locally decoded image R obtained in step S206. local , and the locally decoded image R obtained by the process of step S207 after the filtering process. local At least one of the above is stored.

[0221] When the process of step S209 ends, the image decoding process ends.

[0222] <Decryption process flow> Next, an example of the flow of the decoding process executed in step S202 in FIG. 8 will be described with reference to the flowchart in FIG.

[0223] When the decoding process starts, in step S251, the transform mode information decoding unit 250 of the decoding unit 212 decodes the transform mode information (transform_mode) of the transform block corresponding to the component identifier cIdx.

[0224] In step S252, the control unit 251 derives Condition1 by the above-mentioned formula (1). That is, the control unit 251 generates Condition1 by using the transform skip flag (transform_skip_flag[cIdx]) corresponding to the component.

[0225] In step S253, the control unit 251 determines whether or not this Condition1 is true. If Condition1 is true, the transform skip flag (transform_skip_flag[cIdx]) corresponding to the component is true (IS_SKIP). Therefore, the selection unit 252 and the selection unit 255 are connected to the TS residual decoding unit 253 under the control of the control unit 251. As a result, the process proceeds to step S254.

[0226] In step S254, the TS residual decoding unit 253 decodes the encoded data in the TS residual decoding mode to generate quantized coefficients. When the quantized coefficients are generated, the decoding process ends.

[0227] Also, in step S253, if Condition1 is false, the transform skip flag (transform_skip_flag[cIdx]) corresponding to the component is false. Therefore, the selection unit 252 and the selection unit 255 connect to the non-TS residual decoding unit 254 under the control of the control unit 251. As a result, the process proceeds to step S255.

[0228] In step S255, the non-TS residual decoding unit 254 encodes the encoded data in the non-TS residual decoding mode to generate quantized coefficients. When the quantized coefficients are generated, the decoding process ends.

[0229] By performing each process as described above, the image decoding device 200 can apply the TS residual decoding mode to color components as well, and can suppress a decrease in coding efficiency.

[0230] In the above, it has been described that whether or not a transform skip is performed is notified by the transform skip flag (transform_skip_flag), but this is not limiting, and it may be notified as one mode of the identifier mts_idx related to the primary transform. The identifier mts_idx is an identifier indicating the horizontal and vertical transform types of the primary transform. In this case, the control unit 251 can derive Condition1 as in the above-mentioned formula (2).

[0231] <3-3. Syntax and semantics> An example of the syntax of TU (transform-unit) in this case is shown in Figure 10. In the example of Figure 10, the transform mode information (transform_mode(...,0), transform_mode(...,0), transform_mode(...,2)) corresponding to each component is derived, and whether or not to apply transform skip (residual_coding(...,cIdx) or residual_ts_coding(...,cIdx)) is determined for each component (tu_cbf_luma, tu_cbf_cb, tu_cbf_cr).

[0232] An example of the syntax of the transformation mode information is shown in FIG. 11. In the example of FIG. 11, a transformation skip flag (transform_skip_flag[x0][y0][cIdx]) for each component and a transformation type identifier (mts_idx[x0][y0][cIdx]) of the primary transformation for each component are set as the transformation mode information (transform_mode(..., cIdx)) for each component. In this case, the conditions for generating this information for each component are the same. The contents of this condition are arbitrary. For example, a transformation skip enable flag (sps_transform_skip_enabled_flag) set in the sequence parameter set (SPS) is used as the condition.

[0233] FIG. 12 shows an example of the syntax of transform skip (residual_ts_coding(..., cIdx)). As shown in FIG. 12, the setting of the transform skip is performed for each component (cIdx). Also, an example of the semantics of the transform skip flag (transform_skip_flag[x0][y0][cIdx]) of the transform mode information is shown in FIG. 13A. Furthermore, an example of the semantics of the transform type identifier (mts_idx[x0][y0][cIdx]) of the primary transform is shown in FIG. 13B. In this way, information about the transform skip is set for each component. Therefore, as described above, the transform skip can be applied to the color component, and a decrease in coding efficiency can be suppressed.

[0234] In addition, in the syntax of the transformation mode information, the conditions for each component for generating the transformation skip flag (transform_skip_flag[x0][y0][cIdx]) for each component and the transformation type identifier (mts_idx[x0][y0][cIdx]) of the primary transformation for each component may be different from each other as in the example of Fig. 14. By preparing conditions for each component in this way, it is possible to suppress redundancy of the judgment conditions and an increase in the processing load.

[0235] <3-4. Transform skip residual coding use flag> Note that a transform skip residual coding usage flag may be applied, which indicates the selection between a TS residual coding mode (TS residual decoding mode) and a non-TS residual coding mode (non-TS residual decoding mode).

[0236] An example of the syntax of the sequence parameter set (SPS) is shown in Fig. 15. For example, the control unit 101 may set a transform skip residual coding use flag sps_ts_residual_coding_use_flag in this sequence parameter set.

[0237] This sps_ts_residual_coding_use_flag is a transform skip residual coding use flag notified at the sequence parameter set level. When the value of this flag is "1", it indicates that the TS residual coding mode is applied when the transform skip occurs. When the value of this flag is "0", it indicates that the non-TS residual coding mode is applied when the transform skip occurs.

[0238] An example of the syntax of a TU in this case is shown in Fig. 16. In the example of Fig. 16, the transform skip residual coding use flag is used as a condition for determining whether to apply the transform skip. The semantics of sps_ts_residual_coding_use_flag are shown in Fig. 17. In this case, Condition1 is derived using, for example, the following formula (3).

[0239] TIFF2025026585000004.tif22131

[0240] When using the conversion type identifier of the primary conversion, it is derived as shown in (4) below.

[0241] TIFF2025026585000005.tif24133

[0242] By using such a high-level flag, even an encoder or decoder that does not support transform skip can correctly perform encoding and decoding based on this flag. Therefore, it is possible to suppress a decrease in encoding efficiency. In other words, it is possible to omit the implementation of the TS residual encoding mode (TS residual decoding mode) in the encoder or decoder, and it is possible to suppress an increase in the circuit scale.

[0243] The data unit for setting the transform skip residual coding use flag is arbitrary and may be other than the sequence parameter set. For example, the transform skip residual coding use flag may be notified at the CU level, slice level, picture level, etc. The finer the data unit granularity, the more freedom there is in switching modes, and the more room there is for improving coding efficiency.

[0244] <3-5. Transform skip residual coding use specific mode flag> Also, in a particular mode, a flag may be applied that indicates whether to apply a TS residual coding mode (TS residual decoding mode) or a non-TS residual coding mode (non-TS residual decoding mode).

[0245] An example of the syntax of the sequence parameter set (SPS) is shown in Fig. 18. For example, the control unit 101 may set a transform skip residual coding use specific mode flag sps_ts_residual_coding_use_for_bdpcm_flag in this sequence parameter set.

[0246] This sps_ts_residual_coding_use_for_bdpcm_flag is a flag notified at the sequence parameter set level, and is flag information indicating that coding mode selection is enabled in BDPCM (Block_based Differential Pulse Code Modulation). When the value of this flag is "1", it indicates that the TS residual coding mode is applied in the case of BDPCM. When the value of this flag is "0", it indicates that the non-TS residual coding mode is applied in the case of BDPCM.

[0247] An example of the syntax of the TU in this case is shown in Fig. 19. In the example of Fig. 19, the transform skip residual coding use specific mode flag is used as a condition for determining whether to apply the transform skip. The semantics of sps_ts_residual_coding_use_for_bdpcm_flag are shown in Fig. 20. In this case, Condition1 is derived, for example, by using the following formula (5).

[0248] TIFF2025026585000006.tif17141

[0249] When using the conversion type identifier of the primary conversion, it is derived as shown in (6) below.

[0250] TIFF2025026585000007.tif20141

[0251] By using such a high-level flag, it becomes possible to switch between TS residual coding mode and non-TS residual coding mode in BDPCM.

[0252] The data unit for setting the transform skip residual coding use specific mode flag is arbitrary and may be other than the sequence parameter set. For example, the transform skip residual coding use specific mode flag may be notified at the CU level, slice level, picture level, etc. The finer the data unit granularity, the greater the freedom of mode switching and the greater the room for improving coding efficiency.

[0253] <4. Second embodiment> <4-1. Correction of quantization parameters in quantization> For each value of the component identifier, in the case of a transform skip, the quantization parameter may be corrected. That is, when a transform skip is applied, the quantization parameter may be corrected. And, such control may be performed for each component.

[0254] <Quantization part> In this case, the image encoding device 100 is similar to the example in Fig. 2. Fig. 21 is a block diagram showing an example of the main configuration of the quantization unit 114 in this case. As shown in Fig. 21, the quantization unit 114 in this case has a QP correction unit 311 and a quantization processing unit 312.

[0255] The QP correction unit 311 performs processing related to correction of the quantization parameter. For example, the QP correction unit 311 can derive a quantization parameter Qp to be applied to a processing target transform block corresponding to the component identifier cIdx by referring to a transform skip flag corresponding to the component identifier cIdx, joint chrominance encoding mode information TuResMode, a QP (Qp') of a CU level corresponding to the component identifier cIdx, and a minimum QP (QpPrimeTsMin) of the transform skip. The types of this quantization parameter include Qp'y, Qp'cb, Qp'cr, and Qp'cbcr. In addition, the minimum quantization parameter QpPrimeTsMin of the transform skip is notified by a parameter set.

[0256] The joint chrominance encoding mode is a mode in which only one of the chrominance components (Cb, Cr) is transmitted, and the other is derived from the other and not transmitted. For example, only the residual of Cr is encoded and transmitted, and the residual of Cb is derived from the residual of Cr and not transmitted. The joint chrominance encoding mode information TuResMode is information related to such a joint chrominance encoding mode.

[0257] The QP correction unit 311 supplies the corrected quantization parameter (corrected QP(qP)) to the quantization processing unit 312.

[0258] The quantization processing unit 312 quantizes the coefficient data (transform coefficients) using the quantization parameter (corrected QP(qP)) supplied from the QP correction unit 311 to generate quantized coefficients. The quantization processing unit 312 supplies the generated quantized coefficients to the encoding unit 115 and the inverse quantization unit 117.

[0259] <Quantization process flow> An example of the flow of the quantization process executed in step S108 in Fig. 4 will be described with reference to the flowchart in Fig. 22. When the quantization process starts, the QP correction unit 311 of the quantization unit 114 derives Condition2 by the following formula (7) in step S301. That is, the QP correction unit 311 generates Condition2 by using a transform skip flag (transform_skip_flag[cIdx]) corresponding to the component.

[0260] TIFF2025026585000008.tif17130

[0261] When using the conversion type identifier of the primary conversion, it is derived as shown in (8) below.

[0262] TIFF2025026585000009.tif20113

[0263] In step S302, the QP correction unit 311 determines whether or not this Condition 2 is true. If Condition 2 is true, that is, if the transform skip flag (transform_skip_flag[cIdx]) corresponding to the component is true (IS_SKIP), the process proceeds to step S303.

[0264] In step S303, the QP correction unit 311 corrects the quantization parameter QP. In this case, the QP correction unit 311 sets, for example, the larger value of the minimum quantization parameter (QpPrimeTsMin) of the transform skip and the quantization parameter QP'x of the CU level as the quantization parameter (corrected QP(qP)) to be applied to the target transform block corresponding to the component identifier. When the process of step S303 ends, the process proceeds to step S305.

[0265] Also, if it is determined that Condition2 is false, the process proceeds to step S304. In this case, the QP correction unit 311 sets, for example, a quantization parameter QP'x at the CU level as a quantization parameter (corrected QP(qP)) to be applied to the target transform block corresponding to the component identifier. When the process of step S304 ends, the process proceeds to step S305.

[0266] In step S305, the quantization processor 312 quantizes the transform coefficients using the quantization parameters updated in step S303 or step S304. When the process of step S305 ends, the quantization process ends, and the process returns to FIG.

[0267] In this way, the reduction in PSNR can be suppressed.

[0268] <4-2. Correction of quantization parameters in inverse quantization> Similarly, in the inverse quantization, for each value of the component identifier, in the case of a transform skip, the quantization parameter may be corrected. That is, when a transform skip is applied, the quantization parameter may be corrected. And, such control may be performed for each component.

[0269] <Inverse quantization part> In this case, the image decoding device 200 is similar to the example in Fig. 6. Fig. 23 is a block diagram showing an example of the main configuration of the inverse quantization unit 213 in this case. As shown in Fig. 23, the inverse quantization unit 213 in this case has a QP correction unit 411 and an inverse quantization processing unit 412.

[0270] The QP correction unit 411 performs processing related to correction of the quantization parameter. For example, the QP correction unit 411 can derive a quantization parameter Qp to be applied to a processing target transform block corresponding to the component identifier cIdx by referring to a transform skip flag corresponding to the component identifier cIdx, joint chrominance encoding mode information TuResMode, a QP (Qp') of a CU level corresponding to the component identifier cIdx, and a minimum QP (QpPrimeTsMin) of the transform skip. The types of this quantization parameter include Qp'y, Qp'cb, Qp'cr, and Qp'cbcr. In addition, the minimum quantization parameter QpPrimeTsMin of the transform skip is notified by a parameter set.

[0271] The QP correction unit 411 supplies the corrected quantization parameter (corrected QP(qP)) to the inverse quantization processing unit 412.

[0272] The inverse quantization unit 412 inverse quantizes the quantized coefficients using the quantization parameter (corrected QP(qP)) supplied from the QP correction unit 411, and generates coefficient data (transformation coefficients). The inverse quantization unit 412 supplies the generated coefficient data to the inverse orthogonal transformation unit 214.

[0273] <Flow of inverse quantization process> An example of the flow of the inverse quantization process executed in step S203 in Fig. 8 in this case will be described with reference to the flowchart in Fig. 24. When the inverse quantization process is started, the QP correction unit 411 of the inverse quantization unit 213 derives Condition2 by the above-mentioned equation (7) in step S401. That is, the QP correction unit 411 generates Condition2 by using a transform skip flag (transform_skip_flag[cIdx]) corresponding to the component. This transform skip flag is obtained by decoding by the transform mode information decoding unit 250 as described in the first embodiment.

[0274] In step S402, the QP corrector 411 determines whether or not Condition 2 is true. If Condition 2 is true, that is, if the transform skip flag (transform_skip_flag[cIdx]) corresponding to the component is true (IS_SKIP), the process proceeds to step S403.

[0275] In step S403, the QP correction unit 411 corrects the quantization parameter QP. In this case, the QP correction unit 411 sets, for example, the larger value of the minimum quantization parameter (QpPrimeTsMin) of the transform skip and the quantization parameter QP'x of the CU level as the quantization parameter (corrected QP(qP)) to be applied to the target transform block corresponding to the component identifier. When the process of step S403 ends, the process proceeds to step S405.

[0276] Also, if it is determined that Condition2 is false, the process proceeds to step S404. In this case, the QP correction unit 411 sets, for example, a quantization parameter QP'x at the CU level as a quantization parameter (corrected QP(qP)) to be applied to the target transform block corresponding to the component identifier. When the process of step S404 ends, the process proceeds to step S405.

[0277] In step S405, the inverse quantization unit 412 inverse quantizes the quantized coefficients using the quantization parameters updated in step S403 or step S404. When the process of step S405 ends, the inverse quantization process ends, and the process returns to FIG.

[0278] In this way, the reduction in PSNR can be suppressed.

[0279] <4-3.Syntax> An example of the syntax of the quantization parameter in this case is shown in Fig. 25. As described above, the quantization parameter qP is corrected for each component using the transform skip flag transform_skip_flag.

[0280] <5. Third embodiment> <Sharing context variables> When applying the transform skip as described above, the context variables corresponding to each binIdx in the bin string of each syntax may be made to be shared between the luminance component and the chrominance component.

[0281] For example, in the case of skipping the transform process in encoding, the context variables may be shared between the encoding of the luminance component of the coefficient data and the encoding of the chrominance component.

[0282] Furthermore, for example, when the inverse transform process is skipped during decoding, the context variables may be shared between the decoding of the coded data of the luminance component of the coefficient data and the decoding of the coded data of the chrominance component.

[0283] An example of the syntax related to the context variables in this case is shown in Fig. 26. When transform skip is applied as in the example of Fig. 26, the context variables may be derived using a method common to the luminance component and the chrominance component.

[0284] Moreover, another example of the syntax related to the context variables is shown in Fig. 27. When a transform skip is applied as in the example of Fig. 27, the context variables may be derived for the luminance component and the chrominance component using methods independent of each other.

[0285] As described above, by sharing the context variables corresponding to each binIdx of the bin string of each syntax between the luminance component and the color component, it is possible to suppress an increase in the memory size for storing the context variables, thereby suppressing an increase in the hardware cost.

[0286] <6. Fourth embodiment> <Sign code encoding / decoding mode control> Also, the encoding / decoding method of the sign code may be switched depending on the transform skip flag corresponding to the component identifier. An example of the syntax in this case is shown in FIG.

[0287] As shown in the column of binIdx = 0 in the table shown in Fig. 28, for example, in the case of no transform skip, bypass coding / decoding may be applied to the coding / decoding of the sign code. Also, for example, in the case of transform skip, if the number of remaining context coding bins is equal to or greater than a threshold, context coding / decoding may be applied to the coding / decoding of the sign code, and otherwise bypass coding / decoding may be applied to the coding / decoding of the sign code ((MaxCcbs > 0) ? (0...5) : bypass).

[0288] In this way, it is possible to suppress a decrease in coding efficiency.

[0289] <7. Notes> <Computer> The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed in a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, etc., capable of executing various functions by installing various programs.

[0290] FIG. 29 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes by a program.

[0291] In a computer 900 shown in FIG. 29, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are connected to each other via a bus 904.

[0292] An input / output interface 910 is also connected to the bus 904. To the input / output interface 910, an input unit 911, an output unit 912, a storage unit 913, a communication unit 914, and a drive 915 are connected.

[0293] The input unit 911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, an input terminal, etc. The output unit 912 includes, for example, a display, a speaker, an output terminal, etc. The storage unit 913 includes, for example, a hard disk, a RAM disk, a non-volatile memory, etc. The communication unit 914 includes, for example, a network interface. The drive 915 drives removable media 921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0294] In the computer configured as above, the series of processes described above are performed by the CPU 901, for example, loading a program stored in the storage unit 913 into the RAM 903 via the input / output interface 910 and the bus 904 and executing the program. The RAM 903 also stores data and the like necessary for the CPU 901 to execute various processes as appropriate.

[0295] The program executed by the computer can be applied by recording it on a removable medium 921 such as a package medium, for example. In this case, the program can be installed in the storage unit 913 via the input / output interface 910 by mounting the removable medium 921 on the drive 915.

[0296] This program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program can be received by the communication unit 914 and installed in the storage unit 913.

[0297] In addition, this program can also be installed in advance in the ROM 902 or the storage unit 913.

[0298] <Unit of information processing> The data units in which the various information described above is set and the data units targeted by the various processes are each arbitrary and are not limited to the above examples. For example, these pieces of information and processes may be set for each TU (Transform Unit), TB (Transform Block), PU (Prediction Unit), PB (Prediction Block), CU (Coding Unit), LCU (Largest Coding Unit), sub-block, block, tile, slice, picture, sequence, or component, or may target data of these data units. Of course, this data unit may be set for each piece of information or process, and the data units of all pieces of information and processes do not need to be unified. The storage location of these pieces of information is arbitrary, and may be stored in the header or parameter set of the above-mentioned data unit. Also, they may be stored in multiple locations.

[0299] <Control information> Control information related to the present technology described in each of the above embodiments may be transmitted from the encoding side to the decoding side. For example, control information (e.g., enabled_flag) that controls whether or not to permit (or prohibit) application of the present technology described above may be transmitted. In addition, for example, control information (e.g., present_flag) indicating a target to which the present technology described above is applied (or a target to which it is not applied) may be transmitted. For example, control information that specifies a block size (upper limit or lower limit, or both), frame, component, or layer to which the present technology is applied (or permitted or prohibited to be applied) may be transmitted.

[0300] <Target of application of this technology> This technology can be applied to any image encoding / decoding method. In other words, as long as it does not conflict with the above-mentioned technology, the specifications of various processes related to image encoding / decoding, such as transform (inverse transform), quantization (inverse quantization), encoding (decoding), and prediction, are arbitrary and are not limited to the above-mentioned examples. In addition, as long as it does not conflict with the above-mentioned technology, some of these processes may be omitted.

[0301] This technology can also be applied to a multi-viewpoint image encoding / decoding system that encodes and decodes multi-viewpoint images that include images from multiple views. In that case, this technology can be applied to the encoding / decoding of each view.

[0302] Furthermore, this technology can be applied to a hierarchical image coding (scalable coding) and decoding system that performs coding and decoding of hierarchical images that are layered to have a scalability function for a predetermined parameter. In that case, this technology can be applied to the coding and decoding of each layer.

[0303] Furthermore, although the image encoding device 100 and the image decoding device 200 have been described above as application examples of the present technology, the present technology can be applied to any configuration.

[0304] For example, the present technology can be applied to various electronic devices, such as transmitters and receivers (e.g., television sets and mobile phones) in satellite broadcasting, wired broadcasting such as cable TV, distribution over the Internet, and distribution to terminals via cellular communication, or devices (e.g., hard disk recorders and cameras) that record images on media such as optical disks, magnetic disks, and flash memories, and play images from these storage media.

[0305] Furthermore, for example, the present technology can also be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module (e.g., a video module) using multiple processors, a unit (e.g., a video unit) using multiple modules, or a set in which other functions are added to a unit (e.g., a video set).

[0306] In addition, for example, the present technology can also be applied to a network system composed of multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and cooperate with each other to perform processing via a network. For example, the present technology may be implemented in a cloud service that provides services related to images (video images) to any terminal such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.

[0307] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

[0308] <Fields and applications where this technology can be applied> Systems, devices, processing units, etc. to which the present technology is applied can be used in any field, such as transportation, medical care, crime prevention, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. In addition, the applications are also arbitrary.

[0309] For example, the present technology can be applied to systems and devices used for providing ornamental content, etc. Furthermore, for example, the present technology can be applied to systems and devices used for traffic, such as monitoring traffic conditions and automatic driving control. Furthermore, for example, the present technology can be applied to systems and devices used for security. Furthermore, for example, the present technology can be applied to systems and devices used for automatic control of machines, etc. Furthermore, for example, the present technology can be applied to systems and devices used for agriculture and livestock farming. Furthermore, for example, the present technology can be applied to systems and devices that monitor natural conditions, such as volcanoes, forests, and oceans, and wildlife, etc. Furthermore, for example, the present technology can be applied to systems and devices used for sports.

[0310] <Other> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Thus, the value that this "flag" can take may be, for example, a binary value of 1 / 0, or may be three or more values. In other words, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, it is assumed that identification information (including flags) is not only included in the bit stream, but also includes difference information of the identification information with respect to certain reference information in the bit stream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also difference information with respect to the reference information.

[0311] Also, various information (metadata, etc.) related to the coded data (bit stream) may be transmitted or recorded in any form as long as it is associated with the coded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. That is, data associated with each other may be combined into one piece of data, or each piece of data may be made into separate data. For example, information associated with coded data (image) may be transmitted on a transmission path different from that of the coded data (image). Also, for example, information associated with coded data (image) may be recorded on a recording medium different from that of the coded data (image) (or on a different recording area of ​​the same recording medium). Note that this "association" may be a part of the data, not the entire data. For example, an image and information corresponding to the image may be associated with each other in any unit, such as multiple frames, one frame, or a part of a frame.

[0312] In this specification, the terms "combine," "multiplex," "append," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0313] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0314] For example, the configuration described above as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or processing unit). Furthermore, as long as the configuration and operation of the system as a whole are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0315] Also, for example, the above-mentioned program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.

[0316] Also, for example, each step of one flowchart may be executed by one device, or may be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes may be executed by one device, or may be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.

[0317] Also, for example, the processing of the steps of a program executed by a computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel, or individually at a required timing such as when a call is made. In other words, the processing of each step may be executed in an order different from the order described above, so long as no contradiction occurs. Furthermore, the processing of the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0318] Also, for example, a plurality of techniques related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any of a plurality of the present technologies can be implemented in combination. For example, a part or all of the present technology described in any of the embodiments can be implemented in combination with a part or all of the present technology described in other embodiments. Also, a part or all of any of the above-mentioned present technologies can be implemented in combination with other technologies not described above.

[0319] The present technology can also be configured as follows. (1) a flag generation unit that generates a transformation skip flag, which is flag information indicating, for each component, whether or not to skip a transformation process for transforming a residual between an image and a prediction image of the image into coefficient data, in encoding an image; a flag encoding unit that encodes the transform skip flag generated by the flag generation unit and generates encoded data of the transform skip flag; a bit stream generating unit that generates a bit stream including encoded data of the transform skip flag generated by the flag encoding unit; An image processing device comprising: (2) a mode control unit that controls, based on the transform skip flag corresponding to the component identifier generated by the flag generation unit, whether a coding mode of coefficient data corresponding to the component identifier is a TS residual coding mode, which is a mode for skipping the transform process, or a non-TS residual coding mode, which is a mode for not skipping the transform process; and a coefficient data encoding unit that encodes coefficient data corresponding to the component identifier in the encoding mode set by the mode control unit, and generates encoded data of the coefficient data; Further equipped with The bit stream generation unit generates a bit stream including the encoded data of the transform skip flag generated by the flag encoding unit and the encoded data of the coefficient data generated by the coefficient data encoding unit. An image processing device as described in (1). (3) When the conversion process is skipped, the coefficient data encoding unit shares a context variable between encoding a luminance component and encoding a chrominance component of the coefficient data. An image processing device as described in (2). (4) The coefficient data encoding unit applies a sign code encoding method according to the transform skip flag generated by the flag generating unit. An image processing device according to (2) or (3). (5) The flag generation unit further generates a transform skip residual coding use flag, which is flag information indicating whether to apply a TS residual coding mode, which is a mode for skipping the transform process, or a non-TS residual coding mode, which is a mode for not skipping the transform process; and The flag encoding unit further encodes the transform skip residual encoding use flag generated by the flag generation unit to generate encoded data of the transform skip residual encoding use flag; The bitstream generation unit generates the bitstream further including encoded data of the transform skip residual encoding use flag generated by the flag encoding unit. An image processing device according to any one of (1) to (4). (6) The flag generation unit further generates a transform skip residual coding use specific mode flag, which is flag information indicating whether to apply a TS residual coding mode, which is a mode for skipping the transform process, or a non-TS residual coding mode, which is a mode for not skipping the transform process, in a specific mode; The flag encoding unit further encodes the transform skip residual encoding use specific mode flag generated by the flag generation unit to generate encoded data of the transform skip residual encoding use specific mode flag; The bit stream generating unit generates the bit stream further including encoded data of the transform skip residual encoding use specific mode flag generated by the flag encoding unit. An image processing device according to any one of (1) to (5). (7) generating a transform skip flag which is flag information indicating, for each component, whether or not to skip a transform process for transforming a residual between the image and a predicted image of the image into coefficient data in encoding the image; encoding the generated transform skip flag to generate encoded data of the transform skip flag; A bitstream generating method for generating a bitstream including encoded data of the generated transform skip flag.

[0320] (8) In encoding an image, when a transform skip flag corresponding to a component identifier indicates a transform skip for skipping a transform process for converting a residual between the image and a predicted image of the image into a transform coefficient, a quantization parameter correction unit corrects a quantization parameter to be applied to a transform block to be processed corresponding to the component identifier. a quantization unit that quantizes a target transform block corresponding to the component identifier by using the quantization parameter corrected by the quantization parameter correction unit; An image processing device comprising: (9) The quantization parameter correction unit When the transform skip flag corresponding to the component identifier indicates the transform skip, a larger value of the minimum quantization parameter of the transform skip and the quantization parameter corresponding to the component identifier is set as a quantization parameter to be applied to the transform block to be processed corresponding to the component identifier; When the transform skip flag corresponding to the component identifier indicates a non-transform skip in which the transform skip is not performed, the quantization parameter corresponding to the component identifier is set as the quantization parameter to be applied to the target transform block corresponding to the component identifier. An image processing device according to (8). (10) In encoding an image, when a transform skip flag corresponding to a component identifier indicates a transform skip for skipping a transform process for converting a residual between the image and a predicted image of the image into a transform coefficient, a quantization parameter to be applied to a transform block to be processed corresponding to the component identifier is corrected; Using the corrected quantization parameter, the transform block to be processed corresponding to the component identifier is quantized to generate a quantized coefficient corresponding to the component identifier. Quantization coefficient generation method.

[0321] (11) a flag decoding unit that decodes encoded data of a transform skip flag corresponding to a component identifier to obtain the transform skip flag corresponding to the component identifier; a mode control unit that controls, based on the transform skip flag corresponding to the component identifier obtained by the flag decoding unit, whether a decoding mode of encoded data of coefficient data corresponding to the component identifier is set to a TS residual decoding mode, which is a mode for skipping an inverse transform process for transforming coefficient data into a residual between an image and a predicted image, or a non-TS residual decoding mode, which is a mode for not skipping the inverse transform process; a coefficient data decoding unit that decodes the encoded data of the coefficient data corresponding to the component identifier in accordance with the decoding mode set by the mode control unit, and generates the coefficient data corresponding to the component identifier; An image processing device comprising: (12) When the inverse transform process is skipped, the coefficient data decoding unit shares a context variable between the decoding of the coded data of a luminance component of the coefficient data and the decoding of the coded data of a chrominance component of the coefficient data. The image processing device according to (11). (13) The image processing device according to (11) or (12), wherein the coefficient data decoding unit applies a sign code decoding method according to the transform skip flag corresponding to the component identifier obtained by the flag decoding unit. (14) The flag decoding unit further includes a flag decoding unit that decodes encoded data of a transform skip residual coding usage flag, which is flag information indicating whether to apply the TS residual decoding mode or the non-TS residual decoding mode when skipping the inverse transform process, and obtains the transform skip residual coding usage flag corresponding to the component identifier; and the mode control unit further controls, based on the transform skip residual coding usage flag corresponding to the component identifier generated by the flag decoding unit, whether to set the decoding mode of the encoded data of the coefficient data corresponding to the component identifier to the TS residual decoding mode or the non-TS residual decoding mode. An image processing device according to any one of (11) to (13). (15) The flag decoding unit further decodes encoded data of a transform skip residual coding use specific mode flag, which is flag information indicating whether to apply the TS residual decoding mode or the non-TS residual decoding mode in a specific mode, to obtain the transform skip residual coding use specific mode flag corresponding to the component identifier, The mode control unit further controls whether the decoding mode of the encoded data of the coefficient data corresponding to the component identifier is the TS residual decoding mode or the non-TS residual decoding mode, based on the transform skip residual encoding use specific mode flag corresponding to the component identifier generated by the flag decoding unit. An image processing device according to any one of (11) to (14). (16) Decoding encoded data of a transform skip flag corresponding to a component identifier to obtain the transform skip flag corresponding to the component identifier; based on the transform skip flag corresponding to the obtained component identifier, control whether a decoding mode of the encoded data of the coefficient data corresponding to the component identifier is set to a TS residual decoding mode, which is a mode for skipping an inverse transform process for transforming the coefficient data into a residual between an image and a predicted image, or a non-TS residual decoding mode, which is a mode for not skipping the inverse transform process; The encoded data of the coefficient data corresponding to the component identifier is decoded according to the set decoding mode, and the coefficient data corresponding to the component identifier is generated. Coefficient data generation method.

[0322] (17) A quantization parameter correction unit that corrects a quantization parameter to be applied to a target transform block corresponding to a component identifier when a transform skip flag corresponding to the component identifier indicates a transform skip that skips an inverse transform process that transforms coefficient data into a residual between an image and a predicted image; an inverse quantization unit that performs inverse quantization of a target transform block corresponding to the component identifier, using the quantization parameter corrected by the quantization parameter correction unit; An image processing device comprising: (18) The quantization parameter correction unit When the transform skip flag corresponding to the component identifier indicates the transform skip, a larger value of the minimum quantization parameter of the transform skip and the quantization parameter corresponding to the component identifier is set as the quantization parameter to be applied to the target transform block corresponding to the component identifier; When the transform skip flag corresponding to the component identifier indicates a non-transform skip in which the transform skip is not performed, the quantization parameter corresponding to the component identifier is set as the quantization parameter to be applied to the target transform block corresponding to the component identifier. The image processing device according to (17). (19) A flag decoding unit that decodes encoded data of the transform skip flag corresponding to the component identifier to obtain the transform skip flag corresponding to the component identifier, The quantization parameter correction unit corrects the quantization parameter to be applied to the target transform block corresponding to the component identifier when the transform skip flag corresponding to the component identifier obtained by the flag decoding unit indicates the transform skip. The image processing device according to (17) or (18). (20) When a transform skip flag corresponding to a component identifier indicates a transform skip for skipping an inverse transform process for transforming coefficient data into a residual between an image and a predicted image, a quantization parameter to be applied to a transform block to be processed corresponding to the component identifier is corrected; Using the corrected quantization parameter, inverse quantization is performed on the transform block to be processed corresponding to the component identifier, and the coefficient data corresponding to the component identifier is generated. Coefficient data generation method. [Explanation of symbols]

[0323] 100 image encoding device, 101 control unit, 114 quantization unit, 115 encoding unit, 150 transform mode information encoding unit, 151 control unit, 152 selection unit, 153 TS residual encoding unit, 154 non-TS residual encoding unit, 155 selection unit, 200 image decoding device, 212 decoding unit, 213 inverse quantization unit, 250 transform mode information decoding unit, 251 control unit, 252 selection unit, 253 TS residual decoding unit, 254 non-TS residual decoding unit, 255 selection unit, 311 QP correction unit, 312 quantization processing unit, 411 QP correction unit, 412 inverse quantization processing unit

Claims

1. a flag generating unit that generates a transform skip flag which is flag information indicating whether or not a transform process for transforming a residual between an image and a predicted image of the image into coefficient data is to be skipped for each component in an image encoding process; a flag encoding unit that encodes the transform skip flag generated by the flag generation unit and generates encoded data of the transform skip flag; a bit stream generating unit that generates a bit stream including encoded data of the transform skip flag generated by the flag encoding unit; An image processing device comprising:

2. a mode control unit that controls, based on the transform skip flag corresponding to the component identifier generated by the flag generation unit, whether a coding mode of coefficient data corresponding to the component identifier is set to a TS residual coding mode, which is a mode for skipping the transform process, or a non-TS residual coding mode, which is a mode for not skipping the transform process; a coefficient data encoding unit that encodes coefficient data corresponding to the component identifier in the encoding mode set by the mode control unit, and generates encoded data of the coefficient data; Further equipped with The bit stream generation unit generates a bit stream including the encoded data of the transform skip flag generated by the flag encoding unit and the encoded data of the coefficient data generated by the coefficient data encoding unit. The image processing device according to claim 1 .

3. When the conversion process is skipped, the coefficient data encoding unit shares a context variable between encoding a luminance component and encoding a chrominance component of the coefficient data. The image processing device according to claim 2 .

4. The coefficient data encoding unit applies a sign code encoding method according to the transform skip flag generated by the flag generation unit. The image processing device according to claim 2 .

5. the flag generation unit further generates a transform skip residual coding usage flag, which is flag information indicating whether to apply a TS residual coding mode, which is a mode for skipping the transform process, or a non-TS residual coding mode, which is a mode for not skipping the transform process; The flag encoding unit further encodes the transform skip residual encoding use flag generated by the flag generation unit to generate encoded data of the transform skip residual encoding use flag; The bitstream generation unit generates the bitstream further including encoded data of the transform skip residual encoding use flag generated by the flag encoding unit. The image processing device according to claim 1 .

6. the flag generation unit further generates a transform skip residual coding use specific mode flag, which is flag information indicating whether to apply a TS residual coding mode, which is a mode for skipping the transform process, or a non-TS residual coding mode, which is a mode for not skipping the transform process, in a specific mode; The flag encoding unit further encodes the transform skip residual encoding use specific mode flag generated by the flag generation unit to generate encoded data of the transform skip residual encoding use specific mode flag; The bit stream generating unit generates the bit stream further including encoded data of the transform skip residual encoding use specific mode flag generated by the flag encoding unit. The image processing device according to claim 1 .

7. generating a transform skip flag which is flag information indicating, for each component, whether or not a transform process for transforming a residual between the image and a predicted image of the image into coefficient data is to be skipped in encoding the image; encoding the generated transform skip flag to generate encoded data of the transform skip flag; A bitstream generating method for generating a bitstream including encoded data of the generated transform skip flag.

8. In encoding an image, when a transform skip flag corresponding to a component identifier indicates a transform skip for skipping a transform process for transforming a residual between the image and a predicted image of the image into a transform coefficient, a quantization parameter correction unit corrects a quantization parameter to be applied to a transform block to be processed corresponding to the component identifier; a quantization unit that quantizes a target transform block corresponding to the component identifier by using the quantization parameter corrected by the quantization parameter correction unit; An image processing device comprising:

9. The quantization parameter correction unit When the transform skip flag corresponding to the component identifier indicates the transform skip, a larger value of the minimum quantization parameter of the transform skip and the quantization parameter corresponding to the component identifier is set as a quantization parameter to be applied to the transform block to be processed corresponding to the component identifier; When the transform skip flag corresponding to the component identifier indicates a non-transform skip in which the transform skip is not performed, the quantization parameter corresponding to the component identifier is set as the quantization parameter to be applied to the target transform block corresponding to the component identifier. The image processing device according to claim 8.

10. In encoding an image, when a transform skip flag corresponding to a component identifier indicates a transform skip for skipping a transform process for converting a residual between the image and a predicted image of the image into a transform coefficient, a quantization parameter to be applied to a transform block to be processed corresponding to the component identifier is corrected; Using the corrected quantization parameter, the transform block to be processed corresponding to the component identifier is quantized to generate a quantized coefficient corresponding to the component identifier. Quantization coefficient generation method.

Images