Image processing device and image processing method
By adaptively transforming the color space only when the maximum block size is 32, the image processing device manages memory efficiently, addressing the memory increase issue caused by ACT processing in image encoding.
Patent Information
- Application Number
- JP2024152944
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2019-12-27
- Filing Date
- 2024-09-05
- Publication Date
- 2024-11-28
- Estimated Expiration
- 2040-12-18
AI Technical Summary
The application of Adaptive Color Transform (ACT) processing in image encoding leads to a temporary accumulation of YCgCo residual signals, necessitating an increase in memory size due to varying block sizes in orthogonal transform blocks.
An image processing device and method that adaptively transforms the color space only when the maximum block size is identified as 32, controlling the application of ACT and its inverse process to manage memory usage efficiently.
This approach effectively reduces memory requirements and implementation costs by limiting ACT processing to specific block sizes, thereby avoiding unnecessary memory allocation.
Smart Images

Figure 2024166234000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an image processing device and an image processing method, and more particularly to an image processing device and an image processing method that can prevent an increase in memory size. [Background technology]
[0002] Conventionally, image information has been handled digitally, and in order to transmit and store information efficiently, devices that compress and code images using a coding method that utilizes the redundancy inherent in image information and compresses the image using orthogonal transform such as discrete cosine transform and motion compensation are becoming widespread.
[0003] Examples of such encoding methods include MPEG (Moving Picture Experts Group), H.264 and MPEG-4 Part 10 (Advanced Video Coding, hereinafter referred to as H.264 / AVC), and H.265 and MPEG-H Part 2 (High Efficiency Video Coding, hereinafter referred to as H.265 / HEVC).
[0004] In addition, in order to further improve the coding efficiency of AVC (Advanced Video Coding) and HEVC (High Efficiency Video Coding), standardization of a coding method called VVC (Versatile Video Coding) is underway (see support in the embodiments described later).
[0005] As disclosed in Non-Patent Document 1, in VVC, a technique related to ACT (Adaptive Color Transform) that adaptively transforms the color space of an image is disclosed. [Prior art documents] [Non-patent literature]
[0006] [Non-Patent Document 1] Xiaoyu Xiu, Yi-Wen Chen, Tsung-Chuan Ma, Hong-Jheng Jhu, Xianglin Wang, Support of adaptive color transform for 444 video coding in VVC, JVET-P0517_r1 (version 3 - date 2019-10-11) Summary of the Invention [Problem to be solved by the invention]
[0007] By the way, when applying ACT processing to convert, for example, an RGB color space to a YCgCo color space, it is necessary to temporarily store the YCgCo residual signal output as a result of the processing. Therefore, depending on the block size of the orthogonal transform block in the orthogonal transform processing performed after the ACT processing, it may be necessary to increase the memory size for storing the YCgCo residual signal.
[0008] The present disclosure has been made in consideration of such circumstances, and makes it possible to avoid an increase in memory size. [Means for solving the problem]
[0009] An image processing device according to a first aspect of the present disclosure includes a parsing unit that parses size identification data from a bitstream that identifies whether a maximum block size of an orthogonal transform block is 64 or 32, and an inverse adaptive color transform unit that applies the inverse adaptive color transform processing to a residual signal generated by applying an inverse orthogonal transform processing to transform coefficients obtained by decoding the bitstream for each orthogonal transform block, which is a processing unit, in accordance with identification data that identifies whether an adaptive color transform processing that adaptively transforms a color space of an image or an inverse adaptive color transform processing that adaptively inverse transforms the color space of an image is enabled only when the size identification data parsed by the parsing unit indicates that 32 is to be applied as the maximum block size of the orthogonal transform block.
[0010] An image processing method according to a first aspect of the present disclosure includes parsing size identification data from a bitstream that identifies whether a maximum block size of an orthogonal transform block is 64 or 32, and only if the size identification data indicates that 32 is to be applied as the maximum block size of the orthogonal transform block, applying the inverse adaptive color transform processing to a residual signal generated by applying an inverse orthogonal transform processing to transform coefficients obtained by decoding the bitstream for each orthogonal transform block, which serves as a processing unit, in accordance with identification data that identifies whether an adaptive color transform processing that adaptively transforms a color space of an image or an inverse adaptive color transform processing that adaptively inverse transforms the color space of an image is enabled.
[0011] In a first aspect of the present disclosure, size identification data identifying whether the maximum block size of an orthogonal transform block is 64 or 32 is parsed from the bitstream, and only if the size identification data indicates that 32 is to be applied as the maximum block size of the orthogonal transform block, an inverse adaptive color transform process is applied to a residual signal generated by applying an inverse orthogonal transform process to the transform coefficients obtained by decoding the bitstream for each orthogonal transform block, which serves as a processing unit, in accordance with identification data identifying whether an adaptive color transform process that adaptively transforms the color space of an image or an inverse adaptive color transform process that adaptively inverse transforms the color space of an image is enabled.
[0012] An image processing device according to a second aspect of the present disclosure includes a setting unit that sets identification data identifying whether an adaptive color conversion process that adaptively converts the color space of an image or an inverse adaptive color conversion process that adaptively inversely converts the color space of an image is enabled only when size identification data that identifies whether the maximum block size of an orthogonal transform block is 64 or 32 is set to be 32 as the maximum block size of the orthogonal transform block, and an encoding unit that encodes the image to generate a bit stream including the identification data set by the setting unit.
[0013] An image processing method of a second aspect of the present disclosure includes setting identification data that identifies whether an adaptive color conversion process that adaptively converts a color space of an image or an inverse adaptive color conversion process that adaptively inversely converts the color space of an image is enabled only when size identification data that identifies whether the maximum block size of an orthogonal transform block is 64 or 32 is set to be 32 as the maximum block size of the orthogonal transform block, and encoding the image to generate a bitstream including the identification data.
[0014] In a second aspect of the present disclosure, only when size identification data identifying whether the maximum block size of the orthogonal transform block is 64 or 32 is set to be 32 as the maximum block size of the orthogonal transform block, identification data identifying whether an adaptive color transform process that adaptively transforms the color space of an image or an inverse adaptive color transform process that adaptively inverse transforms the color space of an image is enabled is set, and the image is encoded to generate a bitstream including the identification data. [Brief description of the drawings]
[0015] [Figure 1] 1 is a block diagram showing an example of the configuration of an embodiment of an image processing system to which the present technology is applied. [Diagram 2] FIG. 1 illustrates an example of the configuration of an image encoding device. [Diagram 3] FIG. 1 is a diagram illustrating an example of the configuration of an image decoding device. [Figure 4] FIG. 13 is a diagram showing an example of a parameter set of high-level syntax. [Diagram 5] A diagram showing an example of a coding unit in high level syntax. [Figure 6] FIG. 13 is a diagram showing an example of a parameter set of high-level syntax. [Figure 7] 1 is a block diagram showing an example of the configuration of an embodiment of a computer-based system to which the present technology is applied; [Figure 8] 1 is a block diagram showing an example of the configuration of an embodiment of an image encoding device; [Figure 9] 11 is a flowchart illustrating an encoding process. [Figure 10] FIG. 1 is a block diagram showing an example of the configuration of an embodiment of an image decoding device. [Figure 11] 13 is a flowchart illustrating a decoding process. [Figure 12] 1 is a block diagram showing an example of the configuration of an embodiment of a computer to which the present technology is applied. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0016] <References supporting technical content and technical terms> The scope of the disclosure in this specification is not limited to the contents of the Examples, and the contents of the following reference documents REF1 to REF5, which were publicly known at the time of filing, are also incorporated by reference into this specification. In other words, the contents described in the reference documents REF1 to REF5 also serve as the basis for determining the support requirements. Furthermore, the documents referenced in the reference documents REF1 to REF5 also serve as the basis for determining the support requirements.
[0017] For example, even if Quad-Tre Block Structure, QTBT (Quad Tree Plus Binary Tree), Block Structure, MTT (Multi-type Tree) Block Structure, etc. are not directly defined in the detailed description of the invention, they are within the scope of this disclosure and meet the support requirements of the claims. Similarly, even if technical terms such as parsing, syntax, and semantics are not directly defined in the detailed description of the invention, they are within the scope of this disclosure and meet the support requirements of the claims. Similarly, even if technical applications such as ACT (Adaptive Color Transform) are not directly defined in the detailed description of the invention, they are within the scope of this disclosure and meet the support requirements of the claims.
[0018] REF1:Recommendation ITU-T H.264 (04 / 2017) “Advanced video coding for generic audiovisual services”, April 2017 REF2:Recommendation ITU-T H.265 (02 / 2018) “High efficiency video coding”, February 2018 REF3:Benjamin Bross, Jianle Chen, Shan Liu, Versatile Video Coding (Draft 7), JVET-P2001-v14 (version 14 - date 2019-11-14) REF4: Jianle Chen, Yan Ye, Seung Hwan Kim, Algorithm description for Versatile Video Coding and Test Model 7 (VTM 7), JVET-P2002-v1 (version 1 - date 2019-11-10) REF5: Xiaoyu Xiu, Yi-Wen Chen, Tsung-Chuan Ma, Hong-Jheng Jhu, Xianglin Wang, Support of adaptive color transform for 444 video coding in VVC, JVET-P0517_r1 (version 3 - date 2019-10-11)
[0019] <Terminology> As used herein, the following terms are defined as follows.
[0020] <Block> A "block" (not a block indicating a processing unit) used in the description as a partial region or processing unit of an image (picture) indicates any partial region in a picture, and its size, shape, characteristics, etc. are not limited, unless otherwise specified. For example, a "block" includes any partial region (processing unit) such as a TB (Transform Block), TU (Transform Unit), PB (Prediction Block), PU (Prediction Unit), SCU (Smallest Coding Unit), CU (Coding Unit), LCU (Largest Coding Unit), CTB (Coding Tree Block), CTU (Coding Tree Unit), transform block, sub-block, macroblock, tile, or slice.
[0021] <Block size specification> In addition, when specifying the size of such a block, the block size may be specified indirectly in addition to directly. For example, the block size may be specified using identification information for identifying the size. For example, the block size may be specified by a ratio or difference with respect to the size of a reference block (for example, LCU or SCU). For example, when transmitting information specifying a block size as a syntax element or the like, the information may be used to indirectly specify the size as described above. In this way, the amount of information can be reduced, and coding efficiency may be improved. The specification of the block size also includes specification of a range of block sizes (for example, specification of a range of allowable block sizes, etc.).
[0022] <Unit of information processing> The data units in which various information is set and the data units targeted by various processes are arbitrary and are not limited to the above-mentioned examples. For example, these information and processes may be set for each TU (Transform Unit), TB (Transform Block), PU (Prediction Unit), PB (Prediction Block), CU (Coding Unit), LCU (Largest Coding Unit), sub-block, block, tile, slice, picture, sequence, or component, or may target data of these data units. Of course, this data unit may be set for each information or process, and the data units of all information and processes do not need to be unified. The storage location of these pieces of information is arbitrary, and may be stored in the header or parameter set of the above-mentioned data unit. Also, they may be stored in multiple locations.
[0023] <Control information> Control information related to the present technology may be transmitted from the encoding side to the decoding side. For example, control information (e.g., enabled_flag) that controls whether or not to permit (or prohibit) application of the present technology described above may be transmitted. In addition, for example, control information indicating a target to which the present technology described above is applied (or a target to which it is not applied) may be transmitted. For example, control information that specifies a block size (upper or lower limit, or both), frame, component, or layer to which the present technology is applied (or permitted or prohibited to be applied) may be transmitted.
[0024] <flag> In this specification, a "flag" refers to information for identifying multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Thus, the value that this "flag" can take may be, for example, a binary value of 1 / 0, or may be three or more values. In other words, the number of bits constituting this "flag" is arbitrary, and may be one bit or multiple bits. Furthermore, it is assumed that identification information (including flags) is not only included in the bit stream, but also includes difference information of the identification information with respect to certain reference information in the bit stream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also difference information with respect to the reference information.
[0025] <Associate metadata> Also, various information (metadata, etc.) related to the coded data (bit stream) may be transmitted or recorded in any form as long as it is associated with the coded data. Here, the term "associate" means, for example, that one piece of data can be used (linked) when processing the other piece of data. That is, data associated with each other may be combined into one piece of data, or each piece of data may be made into separate data. For example, information associated with coded data (image) may be transmitted on a transmission path different from that of the coded data (image). Also, for example, information associated with coded data (image) may be recorded on a recording medium different from that of the coded data (image) (or on a different recording area of the same recording medium). Note that this "association" may be a part of the data, not the entire data. For example, an image and information corresponding to the image may be associated with each other in any unit, such as multiple frames, one frame, or a part of a frame.
[0026] In this specification, the terms "composite", "multiplex", "add", "integrate", "include", "store", "put in", "insert", "insert" and the like mean combining multiple things into one, for example combining encoded data and metadata into one data, and mean one method of "associating" as described above. In this specification, encoding includes not only the whole process of converting an image into a bitstream, but also a part of the process. For example, it includes not only a process that includes prediction processing, orthogonal transformation, quantization, arithmetic coding, etc., but also a process that collectively refers to quantization and arithmetic coding, a process that includes prediction processing, quantization, and arithmetic coding, and the like. Similarly, decoding includes not only the whole process of converting a bitstream into an image, but also a part of the process. For example, it includes not only a process that includes inverse arithmetic decoding, inverse quantization, inverse orthogonal transformation, prediction processing, etc., but also a process that includes inverse arithmetic decoding and inverse quantization, a process that includes inverse arithmetic decoding, inverse quantization, and prediction processing, and the like.
[0027] The prediction block means a block that is a processing unit when performing inter prediction, and includes sub-blocks within the prediction block. In addition, when the processing unit is the orthogonal transform block that is the processing unit when performing orthogonal transform and the coding block that is the processing unit when performing coding processing, the prediction block means the same block as the orthogonal transform block and the coding block.
[0028] Inter prediction is a general term for processing involving prediction between frames (prediction blocks), such as derivation of a motion vector by motion detection (Motion Prediction / Motion Estimation) and motion compensation using a motion vector (Motion Compensation), and includes some processing (for example, only motion compensation processing) or all processing (for example, motion detection processing + motion compensation processing) used when generating a predicted image. Inter prediction mode refers to variables (parameters) referenced when deriving an inter prediction mode, such as a mode number when performing inter prediction, an index of the mode number, a block size of a prediction block, and a size of a sub-block that is a processing unit within a prediction block.
[0029] In the present disclosure, identification data for identifying multiple patterns can also be set as a syntax of the bit stream. In this case, the decoder can parse and refer to the identification data to perform processing more efficiently. Methods (data) for identifying block sizes include not only converting the block size itself into a numerical value (bit conversion), but also methods (data) for identifying a difference value from a reference block size (maximum block size, minimum block size, etc.).
[0030] Hereinafter, specific embodiments to which the present technology is applied will be described in detail with reference to the drawings.
[0031] <Image processing system configuration example> FIG. 1 is a block diagram showing an example of the configuration of an embodiment of an image processing system to which the present technology is applied.
[0032] 1, an image processing system 11 includes an image encoding device 12 and an image decoding device 13. For example, in the image processing system 11, an image input to the image encoding device 12 is encoded, a bit stream obtained by the encoding is transmitted to the image decoding device 13, and a decoded image decoded from the bit stream in the image decoding device 13 is output.
[0033] As shown in FIG. 1, the image encoding device 12 has a prediction unit 21, an encoding unit 22, a memory unit 23, and a control unit 24, and the image decoding device 13 has a prediction unit 31, a decoding unit 32, a memory unit 33, and a control unit 34.
[0034] The prediction unit 21 generates a predicted image by performing inter prediction or intra prediction. For example, when performing inter prediction, the prediction unit 21 generates a predicted image with a prediction block (Prediction Unit) of a predetermined block size as a processing unit.
[0035] The encoding unit 22 encodes the image input to the image encoding device 12 according to a predetermined encoding method, using a coding block (coding unit) of a predetermined block size as a processing unit, and transmits a bit stream of the encoded data to the image decoding device 13. In addition, this bit stream includes parameters related to the blocks, which will be described later with reference to Figs. 4 to 6.
[0036] The storage unit 23 stores various data that needs to be stored when encoding an image in the image encoding device 12. For example, as described later with reference to Fig. 2, the storage unit 23 temporarily stores a YCgCo residual signal 1 output by ACT processing and a YCgCo residual signal 2 that is the target of IACT processing.
[0037] The control unit 24 performs control related to application of the ACT process and the IACT process, which will be described later with reference to FIG.
[0038] The prediction unit 31 generates a predicted image by performing inter prediction or intra prediction. For example, when performing inter prediction, the prediction unit 21 generates a predicted image by using a prediction block of a predetermined block size as a processing unit.
[0039] The decoding unit 32 decodes the bit stream transmitted from the image encoding device 12 in accordance with the encoding method used by the encoding unit 22, and outputs the decoded image.
[0040] The storage unit 33 stores various data that needs to be stored when decoding an image in the image decoding device 13. For example, the storage unit 33 temporarily accumulates a YCgCo residual signal 2 that is to be subjected to IACT processing, as will be described later with reference to FIG.
[0041] The control unit 34 performs control related to the application of the IACT process, which will be described later with reference to FIG.
[0042] In the image processing system 11 configured as above, by appropriately controlling the ACT processing and the IACT processing, it is possible to prevent the memory sizes of the storage units 23 and 33 from increasing.
[0043] The configuration of the image encoding device 12 will be further described with reference to the block diagram shown in FIG.
[0044] As shown in FIG. 2, the image encoding device 12 is configured to include a calculation unit 41, an adaptive color transformation unit 42, an orthogonal transformation unit 43, a quantization unit 44, an inverse quantization unit 45, an inverse orthogonal transformation unit 46, an inverse adaptive color transformation unit 47, a calculation unit 48, a prediction unit 49, and an encoding unit 50.
[0045] The calculation unit 41 performs a calculation to subtract the predicted image supplied from the prediction unit 49 from the image input to the image encoding device 12, and supplies the RGB residual signal 1, which is the difference information obtained by the calculation, to the adaptive color conversion unit 42.
[0046] The adaptive color conversion unit 42 performs ACT processing for adaptively converting the color space of the image to be encoded, on the RGB residual signal 1 supplied from the calculation unit 41. For example, the adaptive color conversion unit 42 obtains a YCgCo residual signal 1 from the RGB residual signal 1 by performing ACT processing for converting the RGB color space into a YCgCo color space, and supplies the YCgCo residual signal 1 to the orthogonal transformation unit 43.
[0047] The orthogonal transform unit 43 obtains transform coefficients by performing orthogonal transform processing on the YCgCo residual signal 1 supplied from the adaptive color transform unit 42, for each orthogonal transform block that serves as a processing unit, and supplies the transform coefficients to the quantization unit 44. Furthermore, when the adaptive color transform unit 42 is controlled so that ACT processing is not performed, the orthogonal transform unit 43 can perform orthogonal transform processing on the RGB residual signal 1 supplied from the calculation unit 41.
[0048] The quantization unit 44 quantizes the transform coefficients supplied from the orthogonal transformation unit 43, and supplies the result to the inverse quantization unit 45 and the encoding unit 50. The inverse quantization unit 45 inverse quantizes the transform coefficients quantized by the quantization unit 44, and supplies the result to the inverse orthogonal transformation unit 46.
[0049] The inverse orthogonal transform unit 46 performs inverse orthogonal transform on the transform coefficients supplied from the inverse quantization unit 45 for each orthogonal transform block, which is a processing unit, to obtain an YCgCo residual signal 2, which is supplied to the inverse adaptive color transform unit 47. Furthermore, when the inverse adaptive color transform unit 47 is controlled so that IACT processing is not performed, the inverse orthogonal transform unit 46 can obtain an RGB residual signal 2 by inverse orthogonal transform processing and supply it to the calculation unit 48.
[0050] The inverse adaptive color transform unit 47 performs IACT processing, which adaptively inversely transforms the color space of the image, on the YCgCo residual signal 2 supplied from the inverse orthogonal transform unit 46. For example, the inverse adaptive color transform unit 47 obtains an RGB residual signal 2 from the YCgCo residual signal 2 by performing IACT processing, which inversely transforms the YCgCo color space into an RGB color space, and supplies the RGB residual signal 2 to the calculation unit 48.
[0051] The calculation unit 48 locally reconstructs (decodes) an image by performing a calculation to add the RGB residual signal 2 supplied from the inverse adaptive color transform unit 47 to a predicted image supplied from the prediction unit 49, and outputs a reconstruction signal representing the reconstructed image. Furthermore, when the inverse adaptive color transform unit 47 is controlled so as not to perform IACT processing, the calculation unit 48 can reconstruct (decode) an image from the RGB residual signal 2 supplied from the inverse orthogonal transform unit 46.
[0052] The prediction unit 49 corresponds to the prediction unit 21 in FIG. 1, and generates a predicted image predicted from the image reconstructed in the calculation unit 48 , and supplies the predicted image to the calculation units 41 and 48 .
[0053] The encoding unit 50 corresponds to the encoding unit 22 in Fig. 1 and performs encoding processing on the transform coefficients quantized by the quantization unit 44 using, for example, CABAC (Context-based Adaptive Binary Arithmetic Coding), which is an encoding method with high encoding efficiency for a series of identical values. As a result, the encoding unit 50 obtains a bit stream of encoded data and transmits it to the image decoding device 13.
[0054] In the image encoding device 12 configured in this manner, the adaptive color conversion unit 42 performs ACT processing to convert the RGB residual signal 1 into a YCgCo residual signal 1, thereby improving the energy concentration of the signal. By improving the energy concentration of the signal in this manner, the image encoding device 12 can express the image signal with a smaller amount of code, and improvement in encoding efficiency is expected.
[0055] The configuration of the image decoding device 13 will be further described with reference to the block diagram shown in FIG.
[0056] As shown in FIG. 3, the image decoding device 13 includes a decoding unit 61, an inverse quantization unit 62, an inverse orthogonal transformation unit 63, an inverse adaptive color transformation unit 64, a calculation unit 65, and a prediction unit 66.
[0057] The decoding unit 61 corresponds to the decoding unit 32 in Fig. 1, and performs a decoding process on the bit stream of coded data transmitted from the image coding device 12 using a coding method (e.g., CABAC) corresponding to the coding by the coding unit 50 in Fig. 2. As a result, the decoding unit 61 obtains quantized transform coefficients from the bit stream of coded data and supplies them to the inverse quantization unit 62. At this time, the decoding unit 61 also obtains parameters related to blocks, which are included in the bit stream of coded data and will be described later with reference to Figs. 4 to 6.
[0058] The inverse quantization unit 62 inverse quantizes the quantized transformation coefficients supplied from the decoding unit 61 and supplies the result to an inverse orthogonal transformation unit 63 .
[0059] The inverse orthogonal transform unit 63 performs inverse orthogonal transform processing on the transform coefficients supplied from the inverse quantization unit 62, performing inverse orthogonal transform on each orthogonal transform block, which is a processing unit, to obtain an YCgCo residual signal 2, which is supplied to the inverse adaptive color transform unit 64. Furthermore, when the inverse adaptive color transform unit 64 is controlled so that IACT processing is not performed, the inverse orthogonal transform unit 63 can supply the RGB residual signal 2 obtained by the inverse orthogonal transform processing to the calculation unit 65.
[0060] 2, the inverse adaptive color transform unit 64 performs IACT processing, which adaptively inversely transforms the color space of the image, on the YCgCo residual signal 2 supplied from the inverse orthogonal transform unit 63. For example, the inverse adaptive color transform unit 64 obtains an RGB residual signal 2 from the YCgCo residual signal 2 by performing IACT processing, which inversely transforms the YCgCo color space into an RGB color space, and supplies the RGB residual signal 2 to the calculation unit 65.
[0061] The calculation unit 65 locally reconstructs (decodes) an image by performing a calculation to add the RGB residual signal 2 supplied from the inverse adaptive color transform unit 64 to a predicted image supplied from the prediction unit 66, and outputs a reconstruction signal representing the reconstructed image. Furthermore, when the inverse adaptive color transform unit 64 is controlled so as not to perform IACT processing, the calculation unit 65 can reconstruct (decode) an image from the RGB residual signal 2 supplied from the inverse orthogonal transform unit 63.
[0062] The prediction unit 66 corresponds to the prediction unit 31 in FIG. 1, and similarly to the prediction unit 49 in FIG. 2, generates a predicted image predicted from the image reconstructed in the calculation unit 65, and supplies the predicted image to the calculation unit 65.
[0063] Like the image encoding device 12, the image decoding device 13 configured in this manner can contribute to improving encoding efficiency.
[0064] The image processing system 11 is configured as described above, with the adaptive color converter 42 performing ACT processing to convert the RGB residual signal 1 into a YCgCo residual signal 1, and the inverse adaptive color converters 47 and 64 performing IACT processing to convert the YCgCo residual signal 2 into an RGB residual signal 2.
[0065] At this time, in the image encoding device 12, the adaptive color transform unit 42 performs ACT processing in units of three components, so that the YCgCo residual signals 1 for three components are temporarily stored in the storage unit 23 of Fig. 1. Therefore, the storage unit 23 stores the YCgCo residual signals 1 for three components corresponding to the block size (e.g., 32 x 32) of the orthogonal transform block in the orthogonal transform unit 43. The storage unit 23 also stores the YCgCo residual signals 2 for three components corresponding to the block size (e.g., 32 x 32) of the orthogonal transform block in the inverse orthogonal transform unit 46.
[0066] Similarly, in the image decoding device 13, the YCgCo residual signal 2 for three components corresponding to the block size of the orthogonal transformation block (for example, 32×32) in the inverse orthogonal transformation unit 63 is stored in the storage unit 33 in FIG. 1.
[0067] Thus, in the image processing system 11, when applying the ACT process and the IACT process, it is necessary to increase the memory size of the storage unit 23 so that the YCgCo residual signal 1 and the YCgCo residual signal 2 corresponding to the block size of the orthogonal transformation block can be stored. Similarly, when applying the ACT process and the IACT process, it is necessary to increase the memory size of the storage unit 33 so that the YCgCo residual signal 2 corresponding to the block size of the orthogonal transformation block can be stored. Therefore, as the memory size increases, the implementation cost of the image processing system 11 increases.
[0068] Therefore, in the image processing system 11, by appropriately controlling, by the control unit 24, the application of the ACT process by the adaptive color conversion unit 42 and the IACT process by the inverse adaptive color conversion unit 47, it is possible to avoid an increase in the memory size of the storage unit 23. Similarly, in the image processing system 11, by appropriately controlling, by the control unit 34, the application of the IACT process by the inverse adaptive color conversion unit 64, it is possible to avoid an increase in the memory size of the storage unit 33. Therefore, as the image processing system 11 avoids an increase in the memory size, it can suppress an increase in the implementation cost.
[0069] <First concept regarding the application of the ACT process and the IACT process>
[0070] For example, in the image processing system 11, when a predetermined restriction (for example, restrictions on size, area, shape, etc.) is provided for the encoding block when encoding an image, control is performed so that the ACT process and the IACT process are applied.
[0071] For example, parameters of the coding block for such restriction include size, long side size, short side size, area, and shape. Size includes 16x16, 16x8, 8x16, etc. Long side size includes 16 for 16x8 blocks. Short side size includes 8 for 16x8 blocks. Area includes 16x16, 16x8, etc. Shape includes square, rectangle, etc.
[0072] Conventionally, the block size of the orthogonal transform block used in orthogonal transform processing and inverse orthogonal transform processing was 32 x 32, which required a memory size capable of storing three components of YCgCo residual signals with a block size of 32 x 32.
[0073] In contrast, in the image processing system 11, the block size of the coding block when the ACT processing and the IACT processing are applied is limited to a predetermined size (for example, 16×16) or less. With such a limit, the memory unit 23 only needs to have a memory size sufficient to store the YCgCo residual signal 1 for three components with a block size of 16×16. Similarly, for the YCgCo residual signal 2, the memory units 23 and 33 only need to have a memory size sufficient to store the YCgCo residual signal 2 for three components with a block size of 16×16.
[0074] For example, with regard to the bitstream syntax, the conditions for applying ACT processing and IACT processing take into consideration the parameters of the coding block (size, area, shape, etc.). That is, when transmitting a flag indicating that ACT processing is applied, the image coding device 12 checks that the block size of the coding block is equal to or smaller than a predetermined size (e.g., 16×16). Then, the syntax is determined so that only when the block size of the coding block is equal to or smaller than the predetermined size (e.g., 16×16), a flag indicating that ACT processing is applied is transmitted, and when the block size is larger than the predetermined size (e.g., 16×16), a flag indicating that ACT processing is applied is not transmitted.
[0075] As a result, in the image processing system 11, when the block size of the coding block is larger than a predetermined size, it becomes unnecessary to transmit a flag indicating that ACT processing is to be applied, and the flag can be removed from the bit stream, so that improvement in coding efficiency can be expected. Also, by not transmitting such a flag that does not need to be transmitted, it is possible to remove signals that are ambiguous in the bit stream syntax.
[0076] In addition, a case where VVC (Versatile Video Coding) is used as the coding method will be described. When VVC is used as the coding method, 64×64 can be used as the block size of the orthogonal transform block in the orthogonal transform process and the inverse orthogonal transform process. Therefore, when VVC is used as the coding method, a memory size capable of storing three components of YCgCo residual signals with a block size of 64×64 is required.
[0077] Therefore, in the image processing system 11, when VVC is used as the encoding method, for example, the block size of the encoding block when ACT processing and IACT processing are applied is limited to 32×32 or less. With such a restriction, the memory unit 23 only needs to have a memory size of 32×32 block size to store three components of YCgCo residual signal 1. Similarly, for the YCgCo residual signal 2, the memory units 23 and 33 only need a memory size of 32×32 block size to store three components of YCgCo residual signal 2.
[0078] In addition, even when HEVC (High Efficiency Video Coding) is used as the encoding method, a mechanism for performing ACT processing and IACT processing is provided, and in the HEVC standard, the maximum block size of the orthogonal transform block in the orthogonal transform processing and the inverse orthogonal transform processing is 32 x 32. In contrast, VVC can support a maximum block size of 64 x 64 for the orthogonal transform block, so that the increase in memory size is greater than that of HEVC. Therefore, in the image processing system 11, when VVC is used as the encoding method, when ACT processing and IACT processing are applied, the orthogonal transform block in the orthogonal transform processing and the inverse orthogonal transform processing is limited, thereby making it possible to perform processing with the same memory size as HEVC.
[0079] FIG. 4 shows an example of a parameter set in high-level syntax.
[0080] In the parameter set shown in Figure 4, when sps_act_enabled_flag is 1, it specifies that ACT processing and IACT processing can be applied, and cu_act_enabled_flag may be present in the coding unit syntax. On the other hand, when sps_act_enabled_flag is 0, it specifies that ACT processing and IACT processing are not applied, and cu_act_enabled_flag is not present in the coding unit syntax. Note that when sps_act_enabled_flag is not present in the parameter set, sps_act_enabled_flag is inferred to be 0.
[0081] 4, sps_log2_act_max_size_minus2 specifies the maximum block size used in the ACT process and the IACT process in the range of 0 to 7. If sps_log2_act_max_size_minus2 does not exist in the parameter set, sps_log2_act_max_size_minus2 is inferred to be 0.
[0082] Also, when the MaxActSize variable is set to 1, it becomes sufficiently smaller than sps_log2_act_max_size_minus2 + 2 (The variable MaxActSize is set equal to 1 << (sps_log2_act_max_size_minus2 + 2)). For example, if sps_log2_act_max_size_minus2 is set to 2, the MaxActSize variable becomes 16 (=1<<4). This makes it possible to prohibit ACT processing and IACT processing from being performed with a size larger than MaxActSize.
[0083] FIG. 5 shows an example of a coding unit in high level syntax.
[0084] In the coding unit shown in Figure 5, cu_act_enabled_flag specifies that the residual of the current coding unit is coded in the YCgCo color space when it is 1. On the other hand, cu_act_enabled_flag specifies that the residual of the current coding unit is coded in the original color space when it is 0. Note that if cu_act_enabled_flag is not present, cu_act_enabled_flag is inferred to be 0.
[0085] Here, by adding the conditions that the width of the coding block is less than or equal to the maximum block size in ACT processing and IACT processing, and the height of the coding block is less than or equal to the maximum block size in ACT processing and IACT processing (&& cbWidth <= MaxActSize && cbHeight <= MaxActSize) to the conditions for sending cu_act_enabled_flag, the memory size used in ACT processing and IACT processing can be limited.
[0086] In addition, by adding the condition that the width of the encoded block or the height of the encoded block is less than or equal to the maximum size of the block size in the ACT process and the IACT process (&& (cbWidth * cbHeight) <= (MaxActSize * MaxActSize)) to the condition for transmitting the cu_act_enabled_flag, the memory size can also be restricted. In this case, even if one side exceeds MaxActSize, if the other side is small and satisfies the above condition, the ACT process and the IACT process can be applied.
[0087] <Second Concept Regarding Application of ACT Process and IACT Process> In the second concept, even when the block size of the encoded block or the predicted block used in the image processing system 11 is large, control is performed so that orthogonal transformation processing and inverse orthogonal transformation processing are performed using orthogonal transformation blocks with a small size obtained by dividing the block size of the encoded block or the predicted block, and control is performed so that the ACT process and the IACT process are applied. That is, even when the block size of the encoded block or the predicted block used in the image processing system 11 is large, when the ACT process and the IACT process are applied, orthogonal transformation processing and inverse orthogonal transformation processing are performed using orthogonal transformation blocks smaller than the block size of the encoded block or the predicted block.
[0088] For example, when the block size of the encoded block is 64×64 and the block size of the prediction block for inter prediction is 64×64, usually, a block size of 64×64 is also used in the orthogonal transformation block. In contrast, in the image processing system 11, even in the same case, when the ACT process and the IACT process are applied, orthogonal transformation blocks with a block size smaller than 64×64 (for example, four blocks with a block size of 32×32) are automatically used.
[0089] For example, by referring to a control signal indicating that the ACT process is applied, when the ACT process is applied, control is performed such that orthogonal transformation processing and inverse orthogonal transformation processing are performed using an orthogonal transformation block obtained by dividing the block size of the encoding block or prediction block at that time into a small block size.
[0090] Therefore, in the image processing system 11, by performing control according to such a second concept, the block size of the orthogonal transformation block in the orthogonal transformation processing and the inverse orthogonal transformation processing is reduced, and an increase in the memory size required when applying the ACT process and the IACT process can be avoided. For example, the storage unit 23 that stores the YCbCo residual signal 1 can have a memory size for three components with a block size of 32×32 instead of a memory size for three components with a block size of 64×64. As a result, in the image processing system 11, an increase in the implementation cost can be suppressed.
[0091] <Third Concept Regarding Application of ACT Process and IACT Process> In the third concept, when a small size is used as the maximum block size of the orthogonal transformation block in the orthogonal transformation processing and the inverse orthogonal transformation processing, control is performed so that the ACT process and the IACT process are applied.
[0092] For example, in the image processing system 11, 32 and 64 are defined as the maximum block sizes of the orthogonal transformation block. Then, the control unit 24 restricts the case where 32 is used as the maximum block size of the orthogonal transformation block, and causes the adaptive color conversion unit 42 to perform the ACT process and the inverse adaptive color conversion unit 47 to perform the IACT process. Similarly, the control unit 34 restricts the case where 32 is used as the maximum block size of the orthogonal transformation block, and causes the inverse adaptive color conversion unit 64 to perform the IACT process.
[0093] Control according to this third concept can be achieved by using the sps_max_luma_transform_size_64_flag included in the parameter set of the high-level syntax.
[0094] FIG. 6 shows an example of a parameter set in a high-level syntax used in the image processing system 11.
[0095] For example, when sps_max_luma_transform_size_64_flag is 0, the maximum block size of the orthogonal transform block is set to 32. On the other hand, when sps_max_luma_transform_size_64_flag is 1, the maximum block size of the orthogonal transform block is set to 64.
[0096] Therefore, in the image processing system 11, control is performed so that the ACT processing and the IACT processing are applied only when sps_max_luma_transform_size_64_flag is 0. That is, in the image processing system 11, when sps_max_luma_transform_size_64_flag is 1, the ACT processing and the IACT processing are not applied.
[0097] Then, when sps_max_luma_transform_size_64_flag is 0, the control unit 24 sets sps_act_enabled_flag, which indicates whether or not adaptive color conversion processing is applied, to 1, which indicates that adaptive color conversion processing is applied, and transmits it to the image decoding device 13. In this case, cu_act_enabled_flag may also be included in the coding unit syntax. Note that when sps_act_enabled_flag is 0, it indicates that adaptive color conversion processing is not applied, and in this case, cu_act_enabled_flag is not included in the coding unit syntax. Here, when sps_act_enabled_flag is not included in the parameter set, sps_act_enabled_flag is inferred to be 0.
[0098] In this way, when sps_max_luma_transform_size_64_flag is 0, the maximum block size of the orthogonal transform block is limited to 32, so that the image processing system 11 can avoid an increase in memory size required when applying ACT processing and IACT processing.
[0099] For example, when such control is not performed, that is, when the maximum block size of the orthogonal transform block can be 64, the storage units 23 and 33 require a memory size capable of storing three components of YCgCo residual signals with a block size of 64 × 64. In contrast, by performing control to apply ACT processing and IACT processing only when the maximum block size of the orthogonal transform block is limited to 32, the storage units 23 and 33 only need a memory size sufficient to store three components of YCgCo residual signals with a block size of 32 × 32.
[0100] Therefore, in the image processing system 11, by performing control according to the third concept, it is possible to avoid an increase in memory size, and as a result, it is possible to suppress an increase in implementation costs.
[0101] <Example of computer-based system configuration> FIG. 7 is a block diagram showing an example of the configuration of an embodiment of a computer-based system to which the present technology is applied.
[0102] Fig. 7 is a block diagram showing a configuration example of a network system in which one or more computers, servers, etc. are connected via a network. Note that the hardware and software environment shown in the embodiment of Fig. 7 is shown as an example that can provide a platform for implementing the software and / or method according to the present disclosure.
[0103] 7, a network system 101 includes a computer 102, a network 103, a remote computer 104, a web server 105, a cloud storage server 106, and a computer server 107. In this embodiment, multiple instances are executed by one or more of the functional blocks shown in FIG.
[0104] 7 illustrates a detailed configuration of the computer 102. Note that the functional blocks illustrated in the computer 102 are illustrated to establish exemplary functions, and are not limited to such a configuration. Also, detailed configurations of the remote computer 104, the web server 105, the cloud storage server 106, and the computer server 107 are not illustrated, but these include configurations similar to the functional blocks illustrated in the computer 102.
[0105] The computer 102 may be a personal computer, a desktop computer, a laptop computer, a tablet computer, a netbook computer, a personal digital assistant, a smart phone, or any other programmable electronic device capable of communicating with other devices on a network.
[0106] Computer 102, in turn, is configured with bus 111, processor 112, memory 113, non-volatile storage 114, network interface 115, peripherals interface 116, and display interface 117. Each of these functions may, in some embodiments, be implemented in individual electronic subsystems (integrated circuit chips or combinations of chips and related devices), or in other embodiments, some of the functions may be combined and implemented on a single chip (System on Chip or SoC).
[0107] Bus 111 may employ any of a variety of proprietary or industry standard high speed parallel or serial peripheral interconnect buses.
[0108] Processor 112 may be designed and / or manufactured as one or more single or multi-chip microprocessors.
[0109] The memory 113 and the non-volatile storage 114 are storage media that can be read by the computer 102. For example, the memory 113 can employ any suitable volatile storage device such as a dynamic random access memory (DRAM) or a static RAM (SRAM). The non-volatile storage 114 can employ at least one of a flexible disk, a hard disk, a solid state drive (SSD), a read only memory (ROM), an erasable and programmable read only memory (EPROM), a flash memory, a compact disk (CD or CD-ROM), a digital versatile disk (DVD), a card-type memory, or a stick-type memory.
[0110] Also stored in non-volatile storage 114 are programs 121. Programs 121 are, for example, collections of machine-readable instructions and / or data used to create, manage, and control a particular software function. Note that in configurations where memory 113 is significantly faster than non-volatile storage 114, programs 121 can be transferred from non-volatile storage 114 to memory 113 before being executed by processor 112.
[0111] Through network interface 115, computer 102 can communicate and interact with other computers over network 103. Network 103 can be, for example, a Local Area Network (LAN), a Wide Area Network (WAN) such as the Internet, or a combination of LANs and WANs, including wired, wireless, or fiber optic connections. In general, network 103 consists of any combination of connections and protocols that support communication between two or more computers and associated devices.
[0112] The peripherals interface 116 can input and output data to and from other devices that may be locally connected to the computer 102. For example, the peripherals interface 116 provides a connection to external devices 131, which may include a keyboard, a mouse, a keypad, a touch screen, and / or other suitable input devices. The external devices 131 may also include portable computer-readable storage media, such as, for example, thumb drives, portable optical or magnetic disks, and memory cards.
[0113] In embodiments of the present disclosure, for example, the software and data used to implement program 121 may be stored on such a portable computer-readable storage medium. In such an embodiment, the software may be loaded directly into non-volatile storage 114 or into memory 113 via peripheral interface 116. Peripheral interface 116 may use industry standards such as, for example, RS-232 or Universal Serial Bus (USB) to connect to external devices 131.
[0114] Display interface 117 may connect computer 102 to a display 132 that may be used to present a command line or a graphical user interface to a user of computer 102. For example, display interface 117 may employ industry standards such as Video Graphics Array (VGA), Digital Visual Interface (DVI), DisplayPort, High-Definition Multimedia Interface (HDMI), or the like.
[0115] <Example of the configuration of an image encoding device> FIG. 8 shows the configuration of an embodiment of an image encoding device as an image processing device to which the present disclosure is applied.
[0116] 8 encodes image data using a predictive process. Here, as an encoding method, for example, a Versatile Video Coding (VVC) method or a High Efficiency Video Coding (HEVC) method is used.
[0117] 8 includes an A / D conversion unit 202, a screen rearrangement buffer 203, a calculation unit 204, an orthogonal transformation unit 205, a quantization unit 206, a lossless coding unit 207, and an accumulation buffer 208. The image coding device 201 also includes an inverse quantization unit 209, an inverse orthogonal transformation unit 210, a calculation unit 211, a deblocking filter 212, an adaptive offset filter 213, an adaptive loop filter 214, a frame memory 215, a selection unit 216, an intra prediction unit 217, a motion prediction and compensation unit 218, a predicted image selection unit 219, and a rate control unit 220.
[0118] The A / D conversion unit 202 A / D converts the input image data (Picture(s)) and supplies the converted data to a screen rearrangement buffer 203. Note that a configuration may be adopted in which an image in the form of digital data is input without providing the A / D conversion unit 202.
[0119] The screen rearrangement buffer 203 stores image data supplied from the A / D conversion unit 202, and rearranges the stored frame images in display order into a frame order for encoding in accordance with a GOP (Group of Picture) structure. The screen rearrangement buffer 203 outputs the image with the rearranged frame order to the calculation unit 204, the intra prediction unit 217, and the motion prediction / compensation unit 218.
[0120] The calculation unit 204 subtracts a predicted image supplied from the intra prediction unit 217 or the motion prediction / compensation unit 218 via a predicted image selection unit 219 from the image output from the screen rearrangement buffer 203, and outputs the difference information to the orthogonal transformation unit 205.
[0121] For example, in the case of an image to be intra-coded, the calculation unit 204 subtracts a predicted image supplied from the intra prediction unit 217 from the image output from the screen rearrangement buffer 203. Also, in the case of an image to be inter-coded, the calculation unit 204 subtracts a predicted image supplied from the motion prediction / compensation unit 218 from the image output from the screen rearrangement buffer 203.
[0122] The orthogonal transformation unit 205 performs orthogonal transformation such as discrete cosine transformation or Karhunen-Loeve transformation on the difference information supplied from the calculation unit 204 , and supplies the resulting transformation coefficients to a quantization unit 206 .
[0123] The quantization unit 206 quantizes the transform coefficients output by the orthogonal transformation unit 205. The quantization unit 206 supplies the quantized transform coefficients to the lossless encoding unit 207.
[0124] The lossless encoding unit 207 performs lossless encoding, such as variable length encoding or arithmetic encoding, on the quantized transform coefficients.
[0125] The lossless encoding unit 207 obtains parameters such as information indicating an intra prediction mode from the intra prediction unit 217, and obtains parameters such as information indicating an inter prediction mode and motion vector information from the motion prediction / compensation unit 218.
[0126] The lossless encoding unit 207 encodes the quantized transform coefficients and also encodes each acquired parameter (syntax element) to form (multiplex) the encoded data as part of the header information of the encoded data. The lossless encoding unit 207 supplies the encoded data obtained by encoding to the accumulation buffer 208 to accumulate it.
[0127] For example, the lossless encoding unit 207 performs lossless encoding processing such as variable length encoding or arithmetic encoding. An example of variable length encoding is CAVLC (Context-Adaptive Variable Length Coding). An example of arithmetic encoding is CABAC (Context-Adaptive Binary Arithmetic Coding).
[0128] The accumulation buffer 208 temporarily holds the encoded stream (encoded data) supplied from the lossless encoding unit 207, and outputs the encoded stream as an encoded image at a predetermined timing, for example, to a downstream recording device or a transmission path (not shown). In other words, the accumulation buffer 208 also functions as a transmission unit that transmits the encoded stream.
[0129] The transform coefficients quantized by the quantization unit 206 are also supplied to the inverse quantization unit 209. The inverse quantization unit 209 inverse quantizes the quantized transform coefficients using a method corresponding to the quantization by the quantization unit 206. The inverse quantization unit 209 supplies the obtained transform coefficients to the inverse orthogonal transformation unit 210.
[0130] The inverse orthogonal transform unit 210 performs inverse orthogonal transform on the supplied transform coefficients using a method corresponding to the orthogonal transform processing by the orthogonal transform unit 205. The inverse orthogonal transformed output (reconstructed difference information) is supplied to the calculation unit 211.
[0131] The calculation unit 211 adds a predicted image supplied from the intra prediction unit 217 or the motion prediction / compensation unit 218 via the predicted image selection unit 219 to the inverse orthogonal transform result supplied from the inverse orthogonal transform unit 210, i.e., the restored differential information, to obtain a locally decoded image (decoded image).
[0132] For example, when the difference information corresponds to an image to be intra-coded, the calculation unit 211 adds to the difference information a predicted image supplied from the intra prediction unit 217. Also, when the difference information corresponds to an image to be inter-coded, the calculation unit 211 adds to the difference information a predicted image supplied from the motion prediction / compensation unit 218.
[0133] The decoded image resulting from this addition is supplied to a deblocking filter 212 and a frame memory 215.
[0134] The deblocking filter 212 suppresses block distortion of the decoded image by appropriately performing deblocking filter processing on the image from the calculation unit 211, and supplies the filter processing result to the adaptive offset filter 213. The deblocking filter 212 has parameters β and Tc that are determined based on the quantization parameter QP. The parameters β and Tc are thresholds (parameters) used in determining the deblocking filter.
[0135] Note that the parameters β and Tc of the deblocking filter 212 are extended from β and Tc defined in the HEVC format. The offsets of the parameters β and Tc are coded in the lossless coding unit 207 as parameters of the deblocking filter and transmitted to the image decoding device 301 in Fig. 10, which will be described later.
[0136] The adaptive offset filter 213 performs an offset filter (SAO: Sample Adaptive Offset) process on the image filtered by the deblocking filter 212, mainly suppressing ringing.
[0137] There are nine types of offset filters: two types of band offset, six types of edge offset, and no offset. The adaptive offset filter 213 performs filtering on the image filtered by the deblocking filter 212, using a quad-tree structure in which the type of offset filter is determined for each divided region, and an offset value for each divided region. The adaptive offset filter 213 supplies the filtered image to the adaptive loop filter 214.
[0138] In the image encoding device 201, the quad-tree structure and the offset value for each divided region are calculated and used by the adaptive offset filter 213. The calculated quad-tree structure and the offset value for each divided region are coded as adaptive offset parameters in the lossless coding unit 207 and transmitted to the image decoding device 301 in Fig. 10, which will be described later.
[0139] The adaptive loop filter 214 performs adaptive loop filter (ALF) processing for each processing unit using a filter coefficient on the image filtered by the adaptive offset filter 213. In the adaptive loop filter 214, for example, a two-dimensional Wiener filter is used as a filter. Of course, a filter other than a Wiener filter may be used. The adaptive loop filter 214 supplies the filter processing result to the frame memory 215.
[0140] Although not shown in the example of Fig. 8, in the image encoding device 201, the filter coefficients are calculated and used by the adaptive loop filter 214 for each processing unit so as to minimize a residual with respect to the original image from the screen rearrangement buffer 203. The calculated filter coefficients are coded as adaptive loop filter parameters in the lossless coding unit 207 and transmitted to the image decoding device 301 in Fig. 10, which will be described later.
[0141] The frame memory 215 outputs the stored reference image to the intra prediction unit 217 or the motion prediction / compensation unit 218 via the selection unit 216 at a predetermined timing.
[0142] For example, in the case of an image to be intra-coded, the frame memory 215 supplies the reference image to the intra prediction unit 217 via the selection unit 216. Also, in the case of an image to be inter-coded, the frame memory 215 supplies the reference image to the motion prediction / compensation unit 218 via the selection unit 216.
[0143] When the reference image supplied from the frame memory 215 is an image to be intra-coded, the selection unit 216 supplies the reference image to the intra prediction unit 217. In addition, when the reference image supplied from the frame memory 215 is an image to be inter-coded, the selection unit 216 supplies the reference image to the motion prediction / compensation unit 218.
[0144] The intra prediction unit 217 performs intra prediction (intra-screen prediction) that generates a predicted image using pixel values within a screen. The intra prediction unit 217 performs intra prediction in a plurality of modes (intra prediction modes).
[0145] The intra prediction unit 217 generates prediction images in all intra prediction modes, evaluates each prediction image, and selects an optimal mode. When the intra prediction unit 217 selects the optimal intra prediction mode, the intra prediction unit 217 supplies the prediction image generated in the optimal mode to the calculation unit 204 or the calculation unit 211 via the prediction image selection unit 219.
[0146] Furthermore, as described above, the intra prediction unit 217 appropriately supplies parameters such as intra prediction mode information indicating the adopted intra prediction mode to the lossless encoding unit 207.
[0147] The motion prediction and compensation unit 218 performs motion prediction for an image to be inter-coded, using an input image supplied from the screen rearrangement buffer 203 and a reference image supplied from the frame memory 215 via a selection unit 216. The motion prediction and compensation unit 218 also performs motion compensation processing according to a motion vector detected by motion prediction, and generates a predicted image (inter-predicted image information).
[0148] The motion prediction and compensation unit 218 performs inter prediction processing for all candidate inter prediction modes to generate predicted images. The motion prediction and compensation unit 218 supplies the generated predicted images to the calculation unit 204 and the calculation unit 211 via the predicted image selection unit 219. In addition, the motion prediction and compensation unit 218 supplies parameters such as inter prediction mode information indicating the adopted inter prediction mode and motion vector information indicating the calculated motion vector to the lossless encoding unit 207.
[0149] In the case of an image to be intra-coded, the prediction image selection unit 219 supplies the output of the intra prediction unit 217 to the calculation unit 204 or the calculation unit 211, and in the case of an image to be inter-coded, the prediction image selection unit 219 supplies the output of the motion prediction / compensation unit 218 to the calculation unit 204 or the calculation unit 211.
[0150] The rate control unit 220 controls the rate of the quantization operation of the quantization unit 206 based on the compressed image stored in the storage buffer 208 so that overflow or underflow does not occur.
[0151] The image encoding device 201 is configured in this manner, with an adaptive color transform unit 42 (FIG. 2) provided between the calculation unit 204 and the orthogonal transform unit 205, and an inverse adaptive color transform unit 47 (FIG. 2) provided between the inverse orthogonal transform unit 210 and the calculation unit 211. In the image encoding device 201, control is performed in accordance with the first to third concepts described above, thereby making it possible to avoid an increase in memory size.
[0152] <Operation of the image encoding device> The flow of the encoding process executed by the image encoding device 201 described above will be described with reference to FIG.
[0153] In step S101, the A / D conversion unit 202 performs A / D conversion on an input image.
[0154] In step S102, the screen rearrangement buffer 203 stores the images A / D converted by the A / D conversion unit 202, and rearranges the pictures from the display order to the coding order.
[0155] When the image to be processed supplied from the screen rearrangement buffer 203 is an image of a block to be intra-processed, a decoded image to be referenced is read from the frame memory 215 and supplied to the intra prediction unit 217 via the selection unit 216.
[0156] Based on these images, in step S103, the intra prediction unit 217 intra predicts pixels of the block to be processed in all candidate intra prediction modes. Note that pixels that have not been filtered by the deblocking filter 212 are used as decoded pixels to be referenced.
[0157] Through this process, intra prediction is performed in all candidate intra prediction modes, and cost function values are calculated for all candidate intra prediction modes. Then, an optimal intra prediction mode is selected based on the calculated cost function value, and a predicted image generated by intra prediction in the optimal intra prediction mode and its cost function value are supplied to the predicted image selection unit 219.
[0158] When the image to be processed supplied from the screen rearrangement buffer 203 is an image to be inter-processed, the image to be referenced is read from the frame memory 215 and supplied to the motion prediction and compensation unit 218 via the selection unit 216. In step S104, the motion prediction and compensation unit 218 performs motion prediction and compensation processing based on these images.
[0159] Through this process, motion prediction processing is performed in all candidate inter prediction modes, cost function values are calculated for all candidate inter prediction modes, and an optimal inter prediction mode is determined based on the calculated cost function values. Then, a predicted image generated in the optimal inter prediction mode and its cost function value are supplied to a predicted image selection unit 219.
[0160] In step S105, the prediction image selection unit 219 determines one of the optimal intra prediction mode and the optimal inter prediction mode as an optimal prediction mode, based on each cost function value output from the intra prediction unit 217 and the motion prediction / compensation unit 218. Then, the prediction image selection unit 219 selects a prediction image of the determined optimal prediction mode, and supplies it to the calculation units 204, 211. This prediction image is used in the calculations of steps S106 and S111, which will be described later.
[0161] It should be noted that the selection information of this predicted image is supplied to the intra prediction unit 217 or the motion prediction / compensation unit 218. When a predicted image of the optimal intra prediction mode is selected, the intra prediction unit 217 supplies information indicating the optimal intra prediction mode (i.e., parameters related to intra prediction) to the lossless encoding unit 207.
[0162] When a predicted image in the optimal inter prediction mode is selected, the motion prediction / compensation unit 218 outputs information indicating the optimal inter prediction mode and information corresponding to the optimal inter prediction mode (i.e., parameters related to motion prediction) to the lossless encoding unit 207. Examples of the information corresponding to the optimal inter prediction mode include motion vector information and reference frame information.
[0163] In step S106, the calculation unit 204 calculates the difference between the image rearranged in step S102 and the predicted image selected in step S105. The predicted image is supplied to the calculation unit 204 via a predicted image selection unit 219 from a motion prediction and compensation unit 218 in the case of inter prediction, or from an intra prediction unit 217 in the case of intra prediction.
[0164] The differential data has a smaller data volume than the original image data, and therefore the data volume can be compressed compared to when the image is encoded as is.
[0165] In step S107, the orthogonal transformation unit 205 performs an orthogonal transformation on the difference information supplied from the calculation unit 204. Specifically, an orthogonal transformation such as a discrete cosine transform or a Karhunen-Loeve transform is performed, and a transform coefficient is output.
[0166] In step S108, the quantization unit 206 quantizes the transform coefficients. During this quantization, the rate is controlled as will be described in the process of step S118 below.
[0167] The difference information quantized as described above is locally decoded as follows. That is, in step S109, the inverse quantization unit 209 inverse quantizes the transform coefficients quantized by the quantization unit 206 with characteristics corresponding to the characteristics of the quantization unit 206. In step S110, the inverse orthogonal transformation unit 210 inverse orthogonally transforms the transform coefficients inverse quantized by the inverse quantization unit 209 with characteristics corresponding to the characteristics of the orthogonal transformation unit 205.
[0168] In step S111, the calculation unit 211 adds the predicted image input via the predicted image selection unit 219 to the locally decoded difference information to generate a locally decoded (i.e., locally decoded) image (an image corresponding to the input to the calculation unit 204).
[0169] In step S112, the deblocking filter 212 performs a deblocking filter process on the image output from the calculation unit 211. At this time, parameters β and Tc extended from β and Tc defined in the HEVC method are used as thresholds for determining the deblocking filter. The filtered image from the deblocking filter 212 is output to the adaptive offset filter 213.
[0170] Note that the parameters β and each offset of Tc, which are input by the user operating an operation unit or the like and used in the deblocking filter 212, are supplied to the lossless encoding unit 207 as parameters of the deblocking filter.
[0171] In step S113, the adaptive offset filter 213 performs adaptive offset filter processing. With this processing, a filter processing is performed on the image filtered by the deblocking filter 212 using a quad-tree structure in which the type of offset filter is determined for each divided region and an offset value for each divided region. The filtered image is supplied to the adaptive loop filter 214.
[0172] The determined quad-tree structure and offset value for each divided region are supplied to the lossless encoding unit 207 as adaptive offset parameters.
[0173] In step S114, the adaptive loop filter 214 performs adaptive loop filter processing on the image filtered by the adaptive offset filter 213. For example, the image filtered by the adaptive offset filter 213 is subjected to filtering processing for each processing unit using a filter coefficient, and the filtering result is supplied to the frame memory 215.
[0174] In step S115, the frame memory 215 stores the filtered image. Note that the frame memory 215 also receives images that have not been filtered by the deblocking filter 212, the adaptive offset filter 213, and the adaptive loop filter 214 from the calculation unit 211 and stores them.
[0175] Meanwhile, the transform coefficients quantized in step S108 described above are also supplied to the lossless encoding unit 207. In step S116, the lossless encoding unit 207 encodes the quantized transform coefficients output from the quantization unit 206 and the supplied parameters. That is, the difference image is losslessly encoded by variable length encoding, arithmetic encoding, or the like, and compressed. Here, the parameters to be encoded include parameters of a deblocking filter, parameters of an adaptive offset filter, parameters of an adaptive loop filter, quantization parameters, motion vector information, reference frame information, prediction mode information, and the like.
[0176] In step S117, the accumulation buffer 208 accumulates the encoded differential image (i.e., the encoded stream) as a compressed image. The compressed image accumulated in the accumulation buffer 208 is read out as appropriate and transmitted to the decoding side via a transmission path.
[0177] In step S118, the rate control unit 220 controls the rate of the quantization operation of the quantization unit 206 based on the compressed image stored in the storage buffer 208 so that an overflow or underflow does not occur.
[0178] When the process of step S118 ends, the encoding process ends.
[0179] In the encoding process as described above, the adaptive color converter 42 (FIG. 2) performs ACT processing between steps S106 and S107, and the inverse adaptive color converter 47 (FIG. 2) performs IACT processing between steps S110 and S111. In the encoding process, control regarding application of the ACT processing and IACT processing is performed according to the first to third concepts described above.
[0180] <Configuration example of an image decoding device> 10 shows the configuration of an embodiment of an image decoding device as an image processing device to which the present disclosure is applied. An image decoding device 301 shown in Fig. 10 is a decoding device corresponding to the image encoding device 201 in Fig. 8.
[0181] The encoded stream (encoded data) encoded by the image encoding device 201 is transmitted via a predetermined transmission path to an image decoding device 301 corresponding to this image encoding device 201, and is decoded.
[0182] As shown in FIG. 10, the image decoding device 301 includes an accumulation buffer 302, a lossless decoding unit 303, an inverse quantization unit 304, an inverse orthogonal transformation unit 305, a calculation unit 306, a deblocking filter 307, an adaptive offset filter 308, an adaptive loop filter 309, a screen rearrangement buffer 310, a D / A conversion unit 311, a frame memory 312, a selection unit 313, an intra prediction unit 314, a motion prediction and compensation unit 315, and a selection unit 316.
[0183] The accumulation buffer 302 also serves as a receiving unit that receives the transmitted encoded data. The accumulation buffer 302 receives and accumulates the transmitted encoded data. This encoded data is encoded by the image encoding device 201. The lossless decoding unit 303 decodes the encoded data read from the accumulation buffer 302 at a predetermined timing using a method corresponding to the encoding method of the lossless encoding unit 207 in Fig. 8.
[0184] The lossless decoding unit 303 supplies parameters such as information indicating the decoded intra prediction mode to an intra prediction unit 314, and supplies parameters such as information indicating the inter prediction mode and motion vector information to a motion prediction / compensation unit 315. In addition, the lossless decoding unit 303 supplies decoded deblocking filter parameters to a deblocking filter 307, and supplies decoded adaptive offset parameters to an adaptive offset filter 308.
[0185] The inverse quantization unit 304 inverse quantizes the coefficient data (quantized coefficients) obtained by decoding by the lossless decoding unit 303, using a method corresponding to the quantization method of the quantization unit 206 in Fig. 8. That is, the inverse quantization unit 304 uses the quantization parameter supplied from the image encoding device 201 to inverse quantize the quantized coefficients in a method similar to that of the inverse quantization unit 209 in Fig. 8.
[0186] The inverse quantization unit 304 supplies the inverse quantized coefficient data, i.e., the orthogonal transform coefficients, to the inverse orthogonal transformation unit 305. The inverse orthogonal transformation unit 305 performs inverse orthogonal transform on the orthogonal transform coefficients using a method corresponding to the orthogonal transform method of the orthogonal transformation unit 205 in Fig. 8, and obtains decoded residual data corresponding to the residual data before being orthogonally transformed in the image encoding device 201.
[0187] The decoded residual data obtained by the inverse orthogonal transform is supplied to the calculation unit 306. In addition, a predicted image is supplied to the calculation unit 306 from the intra prediction unit 314 or the motion prediction / compensation unit 315 via a selection unit 316.
[0188] The calculation unit 306 adds the decoded residual data and the predicted image to obtain decoded image data that corresponds to the image data before the predicted image is subtracted by the calculation unit 204 of the image encoding device 201. The calculation unit 306 supplies the decoded image data to a deblocking filter 307.
[0189] The deblocking filter 307 appropriately performs deblocking filter processing on the image from the calculation unit 306 to suppress block distortion of the decoded image, and supplies the filter processing result to the adaptive offset filter 308. The deblocking filter 307 is basically configured in the same manner as the deblocking filter 212 in Fig. 8. That is, the deblocking filter 307 has parameters β and Tc that are determined based on the quantization parameter. The parameters β and Tc are thresholds used in determining the deblocking filter.
[0190] Note that the parameters β and Tc of the deblocking filter 307 are extended from β and Tc defined in the HEVC format. The offsets of the deblocking filter parameters β and Tc encoded by the image encoding device 201 are received by the image decoding device 301 as deblocking filter parameters, decoded by the lossless decoding unit 303, and used by the deblocking filter 307.
[0191] The adaptive offset filter 308 performs offset filter (SAO) processing, which mainly suppresses ringing, on the image filtered by the deblocking filter 307.
[0192] The adaptive offset filter 308 performs filtering on the image filtered by the deblocking filter 307, using a quad-tree structure in which the type of offset filter is determined for each divided region, and an offset value for each divided region. The adaptive offset filter 308 supplies the filtered image to the adaptive loop filter 309.
[0193] Note that this quad-tree structure and the offset value for each divided region are calculated by the adaptive offset filter 213 of the image encoding device 201, and are encoded and transmitted as adaptive offset parameters. The quad-tree structure and the offset value for each divided region encoded by the image encoding device 201 are received by the image decoding device 301 as adaptive offset parameters, decoded by the lossless decoding unit 303, and used by the adaptive offset filter 308.
[0194] The adaptive loop filter 309 performs filtering on the image filtered by the adaptive offset filter 308 for each processing unit using the filter coefficients, and supplies the filtering results to a frame memory 312 and a screen rearrangement buffer 310 .
[0195] Although not shown in the example of Figure 10, in the image decoding device 301, the filter coefficients are calculated for each LUC by the adaptive loop filter 214 of the image encoding device 201, and the encoded and transmitted filter coefficients are decoded by the lossless decoding unit 303 and used as adaptive loop filter parameters.
[0196] The screen rearrangement buffer 310 rearranges the images and supplies them to the D / A conversion unit 311. That is, the order of the frames rearranged for the encoding order by the screen rearrangement buffer 203 in Fig. 8 is rearranged to the original display order.
[0197] The D / A conversion unit 311 D / A converts the images (decoded picture(s)) supplied from the screen rearrangement buffer 310, outputs them to a display (not shown), and displays them. Note that a configuration may be adopted in which the images are output as digital data without providing the D / A conversion unit 311.
[0198] The output of the adaptive loop filter 309 is further provided to a frame memory 312 .
[0199] The frame memory 312, the selection unit 313, the intra prediction unit 314, the motion prediction / compensation unit 315, and the selection unit 316 correspond to the frame memory 215, the selection unit 216, the intra prediction unit 217, the motion prediction / compensation unit 218, and the predicted image selection unit 219 of the image encoding device 201, respectively.
[0200] The selection unit 313 reads out an image to be inter-processed and an image to be referenced from the frame memory 312, and supplies them to the motion prediction and compensation unit 315. The selection unit 313 also reads out an image to be used in intra prediction from the frame memory 312, and supplies them to the intra prediction unit 314.
[0201] The intra prediction unit 314 is appropriately supplied with information indicating an intra prediction mode obtained by decoding the header information, etc. from the lossless decoding unit 303. The intra prediction unit 314 generates a predicted image from a reference image obtained from the frame memory 312 based on this information, and supplies the generated predicted image to the selection unit 316.
[0202] The motion prediction / compensation unit 315 receives information obtained by decoding the header information (prediction mode information, motion vector information, reference frame information, flags, various parameters, and the like) from the lossless decoding unit 303 .
[0203] The motion prediction / compensation unit 315 generates a predicted image from the reference image obtained from the frame memory 312 based on the information supplied from the lossless decoding unit 303 , and supplies the generated predicted image to the selection unit 316 .
[0204] The selection unit 316 selects the predicted image generated by the motion prediction / compensation unit 315 or the intra prediction unit 314 , and supplies the selected image to the calculation unit 306 .
[0205] The image decoding device 301 is configured as described above, and the inverse adaptive color transform unit 64 (FIG. 3) is provided between the inverse orthogonal transform unit 305 and the calculation unit 306. In the image decoding device 301, control is performed according to the first to third concepts described above, thereby making it possible to avoid an increase in memory size.
[0206] <Operation of the image decoding device> An example of the flow of the decoding process executed by the image decoding device 301 described above will be described with reference to FIG.
[0207] When the decoding process starts, in step S201, the accumulation buffer 302 receives and accumulates the transmitted encoded stream (data). In step S202, the lossless decoding unit 303 decodes the encoded data supplied from the accumulation buffer 302. The I-pictures, P-pictures, and B-pictures encoded by the lossless encoding unit 207 in Fig. 8 are decoded.
[0208] Prior to decoding a picture, parameter information such as motion vector information, reference frame information, and prediction mode information (intra prediction mode or inter prediction mode) is also decoded.
[0209] When the prediction mode information is intra prediction mode information, the prediction mode information is supplied to the intra prediction unit 314. When the prediction mode information is inter prediction mode information, the prediction mode information and corresponding motion vector information are supplied to the motion prediction and compensation unit 315. In addition, the parameters of the deblocking filter and the adaptive offset parameters are also decoded and supplied to the deblocking filter 307 and the adaptive offset filter 308, respectively.
[0210] In step S203 , the intra prediction unit 314 or the motion prediction / compensation unit 315 performs a prediction image generation process in accordance with the prediction mode information supplied from the lossless decoding unit 303 .
[0211] That is, when intra prediction mode information is supplied from the lossless decoding unit 303, the intra prediction unit 314 generates an intra prediction image in the intra prediction mode. When inter prediction mode information is supplied from the lossless decoding unit 303, the motion prediction / compensation unit 315 performs motion prediction / compensation processing in the inter prediction mode to generate an inter prediction image.
[0212] Through this process, the predicted image (intra predicted image) generated by the intra prediction unit 314 or the predicted image (inter predicted image) generated by the motion prediction / compensation unit 315 is supplied to the selection unit 316 .
[0213] In step S204, the selection unit 316 selects a predicted image. That is, the predicted image generated by the intra prediction unit 314 or the predicted image generated by the motion prediction / compensation unit 315 is supplied. Therefore, the supplied predicted image is selected and supplied to the calculation unit 306, and is added to the output of the inverse orthogonal transformation unit 305 in step S207 described later.
[0214] In the above-mentioned step S202, the transform coefficients decoded by the lossless decoding unit 303 are also supplied to the inverse quantization unit 304. In step S205, the inverse quantization unit 304 inverse quantizes the transform coefficients decoded by the lossless decoding unit 303 with characteristics corresponding to the characteristics of the quantization unit 206 in FIG.
[0215] In step S206, the inverse orthogonal transform unit 305 performs inverse orthogonal transform on the transform coefficients dequantized by the inverse quantization unit 304 with characteristics corresponding to the characteristics of the orthogonal transform unit 205 in Fig. 8. As a result, difference information corresponding to the input (output of the calculation unit 204) of the orthogonal transform unit 205 in Fig. 8 is decoded.
[0216] In step S207, the calculation unit 306 adds the predicted image selected in the process of step S204 described above and input via the selection unit 316 to the difference information, thereby decoded the original image.
[0217] In step S208, the deblocking filter 307 performs a deblocking filter process on the image output from the calculation unit 306. At this time, parameters β and Tc extended from β and Tc defined in the HEVC format are used as thresholds for determining the deblocking filter. The filtered image from the deblocking filter 307 is output to the adaptive offset filter 308. Note that in the deblocking filter process, the offsets of the parameters β and Tc of the deblocking filter supplied from the lossless decoding unit 303 are also used.
[0218] In step S209, the adaptive offset filter 308 performs adaptive offset filter processing. With this processing, a filter process is performed on the image filtered by the deblocking filter 307 using a quad-tree structure in which the type of offset filter is determined for each divided region and an offset value for each divided region. The filtered image is supplied to the adaptive loop filter 309.
[0219] In step S210, the adaptive loop filter 309 performs adaptive loop filtering on the image filtered by the adaptive offset filter 308. The adaptive loop filter 309 performs filtering on the input image for each processing unit by using the filter coefficients calculated for each processing unit, and supplies the filtering results to the screen rearrangement buffer 310 and the frame memory 312.
[0220] In step S211, the frame memory 312 stores the filtered image.
[0221] In step S212, the screen rearrangement buffer 310 rearranges the images after the adaptive loop filter 309, and then supplies the images to the D / A conversion unit 311. That is, the order of the frames rearranged for encoding by the screen rearrangement buffer 203 of the image encoding device 201 is rearranged to the original display order.
[0222] In step S213, the D / A conversion unit 311 performs D / A conversion on the image rearranged by the screen rearrangement buffer 310 and outputs the image to a display (not shown), where the image is displayed.
[0223] When the process of step S213 ends, the decoding process ends.
[0224] In the above-described decoding process, the IACT process is performed by the inverse adaptive color converter 64 (FIG. 3) between steps S206 and S207. In the decoding process, control regarding application of the ACT process and the IACT process is performed according to the first to third concepts described above.
[0225] <Computer configuration example> Next, the above-mentioned series of processes (image processing method) can be performed by hardware or software. When the series of processes is performed by software, a program constituting the software is installed in a general-purpose computer or the like.
[0226] FIG. 12 is a block diagram showing an example of the configuration of an embodiment of a computer in which a program for executing the series of processes described above is installed.
[0227] The program can be recorded in advance on a hard disk 1005 or a ROM 1003 as a recording medium built into the computer.
[0228] Alternatively, the program can be stored (recorded) in a removable recording medium 1011 driven by the drive 1009. Such a removable recording medium 1011 can be provided as a so-called package software. Here, examples of the removable recording medium 1011 include a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, and a semiconductor memory.
[0229] The program can be installed in the computer from the removable recording medium 1011 as described above, or can be downloaded to the computer via a communication network or a broadcasting network and installed in the built-in hard disk 1005. That is, the program can be transferred to the computer wirelessly from a download site via an artificial satellite for digital satellite broadcasting, or transferred to the computer by wire via a network such as a LAN (Local Area Network) or the Internet.
[0230] The computer includes a CPU (Central Processing Unit) 1002 , to which an input / output interface 1010 is connected via a bus 1001 .
[0231] When a user inputs a command by operating an input unit 1007 via an input / output interface 1010, the CPU 1002 executes a program stored in a ROM (Read Only Memory) 1003 in accordance with the command. Alternatively, the CPU 1002 loads a program stored in a hard disk 1005 into a RAM (Random Access Memory) 1004 and executes the program.
[0232] As a result, the CPU 1002 performs processing according to the above-mentioned flowchart or processing performed by the configuration of the above-mentioned block diagram. Then, the CPU 1002 outputs the processing result from the output unit 1006 via the input / output interface 1010, or transmits it from the communication unit 1008, or further records it on the hard disk 1005, as necessary.
[0233] The input unit 1007 includes a keyboard, a mouse, a microphone, etc. The output unit 1006 includes an LCD (Liquid Crystal Display), a speaker, etc.
[0234] Here, in this specification, the processing performed by a computer according to a program does not necessarily have to be performed in chronological order according to the order described in the flowchart. In other words, the processing performed by a computer according to a program also includes processing executed in parallel or individually (for example, parallel processing or processing by objects).
[0235] The program may be processed by one computer (processor), or may be distributed among multiple computers. Furthermore, the program may be transferred to a remote computer for execution.
[0236] Furthermore, in this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.
[0237] Also, for example, the configuration described above as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Also, it is of course possible to add configurations other than those described above to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the system as a whole are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).
[0238] Furthermore, for example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.
[0239] Furthermore, for example, the above-mentioned program can be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and can obtain the necessary information.
[0240] Also, for example, each step described in the above flow chart can be executed by one device, or can be shared and executed by multiple devices. Furthermore, when one step includes multiple processes, the multiple processes included in the one step can be executed by one device, or can be shared and executed by multiple devices. In other words, multiple processes included in one step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as one step.
[0241] In addition, the processing of the steps of the program executed by the computer may be executed in chronological order according to the order described in this specification, or may be executed in parallel, or individually at a necessary timing such as when a call is made. In other words, the processing of each step may be executed in an order different from the order described above, as long as no contradiction occurs. Furthermore, the processing of the steps of the program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.
[0242] In addition, the present technologies described in this specification can be implemented independently and individually, as long as there is no contradiction. Of course, any of the present technologies can be implemented in combination. For example, a part or all of the present technology described in any of the embodiments can be implemented in combination with a part or all of the present technology described in another embodiment. In addition, any of the present technologies described above can be implemented in combination with other technologies not described above.
[0243] <Examples of configuration combinations> The present technology can also be configured as follows. (1) an adaptive color conversion unit that applies adaptive color conversion processing to a residual signal of an image to be encoded, the adaptive color conversion processing adaptively converting a color space of the image; an orthogonal transform unit that performs an orthogonal transform process on a residual signal of the image or on a residual signal of the image on which the adaptive color transform process has been performed, for each orthogonal transform block that is a processing unit; A control unit that controls application of the adaptive color conversion process; An image processing device comprising: (2) a first block size and a second block size larger than the first block size are defined as the maximum block size of the orthogonal transform block; The control unit controls the adaptive color conversion unit to apply the adaptive color conversion process when the first block size is used as the maximum block size of the orthogonal transformation block. The image processing device according to (1) above. (3) the first block size is 32 and the second block size is 64; The control unit applies the adaptive color conversion process while limiting the maximum block size of the orthogonal transformation block to 32. The image processing device according to (2) above. (4) The control unit transmits, to the decoding side, sps_act_enabled_flag indicating that the adaptive color conversion process is to be applied when sps_max_luma_transform_size_64_flag included in a parameter set of a high-level syntax is 0. The image processing device according to (3) above. (5) The control unit controls the adaptive color conversion unit to apply the adaptive color conversion process when a predetermined restriction is imposed on a coding block when coding the image. An image processing device according to any one of (1) to (4) above. (6) The control unit applies the adaptive color conversion process when a block size of a coding block, which is a processing unit when coding the image, is limited to a predetermined size or less. The image processing device according to (5) above. (7) The control unit applies the adaptive color conversion process when the block size of the coding block is limited to 16×16 or less. The image processing device according to (6) above. (8) When a block size of a coding block, which is a processing unit when coding the image, is larger than a predetermined size, the control unit controls the orthogonal transformation unit to perform the orthogonal transformation processing using the orthogonal transformation block having a smaller block size obtained by dividing the coding block, and controls the adaptive color conversion unit to apply the adaptive color conversion processing. An image processing device according to any one of (1) to (7) above. (9) When the block size of the coding block is 64×64, the control unit causes the orthogonal transform process to be performed with the orthogonal transform block being 32×32, and causes the adaptive color transform process to be applied. The image processing device according to (8) above. (10) an inverse orthogonal transform unit that performs an inverse orthogonal transform process on the transform coefficients obtained by performing the orthogonal transform process for each of the orthogonal transform blocks to obtain the residual signal; an inverse adaptive color transform unit that performs an inverse adaptive color transform process on the residual signal acquired by the inverse orthogonal transform unit to adaptively inversely transform a color space of an image; Further equipped with The control unit performs control related to application of the inverse adaptive color conversion process in correspondence with the adaptive color conversion process. An image processing device according to any one of (1) to (9) above. (11) performing an adaptive color conversion process on a residual signal of an image to be encoded, the adaptive color conversion process converting a color space of the image to be encoded; performing an orthogonal transform process on a residual signal of the image or on a residual signal of the image on which the adaptive color transform process has been performed, for each orthogonal transform block serving as a processing unit; Controlling application of the adaptive color conversion process; An image processing method comprising: (12) an inverse orthogonal transform unit that performs an inverse orthogonal transform process on a transform coefficient obtained by performing an orthogonal transform process on a residual signal of an image to be decoded on an encoding side, for each orthogonal transform block that is a processing unit, to obtain the residual signal; an inverse adaptive color conversion unit that performs an inverse adaptive color conversion process on the residual signal to adaptively inversely convert a color space of an image; A control unit that controls application of the reverse adaptive color conversion process; An image processing device comprising: (13) obtaining the residual signal by performing an inverse orthogonal transform process on each orthogonal transform block, which is a processing unit, on a transform coefficient obtained by performing an orthogonal transform process on a residual signal of an image to be decoded on an encoding side; performing an inverse adaptive color transformation process on the residual signal, which adaptively inversely transforms a color space of an image; controlling application of the reverse adaptive color transformation process; An image processing method comprising:
[0244] In addition, the present embodiment is not limited to the above-described embodiment, and various modifications are possible without departing from the scope of the present disclosure. In addition, the effects described in this specification are merely examples and are not limiting, and other effects may be obtained. [Explanation of symbols]
[0245] 11 image processing system, 12 image encoding device, 13 image decoding device, 21 prediction unit, 22 encoding unit, 23 memory unit, 24 control unit, 31 prediction unit, 32 decoding unit, 33 memory unit, 34 control unit, 41 calculation unit, 42 adaptive color transform unit, 43 orthogonal transform unit, 44 quantization unit, 45 inverse quantization unit, 46 inverse orthogonal transform unit, 47 inverse adaptive color transform unit, 48 calculation unit, 49 prediction unit, 50 encoding unit, 61 decoding unit, 62 inverse quantization unit, 63 inverse orthogonal transform unit, 64 inverse adaptive color transform unit, 65 calculation unit, 66 prediction unit
Claims
1. a parsing unit for parsing size identification data from the bit stream, the size identification data identifying whether a maximum block size of an orthogonal transform block is 64 or 32; an inverse adaptive color transform unit that applies the inverse adaptive color transform processing to a residual signal generated by applying an inverse orthogonal transform processing to transform coefficients obtained by decoding the bitstream for each orthogonal transform block serving as a processing unit, in accordance with identification data that identifies whether or not an adaptive color transform processing that adaptively transforms a color space of an image or an inverse adaptive color transform processing that adaptively inverse transforms a color space of an image is enabled only when the size identification data parsed by the parser indicates that 32 is applied as a maximum block size of an orthogonal transform block; An image processing device comprising:
2. The size identification data is set as a sequence parameter set of the bitstream. The image processing device according to claim 1 .
3. The size identification data is set as the syntax of sps_max_luma_transform_size_64_flag. The image processing device according to claim 2 .
4. The identification data is referenced only when the value of the sps_max_luma_transform_size_64_flag is set to 0. The image processing device according to claim 3 .
5. an arithmetic decoding unit that arithmetically decodes the bitstream to generate quantized transform coefficients; an inverse quantization unit that inverse quantizes the quantized transform coefficients generated by the arithmetic decoding unit to generate the transform coefficients; The image processing device according to claim 1 , further comprising:
6. parsing size identification data from the bitstream that identifies whether a maximum block size of the orthogonal transform block is 64 or 32; only when the size identification data indicates that 32 is to be applied as the maximum block size of the orthogonal transform block, applying the inverse adaptive color transform processing to a residual signal generated by applying an inverse orthogonal transform processing to transform coefficients obtained by decoding the bit stream for each orthogonal transform block serving as a processing unit, in accordance with identification data for identifying whether or not an adaptive color transform processing for adaptively transforming a color space of an image or an inverse adaptive color transform processing for adaptively inverse transforming the color space of an image is Enable; An image processing method comprising:
7. a setting unit which sets identification data for identifying whether an adaptive color conversion process for adaptively converting a color space of an image or an inverse adaptive color conversion process for adaptively inversely converting a color space of an image is enabled only when size identification data for identifying whether a maximum block size of an orthogonal transformation block is 64 or 32 is set to be 32 as the maximum block size of the orthogonal transformation block; an encoding unit that encodes the image to generate a bit stream including the identification data set by the setting unit; An image processing device comprising:
8. The setting unit sets the size identification data as a sequence parameter set of the bit stream. The image processing device according to claim 7.
9. The setting unit sets the size identification data as a syntax of sps_max_luma_transform_size_64_flag. The image processing device according to claim 8.
10. The setting unit sets the identification data only when the value of the sps_max_luma_transform_size_64_flag is set to 0. The image processing device according to claim 9 .
11. an orthogonal transform unit that applies an orthogonal transform process to a residual signal of the image for each orthogonal transform block that is a processing unit; a quantization unit that quantizes the transform coefficients generated by the orthogonal transform unit to generate quantized transform coefficients; an arithmetic coding unit that arithmetically codes the quantized transform coefficients generated by the quantization unit to generate the bit stream; The image processing device according to claim 7 , further comprising:
12. Only when size identification data for identifying whether the maximum block size of the orthogonal transformation block is 64 or 32 is set to be 32 as the maximum block size of the orthogonal transformation block, identification data for identifying whether an adaptive color transformation process for adaptively transforming a color space of an image or an inverse adaptive color transformation process for adaptively inverse transforming a color space of an image is enabled is set; encoding said image to generate a bitstream including said identification data; An image processing method comprising:
