Image processing device, imaging apparatus, image processing method, and program

Abstract

To provide a technique for suppressing noise in a gain map generated on the basis of two images having different dynamic ranges (e.g., an SDR image and an HDR image).SOLUTION: An image processing device comprises: first image generation means configured to generate a first image having a first dynamic range from a captured image; second image generation means configured to generate a second image having a second dynamic range from the captured image; gain map generation means configured to generate a gain map for converting the dynamic range of the second image to the first dynamic range on the basis of the first image and the second image; noise reduction means configured to apply a first noise reduction process to the gain map; and association means configured to associate the gain map with the second image.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an imaging apparatus, an image processing method, and a program.

Background Art

[0002] As the display brightness of a display increases, a high dynamic range (HDR) camera system has been proposed that can obtain an image capable of reproducing the gradations on the high-brightness side, which have been compressed until now, with gradations closer to the visual appearance. HDR can represent a wider dynamic range than standard dynamic range (SDR).

[0003] Also, a technique is known in which a gain map for mutually converting an SDR image and an HDR image is generated based on the SDR image and the HDR image, and the SDR image or the HDR image is stored in a file together with the gain map as a baseline image. According to this technique, for example, when the baseline image is an HDR image, an SDR image can be generated by applying the gain map to the baseline image. Therefore, depending on whether the display for displaying the image supports HDR or not, it becomes possible to select and display an image suitable for the display from among the SDR image and the HDR image.

[0004] Also, Patent Document 1 discloses a technique for performing gradation conversion that emphasizes contrast in consideration of noise. According to Patent Document 1, noise suppression is performed on an input image, and a gain map composed of gain values corresponding to each pixel is generated from the input image after noise suppression and the generated low-frequency image. Then, gain processing is performed on the input image after noise suppression using the gain map.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] If noise is included in the gain map for converting between HDR images and SDR images, the image quality of the image obtained by applying the gain map to the baseline image may deteriorate.

[0007] Since Patent Document 1 does not mention the processing for the gain map for converting between HDR images and SDR images, it is impossible to address the problem regarding the noise included in the gain map for converting between HDR images and SDR images.

[0008] The present invention has been made in view of such a situation, and an object thereof is to provide a technique for suppressing noise in a gain map generated based on two images having different dynamic ranges (for example, an SDR image and an HDR image).

Means for Solving the Problems

[0009] To solve the above problems, the present invention provides an image processing apparatus comprising: first image generation means for generating a first image having a first dynamic range from a captured image; second image generation means for generating a second image having a second dynamic range from the captured image; gain map generation means for generating a gain map for converting the dynamic range of the second image to the first dynamic range based on the first image and the second image; noise reduction means for applying a first noise reduction process to the gain map; and association means for associating the gain map with the second image.

Effects of the Invention

[0010] According to the present invention, it is possible to suppress noise in a gain map generated based on two images with different dynamic ranges (for example, an SDR image and an HDR image).

[0011] In addition, other features and advantages of the present invention will become more apparent from the accompanying drawings and the description in the following embodiments for carrying out the invention.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0014] [First Embodiment] FIG. 1 is a block diagram showing a configuration example of an imaging device 100 to which an image processing apparatus according to the first embodiment is applied. The imaging device 100 includes an optical system 101, an imaging unit 102, an A / D conversion unit 103, an image processing unit 104, a display unit 105, a storage unit 106, an image recording medium 107, a system control unit 108, and an operation unit 109.

[0015] Based on an image (captured image) generated by shooting using the imaging unit 102, the imaging device 100 generates an SDR image and an HDR image in the image processing unit 104, and performs a process of generating a gain map capable of reconstructing HDR representation and SDR representation.

[0016] In this embodiment, the HDR image is an image having a wider dynamic range than the SDR image. For example, it is an image to which the Opto-Electronic Transfer Function (OETF) characteristics described in ST2084 or the like, which is an HDR standard handled by an HDR monitor, are applied. The SDR image is an image having a narrower dynamic range than the HDR image. In this embodiment, the gamma of the HDR image is the OETF characteristic of ST2084, the color gamut is Rec.2020, the gamma of the SDR image is the sRGB gamma, and the color gamut is sRGB, and the description will be made accordingly.

[0017] In FIG. 1, the optical system 101 includes a lens group including a zoom lens and a focus range, a diaphragm adjusting device, and a shutter device. The optical system 101 adjusts the magnification, the focus position, and the amount of light of the subject image that reaches the imaging unit 102. The imaging unit 102 includes an imaging element such as a CCD sensor or a CMOS sensor that converts the light beam of the subject image that has passed through the optical system 101 into an electrical signal (image signal) by photoelectric conversion. The A / D conversion unit 103 generates a digital image by applying analog-digital conversion to the image signal input from the imaging unit 102.

[0018] The image processing unit 104 performs development processing such as pixel interpolation processing, linear gamma conversion for generating a gain map, and processing for converting the color space on the image (captured image) output from the A / D conversion unit 103. Further, the image processing unit 104 performs a predetermined compression process and the like for recording an image on the image recording medium 107 described later. The image processing unit 104 can perform the same image processing not only on the image output from the A / D conversion unit 103 but also on the image read from the image recording medium 107.

[0019] The display unit 105 performs display of the viewfinder image at the time of shooting, display of the captured image, character display for interactive operation, and the like. The display unit 105 displays the image generated by the image processing unit 104 and the image read from the image recording medium 107. The display unit 105 is, for example, a liquid crystal display or an organic Electro Luminescence (EL) display.

[0020] The storage unit 106 stores an image processing program and various information necessary for image processing by the image processing unit 104.

[0021] The image recording medium 107 has a function of recording an image. The image recording medium may include, for example, a memory card equipped with a semiconductor memory, or a recording medium using a package containing a rotating recording body such as a magnetic disk. The image recording medium 107 may be configured to be detachable from the imaging device 100.

[0022] The system control unit 108 controls the entire imaging device 100. The system control unit 108 includes, for example, a CPU (or MPU), a ROM, and a RAM, etc. By expanding the program stored in the ROM to the working area of the RAM and executing it, various controls including the overall control of the imaging device 100 are performed.

[0023] The operation unit 109 is for receiving the user's operations. As the operation unit 109, for example, buttons, levers, touch panels, etc. can be used. The user can set the shooting mode according to the user's preference via the operation unit 109. The setting of the shooting mode includes the setting of the noise reduction intensity. In this specification, "noise reduction" may be denoted as "NR". Also, the setting of the NR intensity may be denoted as "NR setting". As the setting values of the NR intensity setting, there are setting values such as "OFF (no noise reduction)", "weak", "standard", "strong", etc. For example, when an image that emphasizes sharpness at the expense of the amount of noise is desired in landscape photo shooting, the user can obtain more effective image characteristics by changing the NR intensity setting to OFF.

[0024] In the example of FIG. 1, the optical system 101 is configured as a part of the imaging device 100 including the imaging unit 102, but the present embodiment is not limited to this configuration. For example, an interchangeable optical system (interchangeable lens) may be used in an imaging system configured to be detachable from the main body of the imaging device 100 like a single-lens reflex camera.

[0025] FIG. 2 is a block diagram showing a configuration example of the image processing unit 104 according to the first embodiment. The image processing unit 104 includes an SDR development processing unit 201, an HDR development processing unit 202, a linear gamma conversion unit 203, a color space conversion unit 204, a gain map generation unit 205, a gain map NR processing unit 206, a gain map encoding unit 207, and a file storage unit 208.

[0026] Here, it is assumed that the input image (captured image) input to the image processing unit 104 is a digital image generated by A / D converting the signal acquired by the imaging unit 102 using the A / D conversion unit 103. Further, it is assumed that this digital image is a Bayer image composed of three components of red (R), green (G), and blue (B).

[0027] The SDR development processing unit 201 performs a process (image generation process) of performing SDR development processing on the input image to generate an SDR image. The HDR development processing unit 202 performs a process (image generation process) of performing HDR development processing on the input image to generate an HDR image. The development processing refers to a process of generating a YUV image or an RGB image suitable for an output device (not shown) such as a display from the input Bayer image.

[0028] Here, with reference to FIG. 3, a configuration example of the SDR development processing unit 201 and the HDR development processing unit 202 will be described. As shown in FIG. 3, the configuration examples of the SDR development processing unit 201 and the HDR development processing unit 202 can be represented by the same block diagram. Note that FIG. 3 also serves as a configuration example of the baseline HDR development processing unit 801, which will be described in the second embodiment.

[0029] Each of the SDR development processing unit 201 and the HDR development processing unit 202 includes a white balance processing unit 301, an NR processing unit 302, a demosaic processing unit 303, a color matrix processing unit 304, and a gamma processing unit 305. The SDR development processing unit 201 and the HDR development processing unit 202 acquire a Bayer image as an input image and generate a YUV image or an RGB image suitable for an output device (not shown) such as a display as an output image.

[0030] The white balance processing unit 301 performs white balance processing for adjusting the color balance on the input image.

[0031] The NR processing unit 302 performs NR processing on the input image to reduce dark current noise and optical shot noise. As the NR processing, for example, a general method using a low-pass filter (LPF) or a bilateral filter may be applied. Also, the intensity of the NR processing (the intensity of noise reduction in the NR processing) is controlled by the NR setting set by the user via the operation unit 109. Therefore, the operation unit 109 has a function as a reception unit that receives the setting of the intensity of the NR processing (NR setting) from the user. For example, when the NR setting is set to OFF, the NR processing unit 302 passes the input image to the demosaicing processing unit 303 without applying the NR processing. Also, when the NR setting is set to weak, standard, or strong, the NR processing unit 302 can perform the NR processing at the set intensity by using the filter coefficient and the filter matrix according to the NR setting.

[0032] The demosaicing processing unit 303 performs demosaicing processing on the input image to generate a three-plane image of R, G, and B from the Bayer image. As the demosaicing processing, for example, general methods such as linear interpolation and adaptive interpolation can be applied.

[0033] The color matrix processing unit 304 performs color matrix processing on the input image to match the color gamut of the output device according to the spectral characteristics of the imaging device.

[0034] The gamma processing unit 305 performs gamma processing on the input image to convert the signal value with the gamma characteristic (OETF) to generate a signal matching the monitor gamma (OETF) of the output device.

[0035] As described above, the SDR development processing unit 201 and the HDR development processing unit 202 can be represented by the same block diagram. However, as parameters for various processes, different parameters are appropriately used according to the type of output image (SDR image or HDR image). For example, the gamma processing unit 305 of the SDR development processing unit 201 uses the sRGB gamma as the gamma characteristic to be applied to the input image, while the gamma processing unit 305 of the HDR development processing unit 202 uses the OETF characteristic of ST2084 as the gamma characteristic to be applied to the input image.

[0036] Referring again to FIG. 2, the linear gamma conversion unit 203 converts the gamma characteristics of the SDR image and the HDR image generated by the SDR development processing unit 201 and the HDR development processing unit 202 into linear gamma. As a method of linearization, for example, in the case of an SDR image, a method of converting a non-linear SDR signal into a linear signal using the SDR Electro-Optical Transfer Function (EOTF) function can be used. Specifically, the reference EOTF defined in ITU-R BT.709 can be used.

[0037] The color space conversion unit 204 converts the SDR image and the HDR image linearized by the linear gamma conversion unit 203 into a common color space. For example, when the color space is adjusted to Rec.2020, the color space conversion unit 204 performs a color space conversion from sRGB to Rec.2020 on the SDR image. As a method of color space conversion, for example, the method described in ITU-R BT.2087 can be used.

[0038] The gain map generation unit 205 generates a gain map based on the SDR image and the HDR image whose color spaces have been converted by the color space conversion unit 204. In the gain map generation process in the gain map generation unit 205, as shown in the following formula (1), the gain value is calculated by taking the logarithm of the ratio of SDR to HDR for each pixel. In formula (1), SDR, k SDR are the pixel value and the offset value of the SDR image, respectively, and HDR, k HDRare the pixel value and offset value of the HDR image, respectively, and G is the gain value. TIFF2025083921000002.tif1161

[0039] Here, the gain map generation unit 205 may generate a gain value for one-channel grayscale or may generate a gain value for each of the three-channel RGB planes. In the case of one-channel grayscale, the gain map generation unit 205 performs conversion from RGB to YUV and calculates the gain value from the Y value of YUV. Further, the gain map generation unit 205 stores, in the storage unit 106 as metadata, necessary information (for example, the number of channels of the gain value, etc.) for encoding by the gain map encoding unit 207 described later.

[0040] By applying a gain map to one of the SDR image and the HDR image, the other image can be generated. Therefore, if one of the SDR image and the HDR image and the gain map are stored in the image file, both the SDR image and the HDR image can be acquired from the image file. The image stored together with the gain map in the image file is called the baseline image or the main image.

[0041] According to Equation (1), the gain value based on the pixel value of the SDR image is calculated. Therefore, when the gain map calculated according to Equation (1) is stored in the image file, if the baseline image is the SDR image, the HDR image can be generated by multiplying the pixel value of the SDR image by the gain value of the gain map. On the other hand, when the gain map calculated according to Equation (1) is stored in the image file, if the baseline image is the HDR image, the SDR image can be generated by multiplying the pixel value of the HDR image by the reciprocal of the gain value of the gain map.

[0042] The gain map NR processing unit 206 performs NR processing on the gain map generated by the gain map generation unit 205 to reduce random noise included in the input image and noise generated by resolution reduction. As typical methods of NR processing for the gain map, there are two methods described below. The gain map NR processing unit 206 may use either one of the methods or a combination of the two methods. Also, there are various other methods of NR processing for the gain map, which will be described later. The gain map NR processing unit 206 can use any one of these two methods and various other methods, or a combination of any two or more of these multiple methods.

[0043] The first method of NR processing is a method based on the reduction rate of the gain map. In this method, the gain map NR processing unit 206 controls the application amount of noise reduction based on the gain map information 209 including the reduction rate of the resolution reduction of the gain map performed by the gain map encoding unit 207, which will be described later. The reduction rate of the gain map indicates the size after reduction when the original size (resolution) is set to 1. Since the original size of the gain map matches the sizes of the SDR image and the HDR image, the reduction rate of the gain map corresponds to the ratio of the size (resolution) of the gain map to the baseline image.

[0044] For example, the gain map NR processing unit 206 generates a gain map to which an LPF for reducing noise is applied, and controls the application amount of noise reduction by changing the synthesis ratio of the gain maps before and after the application of the LPF according to the resolution reduction ratio. For example, as shown in FIG. 6, the gain map NR processing unit 206 controls the application amount of noise reduction such that the smaller the reduction ratio, the weaker (smaller) the intensity of noise reduction, and the larger the reduction ratio, the stronger (larger) the intensity of noise reduction. Resolution reduction of the gain map has an effect of noise reduction, and the smaller the reduction ratio, the less noise there is. By controlling in this way, the noise remaining in the reduced gain map can be appropriately reduced. Further, the gain map NR processing unit 206 may apply an LPF for removing frequency components exceeding the Nyquist frequency in order to suppress aliasing (aliasing noise) generated by the resolution reduction of the gain map in the gain map encoding unit 207.

[0045] The second NR processing method is a method based on the NR setting of the shooting mode. In this method, the gain map NR processing unit 206 controls the intensity of noise reduction based on the user setting information 210 including the information on the NR setting of the shooting mode set by the user via the operation unit 109.

[0046] For example, when the NR setting is OFF, the NR processing unit 302 does not perform NR processing on the SDR image and the HDR image. Therefore, when the NR setting is OFF, it is considered that the noise in the gain map is larger compared to the case of other setting values (such as standard). Thus, the gain map NR processing unit 206 performs NR processing on the gain map using a filter coefficient and a filter matrix that enhance the intensity of noise reduction. Conversely, when the NR setting is strong, it is considered that the noise in the gain map is smaller compared to the case of other setting values (such as standard). Thus, the gain map NR processing unit 206 performs NR processing on the gain map using a filter coefficient and a filter matrix that weaken the intensity of noise reduction. In this way, the gain map NR processing unit 206 controls such that the smaller the intensity of the NR setting (the intensity of the NR processing applied to the SDR image and the HDR image), the larger the intensity of the NR processing on the gain map. FIG. 7 is a diagram showing an example of table data that defines the filter coefficient and the filter matrix corresponding to each setting value of the NR setting. The table data in FIG. 7 is stored in advance in the storage unit 106.

[0047] Also, as various other NR processing methods, there are a method based on the bit depth of the gain map included in the gain map information 209, a method based on the minimum value and the maximum value of the luminance value of the baseline image, a method based on the shooting sensitivity (ISO sensitivity) of the captured image, a method based on the shape of the gamma applied to the baseline image, and a method based on the variance in the flat part of the image.

[0048] In the method based on the bit depth, the gain map NR processing unit 206 controls to reduce the intensity of noise reduction as the bit depth of the gain map is smaller (as the number of bits of each gain value in the gain map is smaller). Since it becomes more difficult to distinguish between the signal and the noise as the bit depth is smaller, by controlling in this way, the noise in the gain map can be reduced with an appropriate intensity.

[0049] In the method based on the minimum and maximum luminance values of the baseline image, the gain map NR processing unit 206 performs conversion from RGB to YUV and obtains the maximum and minimum values of the Y value (luminance value) of YUV. The difference between the maximum and minimum luminance values is considered to be an index of the amount of noise amplified when applying the gain map to the baseline image. Therefore, the gain map NR processing unit 206 controls so that the intensity of noise reduction increases as the difference between the maximum and minimum luminance values increases.

[0050] In the method based on the shooting sensitivity (ISO sensitivity) of the captured image, the gain map NR processing unit 206 controls so that the intensity of noise reduction increases as the shooting sensitivity increases. When the shooting sensitivity is high, it is considered that the amount of noise in the SDR image and HDR image used for generating the gain map increases, and thus the amount of noise in the gain map also increases. Therefore, by controlling in this way, the noise in the gain map can be reduced with an appropriate intensity.

[0051] Regarding the method based on the shape of the gamma applied to the baseline image, depending on the shape of the gamma, there is a value range where the contrast becomes strong, and low-amplitude noise that was difficult to visually recognize before gamma application may be emphasized after gamma application. Therefore, when the slope of the input-output characteristics related to this value range is steep, the gain map NR processing unit 206 controls to increase the intensity of noise reduction, and conversely, when the slope of the input-output characteristics is gentle, it controls to decrease the intensity of noise reduction.

[0052] In the method based on the variance in the flat part of the image, the gain map NR processing unit 206 calculates the variance of the pixel values in the flat part for at least one of the SDR image or HDR image used for generating the gain map. The magnitude of the calculated variance is considered to be an index of the amount of noise in the gain map. Therefore, the gain map NR processing unit 206 controls so that the intensity of noise reduction increases as the variance increases.

[0053] The gain map encoding unit 207 encodes the gain map NR-processed by the gain map NR processing unit 206 into a format that conforms to the specifications of the output image file. Quantization is performed during encoding, but the bit depth of the gain map does not have to match the bit depth of the baseline image, and it may match or exceed the bit depth of the HDR image. For example, if the baseline image is an 8-bit SDR image and the HDR image obtained by combining the SDR image and the gain map is 10-bit, the bit depth of the gain map needs to be 10 bits or more.

[0054] Also, during encoding, in order to reduce the file size, a process of downsampling the gain map to a low resolution is performed. For example, the gain map encoding unit 207 reduces the resolution of the gain map corresponding to the resolution of the input image, such as by a factor of 1 / 4 (1 / 2 in the horizontal direction and 1 / 2 in the vertical direction). In addition, the gain map encoding unit 207 calculates the minimum and maximum values of each RGB plane of the gain map.

[0055] The file storage unit 208 stores the gain map encoded by the gain map encoding unit 207, the metadata stored in the storage unit 106 by the gain map generation unit 205, and the baseline image in an image file. By storing the gain map and the baseline image in the image file, the gain map is associated with the baseline image.

[0056] Figure 4 is a flowchart showing an operation example of the image processing unit 104 according to the first embodiment. Each process shown in this flowchart may be realized, for example, by a CPU or the like executing an image processing program according to this embodiment. Alternatively, part or all of the processes shown in this flowchart may be realized by hardware such as an electronic circuit. Here, the case where the baseline image is an HDR image will be described, but the baseline image of this embodiment is not limited to an HDR image and may be an SDR image.

[0057] In S401, the SDR development processing unit 201 and the HDR development processing unit 202 of the image processing unit 104 generate an SDR image and an HDR image from the input image (Bayer image) from the A / D conversion unit 103.

[0058] In S402, the linear gamma conversion unit 203 performs conversion processing to linear gamma on the SDR image and the HDR image generated in S401. As a linearization method, a method of converting a non-linear signal into a linear signal using an EOTF function as shown in FIGS. 5(a) and 5(b) can be used. Note that FIG. 5(a) shows an example of an EOTF function for an SDR image, and FIG. 5(b) shows an example of an EOTF function for an HDR image.

[0059] In S403, the color space conversion unit 204 converts the SDR image and the HDR image converted to linear gamma in S402 into a common color space. Here, it is assumed that color space conversion from sRGB to Rec. 2020 is performed on the SDR image.

[0060] In S404, the gain map generation unit 205 generates a gain map based on the SDR image and the HDR image whose color spaces have been converted in S403. The details of the generation method are as described above with reference to equation (1). Here, it is assumed that the gain map generation unit 205 generates a gain map for each of the three RGB planes. The gain map generation unit 205 stores, as metadata, necessary information (for example, information regarding the number of channels and the gain map information 209, etc.) for encoding by the gain map encoding unit 207 in the storage unit 106.

[0061] In S405, the gain map NR processing unit 206 performs NR processing on the gain map generated in S404 to reduce random noise included in the input image and noise generated by resolution reduction. As described above, as methods for NR processing of the gain map, there are a method based on the reduction ratio of the gain map, a method based on the NR setting of the shooting mode, and various other methods. The gain map NR processing unit 206 can use any of these methods, or a combination of any two or more of these methods.

[0062] In S406, the gain map encoding unit 207 encodes the gain map NR processed in S405 into a format that conforms to the specification of the output image file. Here, the resolution is reduced to 1 / 4 (1 / 2 in the horizontal direction and 1 / 2 in the vertical direction) with respect to the resolution of the input image. Also, the gain map encoding unit 207 calculates the minimum value and the maximum value of each plane (R, G, B) of the gain map.

[0063] In S407, the file storage unit 208 stores the HDR image generated in S401 and the gain map encoded in S406 in the output image file. Also, the file storage unit 208 stores the metadata stored in the storage unit 106 in S404 in the output image file. Here, the HDR image is stored in the file as the baseline image (main image), but when the baseline image is an SDR image, the file storage unit 208 stores the SDR image generated in S401 in the output image file instead of the HDR image.

[0064] As described above, according to the first embodiment, the imaging device 100 generates, from a captured image, a first image having a first dynamic range and a second image having a second dynamic range. For example, when using an HDR image as a baseline image, the first image is an SDR image and the second image is an HDR image. Also, when using an SDR image as a baseline image, the first image is an HDR image and the second image is an SDR image. The imaging device 100 generates a gain map for converting the dynamic range of the second image (baseline image) to the first dynamic range based on the first image and the second image. The imaging device 100 applies NR processing (first noise reduction processing) to the gain map and associates the gain map with the second image (baseline image).

[0065] Thus, according to this embodiment, the noise of the gain map generated based on two images with different dynamic ranges (for example, an SDR image and an HDR image) is suppressed. Therefore, according to this embodiment, it is possible to improve the image quality of the image generated by applying the gain map to the baseline image.

[0066] [Second Embodiment] Next, the second embodiment will be described. The basic configuration of the imaging device 100 according to the second embodiment is the same as that of the first embodiment. Hereinafter, mainly the differences from the first embodiment will be described.

[0067] FIG. 8 is a block diagram showing a configuration example of the image processing unit 104 according to the second embodiment. The image processing unit 104 includes an SDR development processing unit 201, an HDR development processing unit 202, a linear gamma conversion unit 203, a color space conversion unit 204, a gain map generation unit 205, a gain map encoding unit 207, a file storage unit 208, and a baseline HDR development processing unit 801.

[0068] The baseline HDR development processing unit 801 generates an HDR image as a baseline image by performing HDR development processing on the input image.

[0069] Here, with reference to FIG. 3, a configuration example of the SDR development processing unit 201, the HDR development processing unit 202, and the baseline HDR development processing unit 801 will be described.

[0070] The configuration examples of the SDR development processing unit 201 and the HDR development processing unit 202 can be represented by the block diagram of FIG. 3, similar to the first embodiment. However, the method of controlling the intensity of noise reduction in the NR processing unit 302 of the SDR development processing unit 201 and the HDR development processing unit 202 is different from that of the first embodiment. In the first embodiment, the NR processing was performed with an intensity according to the NR setting of the shooting mode. On the other hand, in the second embodiment, the NR processing unit 302 of the SDR development processing unit 201 and the HDR development processing unit 202 performs the NR processing with an intensity suitable for reducing the noise of the gain map generated later, regardless of the NR setting. For example, the NR processing unit 302 of the SDR development processing unit 201 and the HDR development processing unit 202 may control the intensity of noise reduction based on at least one of the shooting sensitivity (ISO sensitivity) of the captured image and the variance of the pixel values in the flat part of the captured image. As can be understood from FIG. 8, in the second embodiment, the SDR image and the HDR image input to the gain map generation unit 205 have been subjected to the NR processing with an intensity suitable for reducing the noise of the gain map in the NR processing unit 302 of the SDR development processing unit 201 and the HDR development processing unit 202. Therefore, unlike the first embodiment, it is not necessary to perform the NR processing on the gain map after the generation of the gain map, and the image processing unit 104 in FIG. 8 does not need to include the gain map NR processing unit 206.

[0071] The configuration example of the baseline HDR development processing unit 801 can be represented by the block diagram of FIG. 3, similar to the SDR development processing unit 201 and the HDR development processing unit 202. However, the method for controlling the intensity of noise reduction in the NR processing unit 302 of the baseline HDR development processing unit 801 is different from that of the SDR development processing unit 201 and the HDR development processing unit 202 according to the second embodiment. The NR processing unit 302 of the baseline HDR development processing unit 801 performs NR processing with an intensity according to the NR setting received from the user, similar to the SDR development processing unit 201 and the HDR development processing unit 202 according to the first embodiment. The operation unit 109 has a function as a reception unit that receives the setting of the intensity of the NR processing (NR setting) from the user.

[0072] FIG. 9 is a flowchart showing an operation example of the image processing unit 104 according to the second embodiment. Each process shown in this flowchart may be realized, for example, when a CPU or the like executes the image processing program according to this embodiment. Alternatively, part or all of the processes shown in this flowchart may be realized by hardware such as an electronic circuit.

[0073] In S901, the SDR development processing unit 201 and the HDR development processing unit 202 of the image processing unit 104 generate an SDR image and an HDR image used for generating a gain map from the input image (Bayer image) from the A / D conversion unit 103.

[0074] In S902, the baseline HDR development processing unit 801 of the image processing unit 104 generates an HDR image used as a baseline image from the input image (Bayer image) from the A / D conversion unit 103.

[0075] The processes of S903 to S905 are the same as the processes of S402 to S404 in FIG. 4.

[0076] In S906, the gain map encoding unit 207 encodes the gain map generated in S905 into a format that conforms to the specifications of the output image file. Here, the resolution is reduced to 1 / 4 (1 / 2 in the horizontal direction and 1 / 2 in the vertical direction) with respect to the resolution of the input image. Also, the gain map encoding unit 207 calculates the minimum value and the maximum value of each plane (R, G, B) of the gain map.

[0077] In S907, the file storage unit 208 stores the HDR image as the baseline image generated in S902 and the gain map encoded in S906 in the output image file. Also, the file storage unit 208 stores the metadata stored in the storage unit 106 in S905 in the output image file.

[0078] Note that in the above description, the baseline image is assumed to be an HDR image, but the baseline image may be an SDR image. In this case, the baseline HDR development processing unit 801 is configured to generate an SDR image as the baseline image. Also, the file storage unit 208 stores the SDR image generated as the baseline image in S902 in the output image file.

[0079] As described above, according to the second embodiment, the imaging device 100 generates a first image having a first dynamic range and to which noise reduction processing (first noise reduction processing) is applied from a captured image. Further, the imaging device 100 generates a second image having a second dynamic range and to which noise reduction processing (first noise reduction processing) is applied from the captured image. Furthermore, the imaging device 100 generates a third image having a second dynamic range and to which second noise reduction processing is applied with an intensity independent of the first noise reduction processing from the captured image. The third image is an image used as a baseline image. When the baseline image is an HDR image, the first image is an SDR image, and the second image is an HDR image as a non-baseline image. When the baseline image is an SDR image, the first image is an HDR image, and the second image is an SDR image as a non-baseline image. The imaging device 100 generates a gain map for converting the dynamic range of the third image (baseline image) to the first dynamic range based on the first image and the second image. Then, the imaging device 100 associates the gain map with the third image (baseline image).

[0080] Thus, according to this embodiment, a gain map is generated based on the first image and the second image to which noise reduction processing is applied with an intensity independent of the noise reduction processing for the baseline image. Therefore, according to this embodiment, it is possible to suppress the noise of the gain map regardless of the intensity of the noise reduction processing for the baseline image, and it is possible to improve the image quality of the image generated by applying the gain map to the baseline image.

[0081] [Other Embodiments] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and causing one or more processors in a computer of the system or device to read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0082] [Summary] The above-described embodiments disclose the inventions shown in at least the following items, but are not limited to these inventions. [Item 1] First image generation means for generating a first image having a first dynamic range from a captured image; Second image generation means for generating a second image having a second dynamic range from the captured image; Gain map generation means for generating a gain map for converting the dynamic range of the second image to the first dynamic range based on the first image and the second image; Noise reduction means for applying a first noise reduction process to the gain map; Association means for associating the gain map with the second image; An image processing apparatus comprising the same. [Item 2] The resolution of the gain map is smaller than the resolution of the second image, The noise reduction means controls the intensity of the first noise reduction process based on the ratio of the resolution of the gain map to the resolution of the second image. The image processing apparatus according to Item 1, characterized in that. [Item 3] The noise reduction means controls so that the intensity of the first noise reduction process increases as the ratio increases. The image processing apparatus according to Item 2, characterized in that. [Item 4] The first image generation means generates the first image to which a second noise reduction process is applied, The second image generation means generates the second image to which the second noise reduction process is applied, The noise reduction means controls the intensity of the first noise reduction process based on the intensity of the second noise reduction process. The image processing apparatus according to any one of Items 1 to 3, characterized in that. [Item 5] The noise reduction means controls such that the intensity of the first noise reduction process increases as the intensity of the second noise reduction process decreases. The image processing apparatus according to item 4, characterized in that. [Item 6] The image processing apparatus further comprises reception means for receiving a setting of the intensity of the second noise reduction process from a user. The image processing apparatus according to item 4 or 5, characterized in that. [Item 7] The noise reduction means controls the intensity of the first noise reduction process based on at least any one of the bit depth of the gain map, the difference between the maximum value and the minimum value of the luminance values of the second image, the shooting sensitivity of the captured image, the shape of the gamma applied to the second image, and the variance of the pixel values in the flat part of the first image or the second image. The image processing apparatus according to any one of items 1 to 6, characterized in that. [Item 8] First image generation means for generating a first image having a first dynamic range and to which a first noise reduction process is applied from a captured image; Second image generation means for generating a second image having a second dynamic range and to which the first noise reduction process is applied from the captured image; Third image generation means for generating a third image having the second dynamic range and to which a second noise reduction process is applied with an intensity independent of the first noise reduction process from the captured image; Gain map generation means for generating a gain map for converting the dynamic range of the third image to the first dynamic range based on the first image and the second image; Association means for associating the gain map with the third image; An image processing apparatus comprising the above. [Item 9] The image processing apparatus further comprises first control means for controlling the intensity of the first noise reduction process based on at least one of the ISO sensitivity of the captured image and the variance of the pixel values in the flat part of the captured image. The image processing apparatus according to item 8, characterized in that... [Item 10] The image processing apparatus further comprises receiving means for receiving from the user a setting of the intensity of the second noise reduction process. The image processing apparatus according to item 8 or 9, characterized in that... [Item 11] An image processing apparatus according to any one of items 1 to 10, Imaging means for generating the captured image, An imaging apparatus characterized by comprising the same. [Item 12] An image processing method executed by an image processing apparatus, A first image generation step of generating a first image having a first dynamic range from a captured image, A second image generation step of generating a second image having a second dynamic range from the captured image, A gain map generation step of generating a gain map for converting the dynamic range of the second image to the first dynamic range based on the first image and the second image, A noise reduction step of applying a first noise reduction process to the gain map, An association step of associating the gain map with the second image, An image processing method characterized by comprising the same. [Item 13] An image processing method executed by an image processing apparatus, A first image generation step of generating a first image having a first dynamic range and to which a first noise reduction process is applied from a captured image, A second image generation step of generating a second image having a second dynamic range and to which the first noise reduction process is applied from the captured image, A third image generation step of generating a third image having the second dynamic range and to which a second noise reduction process is applied at an intensity independent of the first noise reduction process from the captured image, A gain map generation step of generating a gain map for converting the dynamic range of the third image to the first dynamic range based on the first image and the second image; An association step of associating the gain map with the third image; An image processing method characterized by comprising the above. [Item 14] A program for causing a computer to function as each means of the image processing apparatus according to any one of Items 1 to 10.

[0083] The invention is not limited to the above embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, claims are attached to disclose the scope of the invention.

Explanation of Signs

[0084] 100... Imaging device, 101... Optical system, 102... Imaging unit, 103... A / D conversion unit, 104... Image processing unit, 105... Display unit, 106... Storage unit, 107... Image recording medium, 108... System control unit, 109... Operation unit

Claims

1. First image generation means for generating a first image having a first dynamic range from a captured image; Second image generation means for generating a second image having a second dynamic range from the captured image; Gain map generation means for generating a gain map for converting the dynamic range of the second image to the first dynamic range based on the first image and the second image; Noise reduction means for applying a first noise reduction process to the gain map; Association means for associating the gain map with the second image; An image processing apparatus comprising the same.

2. The resolution of the gain map is smaller than the resolution of the second image, The noise reduction means controls the intensity of the first noise reduction process based on the ratio of the resolution of the gain map to the resolution of the second image. The image processing apparatus according to claim 1, characterized in that.

3. The noise reduction means controls so that the intensity of the first noise reduction process increases as the ratio increases. The image processing apparatus according to claim 2, characterized in that.

4. The first image generation means generates the first image to which a second noise reduction process is applied, The second image generation means generates the second image to which the second noise reduction process is applied, The noise reduction means controls the intensity of the first noise reduction process based on the intensity of the second noise reduction process. The image processing apparatus according to claim 1, characterized in that.

5. The noise reduction means controls so that the intensity of the first noise reduction process increases as the intensity of the second noise reduction process decreases. The image processing apparatus according to claim 4, characterized in that.

6. Further comprising reception means for receiving from the user a setting of the intensity of the second noise reduction process. The image processing apparatus according to claim 4, characterized in that.

7. The noise reduction means is based on at least any one of the bit depth of the gain map, the difference between the maximum value and the minimum value of the luminance values of the second image, the shooting sensitivity of the captured image, the shape of the gamma applied to the second image, and the variance of the pixel values in the flat portion of the first image or the second image. Controls the intensity of the first noise reduction process. The image processing apparatus according to claim 1, characterized in that.

8. First image generation means for generating a first image having a first dynamic range and to which first noise reduction processing is applied from a captured image; Second image generation means for generating a second image having a second dynamic range and to which the first noise reduction processing is applied from the captured image; Third image generation means for generating a third image having the second dynamic range and to which second noise reduction processing is applied with an intensity independent of the first noise reduction processing from the captured image; Gain map generation means for generating a gain map for converting the dynamic range of the third image to the first dynamic range based on the first image and the second image; Association means for associating the gain map with the third image; An image processing apparatus comprising the above.

9. The image processing apparatus according to claim 8, further comprising first control means for controlling the intensity of the first noise reduction processing based on at least one of the ISO sensitivity of the captured image and the variance of the pixel values in the flat part of the captured image. The image processing apparatus according to claim 8, characterized by the above.

10. The image processing apparatus according to claim 8, further comprising reception means for receiving a setting of the intensity of the second noise reduction processing from a user. The image processing apparatus according to claim 8, characterized by the above.

11. An imaging apparatus comprising the image processing apparatus according to any one of claims 1 to 10, and imaging means for generating the captured image. Imaging means for generating the captured image; An imaging apparatus comprising the above.

12. An image processing method executed by an image processing apparatus, comprising: A first image generation step of generating a first image having a first dynamic range from a captured image; A second image generation step of generating a second image having a second dynamic range from the captured image; A gain map generation step of generating a gain map for converting the dynamic range of the second image to the first dynamic range based on the first image and the second image; A noise reduction step of applying first noise reduction processing to the gain map; An association step of associating the gain map with the second image. An image processing method comprising the above.

13. An image processing method executed by an image processing apparatus, comprising: A first image generation step of generating a first image having a first dynamic range and to which first noise reduction processing is applied from a captured image; A second image generation step of generating a second image having a second dynamic range from the captured image and to which the first noise reduction process is applied; A third image generation step of generating a third image having the second dynamic range from the captured image and to which a second noise reduction process is applied with an intensity independent of the first noise reduction process; A gain map generation step of generating a gain map for converting the dynamic range of the third image to the first dynamic range based on the first image and the second image; An association step of associating the gain map with the third image; An image processing method characterized by comprising:

14. A program for causing a computer to function as each means of the image processing apparatus according to any one of Claims 1 to 10.

Images