Image processing apparatus, imaging apparatus, image processing method, storage medium, and computer program product

Abstract

The invention provides an image processing apparatus, an imaging apparatus, an image processing method, a storage medium, and a computer program product. The first image generation unit generates a first image having a first dynamic range from a captured image. The second image generation unit generates a second image having a second dynamic range from the captured image. A gain map generation unit generates a gain map for converting a dynamic range of the second image into a first dynamic range based on the first image and the second image. The noise reduction unit applies a first noise reduction process to the gain map. The gain map is associated with the second image.

Description

Technical Field

[0001] The invention relates to an image processing device, an image pickup device, an image processing method, a storage medium and a computer program product. Background Art

[0002] As the display brightness of displays increases, a high dynamic range (HDR) camera system has been proposed to obtain an image capable of reproducing the tones of the high brightness side that have been traditionally compressed as tones more similar to the appearance. HDR can express a wider dynamic range than the standard dynamic range (SDR).

[0003] In addition, a technique is known in which a gain map is generated for mutual conversion between an SDR image and an HDR image based on an SDR image and an HDR image, and the SDR image or the HDR image is used as a baseline image and the gain map is used as a base line image. Figure 1 According to this technology, for example, in the case where the baseline image is an HDR image, an SDR image can be generated by applying the gain map to the baseline image. This makes it possible to select and display an image suitable for the display, of the SDR image and the HDR image, depending on whether the display used to display the image supports HDR.

[0004] In addition, Japanese Patent Laid-Open No. 2018-128764 discloses a technique for performing tone conversion for emphasizing contrast while taking noise into consideration. According to Japanese Patent Laid-Open No. 2018-128764, noise suppression is performed on an input image, and a gain map consisting of gain values ​​corresponding to each pixel is generated from the input image after noise suppression and the low-frequency image that has been generated. Then, using the gain map, gain processing is performed on the input image after noise suppression.

[0005] If a gain map used for mutual conversion between an HDR image and an SDR image includes noise, the image quality of an image obtained by applying the gain map to a baseline image is likely to be degraded.

[0006] Since Japanese Patent Laid-Open No. 2018-128764 does not mention processing for a gain map for mutual conversion between HDR images and SDR images, it cannot solve problems related to noise included in the gain map for mutual conversion between HDR images and SDR images. Summary of the invention

[0007] The present invention has been made in view of the above-described circumstances, and provides a technique for suppressing noise in a gain map generated based on two images having different dynamic ranges (eg, an SDR image and an HDR image).

[0008] According to a first aspect of the present invention, there is provided an image processing device, comprising: a first image generating unit, configured to generate a first image having a first dynamic range from a captured image; a second image generating unit, configured to generate a second image having a second dynamic range from the captured image; a gain map generating unit, configured to generate a gain map for converting the dynamic range of the second image into the first dynamic range based on the first image and the second image; a noise reduction unit, configured to apply a first noise reduction process to the gain map; and an association unit, configured to associate the gain map with the second image.

[0009] According to a second aspect of the present invention, there is provided an image processing device, comprising: a first image generating unit, configured to generate a first image having a first dynamic range and to which a first noise reduction process is applied from a captured image; a second image generating unit, configured to generate a second image having a second dynamic range and to which the first noise reduction process is applied from the captured image; a third image generating unit, configured to generate a third image having the second dynamic range and to which the second noise reduction process is applied at an intensity independent of the first noise reduction process from the captured image; a gain map generating unit, configured to generate a gain map for converting the dynamic range of the third image into the first dynamic range based on the first image and the second image; and an associating unit, configured to associate the gain map with the third image.

[0010] According to a third aspect of the present invention, there is provided an image pickup apparatus including: the image processing apparatus according to the first aspect; and a pickup unit configured to generate the pickup image.

[0011] According to a fourth aspect of the present invention, there is provided an image processing method, which is performed by an image processing device, and the image processing method includes: generating a first image having a first dynamic range from a captured image; generating a second image having a second dynamic range from the captured image; generating a gain map for converting the dynamic range of the second image into the first dynamic range based on the first image and the second image; applying a first noise reduction process to the gain map; and associating the gain map with the second image.

[0012] According to a fifth aspect of the present invention, there is provided an image processing method, which is performed by an image processing device, and the image processing method includes: generating a first image with a first dynamic range and a first noise reduction process applied from a captured image; generating a second image with a second dynamic range and a first noise reduction process applied from the captured image; generating a third image with the second dynamic range and a second noise reduction process applied at an intensity independent of the first noise reduction process from the captured image; generating a gain map for converting the dynamic range of the third image into the first dynamic range based on the first image and the second image; and associating the gain map with the third image.

[0013] According to a sixth aspect of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein when the computer program is executed by a processor, the image processing method according to the fourth aspect or the fifth aspect is performed.

[0014] According to a seventh aspect of the present invention, a computer program product is provided, which includes a computer program, wherein when the computer program is executed by a processor, the image processing method according to the fourth aspect or the fifth aspect is performed.

[0015] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a block diagram showing an exemplary configuration of an image pickup apparatus 100 to which the image processing apparatus is applied.

[0017] Figure 2 : is a block diagram showing an exemplary configuration of the image processing unit 104 according to the first embodiment.

[0018] Figure 3 is a block diagram showing an exemplary configuration of the SDR development processing unit 201 , the HDR development processing unit 202 , and the baseline HDR development processing unit 801 .

[0019] Figure 4 is a flowchart illustrating an exemplary operation of the image processing unit 104 according to the first embodiment.

[0020] Figure 5A and Figure 5B is a diagram showing an example of the EOTF used in the linear gamma conversion unit 203 .

[0021] Figure 6 is a diagram showing an example of the relationship between the reduction rate of the gain map and the amount of noise reduction application.

[0022] Figure 7: is a diagram showing an example of table data defining filter coefficients and a filter matrix corresponding to respective setting values ​​of NR settings.

[0023] Figure 8 is a block diagram showing an exemplary configuration of the image processing unit 104 according to the second embodiment.

[0024] Fig. 9 is a flowchart illustrating an exemplary operation of the image processing unit 104 according to the second embodiment. DETAILED DESCRIPTION

[0025] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. A plurality of features are described in the embodiments, but are not limited to inventions requiring all such features, and a plurality of such features may be appropriately combined. In addition, in the accompanying drawings, the same reference numerals are given to the same or similar configurations, and redundant descriptions thereof are omitted.

[0026] [First embodiment]

[0027] Figure 1 1 is a block diagram showing an exemplary configuration of an image pickup apparatus 100 to which the image processing apparatus according to the first embodiment is applied. The image pickup apparatus 100 includes an optical system 101, an image pickup unit 102, an A / D conversion unit 103, an image processing unit 104, a display unit 105, a storage unit 106, an image recording medium 107, a system control unit 108, and an operation unit 109.

[0028] The imaging apparatus 100 performs processing to generate an SDR image and an HDR image based on an image (captured image) generated by capturing using the imaging unit 102 , using the image processing unit 104 , and generates a gain map for enabling reconstruction of HDR representation and SDR representation.

[0029] Note that in the present embodiment, an HDR image is an image having a wider dynamic range than an SDR image, for example, an image incorporating an optoelectronic transfer function (OETF) described in ST 2084, for example, which is an HDR standard handled on an HDR monitor. In addition, an SDR image is an image having a narrower dynamic range than an HDR image. The present embodiment will be described under the assumption that the gamma and color gamut of the HDR image are the OETF characteristics of ST 2084 and Rec. 2020, respectively, and the gamma and color gamut of the SDR image are sRGB gamma and sRGB, respectively.

[0030] exist Figure 1, the optical system 101 includes a lens assembly including a zoom lens and a focus lens, an aperture adjustment device, and a shutter device. The optical system 101 adjusts the magnification, focus position, and light amount of the subject image reaching the imaging unit 102. The imaging unit 102 includes an image sensor such as a CCD sensor and a CMOS sensor for converting a light beam of the subject image that has passed through the optical system 101 into an electrical signal (image signal) by photoelectric conversion. The A / D conversion unit 103 generates a digital image by applying analog-to-digital conversion to the image signal input from the imaging unit 102.

[0031] The image processing unit 104 performs development processing such as pixel interpolation processing, linear gamma conversion for generating a gain map, and processing for converting a color space, etc., for the image (captured image) output from the A / D conversion unit 103. In addition, the image processing unit 104 performs, for example, predetermined compression processing for recording the image in the image recording medium 107 to be described later. The image processing unit 104 can perform similar image processing not only for the image output from the A / D conversion unit 103 but also for the image that has been read out from the image recording medium 107.

[0032] The display unit 105 displays a viewfinder image at the time of shooting, displays a shot image, displays characters for interactive operation, etc. The display unit 105 displays an image generated by the image processing unit 104 and an image that has been read from the image recording medium 107. The display unit 105 is, for example, a liquid crystal display or an organic electroluminescent (EL) display.

[0033] The storage unit 106 stores, for example, an image processing program and various types of information necessary for image processing performed by the image processing unit 104 .

[0034] The image recording medium 107 includes a function of recording an image. For example, the image recording medium may include a memory card equipped with a semiconductor memory, or may include a recording medium using, for example, a package containing a rotating recording member such as a magnetic disk, etc. The image recording medium 107 may be configured to be attachable to and removable from the imaging apparatus 100.

[0035] The system control unit 108 controls the entire imaging apparatus 100. The system control unit 108 includes, for example, a CPU (or MPU), a ROM, and a RAM, and performs various types of control including control of the entire imaging apparatus 100 by deploying a program stored in the ROM to a work area of ​​the RAM and executing the program.

[0036] The operation unit 109 is intended to accept user operations. For example, a button, a lever, a touch panel, or the like may be used as the operation unit 109. The user can configure the settings of the shooting mode suitable for the user's preferences via the operation unit 109. The settings of the shooting mode include the settings of the noise reduction intensity. In this specification, "noise reduction" may be referred to as "NR". In addition, the setting of the NR intensity may be referred to as the "NR setting". The setting values ​​of the NR intensity setting include setting values ​​such as "OFF (no noise reduction)", "Low", "Normal", and "High". For example, in the case where the user wants to focus on the sharpness of the image at the expense of the amount of noise in shooting a landscape photo, the user can obtain more effective image characteristics by changing the NR intensity setting to OFF.

[0037] Note that although Figure 1 In the example of FIG. 1 , the optical system 101 is configured as a part of the imaging device 100 including the imaging unit 102, but the present embodiment is not limited to this configuration. For example, it is permissible to use an imaging system configured so that an interchangeable optical system (interchangeable lens) can be attached to and removed from the main body of the imaging device 100, such as a single-lens reflex camera or the like.

[0038] Figure 2 1 is a block diagram showing an exemplary configuration of the image processing unit 104 according to the first embodiment. The image processing unit 104 includes an SDR development processing unit 201, an HDR development processing unit 202, a linear gamma conversion unit 203, a color space conversion unit 204, a gain map generation unit 205, a gain map NR processing unit 206, a gain map encoding unit 207, and a file storage unit 208.

[0039] It is assumed here that the input image (captured image) input to the image processing unit 104 is a digital image generated by causing the A / D conversion unit 103 to apply A / D conversion to the signal obtained by the imaging unit 102. It is also assumed that the digital image is a Bayer image composed of three components, namely, red (R), green (G), and blue (B).

[0040] The SDR development processing unit 201 performs processing for generating an SDR image (image generation processing) by applying the SDR development processing to the input image. The HDR development processing unit 202 performs processing for generating an HDR image (image generation processing) by applying the HDR development processing to the input image. The development processing refers to processing for generating a YUV image or an RGB image suitable for an output device (not shown) such as a display from an input Bayer image.

[0041] refer to Figure 3 Now, a description is given of an exemplary configuration of the SDR image processing unit 201 and the HDR image processing unit 202. Figure 3As shown in FIG. 1 , the exemplary configurations of the SDR image processing unit 201 and the HDR image processing unit 202 can be represented by the same block diagram. Figure 3 An exemplary configuration of the baseline HDR development processing unit 801 is also depicted, but a description thereof will be provided in the second embodiment.

[0042] The SDR development processing unit 201 and the HDR development processing unit 202 each include a white balance processing unit 301, an NR processing unit 302, a demosaic processing unit 303, a color matrix processing unit 304, and a gamma processing unit 305. The SDR development processing unit 201 and the HDR development processing unit 202 obtain a Bayer image as an input image, and generate a YUV image or an RGB image suitable for an output device (not shown) such as a display as an output image.

[0043] The white balance processing unit 301 performs white balance processing for adjusting the color balance on an input image.

[0044] The NR processing unit 302 performs NR processing for reducing dark current noise and photon shot noise for the input image. As the NR processing, it is sufficient to apply a general method using a low-pass filter (LPF) or a bilateral filter, etc. In addition, the intensity of the NR processing (the noise reduction intensity in the NR processing) is controlled according to the NR setting configured by the user via the operation unit 109. Therefore, the operation unit 109 has a function as an acceptance unit for accepting the intensity setting (NR setting) of the NR processing from the user. For example, in the case where the NR setting has been set to OFF, the NR processing unit 302 passes the input image to the demosaic processing unit 303 without applying the NR processing. In addition, in the case where the NR setting has been set to low, normal, or high, the NR processing unit 302 can perform NR processing with the set intensity by using the filter coefficients and filter matrices corresponding to the NR setting.

[0045] For the input image, the demosaic processing unit 303 performs a demosaic process for generating, from the Bayer image, a three-plane image of R, G, and B. For example, as the demosaic process, a general method such as linear interpolation and adaptive interpolation can be applied.

[0046] The color matrix processing unit 304 performs color matrix processing for making the input image deviate from the spectral characteristics of the image sensor and conform to the color gamut of the output device.

[0047] For an input image, the gamma processing unit 305 performs gamma processing for converting a signal value according to a gamma characteristic (OETF) for generating a signal conforming to a monitor gamma (OETF) of an output device.

[0048] Note that, as described above, the SDR development processing unit 201 and the HDR development processing unit 202 may be represented as the same block diagram. However, depending on the type of output image (SDR image or HDR image), different parameters are appropriately used as parameters for various types of processing. For example, the gamma processing unit 305 of the SDR development processing unit 201 uses sRGB gamma as the gamma characteristic to be applied to the input image, while the gamma processing unit 305 of the HDR development processing unit 202 uses the OETF characteristic of ST 2084 as the gamma characteristic to be applied to the input image.

[0049] Reference again Figure 2 , the linear gamma conversion unit 203 converts the gamma characteristics of the SDR image and the HDR image generated by the SDR development processing unit 201 and the HDR development processing unit 202 into linear gamma. For example, in the case of an SDR image, a method of converting a nonlinear SDR signal into a linear signal using an electro-optical transfer function (EOTF) of the SDR can be used as a linearization method. Specifically, a reference EOTF defined by ITU-R BT.709 can be used.

[0050] The color space conversion unit 204 converts the SDR image and the HDR image, which have been linearized by the linear gamma conversion unit 203, into the same color space. For example, in order to make the color space conform to Rec.2020, the color space conversion unit 204 performs color space conversion from sRGB to Rec.2020 for the SDR image. For example, the method described in ITU-R BT.2087 can be used as a method of color space conversion.

[0051] The gain map generation unit 205 generates a gain map based on the SDR image and the HDR image to which the color space conversion is applied by the color space conversion unit 204. In the process for generating the gain map in the gain map generation unit 205, the gain value is calculated by obtaining the logarithm of the ratio between the SDR and HDR of each pixel as indicated by the following equation (1). In equation (1), SDR and k SDR are the pixel values ​​and offset values ​​of the SDR image, HDR and k HDR are the pixel value and offset value of the HDR image respectively, and G is the gain value.

[0052]

[0053] Here, the gain map generation unit 205 may generate a gain value for the grayscale of one channel, or may generate a gain value for each of the RGB planes of the three channels. In the case of the grayscale of one channel, the gain map generation unit 205 performs conversion from RGB to YUV, and calculates the gain value from the Y value of YUV. In addition, the gain map generation unit 205 stores information required for performing encoding in the gain map encoding unit 207 (e.g., the number of channels of the gain value, etc.) as metadata in the storage unit 106.

[0054] Applying the gain map to one of the SDR image and the HDR image enables the other image to be generated. Therefore, when one of the SDR image and the HDR image and the gain map are stored in an image file, both the SDR image and the HDR image can be obtained from the image file. Figure 1 The image stored in the image file will be referred to as the baseline image or master image.

[0055] According to formula (1), a gain value based on the pixel value of the SDR image is calculated. Therefore, in the case where the gain map calculated according to formula (1) is stored in the image file, if the baseline image is an SDR image, an HDR image can be generated by multiplying the pixel value of the SDR image by the gain value of the gain map. On the other hand, in the case where the gain map calculated according to formula (1) is stored in the image file, if the baseline image is an HDR image, an SDR image can be generated by multiplying the pixel value of the HDR image by the inverse of the gain value of the gain map.

[0056] The gain map NR processing unit 206 performs NR processing on the gain map generated by the gain map generating unit 205 to reduce random noise included in the input image and noise generated by resolution reduction. The two methods described below are typical methods for NR processing of the gain map. The gain map NR processing unit 206 can use one of the methods, or can use these two methods in combination. In addition, the method for NR processing of the gain map also includes various other methods to be described later. The gain map NR processing unit 206 can use one of the above two methods and various other methods, or a combination of two or more of these multiple methods.

[0057] The first method of NR processing is a method based on the reduction rate of the gain map. According to this method, the gain map NR processing unit 206 controls the amount of noise reduction applied based on the gain map information 209, which includes the reduction rate of the resolution reduction of the gain map performed by the gain map encoding unit 207 to be described later. The reduction rate of the gain map represents the reduced size assuming that the original size (resolution) is 1. Since the original size of the gain map is consistent with the size of the SDR image and the HDR image, the reduction rate of the gain map is equal to the ratio of the size (resolution) of the gain map to the size (resolution) of the baseline image.

[0058] For example, the gain map NR processing unit 206 controls the amount of noise reduction applied by generating a gain map to which the noise reduction LPF is applied and changing the synthesis ratio of the gain maps before and after the LPF application according to the resolution reduction rate. Figure 6 As shown, the gain map NR processing unit 206 controls the amount of noise reduction applied to reduce (reduce) the intensity of the noise reduction for a smaller reduction ratio and to enhance (increase) the intensity of the noise reduction for a larger reduction ratio. Reducing the resolution of the gain map has a favorable effect on noise reduction, and the noise decreases as the reduction ratio decreases; therefore, by controlling in the aforementioned manner, the noise remaining in the reduced gain map can be appropriately reduced. In addition, since the gain map NR processing unit 206 suppresses aliasing (folded noise) generated by the resolution reduction of the gain map in the gain map encoding unit 207, an LPF for removing frequency components exceeding the Nyquist frequency can be applied.

[0059] The second method of NR processing is a method based on the NR setting of the shooting mode. According to this method, the gain map NR processing unit 206 controls the intensity of noise reduction based on the user setting information 210 including information of the NR setting of the shooting mode configured by the user via the operation unit 109.

[0060] For example, when the NR setting is OFF, the NR processing unit 302 does not perform NR processing for the SDR image and the HDR image. Therefore, when the NR setting is OFF, the gain map is considered to include a large amount of noise compared to other setting values ​​(e.g., normal). In view of this, the gain map NR processing unit 206 performs NR processing on the gain map using filter coefficients and filter matrices that enhance the noise reduction intensity. On the contrary, when the NR setting is high, the gain map is considered to include a small amount of noise compared to other setting values ​​(e.g., normal). In view of this, the gain map NR processing unit 206 performs NR processing on the gain map using filter coefficients and filter matrices for reducing the noise reduction intensity. In this way, the gain map NR processing unit 206 controls so that the lower the intensity of the NR setting (the intensity of the NR processing applied to the SDR image and the HDR image), the higher the intensity of the NR processing of the gain map. Figure 7 : is a diagram showing an example of table data for defining filter coefficients and a filter matrix corresponding to respective setting values ​​of NR settings. Figure 7 The table data is stored in advance in the storage unit 106.

[0061] In addition, various other methods of NR processing include a method based on the bit depth of the gain map included in the gain map information 209, a method based on the minimum and maximum values ​​of the brightness value of the baseline image, a method based on the shooting sensitivity (ISO sensitivity) of the captured image, a method based on the gamma shape applied to the baseline image, and a method based on the variance of the flat area of ​​the image.

[0062] According to the method based on bit depth, the gain map NR processing unit 206 controls so that the smaller the bit depth of the gain map (the smaller the number of bits of each gain value in the gain map), the lower the intensity of noise reduction. The smaller the bit depth, the more difficult it is to distinguish between the signal and the noise; therefore, by controlling in the aforementioned manner, the noise in the gain map can be reduced with an appropriate intensity.

[0063] The gain map NR processing unit 206 performs conversion from RGB to YUV according to a method based on the minimum and maximum values ​​of the brightness value of the baseline image, and obtains the maximum and minimum values ​​of the Y value (brightness value) of the YUV. It is considered that the difference between the maximum and minimum values ​​of the brightness value is used as an indicator of the amount of noise amplified when the gain map is applied to the baseline image. In view of this, the gain map NR processing unit 206 controls so that the greater the difference between the maximum and minimum values ​​of the brightness value, the higher the noise reduction intensity.

[0064] According to the method based on the shooting sensitivity (ISO sensitivity) of the captured image, the gain map NR processing unit 206 controls so that the higher the shooting sensitivity, the higher the noise reduction strength. When the shooting sensitivity is high, the amount of noise in the SDR image and HDR image used in the generation of the gain map increases, so it is considered that the amount of noise in the gain map also increases. For this reason, by controlling in the aforementioned manner, the noise in the gain map can be reduced with an appropriate strength.

[0065] Regarding the method based on the gamma shape applied to the baseline image, there is a value range in which the contrast is increased according to the gamma shape, and low-amplitude noise that is not easily observed before applying the gamma can be enhanced after applying the gamma. In view of this, the gain map NR processing unit 206 controls to enhance the intensity of noise reduction when the slope of the input-output characteristic associated with the value range is steep, and controls to conversely reduce the intensity of noise reduction when the slope of the input-output characteristic is gentle.

[0066] According to the method based on the variance in the flat area of ​​the image, the gain map NR processing unit 206 calculates the variance of the pixel values ​​in the flat area of ​​at least one of the SDR image and the HDR image used in the generation of the gain map. It is considered that the size of the calculated variance is used as an indicator of the amount of noise in the gain map. In view of this, the gain map NR processing unit 206 controls so that the larger the variance, the higher the noise reduction intensity.

[0067] The gain map encoding unit 207 encodes the gain map to which the NR process is applied in the gain map NR processing unit 206 in a format that complies with the output image file specification. Although quantization is performed in the encoding, the bit depth of the gain map does not have to be consistent with the bit depth of the baseline image, and it is sufficient that the bit depth of the gain map is consistent with or exceeds the bit depth of the HDR image. For example, in the case where the baseline image is an 8-bit SDR image and the HDR image obtained by combining the SDR image and the gain map has 10 bits, the bit depth of the gain map must be equal to or greater than 10 bits.

[0068] In addition, in encoding, a process for downsampling the gain map to a low resolution is performed to reduce the file size. For example, the gain map encoding unit 207 reduces the resolution of the gain map corresponding to the resolution of the input image by, for example, 1 / 4 (1 / 2 in the horizontal direction, 1 / 2 in the vertical direction). In addition, the gain map encoding unit 207 calculates the minimum and maximum values ​​of each RGB plane of the gain map.

[0069] File storage unit 208 stores the gain map encoded by gain map encoding unit 207 and the metadata and baseline image stored in storage unit 106 by gain map generation unit 205 into an image file. Storing the gain map and the baseline image into the image file associates the baseline image with the gain map.

[0070] Figure 4 is a flowchart showing an exemplary operation of the image processing unit 104 according to the first embodiment. The various types of processing shown in this flowchart can be implemented by, for example, a CPU or the like executing an image processing program according to the present embodiment. Alternatively, part or all of the processing shown in this flowchart may be implemented by hardware such as an electronic circuit or the like. Although the following describes a case where the baseline image is an HDR image, the baseline image of the present embodiment is not limited to an HDR image and may be an SDR image.

[0071] In step S401 , the SDR development processing unit 201 and the HDR development processing unit 202 of the image processing unit 104 generate an SDR image and an HDR image from the input image (Bayer image) from the A / D conversion unit 103 .

[0072] In step S402, the linear gamma conversion unit 203 performs a process for converting into linear gamma for the SDR image and the HDR image generated in step S401. Figure 5A and Figure 5B The method shown for converting a nonlinear signal into a linear signal using the EOTF function can be used as a linearization method. Note that Figure 5A shows an example of an EOTF function for an SDR image, and Figure 5B An example of an EOTF function for an HDR image is shown.

[0073] In step S403, the color space conversion unit 204 converts the SDR image and the HDR image that have been converted into linear gamma in step S402 into the same color space. It is assumed here that the color space conversion from sRGB to Rec.2020 is performed for the SDR image.

[0074] In step S404, the gain map generation unit 205 generates a gain map based on the SDR image and the HDR image to which the color space conversion is applied in step S403. The details of the generation method are as described above with reference to equation (1). It is assumed here that the gain map generation unit 205 generates a gain map for each of the RGB planes of the three channels. The gain map generation unit 205 stores the information required for encoding in the gain map encoding unit 207 (for example, information related to the number of channels and gain map information 209, etc.) as metadata in the storage unit 106.

[0075] In step S405, the gain map NR processing unit 206 performs NR processing on the gain map generated in step S404 to reduce random noise included in the input image and noise generated by resolution reduction. As described above, the method for NR processing of the gain map includes a method based on the reduction rate of the gain map, a method based on the NR setting of the shooting mode, and various other methods. The gain map NR processing unit 206 can use one of these methods, or a combination of two or more of these methods.

[0076] In step S406, the gain map encoding unit 207 encodes the gain map to which the NR process is applied in step S405 in a format that conforms to the output image file specification. Here, the resolution of the input image is reduced by 1 / 4 (1 / 2 in the horizontal direction, 1 / 2 in the vertical direction). In addition, the gain map encoding unit 207 calculates the minimum and maximum values ​​of each plane (R, G, or B) of the gain map.

[0077] In step S407, the file storage unit 208 stores the HDR image generated in step S401 and the gain map encoded in step S406 in the output image file. In addition, the file storage unit 208 also stores the metadata stored in the storage unit 106 in step S404 in the output image file. Here, the HDR image is stored in the file as a baseline image (main image); however, in the case where the baseline image is an SDR image, the file storage unit 208 stores the SDR image generated in step S401 instead of the HDR image in the output image file.

[0078] As described above, according to the first embodiment, the camera apparatus 100 generates a first image having a first dynamic range and a second image having a second dynamic range from a captured image. For example, in the case where an HDR image is used as a baseline image, the first image is an SDR image, and the second image is an HDR image. In addition, in the case where an SDR image is used as a baseline image, the first image is an HDR image, and the second image is an SDR image. Based on the first image and the second image, the camera apparatus 100 generates a gain map for converting the dynamic range of the second image (baseline image) into the first dynamic range. The camera apparatus 100 applies NR processing (first noise reduction processing) to the gain map, and associates the gain map with the second image (baseline image).

[0079] In this way, the present embodiment suppresses noise in a gain map generated based on two images with different dynamic ranges (e.g., an SDR image and an HDR image). Therefore, the present embodiment makes it possible to improve the image quality of an image generated by applying a gain map to a baseline image.

[0080] [Second embodiment]

[0081] Next, the second embodiment will be described. The basic configuration of the image pickup apparatus 100 according to the second embodiment is similar to that of the first embodiment. The differences from the first embodiment will be mainly described below.

[0082] Figure 8 1 is a block diagram showing an exemplary configuration of the image processing unit 104 according to the second embodiment. The image processing unit 104 includes an SDR development processing unit 201, an HDR development processing unit 202, a linear gamma conversion unit 203, a color space conversion unit 204, a gain map generation unit 205, a gain map encoding unit 207, a file storage unit 208, and a baseline HDR development processing unit 801.

[0083] The baseline HDR development processing unit 801 generates an HDR image as a baseline image by performing HDR development processing on an input image.

[0084] refer to Figure 3 , a description is now given of exemplary configurations of the SDR development processing unit 201, the HDR development processing unit 202, and the baseline HDR development processing unit 801.

[0085] Similar to the first embodiment, an exemplary configuration of the SDR image processing unit 201 and the HDR image processing unit 202 may be composed of Figure 3 . However, the method of controlling the intensity of noise reduction in the NR processing unit 302 of the SDR development processing unit 201 and the HDR development processing unit 202 is different from the method of controlling the intensity of noise reduction in the first embodiment. In the first embodiment, NR processing is performed with an intensity corresponding to the NR setting of the shooting mode. On the other hand, in the second embodiment, the NR processing unit 302 of the SDR development processing unit 201 and the HDR development processing unit 202 performs NR processing with an intensity suitable for reducing noise in a gain map to be generated later, regardless of the NR setting. For example, the NR processing unit 302 of the SDR development processing unit 201 and the HDR development processing unit 202 may control the intensity of noise reduction based on at least one of the shooting sensitivity (ISO sensitivity) of the captured image and the variance of pixel values ​​in a flat area of ​​the captured image. From Figure 8 It can be understood that in the second embodiment, the SDR image and the HDR image input to the gain map generation unit 205 have been subjected to NR processing at an intensity suitable for reducing noise in the gain map in the NR processing unit 302 of the SDR development processing unit 201 and the HDR development processing unit 202. Therefore, unlike the first embodiment, it is not necessary to perform NR processing on the gain map after the gain map is generated, and Figure 8 The image processing unit 104 does not need to include the gain map NR processing unit 206.

[0086] Similar to the SDR image processing unit 201 and the HDR image processing unit 202, the exemplary configuration of the baseline HDR image processing unit 801 can be composed of Figure 3 . However, the method of controlling the noise reduction intensity in the NR processing unit 302 of the baseline HDR development processing unit 801 is different from the method in the SDR development processing unit 201 and the HDR development processing unit 202 according to the second embodiment. Similar to the SDR development processing unit 201 and the HDR development processing unit 202 according to the first embodiment, the NR processing unit 302 of the baseline HDR development processing unit 801 performs NR processing with an intensity corresponding to the NR setting accepted from the user. The operation unit 109 has a function as an acceptance unit for accepting an intensity setting (NR setting) of the NR processing from the user.

[0087] Fig. 9 1 is a flowchart showing an exemplary operation of the image processing unit 104 according to the second embodiment. Each type of processing shown in this flowchart can be implemented by, for example, a CPU, etc. executing an image processing program according to the present embodiment. Alternatively, part or all of the processing shown in this flowchart can be implemented by hardware such as an electronic circuit, etc.

[0088] In step S901 , the SDR development processing unit 201 and the HDR development processing unit 202 of the image processing unit 104 generate an SDR image and an HDR image to be used for generating a gain map from the input image (Bayer image) from the A / D conversion unit 103 .

[0089] In step S902 , the baseline HDR development processing unit 801 of the image processing unit 104 generates an HDR image to be used as a baseline image from the input image (Bayer image) from the A / D conversion unit 103 .

[0090] The processing of steps S903 to S905 is similar to Figure 4 The processing of steps S402 to S404.

[0091] In step S906, the gain map encoding unit 207 encodes the gain map generated in step S905 in a format that complies with the output image file specification. Here, the resolution of the input image is reduced by 1 / 4 (1 / 2 in the horizontal direction and 1 / 2 in the vertical direction). In addition, the gain map encoding unit 207 calculates the minimum and maximum values ​​of each plane (R, G or B) of the gain map.

[0092] In step S907, the file storage unit 208 stores the HDR image generated as the baseline image in step S902 and the gain map encoded in step S906 in the output image file. In addition, the file storage unit 208 also stores the metadata stored in the storage unit 106 in step S905 in the output image file.

[0093] Note that although the above description is provided under the assumption that the baseline image is an HDR image, the baseline image may be an SDR image. In this case, the baseline HDR development processing unit 801 is configured to generate an SDR image as the baseline image. In addition, the file storage unit 208 stores the SDR image generated as the baseline image in step S902 into the output image file.

[0094] As described above, according to the second embodiment, the camera apparatus 100 generates a first image having a first dynamic range to which a noise reduction process (first noise reduction process) is applied from a captured image. In addition, the camera apparatus 100 generates a second image having a second dynamic range to which a noise reduction process (first noise reduction process) is applied from the captured image. In addition, the camera apparatus 100 generates a third image having a second dynamic range to which a second noise reduction process is applied at an intensity independent of the first noise reduction process from the captured image. The third image is an image used as a baseline image. In the case where the baseline image is an HDR image, the first image is an SDR image, and the second image is an HDR image as a non-baseline image. In the case where the baseline image is an SDR image, the first image is an HDR image, and the second image is an SDR image as a non-baseline image. Based on the first image and the second image, the camera apparatus 100 generates a gain map for converting the dynamic range of the third image (baseline image) into the first dynamic range. Then, the camera apparatus 100 associates the gain map with the third image (baseline image).

[0095] In this way, according to the present embodiment, a gain map is generated based on the first image and the second image to which noise reduction processing is applied at an intensity independent of the noise reduction processing for the baseline image. Therefore, the present embodiment makes it possible to suppress noise in the gain map regardless of the intensity of the noise reduction processing for the baseline image, and improve the image quality of the image generated by applying the gain map to the baseline image.

[0096] Other embodiments

[0097] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0098] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

Claims

1. An image processing device, comprising: a first image generating unit configured to generate a first image having a first dynamic range from the captured image; a second image generating unit configured to generate a second image having a second dynamic range from the captured image; a gain map generating unit configured to generate a gain map for converting a dynamic range of the second image into the first dynamic range based on the first image and the second image; a noise reduction unit configured to apply a first noise reduction process to the gain map; as well as An associating unit configured to associate the gain map with the second image.

2. The image processing apparatus according to claim 1, wherein: The resolution of the gain map is lower than the resolution of the second image, and The noise reduction unit controls the intensity of the first noise reduction process based on a ratio of a resolution of the gain map to a resolution of the second image.

3. The image processing apparatus according to claim 2, wherein: The noise reduction unit performs control such that the higher the ratio is, the higher the intensity of the first noise reduction process is.

4. The image processing apparatus according to claim 1, wherein: The first image generation unit generates a first image to which a second noise reduction process is applied, the second image generation unit generates a second image to which the second noise reduction process is applied, and The noise reduction unit controls the intensity of the first noise reduction process based on the intensity of the second noise reduction process.

5. The image processing apparatus according to claim 4, wherein: The noise reduction unit performs control so that the lower the intensity of the second noise reduction processing is, the higher the intensity of the first noise reduction processing is.

6. The image processing device according to claim 4, further comprising: An accepting unit is configured to accept a setting of the intensity of the second noise reduction process from a user.

7. The image processing apparatus according to claim 1, wherein: The noise reduction unit controls the intensity of the first noise reduction processing based on at least one of the bit depth of the gain map, the difference between the maximum and minimum values ​​of the brightness value of the second image, the shooting sensitivity of the captured image, the shape of the gamma applied to the second image, and the variance of the pixel values ​​in the flat area of ​​the first image or the second image.

8. An image processing device, comprising: a first image generating unit configured to generate, from a captured image, a first image having a first dynamic range to which a first noise reduction process is applied; a second image generating unit configured to generate, from the captured image, a second image having a second dynamic range and to which the first noise reduction process is applied; a third image generating unit configured to generate, from the captured image, a third image having the second dynamic range and to which a second noise reduction process is applied at an intensity independent of the first noise reduction process; a gain map generating unit configured to generate a gain map for converting a dynamic range of the third image into the first dynamic range based on the first image and the second image; as well as An associating unit is configured to associate the gain map with the third image.

9. The image processing device according to claim 8, further comprising: The first control unit is configured to control the intensity of the first noise reduction process based on at least one of an ISO sensitivity of the captured image and a variance of pixel values ​​in a flat area of ​​the captured image.

10. The image processing device according to claim 8, further comprising: An accepting unit is configured to accept a setting of the intensity of the second noise reduction process from a user.

11. A camera device, comprising: The image processing device according to any one of claims 1 to 10; as well as A photographing unit is configured to generate the photographed image.

12. An image processing method, which is performed by an image processing device, the image processing method comprising: generating a first image having a first dynamic range from the captured image; generating a second image having a second dynamic range from the captured image; generating a gain map for converting a dynamic range of the second image into the first dynamic range based on the first image and the second image; applying a first noise reduction process to the gain map; as well as The gain map is associated with the second image.

13. An image processing method, which is performed by an image processing device, the image processing method comprising: generating, from the captured image, a first image having a first dynamic range to which a first noise reduction process is applied; generating, from the captured image, a second image having a second dynamic range to which the first noise reduction process is applied; generating, from the captured image, a third image having the second dynamic range to which a second noise reduction process is applied at an intensity independent of the first noise reduction process; generating a gain map for converting a dynamic range of the third image into the first dynamic range based on the first image and the second image; as well as The gain map is associated with the third image.

14. A computer-readable storage medium having a computer program stored therein, wherein: When the computer program is executed by a processor, the image processing method according to claim 12 or 13 is performed.

15. A computer program product comprising a computer program, wherein: When the computer program is executed by a processor, the image processing method according to claim 12 or 13 is performed.

Images