Image processing device, imaging device, image processing method, and program

Abstract

This technology improves the efficiency of generating gain maps based on two images with different dynamic ranges, thereby reducing the processing load. [Solution] An image processing apparatus comprising: an image generation means that generates a first image having a first dynamic range and a second image having a second dynamic range different from the first dynamic range from an image captured by an imaging means; a gain map generation means that generates a gain map for converting the second dynamic range to the first dynamic range based on the first image and the second image; an acquisition means that acquires imaging conditions related to the imaging means; and a reduction means that performs a reduction process on the image from which the gain map is generated, wherein the reduction means modifies the reduction process based on the imaging conditions.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus and an image processing method, and particularly to a dynamic range conversion technique for images.

Background Art

[0002] Conventionally, in order to match the dynamic range that a display device can display, an image in HDR (High Dynamic Range) format and an image in SDR (Standard Dynamic Range) format have been mutually converted using conversion information called a gain map (Non-Patent Document 1).

[0003] Also, a technique is known in which a gain map for mutually converting an SDR image and an HDR image is generated based on the SDR image and the HDR image, and the SDR image or the HDR image is stored in a file together with the gain map as a base image. The gain map is a map of coefficients (gains) applied to each pixel constituting an image. The resolution of the gain map may be less than or equal to the resolution of the image to which the gain map is applied. According to this technique, for example, when the base image is an HDR image, an SDR image can be generated by applying the gain map to the base image. Therefore, depending on whether the display for displaying the image supports HDR or not, it becomes possible to select and display an image suitable for the display from the SDR image and the HDR image.

[0004] Also, in Patent Document 1, it is described that an image area of a RAW image is divided into a plurality of sub-areas, a representative value included in the sub-areas is obtained, a reduced image having the representative value as a pixel value is generated, and tone conversion of the RAW image is performed using a gain map generated based on the reduced image.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006] [Non-Patent Document 1] Eric Chan, “Gain Maps”Version 1.0 draft 15”, [online], February 28, 2024, [Searched on May 15, 2020], Internet <URL:https: / / helpx.adobe.com / content / dam / help / en / camera-raw / using / gain-map / jcr_content / root / content / flex / items / position / position-par / table / row-3u03dx0-column-4a63daf / download_section / download-1 / Gain_Map_1_0d15.pdf> [Overview of the project] [Problems that the invention aims to solve]

[0007] If, for example, the development process is performed simply from the image size of the base image to generate a gain map for converting between HDR and SDR images, the computational load will be higher compared to normal shooting where a gain map is not generated. As a result, when a gain map is generated during imaging, the performance of functions that perform continuous development processing, such as multiple image stacking or high-speed continuous shooting, may not be fully realized.

[0008] Patent Document 1 does not mention processing of gain maps for mutual conversion between HDR and SDR images, and therefore cannot address the issue of computational load during gain map generation.

[0009] The present invention has been made in view of the above problems, and its objective is to provide a technology that improves the efficiency of generating gain maps based on two images with different dynamic ranges (for example, an SDR image and an HDR image) and reduces the processing load. [Means for solving the problem]

[0010] To solve the above problems, the image processing apparatus according to the present invention comprises: an image generation means that generates a first image having a first dynamic range and a second image having a second dynamic range different from the first dynamic range from an image captured by an imaging means; a gain map generation means that generates a gain map for converting the second dynamic range to the first dynamic range based on the first image and the second image; an acquisition means that acquires imaging conditions related to the imaging means; and a reduction means that performs a reduction process on the image from which the gain map is generated, wherein the reduction means modifies the reduction process based on the imaging conditions. [Effects of the Invention]

[0011] According to the present invention, it is possible to improve the efficiency of generating gain maps based on two images with different dynamic ranges and reduce the processing load. [Brief explanation of the drawing]

[0012] [Figure 1] A block diagram showing an example configuration of an imaging device 100 to which an image processing device is applied. [Figure 2] A block diagram showing an example configuration of the image processing unit 104 according to the first embodiment. [Figure 3] A block diagram showing an example configuration of the SDR development processing unit 202 and the HDR development processing unit 203. [Figure 4] A flowchart showing an example of the operation of the image processing unit 104 according to the first embodiment. [Figure 5] A flowchart showing an example of the operation of the shooting and image signal processing S401 according to the first embodiment. [Figure 6] A diagram showing an example of the EOTF function used in the linear gamma transform unit 204. [Figure 7] A flowchart showing an example of the operation of the shooting and image signal processing S401 according to the second embodiment. [Figure 8] A diagram showing a selection example of a user interface in the gain map generation mode.

Embodiments of the Invention

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0014] In this embodiment, an embodiment in which the image processing apparatus of the present invention is applied to an information processing apparatus such as a personal computer (PC) or an information processing terminal such as a smart device or a tablet PC will be described. Note that the present invention is not limited to this, and it may be applied to an imaging apparatus such as a digital camera capable of taking images.

[0015] [First Embodiment] First, referring to FIG. 1, the configuration and functions of the imaging apparatus 100 according to the first embodiment will be described.

[0016] FIG. 1 is a block diagram showing a configuration example of an imaging apparatus 100 to which an image processing apparatus according to an embodiment of the present invention is applied. The imaging apparatus 100 includes an optical system 101, an imaging unit 102, an A / D conversion unit 103, an image processing unit 104, a display unit 105, a storage unit 106, a recording medium 107, a system control unit 108, and an operation unit 109.

[0017] Based on an image (captured image) generated by shooting using the imaging unit 102, the imaging apparatus 100 generates an SDR image and an HDR image in the image processing unit 104, and performs a process of generating a gain map capable of reconstructing HDR representation and SDR representation.

[0018] Each functional block of the imaging device 100 in the first embodiment can be implemented by software, or a combination of software and hardware, except for parts that can only be implemented by hardware. For example, a functional block may be implemented by dedicated hardware such as an ASIC. Alternatively, a functional block may be implemented by a processor such as a CPU executing a program stored in memory. Multiple functional blocks may be implemented by a common configuration (e.g., one ASIC). Furthermore, hardware that implements some functions of one functional block may be included in hardware that implements other functional blocks.

[0019] In this embodiment, an HDR image is an image with a wider dynamic range than an SDR image, and is, for example, an image to which the Opto-Electronic Transfer Function (OETF) characteristics described in ST2084, an HDR standard handled by HDR monitors, are applied. An SDR image is an image with a narrower dynamic range than an HDR image. In this embodiment, the gamma of the HDR image is assumed to be the OETF characteristics of ST2084, the color gamut is assumed to be Rec.2020, and the gamma of the SDR image is assumed to be sRGB gamma, the color gamut is assumed to be sRGB.

[0020] In Figure 1, the optical system 101 includes a zoom lens and a lens group including a focus range, an aperture adjustment device, and a shutter device. The optical system 101 adjusts the magnification, focus position, and light intensity of the subject image that reaches the imaging unit 102. The imaging unit 102 includes an image sensor such as a CCD sensor or CMOS sensor that converts the light beam of the subject image that has passed through the optical system 101 into an electrical signal (image signal) by photoelectric conversion. The A / D conversion unit 103 generates a digital image by applying analog-to-digital conversion to the image signal input from the imaging unit 102.

[0021] The image processing unit 104 performs image processing on the image (captured image) output from the A / D conversion unit 103, including image signal reduction, pixel interpolation, linear gamma conversion for generating a gain map, and color space conversion. The image processing unit 104 also performs predetermined compression processing for recording the image on the recording medium 107, which will be described later. The image processing unit 104 can perform similar image processing not only on the image output from the A / D conversion unit 103, but also on the image read from the recording medium 107.

[0022] The display unit 105 displays the viewfinder image during shooting, displays the captured image, and displays text for interactive operation. The display unit 105 displays images generated by the image processing unit 104 and images read from the recording medium 107. The display unit 105 is, for example, a liquid crystal display or an organic electroluminescence (EL) display.

[0023] The memory unit 106 stores image processing programs and various other information necessary for image processing by the image processing unit 104. The memory unit 106 also stores drive mode information, which indicates the drive mode included in the shooting mode at the time of shooting. Drive modes will be described later.

[0024] The recording medium 107 has the function of recording images. The recording medium may include, for example, a memory card equipped with semiconductor memory, or a recording medium using a package containing a rotating recording element such as a magnetic disk. The recording medium 107 may be configured to be detachable from the imaging device 100.

[0025] The system control unit 108 performs overall control of the imaging device 100. The system control unit 108 includes, for example, a CPU (or MPU), ROM, and RAM, and performs various controls, including overall control of the imaging device 100, by loading a program stored in ROM into the RAM's work area and executing it.

[0026] The control unit 109 is for receiving user input. The control unit 109 can be, for example, a button, a lever, or a touch panel. The user can use the control unit 109 to set modes (operation mode settings, imaging condition settings) such as the shooting mode during shooting, which includes multiple image composite shooting mode, drive mode, and gain map generation mode.

[0027] Next, the drive modes of this embodiment will be described.

[0028] A drive mode is an operating mode related to the operation of the shutter. In this embodiment, the drive modes include a high-speed continuous shooting mode (first continuous shooting mode) and a low-speed continuous shooting mode (second continuous shooting mode) as continuous shooting modes, and a single-shot mode as a single-shot mode.

[0029] The high-speed continuous shooting mode and the low-speed continuous shooting mode are modes in which the system control unit 108 repeats a series of shooting operations after completing a series of shooting operations from reading the signal from the imaging unit 102 to writing the image data to the recording medium 107. The high-speed continuous shooting mode has a high continuous shooting speed, and the low-speed continuous shooting mode has a slower continuous shooting speed compared to the high-speed continuous shooting mode. The single-shot mode is a mode in which the system control unit 108 completes a series of shooting operations from reading the signal from the imaging unit 102 to writing the image data to the recording medium 107.

[0030] The system control unit 108 sets the drive mode by storing the drive mode information of one of the above-mentioned drive modes in the storage unit 106 in response to the user's operation of the operation unit 109.

[0031] In this embodiment, we have described a case where two different continuous shooting speeds, a high-speed continuous shooting mode and a low-speed continuous shooting mode, can be set as the continuous shooting mode. However, the system is not limited to this case, and it may be possible to set three or more different continuous shooting speeds. Furthermore, multiple continuous shooting speeds may be set within the high-speed continuous shooting mode and the low-speed shooting mode.

[0032] In the example shown in Figure 1, the optical system 101 is configured as part of an imaging device 100 equipped with an imaging unit 102, but this embodiment is not limited to this configuration. For example, an imaging system in which interchangeable optical systems (interchangeable lenses) can be attached to and detached from the main body of the imaging device 100 may be used, such as in a single-lens reflex camera.

[0033] Figure 2 is a block diagram showing an example configuration of the image processing unit 104 according to the first embodiment. The image processing unit 104 includes an image reduction unit 201, an SDR development processing unit 202, an HDR development processing unit 203, a linear gamma conversion unit 204, a color space conversion unit 205, a gain map generation unit 206, a gain map encoding unit 207, and a file storage unit 208.

[0034] Here, the input image (captured image) input to the image processing unit 104 is assumed to be a digital image generated by A / D conversion of the signal acquired by the imaging unit 102 in the A / D conversion unit 103. For example, it is a RAW image. Furthermore, this digital image is assumed to be a Bayer image with three components: red (R), green (G), and blue (B).

[0035] The image reduction unit 201 reduces the input image (Bayer image) and then converts the image signal as necessary. The reduction process includes, but is not limited to, processes such as downsampling between pixels, averaging the pixel values ​​of multiple surrounding pixels, or combining the pixel values ​​of multiple pixels into a single pixel value using a certain weight.

[0036] The SDR development processing unit 202 performs SDR development processing on the input image to generate an SDR image (image generation processing). The HDR development processing unit 203 performs HDR development processing on the input image to generate an HDR image (image generation processing). Development processing refers to the process of generating a YUV image or an RGB image suitable for an output device (not shown), such as a display, from the input Bayer image.

[0037] Here, with reference to Figure 3, an example configuration of the SDR development processing unit 202 and the HDR development processing unit 203 will be described. As shown in Figure 3, the example configuration of the SDR development processing unit 202 and the HDR development processing unit 203 can be represented by the same block diagram.

[0038] Each of the SDR development processing unit 202 and HDR development processing unit 203 includes a white balance processing unit 301, an NR processing unit 302, a demosaicing processing unit 303, a color matrix processing unit 304, and a gamma processing unit 305. The SDR development processing unit 202 and HDR development processing unit 203 acquire a Bayer image as an input image and generate a YUV image or an RGB image suitable for an output device (not shown), such as a display, as an output image.

[0039] The white balance processing unit 301 performs white balance processing to adjust the color balance of the input image.

[0040] The NR processing unit 302 performs NR processing on the input image to reduce dark current noise and optical shot noise. For NR processing, general methods such as low-pass filters (LPFs) or bilateral filters can be applied.

[0041] The demosaicing processing unit 303 performs demosaicing on the input image to generate R, G, and B three-plane images from the Bayer image. Common methods such as linear interpolation and adaptive interpolation can be applied to the demosaicing process.

[0042] The color matrix processing unit 304 performs color matrix processing on the input image to match the color gamut of the output device based on the spectral characteristics of the image sensor.

[0043] The gamma processing unit 305 performs gamma processing on the input image, converting the signal value using a gamma characteristic (OETF) to generate a signal that matches the monitor gamma (OETF) of the output device.

[0044] As mentioned above, the SDR development processing unit 202 and the HDR development processing unit 203 can be represented as the same block diagram. However, different parameters are used for various processing depending on the type of output image (SDR image or HDR image). For example, the gamma processing unit 305 of the SDR development processing unit 202 uses sRGB gamma as the gamma characteristic applied to the input image, while the gamma processing unit 305 of the HDR development processing unit 203 uses the OETF characteristic of ST2084 as the gamma characteristic applied to the input image.

[0045] Referring again to Figure 2, the linear gamma conversion unit 204 converts the gamma characteristics of the SDR and HDR images generated by the SDR development processing unit 202 and HDR development processing unit 203 into linear gamma. As a method of linearization, for example, in the case of an SDR image, a method can be used to convert a nonlinear SDR signal into a linear signal using the SDR Electro-Optical Transfer Function (EOTF) function. Specifically, the reference EOTF defined in ITU-R BT.709 can be used.

[0046] The color space conversion unit 205 converts the SDR and HDR images linearized by the linear gamma conversion unit 204 into a common color space. For example, to match the color space to Rec.2020, the color space conversion unit 205 performs a color space conversion from sRGB to Rec.2020 for the SDR image. As a method for color space conversion, for example, the method described in ITU-R BT.2087 can be used.

[0047] The gain map generation unit 206 generates a gain map based on the SDR image and HDR image converted in color space by the color space conversion unit 205. The gain map is conversion information for converting between HDR (High Dynamic Range) format images (HDR images) and SDR (Standard Dynamic Range) format images (SDR images) to match the dynamic range that the display device can display. In the gain map generation process of the gain map generation unit 206, the gain value is calculated by taking the logarithm of the ratio of SDR to HDR for each pixel, as shown in equation (1) below. In equation (1), SDR, k SDR These are the pixel values ​​and offset values ​​of the SDR image, respectively, and HDR, k HDR These are the pixel values ​​and offset values ​​of the HDR image, respectively, and G is the gain value.

[0048]

number

[0049] Here, the gain map generation unit 206 may generate a gain value for a single-channel grayscale image, or it may generate a gain value for each of the three RGB planes. In the case of a single-channel grayscale image, the gain map generation unit 206 performs a conversion from RGB to YUV and calculates the gain value from the Y value of the YUV. The gain map generation unit 206 also stores necessary information (for example, the number of channels for the gain value) as metadata in the storage unit 106 for encoding by the gain map encoding unit 207, which will be described later.

[0050] By applying a gain map to one of the SDR or HDR images, the other image can be generated. Therefore, if an image file contains either an SDR image or an HDR image, along with a gain map, both SDR and HDR images can be obtained from the image file. The image stored in the image file along with the gain map is called the base image or main image.

[0051] According to equation (1), a gain value is calculated based on the pixel values ​​of the SDR image. Therefore, if the gain map calculated according to equation (1) is stored in an image file, and the base image is an SDR image, an HDR image can be generated by multiplying the pixel values ​​of the SDR image by the gain value of the gain map. On the other hand, if the gain map calculated according to equation (1) is stored in an image file, and the base image is an HDR image, an SDR image can be generated by multiplying the pixel values ​​of the HDR image by the reciprocal of the gain value of the gain map.

[0052] The gain map encoding unit 207 encodes the gain map generated by the gain map generation unit 206 into a format that matches the specifications of the output image file. Quantization is performed during encoding, but the bit depth of the gain map does not need to match the bit depth of the base image; it only needs to match or exceed the bit depth of the HDR image. For example, if the base image is an 8-bit SDR image, and the HDR image obtained by combining the SDR image and the gain map is 10-bit, the bit depth of the gain map must be 10 bits or more. In addition, the gain map encoding unit 207 calculates the minimum and maximum values ​​for each RGB plane of the gain map.

[0053] The file storage unit 208 stores the gain map encoded by the gain map encoding unit 207, the metadata stored in the storage unit 106 by the gain map generation unit 206, and the base image in an image file. By storing the gain map and base image in the image file, the gain map is associated with the base image.

[0054] Figure 4 is a flowchart showing an example of the operation of the image processing unit 104 according to the first embodiment. Each process shown in this flowchart may be implemented, for example, by a CPU or the like executing an image processing program according to this embodiment. Alternatively, some or all of the processes shown in this flowchart may be implemented by hardware such as electronic circuits. Here, we will explain the case where the base image is an HDR image, but the base image in this embodiment is not limited to an HDR image and may be an SDR image.

[0055] S401 performs image capture and image signal processing. The image capture and image signal processing in S401 will be explained with reference to the flowchart in Figure 5.

[0056] In S501, the system control unit 108 determines the currently set drive mode. Specifically, the system control unit 108 determines the drive mode by reading the drive mode information stored in the memory unit 106. If it is in high-speed continuous shooting mode, it proceeds to S502; if it is in low-speed continuous shooting mode, it proceeds to S504; and if it is in single-shot mode, it proceeds to S506.

[0057] In S502 (high-speed continuous shooting mode), the system control unit 108 performs a series of shooting processes, from reading the signal from the imaging unit 102 to writing the image data to the recording medium 107.

[0058] In S503, the image reduction unit 201 performs a reduction process on the input image (Bayer image) from the A / D conversion unit 103. Here, the resolution of the input image is reduced by 1 / 16 (1 / 4 horizontally and 1 / 4 vertically). The reduction process averages the pixel values ​​of multiple surrounding pixels, but if there is a subject with high-frequency components in spatial frequency, aliasing (aliasing noise) may occur after the reduction process. To prevent this, a low-pass filter may be applied to the image information before the reduction process, according to the reduction ratio. Specifically, for example, the Nyquist frequency obtained from the reduced image size may be used as the cutoff frequency, and a low-pass filter that cuts out high-frequency components above the Nyquist frequency may be applied. Here, the reduction size is determined in advance from the estimation of the computational processing load of each mode included in the drive mode. In this way, by performing a resolution reduction on the input image, the computational processing load performed on a pixel-by-pixel basis by the SDR development processing unit 202 and HDR development processing unit 203 and beyond can be reduced.

[0059] In S504 (low-speed continuous shooting mode), the system control unit 108 performs a series of shooting processes. This process is the same as that in S502.

[0060] In S505, the image reduction unit 201 performs a reduction process on the input image (Bayer image) from the A / D conversion unit 103. Here, the resolution of the input image is reduced to 1 / 4 (1 / 2 horizontally and 1 / 2 vertically). In addition, to prevent aliasing as in S503, a low-pass filter that cuts out high-frequency components above the Nyquist frequency may be applied. Note that here, the reduction size is determined in advance from the estimation of the computational processing load for each mode included in the drive mode. Generally, the computational processing load is lower in the low-speed continuous shooting mode than in the high-speed continuous shooting mode. Therefore, the reduction process is performed at a size larger than the reduced size in S503, but if there is no significant difference in processing load between the high-speed continuous shooting mode and the low-speed continuous shooting mode, the same resolution reduction may be performed in both modes.

[0061] In S506 (single-shot mode), the system control unit 108 performs a series of shooting processes. This process is the same as that in S502. In single-shot mode, there is no processing load compared to the low-speed continuous shooting mode, so resolution reduction like in S503 and S505 is not performed here.

[0062] In S402, the SDR development processing unit 202 and HDR development processing unit 203 of the image processing unit 104 generate an SDR image and an HDR image used for generating a gain map from the input image generated in S401. Also in S402, the HDR development processing unit 203 of the image processing unit 104 generates an HDR image to be used as a base image from the input image (Bayer image) from the A / D conversion unit 103.

[0063] In S403, the linear gamma conversion unit 204 performs a linear gamma conversion process on the SDR and HDR images generated in S402. As a linearization method, a method can be used that converts nonlinear signals into linear signals using EOTF functions as shown in Figures 6(a) and 6(b). Figure 6(a) shows an example of an EOTF function for SDR images, and Figure 6(b) shows an example of an EOTF function for HDR images.

[0064] In S404, the color space conversion unit 205 converts the SDR image and HDR image, which were converted to linear gamma in S403, to a common color space. Here, the color space conversion from sRGB to Rec.2020 is performed for the SDR image.

[0065] In S405, the gain map generation unit 206 generates a gain map based on the SDR and HDR images that were color-space converted in S404. The details of the generation method are as described above with reference to equation (1). Here, the gain map generation unit 206 generates a gain map for each of the three RGB channels. The gain map generation unit 206 stores the necessary information for encoding by the gain map encoding unit 207 (for example, information regarding the number of channels for gain values) as metadata in the storage unit 106.

[0066] In S406, the gain map encoding unit 207 encodes the gain map generated in S405 into a format that conforms to the specifications of the output image file. Here, the gain map encoding unit 207 calculates the minimum and maximum values ​​of each plane (R, G, B) of the gain map.

[0067] In S407, the file storage unit 208 stores the HDR image used as the base image generated in S402, and the gain map encoded in S406, in the output image file. The file storage unit 208 also stores the metadata stored in the storage unit 106 in S405 in the output image file. Here, the HDR image is stored in the file as the base image, but if the base image is an SDR image, the file storage unit 208 stores the SDR image generated in S402 in the output image file instead of the HDR image.

[0068] As described above, according to this embodiment, the imaging device 100 performs a resolution reduction process on the Bayer image used to generate the gain map based on the drive mode setting. From the image after the resolution reduction process, it generates a first image having a first dynamic range and a second image having a second dynamic range. Furthermore, the imaging device 100 generates a third image having a second dynamic range from the captured image. The third image is used as the base image. If the base image is an HDR image, the first image is an SDR image and the second image is an HDR image as a non-base image. If the base image is an SDR image, the first image is an HDR image and the second image is an SDR image as a non-base image. Based on the first and second images, the imaging device 100 generates a gain map to convert the dynamic range of the third image (base image) to the first dynamic range. Then, the imaging device 100 associates the gain map with the third image (base image).

[0069] Thus, according to this embodiment, by performing a resolution reduction process on the Bayer image used to generate the gain map, it becomes possible to reduce the computational load of the gain map generated based on two images with different dynamic ranges. In particular, by changing the resolution reduction process on the Bayer image used to generate the gain map based on the drive mode setting, it is possible to reduce the computational load.

[0070] In the embodiment described above, the resolution reduction process was controlled based on the continuous shooting speed of each mode included in the drive mode. However, resolution reduction may be performed based on the settings of other shooting modes that involve continuous shooting or shooting modes that have a high computational load, such as those that combine multiple images, not limited to this case.

[0071] [Second Embodiment] Next, a second embodiment will be described. The basic configuration of the imaging device 100 according to the second embodiment is the same as that of the first embodiment. The following will mainly describe the differences from the first embodiment.

[0072] Figure 7 is a flowchart showing an example of the operation of the image processing unit 104 according to the second embodiment. Each process shown in this flowchart may be implemented, for example, by a CPU or the like executing an image processing program according to this embodiment. Alternatively, some or all of the processes shown in this flowchart may be implemented by hardware such as electronic circuits.

[0073] S401 performs image capture and image signal processing. The image capture and image signal processing in S401 will be explained with reference to the flowchart in Figure 7.

[0074] In S701, the system control unit 108 determines the currently set gain map generation mode. Specifically, the system control unit 108 determines the gain map generation mode by reading the gain map generation information stored in the memory unit 106. As shown in Figure 8, the gain map generation mode can be selected via the operation unit 109 from one of the following modes: no gain map generation, speed priority, or image quality priority.

[0075] If speed priority mode is selected, proceed to S702; if image quality priority mode is selected, proceed to S704; and if neither mode is selected, proceed to S706.

[0076] In S702 (speed priority mode), the system control unit 108 performs a series of imaging processes, from reading the signal from the imaging unit 102 to writing the image data to the image recording medium 107.

[0077] In S703, the image reduction unit 201 performs a reduction process on the input image (Bayer image) from the A / D conversion unit 103. Here, the resolution of the input image is reduced by 1 / 16 (1 / 4 horizontally and 1 / 4 vertically). The reduction process averages the pixel values ​​of multiple surrounding pixels, but if there is a subject with high-frequency components in the spatial frequency, aliasing (aliasing noise) may occur after the reduction process. To prevent this, a low-pass filter may be applied to the image information before the reduction process according to the reduction ratio. In speed priority mode, the reduction process is performed at a smaller size than in image quality priority mode to reduce the computational load. In this way, by performing a resolution reduction on the input image, the computational load performed on a pixel-by-pixel basis by the SDR development processing unit 202 and HDR development processing unit 203 and beyond can be reduced.

[0078] In S704 (image quality priority mode), the system control unit 108 performs a series of shooting processes. This process is the same as that in S702.

[0079] In the S705, the image reduction unit 201 performs reduction processing on the input image (Bayer image) from the A / D conversion unit 103. Here, the resolution of the input image is reduced by 1 / 4 (1 / 2 horizontally and 1 / 2 vertically). In addition, to prevent aliasing as in the S703, a low-pass filter that cuts out high-frequency components above the Nyquist frequency may be applied. In image quality priority mode, the image quality after HDR or SDR conversion with the gain map applied is prioritized, so the reduction processing is performed to a size larger than the reduced size of the S703's resolution reduction.

[0080] In S706 (mode without gain map generation), the system control unit 108 performs a series of imaging processes. This process is the same as that in S702. In this mode, there is no need to generate a gain map, so the processes related to gain map generation from S402 onward are skipped.

[0081] As described above, according to this embodiment, the imaging device 100 performs a resolution reduction process on the Bayer image used to generate the gain map based on the setting of the gain map generation mode. From the image after the resolution reduction process, it generates a first image having a first dynamic range and a second image having a second dynamic range. Furthermore, the imaging device 100 generates a third image having a second dynamic range from the captured image. The third image is used as the base image. If the base image is an HDR image, the first image is an SDR image and the second image is an HDR image as a non-base image. If the base image is an SDR image, the first image is an HDR image and the second image is an SDR image as a non-base image. Based on the first and second images, the imaging device 100 generates a gain map to convert the dynamic range of the third image (base image) to the first dynamic range. Then, the imaging device 100 associates the gain map with the third image (base image).

[0082] Thus, according to this embodiment, by performing a resolution reduction process on the Bayer image used to generate the gain map, it becomes possible to reduce the computational processing load of the gain map generated based on two images with different dynamic ranges.

[0083] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0084] The disclosures herein include the following image processing apparatus and its control methods, programs, storage media, and imaging devices.

[0085] (Item 1) From the image captured by the imaging means, a first image having a first dynamic range is obtained, Image generation means for generating a second image having a second dynamic range different from the first dynamic range, A gain map generation means generates a gain map for converting the second dynamic range to the first dynamic range based on the first image and the second image, An acquisition means for acquiring imaging conditions related to the imaging means, The system includes a reduction means that performs a reduction operation on the image used to generate the gain map, The reduction means is characterized by changing the reduction process based on the imaging conditions.

[0086] (Item 2) The image processing apparatus according to item 1, characterized in that the acquisition means acquires information as the imaging condition whether any of the operation modes, which include a plurality of continuous shooting modes with different continuous shooting speeds, is set as an operation mode.

[0087] (Item 3) The image processing apparatus according to item 2, characterized in that the reduction means determines the reduction size in the reduction process based on the continuous shooting speed of the continuous shooting mode.

[0088] (Item 4) The image processing apparatus according to item 3, characterized in that when the continuous shooting mode is a second mode having a faster continuous shooting speed than the first mode, the reduction means reduces the reduction size in the second mode to be smaller than in the first mode.

[0089] (Item 5) The image processing apparatus according to items 1 to 4, characterized in that the acquisition means acquires information as the imaging condition whether it is a mode that prioritizes continuous shooting speed or a mode that prioritizes image quality.

[0090] (Item 6) The image processing apparatus according to item 5, characterized in that the reduction means reduces the reduction size in the reduction process when the imaging conditions are in a mode that prioritizes continuous shooting speed compared to when the imaging conditions are in a mode that prioritizes image quality.

[0091] (Item 7) From the image captured by the imaging means, a first image having a first dynamic range is obtained, Image generation means for generating a second image having a second dynamic range different from the first dynamic range, A gain map generation means generates a gain map for converting the second dynamic range to the first dynamic range based on the first image and the second image, A reduction means that performs a reduction process on the image used to generate the gain map, It has setting means for setting the reduction size in the reduction process, The image processing apparatus is characterized in that the reduction means changes the reduction process based on the setting of the reduction size.

[0092] (Item 8) The image processing apparatus according to any one of items 1 to 7, characterized in that the first image and the second image are each either an SDR image or an HDR image and are different from each other.

[0093] (Item 9) The image processing apparatus according to any one of items 1 to 8, characterized in that the aforementioned image is a RAW image.

[0094] (Item 10) An imaging apparatus having an image processing device described in any one of items 1 to 9.

[0095] (Item 11) Image generation step of generating a first image having a first dynamic range and a second image having a second dynamic range different from the first dynamic range from an image captured by an imaging means, A gain map generation step of generating a gain map for converting the second dynamic range to the first dynamic range based on the first image and the second image, An acquisition step to acquire imaging conditions related to the imaging means, The system includes a reduction step which performs a reduction process on the image used to generate the gain map, A control method for an image processing apparatus, characterized by comprising: a control means for controlling the reduction process based on the imaging conditions in the reduction step.

[0096] (Item 12) Image generation step of generating a first image having a first dynamic range and a second image having a second dynamic range different from the first dynamic range from an image captured by an imaging means, A gain map generation step of generating a gain map for converting the second dynamic range to the first dynamic range based on the first image and the second image, A reduction step which involves reducing the image used to generate the gain map, The system includes a setting step for setting the reduction size in the reduction step, A control method for an image processing apparatus, characterized in that, in the reduction step, the reduction process is changed based on the setting of the reduction size.

[0097] (Item 13) A computer-readable program for causing a computer to execute the control method of the image processing device described in item 11 or 12. The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0098] 100 Imaging device 101 Optical system 102 Imaging Unit 103 A / D Conversion Unit 104 Image Processing Unit 105 Display section 106 Storage section 107 Recording media 108 System Control Unit 109 Operation section

Claims

1. From the image captured by the imaging means, a first image having a first dynamic range is obtained, Image generation means for generating a second image having a second dynamic range different from the first dynamic range, A gain map generation means generates a gain map for converting the second dynamic range to the first dynamic range based on the first image and the second image, An acquisition means for acquiring imaging conditions related to the imaging means, The system includes a reduction means that performs a reduction operation on the image used to generate the gain map, The reduction means is characterized by changing the reduction process based on the imaging conditions.

2. The image processing apparatus according to claim 1, characterized in that the acquisition means acquires information as the imaging condition whether any of a plurality of operating modes, including a plurality of continuous shooting modes with different continuous shooting speeds, is set.

3. The image processing apparatus according to claim 2, characterized in that the reduction means determines the reduction size in the reduction process based on the continuous shooting speed of the continuous shooting mode.

4. The image processing apparatus according to claim 3, characterized in that when the continuous shooting mode is a second mode having a faster continuous shooting speed than the first mode, the reduction means makes the reduction size in the second mode smaller than in the first mode.

5. The image processing apparatus according to claim 1, characterized in that the acquisition means acquires information as the imaging condition whether it is a mode that prioritizes continuous shooting speed or a mode that prioritizes image quality.

6. The image processing apparatus according to claim 5, characterized in that the reduction means reduces the reduction size in the reduction process when the imaging conditions are in a mode that prioritizes continuous shooting speed compared to when the imaging conditions are in a mode that prioritizes image quality.

7. From the image captured by the imaging means, a first image having a first dynamic range is obtained, Image generation means for generating a second image having a second dynamic range different from the first dynamic range, A gain map generation means generates a gain map for converting the second dynamic range to the first dynamic range based on the first image and the second image, A reduction means that performs a reduction process on the image used to generate the gain map, It has setting means for setting the reduction size in the reduction process, The image processing apparatus is characterized in that the reduction means changes the reduction process based on the setting of the reduction size.

8. The image processing apparatus according to any one of claims 1 to 7, characterized in that the first image and the second image are each either an SDR image or an HDR image and are different from each other.

9. The image processing apparatus according to any one of claims 1 to 7, characterized in that the aforementioned image is a RAW image.

10. An imaging apparatus having an image processing apparatus according to any one of claims 1 to 7.

11. Image generation step of generating a first image having a first dynamic range and a second image having a second dynamic range different from the first dynamic range from an image captured by an imaging means, A gain map generation step of generating a gain map for converting the second dynamic range to the first dynamic range based on the first image and the second image, An acquisition step to acquire imaging conditions related to the imaging means, The system includes a reduction step which performs a reduction process on the image used to generate the gain map, A control method for an image processing apparatus, characterized by comprising control means for controlling the reduction process based on the imaging conditions in the reduction step.

12. Image generation step of generating a first image having a first dynamic range and a second image having a second dynamic range different from the first dynamic range from an image captured by an imaging means, A gain map generation step of generating a gain map for converting the second dynamic range to the first dynamic range based on the first image and the second image, A reduction step which involves reducing the image used to generate the gain map, The system includes a setting step for setting the reduction size in the reduction step, A control method for an image processing apparatus, characterized in that, in the reduction step, the reduction process is changed based on the setting of the reduction size.

13. A computer-readable program for causing a computer to execute the control method of the image processing apparatus according to claim 11 or 12.

Images