Imaging apparatus and control method for the same, and image processing device

Abstract

To provide an imaging apparatus and a control method for the same that can more easily capture a low-light scene including a high-luminance subject.SOLUTION: An imaging apparatus performs imaging a plurality of times with underexposure and acquires image data for a plurality of frames. Further, the imaging apparatus corrects the brightness of the plurality of frames of image data and then combines the image data to generate data for a composite image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an imaging device, a control method thereof, and an image processing device, and particularly relates to an image synthesis technique.

Background Art

[0002] When photographing a scene with a wide dynamic range of luminance, such as a low-illuminance scene (for example, a night scene) including a high-luminance subject, with proper exposure, the high-luminance subject is likely to be overexposed. Therefore, a technique of synthesizing images taken with underexposure, proper exposure, and overexposure of the same scene to generate an image with a wide dynamic range is known (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When photographing the same scene with different exposures, in a low-illuminance scene, the exposure time is particularly long in photographing with proper exposure or overexposure, so camera shake is likely to occur. Therefore, it is necessary to fix the imaging device with a tripod or the like for photographing, which is not convenient to use.

[0005] In one aspect of the present invention, there is provided an imaging device and a control method thereof that can more easily photograph a low-illuminance scene including a high-luminance subject.

Means for Solving the Problems

[0006] The above object is achieved by an imaging apparatus having an imaging unit that performs multiple shootings with under-exposure to obtain image data for a plurality of frames, a correction unit that corrects the brightness of the image data for the plurality of frames, and a generation unit that synthesizes the image data for the plurality of frames corrected by the correction unit to generate composite image data.

Effect of the Invention

[0007] According to the present invention, it is possible to provide an imaging apparatus and a control method thereof that can more easily photograph a low-illumination scene including a high-brightness subject.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Modes for Carrying Out the Invention

[0009] Hereinafter, the present invention will be described in detail based on its exemplary embodiments with reference to the accompanying drawings. It should be noted that the following embodiments do not limit the invention according to the claims. Also, although a plurality of features are described in the embodiments, not all of them are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0010] The configurations represented as blocks in the drawings can be implemented by integrated circuits (ICs) such as ASICs or FPGAs, by discrete circuits, or by a combination of a memory and a processor that executes a program stored in the memory. Also, one block may be implemented by a plurality of integrated circuit packages, or a plurality of blocks may be implemented by one integrated circuit package. Further, the same block may be implemented in different configurations depending on the operating environment, required capabilities, etc.

[0011] Note that hereinafter, an embodiment in which the present invention is implemented in an imaging device such as a digital camera will be described. However, the present invention can be implemented in any electronic device having an imaging function. Such electronic devices include, in addition to imaging devices, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game machines, robots, drones, drive recorders, and the like. These are examples, and the present invention can be implemented in other electronic devices as well.

[0012] ●(First Embodiment) FIG. 1 is a block diagram showing a functional configuration example of an imaging device 100 according to an embodiment of the present invention. In FIG. 1, only representative configurations among the configurations of the imaging device 100 are shown. The optical system 101 has a plurality of lenses, a diaphragm combined with a shutter, a diaphragm driving mechanism, etc., and forms an optical image of a subject. The plurality of lenses include movable lenses such as a focus lens for adjusting the focal length of the optical system 101 and a zoom lens for changing the angle of view. Also, the optical system 101 has a driving mechanism for the movable lenses.

[0013] The imaging unit 102 may be a known CCD or CMOS color image sensor having a color filter of a primary color Bayer array, for example. The imaging unit 102 includes a pixel array in which a plurality of pixels are two-dimensionally arranged, and a peripheral circuit for reading signals from each pixel. Each pixel accumulates charges corresponding to the incident light amount by photoelectric conversion. By reading out from each pixel a signal having a voltage corresponding to the amount of charges accumulated during the exposure period, a group of pixel signals (analog image signals) representing the subject image formed by the optical system 101 is obtained.

[0014] The A / D conversion unit 103 A / D-converts the analog image signal read from the imaging unit 102 to convert it into a digital image signal (image data). Note that when the imaging unit 102 has an A / D conversion function, the A / D conversion unit 103 is not necessary.

[0015] The image processing unit 104 applies predetermined image processing to the image data output from the A / D conversion unit 103 or the image data read from the recording unit 109 to generate signals and image data according to the application, or to acquire and / or generate various types of information. The image processing unit 104 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function, for example. Alternatively, the image processing unit 104 may be configured such that a processor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) realizes a specific function by executing software.

[0016] The image processing that the image processing unit 104 can apply to the image data may include, for example, preprocessing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. The preprocessing may include signal amplification, reference level adjustment, defective pixel correction, and the like. The color interpolation process is performed when a color filter is provided in the imaging unit 102, and is a process of interpolating the values of color components not included in the individual pixel data constituting the image data. The color compensation process is also called the demosaicing process. The correction process may include processes such as white balance adjustment, tone correction, correction of image degradation (image restoration) caused by optical aberrations of the optical system 101, correction of the influence of peripheral light reduction of the optical system 101, and color correction. The detection process may include detection of a feature region (for example, a face region or a human body region) and its movement, person recognition processing, and the like. The data processing process may include processes such as region extraction (trimming), synthesis, scaling, encoding and decoding, header information generation (data file generation), etc. Generation of display image data and recording image data is also included in the data processing process. The evaluation value calculation process may include processes such as generation of signals and evaluation values used for autofocus detection (AF), generation of evaluation values used for automatic exposure control (AE), and the like. The special effect process may include processes such as addition of a blur effect, change in color tone, relighting, and the like. Note that these are examples of processes applicable by the image processing unit 104 to the image data, and do not limit the processes applied by the image processing unit 104.

[0017] The exposure control unit 105 determines the shooting conditions (aperture value, shutter speed, ISO sensitivity) according to the evaluation value calculated by the image processing unit 104 and a predetermined program diagram. Further, the exposure control unit 105 controls the operations of the optical system 101 (aperture) and the imaging unit 102 based on the shooting conditions determined at the time of shooting. Further, the exposure control unit 105 drives the optical system 101 (focus lens) based on the evaluation value calculated by the image processing unit 104, and adjusts the focal length of the optical system 101.

[0018] The system control unit 106 includes, for example, a processor (such as a CPU, MPU, or microprocessor) capable of executing programs, a ROM, and a RAM. The system control unit 106 controls the operations of each part of the imaging device 100 and realizes the functions of the imaging device 100 by reading the programs stored in the ROM into the RAM and executing them.

[0019] The ROM is rewritable and stores programs executed by the processor, various setting values of the imaging device 100, GUI data, and the like. The RAM is the main memory used when the system control unit 106 executes programs.

[0020] The operation unit 107 is a general term for input devices (such as buttons, switches, and dials) provided for the user to input various instructions to the imaging device 100. The input devices constituting the operation unit 107 have names corresponding to the assigned functions. For example, the operation unit 107 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, direction keys, a determination key, and the like.

[0021] The release switch is a switch for still image recording, and the system control unit 106 recognizes the half-pressed state of the release switch as an instruction for shooting preparation and the fully pressed state as an instruction for starting shooting. In addition, the system control unit 106 recognizes the pressing of the video recording switch in the shooting standby state as an instruction for starting video recording and the pressing during video recording as an instruction for stopping recording. Note that the functions assigned to the same input device may be variable. Also, the input device may be a software button or key using a touch display. Further, the operation unit 107 may include input devices corresponding to non-contact input methods such as voice input and gaze input.

[0022] The display unit 108 displays an image based on the image data obtained by shooting or the image data read from the recording unit 109. The display unit 108 is, for example, a liquid crystal display or an organic EL display. By displaying the video obtained by shooting while shooting a video on the display unit 108, the display unit 108 can function as an electronic viewfinder (EVF).

[0023] The recording unit 109 is a storage device that stores the recording image data generated by the image processing unit 104. The recording unit 109 may be, for example, a storage device using a non-volatile memory or a magnetic disk. Also, the recording unit 109 may be a storage device using a removable recording medium.

[0024] The memory 111 is, for example, a RAM and is used for the image processing unit 104 and the system control unit 106 to temporarily store various data such as intermediate data. A part of the memory 111 may be used as a video memory.

[0025] The bus 110 is used for communication of data and control signals between the connected blocks.

[0026] FIG. 2 is a diagram representing, in functional blocks, the operations related to image synthesis executed by the image processing unit 104 in the present embodiment. Each pixel data constituting the input image data supplied from the A / D conversion unit 103 to the image processing unit 104 has one color component (R (red), G (green), or B (blue) component) corresponding to the color filter of the primary color Bayer array included in the imaging unit 102.

[0027] The development processing unit 201 generates color image data corresponding to the application from the input image data. The color image data is image data in which each pixel data has color components necessary for representing a color image, such as YUV or RGB.

[0028] FIG. 3 is a block diagram showing a functional configuration example of the development processing unit 201. The input image data is input to the white balance processing unit 301. The white balance processing unit 301 applies a white balance coefficient to the input image data. The white balance coefficient can be calculated by a known method from the input image data, for example. The white balance processing unit 301 outputs the processed image data to the noise reduction processing unit 302.

[0029] The noise reduction processing unit 302 applies noise reduction processing to the image data to reduce the included dark current noise and optical shot noise. The noise reduction processing may be a known processing using, for example, a low-pass filter (LPF) or a bilateral filter. The noise reduction processing unit 302 outputs the processed image data to the demosaicing processing unit 303.

[0030] The demosaicing processing unit 303 applies demosaicing processing to the image data. The demosaicing processing is also called color compensation processing. By complementing the values of the missing color components in each pixel data using, for example, the values of surrounding pixels, each pixel data comes to have the values of three components of R, G, and B. The demosaicing processing can be realized by a known method, for example, by interpolating the values of the surrounding pixels having the values of the missing color components to generate the values of the missing color components. The demosaicing processing unit 303 outputs the processed image data to the color matrix processing unit 304.

[0031] The color matrix processing unit 304 applies color matrix processing to the image data to match the color gamut of the image data to the color gamut of the output device. The color matrix processing can be realized by a known method based on the spectral characteristics of the imaging unit 102 (image sensor) and the color gamut of the device to which the image data is output. The color matrix processing unit 304 outputs the processed image data to the gamma processing unit 305.

[0032] The gamma processing unit 305 applies an opto-electronic transfer function (OETF) to the image data. The OETF applied here is determined according to the electro-optical transfer function (EOTF) of the device (display device) to which the image data is output. Through gamma processing, the value of the image data is converted into a value that can be appropriately displayed on the output device. The image data to which gamma processing has been applied is output to the composition processing unit 202. Returning to FIG. 2, the composition processing unit 202 performs alignment processing on the image data of a plurality of frames generated by the development processing unit 201 and composition processing on the aligned image data of the plurality of frames. For example, the composition processing unit 202 generates composite image data by adding and averaging the image data for a plurality of frames for each corresponding pixel. The composition processing unit 202 outputs the generated composite image data to the local tone mapping processing unit 203.

[0033] The local tone mapping processing unit 203 applies local tone mapping processing to the composite image data. The local tone mapping processing is, for example, a process of changing (brightening here) the brightness of a local area from low brightness to medium brightness.

[0034] Next, a series of operations when the imaging device 100 according to the present embodiment generates a composite image will be described. When the imaging device 100 is set to a mode for shooting a low-illumination scene, such as a night scene shooting mode, or when it is determined through analysis of the captured image that the shooting scene is a low-illumination scene, the imaging device 100 can execute shooting of a plurality of frames assuming composition and generation of image composition. Hereinafter, the process of performing shooting multiple times and synthesizing the obtained image data for a plurality of frames to generate an image of a low-illumination scene is referred to as low-illumination scene shooting processing.

[0035] FIG. 4 is a flowchart regarding the low-illumination scene shooting processing performed by the imaging device 100 in the present embodiment. In S401, when the system control unit 106 detects that a shooting start instruction has been input from the operation unit 107, it instructs the exposure amount control unit 105 to start still image shooting related to the low-illumination scene shooting processing.

[0036] Incidentally, in the shooting standby state, the exposure amount control unit 105 is assumed to sequentially determine shooting conditions for obtaining proper exposure from the evaluation values generated by the image processing unit 104 based on, for example, the moving image data shot for live view display. Further, the exposure amount control unit 105 is assumed to continuously drive the focus lens based on the evaluation values generated by the image processing unit 104 based on, for example, the moving image data shot for live view display.

[0037] In response to an instruction from the system control unit 106, the exposure amount control unit 105 continuously executes the still image shooting process a plurality of times with underexposure. Note that it is assumed that how much underexposure is to be set with respect to the proper exposure and the number of shootings are determined in advance. Here, as an example, it is assumed that shooting is performed three times with underexposure of 1 Ev from the proper exposure. Note that the number of shootings may be dynamically determined according to, for example, the shooting conditions for obtaining proper exposure, the brightness of the scene, or one or more of the magnitudes of the differences between underexposure and proper exposure.

[0038] Further, the difference between underexposure and proper exposure may be dynamically determined based on, for example, the luminance information of the shooting scene, or may be determined by other methods. Further, it may be made possible for the user to arbitrarily specify it.

[0039] By shooting with underexposure, shooting can be performed at a shutter speed faster than when shooting with proper exposure or overexposure. Therefore, the time required for a plurality of shootings can be shortened, and camera shake can be suppressed.

[0040] Each time shooting for one frame is performed, image data is supplied to the image processing unit 104 through the imaging unit 102 and the A / D conversion unit 103.

[0041] In S402, the image processing unit 104 (development processing unit 201) applies demosaicing processing or the like to the image data as described above to generate full-color image data.

[0042] In S403, the image processing unit 104 (composite processing unit 202) applies alignment processing to the full-color image data generated by the development processing unit 201.

[0043] In S404, the image processing unit 104 (composite processing unit 202) synthesizes the image data for three frames aligned in S403 to generate composite image data.

[0044] In S405, the image processing unit 104 (local tone mapping processing unit 203) applies local tone mapping processing to the composite image data generated in S404. The system control unit 106 stores the composite image data output from the image processing unit 104, to which the local tone mapping processing has been applied, in a data file in a predetermined format and records it in the recording unit 109. Note that the system control unit 106 can apply necessary processing such as encoding processing to the composite image data before recording.

[0045] Further, the image processing unit 104 may generate display image data from the generated composite image data, write it to the video memory area of the memory 111, and display the composite image on the display unit 108.

[0046] <Explanation of development processing> Next, the details of the development processing in S402 of FIG. 4 will be described using the flowchart shown in FIG. 5. Here, the processing executed by the development processing unit 201 is called development processing. Each step in FIG. 5 corresponds to the processing performed by each of the function blocks 301 to 305 shown in FIG. 3.

[0047] In S501, the white balance processing unit 301 applies white balance processing for adjusting the color balance according to the color temperature of the ambient light at the time of shooting to the input image data. White balance processing is a process of adjusting the color balance in an image by multiplying each component of R, G, and B by an individual gain as shown in the following formula (1).

[0048] In formula (1), R in , G in, B in are the values of the R, G, and B components before white balance processing. Also, R out , G out , B out are the values of the R, G, and B components after white balance processing. Gain R , Gain G , Gain B are the gains multiplied by the R, G, and B components respectively. R out = Gain R × R in G out = Gain G × G in ··· Equation (1) B out = Gain B × B in

[0049] Note that the gain value can be determined, for example, by estimating the color temperature of the ambient light, extracting a white area from the image according to the color temperature, and using a coefficient value such that the pixels within the white area become achromatic. The white balance processing unit 301 outputs the processed image data to the noise reduction processing unit 302.

[0050] In S502, the noise reduction processing unit 302 applies noise reduction processing for reducing dark current noise and optical shot noise to the input image data. The noise reduction processing can be implemented by a known method using a low-pass filter or a bilateral filter. The noise reduction processing unit 302 outputs the processed image data to the demosaicing processing unit 303.

[0051] In S503, the demosaicing processing unit 303 applies demosaicing processing to the input image data. As described above, by the demosaicing processing, each pixel data constituting the image data comes to have values of three components of R, G, and B. The demosaicing processing unit 303 outputs the processed image data to the color matrix processing unit 304.

[0052] In S504, the color matrix processing unit 304 of the development processing unit 201 applies color matrix processing to the input image data to match the color gamut of the output destination device. Specifically, for each pixel data of the input image data, the color matrix processing unit 304 applies, for example, a matrix composed of coefficients k in to k 33 as shown in the following formula (2) to the color component values R in , G in , and B in . As a result, the color component values of each pixel data are converted into color component values R out , G out , and B out suitable for the color gamut handled by the output destination device. in G in B in to the color component values R 11 ~k 33 constituted by a 3×3 matrix. This converts the color component values of each pixel data into color component values R out , G out , and B out suitable for the color gamut handled by the output destination device. out G out B out

Number

[0053]

[0054] In S505, the gamma processing unit 305 applies an opto-electrical transfer function (OETF) corresponding to the electro-optical transfer function (EOTF) representing the input / output characteristics of the output destination display device to the input image data. The EOTF and OETF are also called a gamma curve or gamma characteristic.

[0055] A specific example of the gamma processing performed by the gamma processing unit 305 will be described. In the present embodiment, the gamma processing unit 305 selectively applies gamma processing using one of a plurality of predetermined OETFs. For example, the gamma processing unit 305 uses either an OETF for prioritizing high-brightness gradations or an OETF for not prioritizing high-brightness gradations.​When giving priority to high-brightness gradation, an EOTF having characteristics that raise the brightness of the middle gradation is applied in the output device. As a result, while maintaining the overall brightness of the finally displayed image equivalent to proper exposure, an effect of suppressing white blooming in the high-brightness part due to imaging with underexposure can be obtained. Note that when giving priority to high-brightness gradation, an OETF that gives priority to high-brightness gradation may be used in the gamma processing unit 305, and an EOTF that does not give priority to high-brightness gradation may be used in the output device.

[0056] Therefore, when executing low-illumination scene shooting processing, the gamma processing unit 305 applies gamma processing using an OETF that gives priority to high-brightness gradation to the image data.

[0057] FIG. 6 shows specific examples of an OETF (high-brightness priority OETF) 602 when giving priority to high-brightness gradation and an OETF (normal OETF) 601 when not giving priority to high-brightness gradation. Here, it is assumed that the image data output from the A / D conversion unit 103 has a gradation value (0 to 4095) of 12 bits per color component. Also, it is assumed that the target gradation value (AE target value) in automatic exposure (AE) is 512. The AE target value is a gradation value obtained when photographing a subject with a specific brightness, specifically an achromatic subject with a reflectance of 18% with proper exposure. That is, in AE processing, shooting conditions are determined such that the gradation value obtained when photographing an achromatic subject with a reflectance of 18% becomes the AE target value.

[0058] Here, in order to record image data with a gradation value (0 to 255) of 8 bits per color component, both OETFs 601 and 602 have input-output characteristics that convert a 12-bit input value into an 8-bit output value. However, the OETFs 601 and 602 may not convert the number of bits, and the conversion of the number of bits may be performed separately.

[0059] Also, the normal OETF 601 (the first gamma curve) is designed such that the AE target value 512 is converted into a gradation value 123 that is the median value of the output gradation range 0 to 255. This is because an achromatic color with a reflectance of 18% is visually recognized as having a brightness in the middle between white and black by the human visual sense.

[0060] Here, in the low-light scene shooting process, shooting is performed with an exposure 1 Ev below the proper exposure. Therefore, the gradation value of an achromatic subject with a reflectance of 18% is 256. And the high-brightness priority OETF602 (the second gamma curve) used in the low-light scene shooting process has a gradation characteristic with a steeper rise in the region with a lower gradation value than the normal OETF601, and is designed such that an input gradation value of 256 is converted to an output gradation value of 123.

[0061] That is, by applying gamma processing using the high-brightness priority OETF602, the brightness of the image obtained by shooting with an exposure 1 Ev below the proper exposure is converted to the same brightness as the image obtained by shooting with the proper exposure.

[0062] However, due to shooting with an exposure below the proper exposure, among the pixels whose gradation values saturate when shooting with the proper exposure, there may be pixels that no longer saturate. That is, by shooting with an exposure below the proper exposure, an effect of suppressing blooming can be obtained.

[0063] In this way, by combining shooting with an exposure below the proper exposure and gamma processing using the high-brightness priority OETF602, it becomes possible to obtain an image in which blooming in the high-brightness part is suppressed while maintaining the overall brightness of the image.

[0064] <Explanation of alignment processing and composition processing> Next, the details of the alignment processing in S403 and the composition processing in S404 in FIG. 4 will be described. First, using the flowchart shown in FIG. 7, the details of the alignment processing in S403 in FIG. 4 will be described.

[0065] In S701, the composition processing unit 202 acquires data of an image (reference image) that serves as a reference for alignment among the image data for a plurality of frames. The data of the reference image may be the image data obtained in the first shooting in the low-light scene shooting process.

[0066] In S702, the composition processing unit 202 acquires data of an image (target image) that is aligned with the reference image. The target image is an image other than the reference image and has not been aligned. When the reference image is the image obtained in the first shooting, the composition processing unit 202 can sequentially acquire the image data obtained in the second and subsequent shootings as the data of the target image.

[0067] In S703, the composition processing unit 202 obtains the amount of misalignment between the reference image and the target image. The composition processing unit 202 can obtain the motion vector of the target image with respect to the reference image as the amount of misalignment. For example, the composition processing unit 202 divides the reference image into a plurality of blocks of the same size, and performs template matching with each block as a template, thereby searching for the region in the target image that is most similar to the template. The similarity between images can be calculated as the sum of absolute differences (SAD), sum of squared differences (SSD), or normalized cross-correlation (NCC) of the values of corresponding pixels. The search range may be the entire target image or a part thereof.

[0068] Then, the composition processing unit 202 detects the difference between the position of the searched region and the position of the block used as a template in the reference image as the motion vector. The composition processing unit 202 obtains the motion vector of the entire target image with respect to the reference image from the motion vectors detected for each block, and uses it as the amount of misalignment.

[0069] In S704, the composition processing unit 202 calculates a conversion coefficient from the amount of misalignment between the reference image and the target image. The composition processing unit 202 calculates, for example, a projective conversion coefficient as the conversion coefficient. An affine conversion coefficient or a simpler horizontal and vertical shift amount may be calculated as the conversion coefficient.

[0070] In S705, the composition processing unit 202 converts the target image using the conversion coefficient calculated in S704. For example, the composition processing unit 202 can deform the target image according to the following formula (3).

Equation

[0071] In Equation (3), (x, y) represents the coordinates before transformation, and (x´, y´) represents the coordinates after transformation. Matrix A represents the transformation coefficients calculated by the synthesis processing unit 202 in S704. The synthesis processing unit 202 stores the data of the target image after transformation in, for example, the memory 111 as the image data that has undergone alignment processing.

[0072] In S706, the synthesis processing unit 202 determines whether alignment processing has been applied to all images other than the reference image. If the synthesis processing unit 202 determines that there are remaining images to which alignment processing has not been applied, it repeatedly executes the processing from S702. If the synthesis processing unit 202 determines that alignment processing has been applied to all images other than the reference image, it ends the processing shown in the flowchart of FIG. 7.

[0073] Next, the synthesis processing performed by the synthesis processing unit 202 in S404 of FIG. 4 will be described. The synthesis processing unit 202 sequentially acquires the aligned image data from the memory 111 and synthesizes it with the data of the reference image. The synthesis processing unit 202 generates the data of the synthesized image, for example, by adding and averaging the pixel data of the corresponding coordinates. Adding and averaging can reduce the random noise superimposed on the image. Although the data of the synthesized image may be generated by a method other than adding and averaging, it is desirable to use a method that reduces random noise.

[0074] <Explanation of Local Tone Mapping Processing> Next, the local tone mapping processing performed by the local tone mapping processing unit 203 in S405 of FIG. 4 will be described. Note that by converting the image obtained by shooting with underexposure to an appropriate exposure equivalent brightness by gamma processing and synthesizing it, white blowout (saturation) in the high-brightness part can be suppressed, and an image with an expanded dynamic range can be obtained. Therefore, local tone mapping processing is not essential. However, by applying local tone mapping processing, a further expansion of the dynamic range can be realized.

[0075] The local tone mapping process in this embodiment is a process of locally brightening the region from low brightness to medium brightness. By applying such local tone mapping processing, it is possible to improve the gradation of the dark region while maintaining the gradation of the high brightness region.

[0076] In addition, in the imaging to development process described above, when imaging is performed with an even more underexposure than 1 Ev and the same development process is performed on the input image, an underexposed image with less deterioration in the gradation of the higher brightness portion can be generated.

[0077] The local tone mapping process can be realized by a known method that generates a gain map whose tone characteristics change locally using discrimination results of images or regions in different frequency bands, and applies tone conversion processing while referring to the gain map.

[0078] FIG. 8(a) shows an example of the gain map used in local tone mapping. The gain map is similar to the gamma characteristic in that it represents the characteristics of tone conversion, and the local tone mapping process can also be said to be local gamma processing.

[0079] The gain map 803 is a straight line with a slope of 1 passing through the origin. The gain map 803 has the characteristic of not changing the tone value. On the other hand, the gain maps 801 and 802 have tone characteristics that correct the input tone values from low brightness to medium brightness to be brighter. The correction amount of the gain map 801 is larger than that of the gain map 802. In this embodiment, the gain map used for the local tone mapping process is generated by weighted addition of the gain maps 801 and 802 with different correction amounts.

[0080] Specifically, the local tone mapping processing unit 203 generates a histogram of the input composite image data. FIG. 8(b) shows an example of the histogram. Then, the local tone mapping processing unit 203 calculates the ratio of the low-luminance to medium-luminance regions in the image by summing the frequencies below the threshold value. The threshold value TH may be, for example, the tone value in the middle of the tone range of the input signal.

[0081] Next, the local tone mapping processing unit 203 uses the calculated ratio to obtain the coefficient α from a preset table as shown in FIG. 8(c). The coefficient α is the weight of the gain map 801, and 0 ≤ α ≤ 1. In the example shown in FIG. 8(c), as the ratio of the low-luminance to medium-luminance regions in the image increases, the coefficient α is determined such that the brightness correction amount for the low-luminance to medium-luminance regions decreases. This is to suppress the influence of local tone mapping on the overall brightness of the image when the proportion of the region to be brightness-corrected in the image is large. Note that the table for obtaining the coefficient α may be changed according to the purpose or intention of the low-illumination scene shooting process. For example, a table of the coefficient α may be prepared when emphasizing the improvement effect of the gradation of the dark region rather than suppressing the influence of local tone mapping on the overall brightness of the image. In this case, the coefficient α can be determined such that as the ratio of the low-luminance to medium-luminance regions occupying the screen increases, the brightness correction amount for the low-luminance to medium-luminance regions also increases.

[0082] Then, the local tone mapping processing unit 203 calculates the gain map to be applied to the composite image data by weighted addition of the gain maps 801 and 802 using the coefficient α. Let the tone value of the input pixel be x and the tone value after conversion be y. If the gain map 801 is y = tm_a(x) and the gain map 802 is y = tm_b(x), the gain map tm(x) used for local tone mapping is obtained by the following formula (4). y = tm(x) = α × tm_a(x) + (1 - α) × tm_b(x) ··· Formula (4)

[0083] The local tone mapping processing unit 203 applies the gain map tm(x) obtained by the formula (4) to the pixel values of the composite image data. The gradation value of the input pixel is p, and the output gradation value p_out after applying the local tone mapping processing is represented by the formula (5). p_out = tm(p) ··· Formula (5)

[0084] By such local tone mapping processing, the black crush in the region from low brightness to medium brightness is suppressed, and the gradation property is improved. Therefore, the dynamic range in the low brightness direction can be further expanded.

[0085] According to the present embodiment, when photographing a low illumination scene, underexposure is used to perform multiple shootings, and the images for a plurality of frames obtained are synthesized. Therefore, it is possible to shoot at a shutter speed faster than when shooting with proper exposure or overexposure, and camera shake and subject blur can be suppressed. In addition, the time required for multiple shootings can be shortened. Further, blooming in the high brightness portion can be suppressed.

[0086] Also, by performing gamma processing on the image photographed with underexposure, the brightness of the entire image is corrected to be the same as that of the image obtained by shooting with proper exposure and then synthesized. Therefore, it is possible to obtain a composite image having an appropriate brightness while suppressing blooming in the high brightness portion. Further, it is possible to reduce random noise during synthesis.

[0087] In addition, by applying local tone mapping processing to the low brightness and medium brightness regions of the composite image, black crush can be suppressed, and the dynamic range of the composite image can be further expanded in the low brightness direction.

[0088] ●(Second Embodiment) Next, a second embodiment of the present invention will be described. FIG. 9 is a flowchart regarding the low-illumination scene shooting process performed by the imaging device 100 in this embodiment. In FIG. 9, the same reference numerals as those in FIG. 4 are given to the steps that perform the same processing as in the first embodiment. In this embodiment, the timing of performing the local tone mapping process is different from that in the first embodiment.

[0089] Specifically, the development processing unit 201 outputs image data to the local tone mapping processing unit 203. Then, the local tone mapping processing unit 203 applies the local tone mapping process to the image data of each frame before the alignment process. The local tone mapping process may be the same as that in the first embodiment, except that the pre-synthesis image is used instead of the synthesized image.

[0090] The local tone mapping processing unit 203 outputs the processed image data to the synthesis processing unit 202. The synthesis processing unit 202 aligns and synthesizes the locally tone-mapped images and outputs them from the image processing unit 104. In FIG. 2, the flow of the image data in the image processing unit 104 in this embodiment is indicated by a dotted line.

[0091] Also in this embodiment, the same effects as those in the first embodiment can be obtained. Further, even when the movement between images is large and alignment and synthesis cannot be performed, it is possible to obtain image data having the effects realized by the processing up to the local tone mapping process.

[0092] (Other Embodiments) Note that the low-illumination scene shooting process described in the above embodiment may be performed during shooting of scenes other than low-illumination scenes. For example, it can be performed to generate an image with an expanded dynamic range regardless of the shooting scene.

[0093] In addition, among the low-illumination scene shooting processes described in the above embodiments, processes other than those related to shooting may be implemented by an electronic device (image processing device) that does not have an imaging function. That is, the image processing device can acquire image data for a plurality of frames shot with underexposure from an external device and apply the processes after S402. Even with such an image processing device, it is possible to obtain a composite image having an appropriate brightness while suppressing white blooming in the high-brightness portion. Furthermore, it is possible to reduce random noise during composition.

[0094] Also, in the first embodiment, when the deviation amount cannot be detected in S703 or the deviation amount is equal to or greater than the threshold value, it may be determined that the alignment process cannot be performed normally, and the local tone mapping process may be applied to the image data of each frame. Thereby, the same effect as in the second embodiment can be obtained.

[0095] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0096] The disclosure of this embodiment includes the following imaging device, image processing device, control method of the imaging device, and program. (Item 1) Imaging means for performing shooting a plurality of times with underexposure and acquiring image data for a plurality of frames, Correction means for correcting the brightness of the image data for the plurality of frames, Generation means for generating composite image data by synthesizing the image data for the plurality of frames corrected by the correction means, An imaging device having (Item 2) The imaging device according to item 1, wherein the correction means corrects the brightness of the image data by applying a gamma curve having a gradation characteristic according to the magnitude of the difference between the appropriate exposure and the under-exposure to the image data. (Item 3) The imaging device according to item 1, wherein the correction means corrects the brightness of the image data obtained by photographing with under-exposure by applying a second gamma curve having a gradation characteristic with a steeper rise in a region where the gradation value is lower than that of a first gamma curve applied to the image data obtained by photographing with appropriate exposure. (Item 4) The imaging device according to item 3, wherein the first gamma curve and the second gamma curve have a gradation characteristic of converting gradation values corresponding to a specific subject into the same gradation value. (Item 5) The imaging device according to item 4, wherein the specific subject is an achromatic subject having a reflectance of 18%. (Item 6) The imaging device according to any one of items 1 to 5, further comprising processing means for applying local tone mapping processing for brightening data having a gradation value equal to or less than a threshold value among the data of the composite image. (Item 7) The imaging device according to any one of items 1 to 5, further comprising processing means for applying local tone mapping processing for brightening data having a gradation value equal to or less than a threshold value to the image data for the plurality of frames corrected by the correction means before the composition. (Item 8) The imaging device according to item 7, wherein the processing means applies the local tone mapping processing when the composition cannot be performed. (Item 9) The imaging device according to any one of items 6 to 8, wherein the threshold value is determined based on a histogram of gradation values of the image data to which the local tone mapping processing is applied. (Item 10) Acquisition means for acquiring image data for a plurality of frames obtained by photographing with under-exposure, Correction means for correcting the brightness of the image data for the plurality of frames, A generating means for generating data of a composite image by synthesizing the image data of the plurality of frames corrected by the correction means; An image processing apparatus characterized by having the same. (Item 11) A control method executed by an imaging apparatus, comprising: Performing multiple shootings with underexposure to obtain image data for a plurality of frames; Correcting the brightness of the image data for the plurality of frames; Generating data of a composite image by synthesizing the corrected image data for the plurality of frames; A control method for an imaging apparatus having the same. (Item 12) A program for causing a computer included in an imaging apparatus to function as each means included in the imaging apparatus according to any one of Items 1 to 9. (Item 13) A program for causing a computer to function as each means included in the image processing apparatus according to Item 10.

[0097] The present invention is not limited to the contents of the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Therefore, claims are attached to disclose the scope of the invention.

Explanation of Signs

[0098] 100... Imaging apparatus, 102... Imaging unit, 104... Image processing unit, 106... System control unit, 201... Development processing unit, 202... Image synthesis unit, 203... Local tone mapping processing unit

Claims

1. Imaging means for performing multiple shootings with underexposure and acquiring image data for a plurality of frames; Correction means for correcting the brightness of the image data for the plurality of frames; Generation means for generating composite image data by synthesizing the image data for the plurality of frames corrected by the correction means; An imaging device having the above.

2. The imaging device according to claim 1, wherein the correction means corrects the brightness of the image data by applying a gamma curve having a gradation characteristic according to the magnitude of the difference between proper exposure and the underexposure to the image data.

3. The imaging device according to claim 1, wherein the correction means applies a second gamma curve having a gradation characteristic with a steeper rise in a region with a lower gradation value than a first gamma curve applied to image data obtained by shooting with proper exposure, thereby correcting the brightness of the image data obtained by shooting with underexposure.

4. The imaging device according to claim 3, wherein the first gamma curve and the second gamma curve have a gradation characteristic of converting gradation values corresponding to a specific subject into the same gradation value.

5. The imaging device according to claim 4, wherein the specific subject is an achromatic subject with a reflectance of 18%.

6. The imaging device according to any one of claims 1 to 5, further comprising processing means for applying local tone mapping processing for brightening data with a gradation value equal to or lower than a threshold value among the data of the composite image.

7. The imaging device according to any one of claims 1 to 5, further comprising processing means for applying local tone mapping processing for brightening data with a gradation value equal to or lower than a threshold value to the image data for the plurality of frames corrected by the correction means before the synthesis.

8. The imaging device according to claim 7, wherein the processing means applies the local tone mapping processing when the synthesis cannot be performed.

9. The imaging device according to claim 6, wherein the threshold value is determined based on a histogram of the gradation values of the image data to which the local tone mapping processing is applied.

10. The imaging device according to claim 7, wherein the threshold value is determined based on a histogram of the gradation values of the image data to which the local tone mapping processing is applied.

11. Acquisition means for acquiring image data for a plurality of frames obtained by shooting with underexposure; Correction means for correcting the brightness of the image data for the plurality of frames; A generating means for generating data of a composite image by synthesizing the image data for the plurality of frames corrected by the correction means; An image processing apparatus characterized by comprising the same. **Claim 12** A control method executed by an imaging apparatus, comprising: Performing imaging a plurality of times with underexposure to obtain image data for a plurality of frames; Correcting the brightness of the image data for the plurality of frames; Generating data of a composite image by synthesizing the image data for the plurality of frames that have been corrected; A control method for an imaging apparatus, comprising the above. **Claim 13** A program for causing a computer included in an imaging apparatus to function as each means included in the imaging apparatus according to Claim 1. **Claim 14** A program for causing a computer to function as each means included in the image processing apparatus according to Claim 11.

Images