Imaging device, control method thereof, and image processing device

Abstract

To provide an imaging apparatus and a control method for the same that can more easily capture a low-light scene including a high-luminance subject.SOLUTION: An imaging apparatus performs imaging a plurality of times with underexposure and acquires image data for a plurality of frames. Further, the imaging apparatus corrects the brightness of the plurality of frames of image data and then combines the image data to generate data for a composite image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an imaging apparatus, a control method thereof, and an image processing apparatus, and more particularly to an image synthesis technique. [Background technology]

[0002] When a scene with a wide dynamic range of brightness, such as a low-light scene (for example, a night scene) containing a high-brightness subject, is photographed with proper exposure, the high-brightness subject is likely to be blown out. For this reason, a technique is known for generating an image with a wide dynamic range by synthesizing images photographed with underexposure, proper exposure, and overexposure of the same scene (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-60578 A Summary of the Invention [Problem to be solved by the invention]

[0004] When shooting the same scene with different exposures, the exposure time is long especially in low-light scenes, when shooting with proper exposure or overexposure, which makes it easy for camera shake to occur. Therefore, it is necessary to fix the imaging device on a tripod or the like, which is inconvenient to use.

[0005] In one aspect, the present invention provides an imaging apparatus capable of more easily capturing an image of a low-illumination scene including a high-brightness subject, and a control method thereof. [Means for solving the problem]

[0006] The above object is to provide an imaging means for capturing images multiple times underexposed to obtain image data for multiple frames, and a correction means for correcting the brightness of the image data for multiple frames captured underexposed to the same brightness as when the image data was captured underexposed by applying a second gamma curve having gradation characteristics with a steeper rise in the low gradation value range than a first gamma curve applied to image data captured under proper exposure, A processing means for applying a local tone mapping process to the image data for a plurality of frames corrected by applying the second gamma curve by the correction means, the processing means brightening data having a gradation value equal to or less than a threshold value; and generating means for averaging the image data of a plurality of frames obtained by the image capturing apparatus to generate data of a composite image. Effect of the Invention

[0007] According to the present invention, it is possible to provide an imaging apparatus capable of more easily capturing an image of a low-illumination scene including a high-brightness subject, and a control method thereof. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of a functional configuration of an imaging apparatus according to an embodiment; [Diagram 2] A block diagram showing an example of the functional configuration of the image processing unit in FIG. 1. [Diagram 3] FIG. 3 is a block diagram showing an example of the functional configuration of the development processing unit in FIG. 2; [Figure 4] 1 is a flowchart illustrating an example of low-illumination scene shooting processing in the first embodiment. [Diagram 5] Flowchart showing an example of the development process performed in S402 of FIG. [Figure 6] FIG. 1 is a diagram showing an example of a gamma curve used in an embodiment. [Figure 7] Flowchart showing an example of the development process performed in S403 of FIG. [Figure 8] FIG. 13 is a diagram showing an example of a gain map used in local tone mapping processing. [Figure 9] 11 is a flowchart illustrating an example of low-illumination scene shooting processing in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0010] The configurations represented as blocks in the drawings may be realized by integrated circuits (ICs) such as ASICs or FPGAs, by discrete circuits, or by a combination of a memory and a processor that executes a program stored in the memory. Also, one block may be realized by multiple integrated circuit packages, or multiple blocks may be realized by one integrated circuit package. Also, the same block may be implemented in different configurations depending on the operating environment, required capabilities, etc.

[0011] In the following, the present invention will be described in terms of an embodiment in an imaging device such as a digital camera. However, the present invention can also be implemented in any electronic device having an imaging function. Such electronic devices include, in addition to imaging devices, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, drive recorders, etc. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0012] ●(First embodiment) Fig. 1 is a block diagram showing an example of the functional configuration of an imaging device 100 according to an embodiment of the present invention. Fig. 1 shows only representative components of the imaging device 100. The optical system 101 has a plurality of lenses, an aperture that also serves as a shutter, an aperture drive mechanism, and the like, and forms an optical image of a subject. The plurality of lenses include movable lenses such as a focus lens that adjusts the focal distance of the optical system 101 and a zoom lens that changes the angle of view. The optical system 101 also has a drive mechanism for the movable lenses.

[0013] The imaging unit 102 may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The imaging unit 102 has a pixel array in which a plurality of pixels are arranged two-dimensionally, and a peripheral circuit for reading out signals from each pixel. Each pixel accumulates electric charge according to the amount of incident light by photoelectric conversion. A group of pixel signals (analog image signals) representing the subject image formed by the optical system 101 is obtained by reading out from each pixel a signal having a voltage according to the amount of electric charge accumulated during the exposure period.

[0014] The A / D conversion unit 103 converts the analog image signal read out from the imaging unit 102 into a digital image signal (image data) by A / D conversion. Note that if the imaging unit 102 has an A / D conversion function, the A / D conversion unit 103 is not necessary.

[0015] The image processing unit 104 applies predetermined image processing to image data output by the A / D conversion unit 103 and image data read from the recording unit 109, generates signals and image data according to the purpose, and acquires and / or generates various information. The image processing unit 104 may be a dedicated hardware circuit such as an ASIC (Application Specific Integrated Circuit) designed to realize a specific function. Alternatively, the image processing unit 104 may be configured to realize a specific function by a processor such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit) executing software.

[0016] Image processing that the image processing unit 104 can apply to image data can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation processing is performed when a color filter is provided in the imaging unit 102, and is processing for interpolating values ​​of color components that are not included in the individual pixel data that constitutes the image data. Color interpolation processing is also called demosaic processing. The correction processing can include white balance adjustment, gradation correction, correction of image degradation caused by optical aberration of the optical system 101 (image restoration), correction of the effect of peripheral light falloff of the optical system 101, color correction, and the like. The detection process may include detection of feature regions (for example, face regions or human body regions) and their movements, person recognition processing, and the like. Data processing may include processes such as area extraction (trimming), synthesis, scaling, encoding and decoding, header information generation (data file generation), etc. Data processing also includes the generation of image data for display or image data for recording. The evaluation value calculation process can include processes such as generation of a signal and evaluation value used in automatic focus detection (AF) and generation of an evaluation value used in automatic exposure control (AE). The special effects processing may include adding a blur effect, changing color tones, relighting, and the like. Note that these are merely examples of processes that the image processing unit 104 can apply to image data, and do not limit the processes that the image processing unit 104 can apply.

[0017] The exposure control unit 105 determines the shooting conditions (aperture value, shutter speed, ISO sensitivity) based on the evaluation value calculated by the image processing unit 104 and a predetermined program diagram. The exposure control unit 105 also controls the operation of the optical system 101 (aperture) and the imaging unit 102 based on the determined shooting conditions during shooting. The exposure control unit 105 also drives the optical system 101 (focus lens) based on the evaluation value calculated by the image processing unit 104, and adjusts the focal distance of the optical system 101.

[0018] The system control unit 106 includes, for example, a processor (such as a CPU, MPU, or microprocessor) capable of executing a program, a ROM, and a RAM. The system control unit 106 loads a program stored in the ROM into the RAM and executes the program, thereby controlling the operation of each unit of the imaging device 100 and realizing the functions of the imaging device 100.

[0019] The ROM is rewritable and stores programs executed by the processor, various setting values ​​of the imaging device 100, GUI data, etc. The RAM is a main memory used by the system control unit 106 when executing programs.

[0020] Operation unit 107 is a general term for input devices (buttons, switches, dials, etc.) provided for a user to input various instructions to imaging device 100. The input devices constituting operation unit 107 have names according to the functions assigned to them. For example, operation unit 107 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, menu buttons, directional keys, an enter key, etc.

[0021] The release switch is a switch for recording still images, and the system control unit 106 recognizes the half-pressed state of the release switch as an instruction to prepare for shooting, and the full-pressed state as an instruction to start shooting. The system control unit 106 also recognizes the video recording switch pressed in a shooting standby state as an instruction to start recording a video, and recognizes the switch pressed during video recording as an instruction to stop recording. Note that the functions assigned to the same input device may be variable. The input device may be a software button or key using a touch display. The operation unit 107 may also include an input device compatible with a non-contact input method, such as voice input or gaze input.

[0022] The display unit 108 displays image data obtained by shooting and images based on image data read from the recording unit 109. The display unit 108 is, for example, a liquid crystal display or an organic EL display. By displaying the moving image obtained by shooting on the display unit 108 while shooting a moving image, the display unit 108 can function as an electronic viewfinder (EVF).

[0023] The recording unit 109 is a storage device that stores image data for recording generated by the image processing unit 104. The recording unit 109 may be, for example, a storage device using a non-volatile memory or a magnetic disk. The recording unit 109 may also be a storage device using a removable recording medium.

[0024] The memory 111 is, for example, a RAM, and is used to temporarily store various data such as intermediate data by the image processing unit 104 and the system control unit 106. A part of the memory 111 may be used as a video memory.

[0025] The bus 110 is used for communicating data and control signals between the connected blocks.

[0026] 2 is a functional block diagram showing the operation of the image synthesis performed by the image processing unit 104 in this embodiment. The individual pixel data constituting the input image data supplied from the A / D conversion unit 103 to the image processing unit 104 is102 The pixel has one color component (R (red), G (green), or B (blue) component) corresponding to the color filter of the primary color Bayer array.

[0027] The development processing unit 201 generates color image data according to the purpose from the input image data. The color image data is image data in which each pixel data has color components necessary to express a color image, such as YUV or RGB.

[0028] FIG. 3 is a block diagram showing an example of the functional configuration of the development processing unit 201. As shown in FIG. Input image data is input to a white balance processing unit 301. The white balance processing unit 301 applies a white balance coefficient to the input image data. The white balance coefficient can be calculated from the input image data by a known method, for example. The white balance processing unit 301 outputs the processed image data to a noise reduction processing unit 302.

[0029] The noise reduction processing unit 302 applies noise reduction processing to the image data to reduce dark current noise and optical shot noise contained therein. The noise reduction processing may be a known processing using, for example, a low pass filter (LPF) or a bilateral filter. The noise reduction processing unit 302 outputs the processed image data to the demosaic processing unit 303.

[0030] The demosaic processing unit 303 applies demosaic processing to the image data. Demosaic processing is also called color complement processing. The values ​​of color components missing from each pixel data are complemented, for example, by using the values ​​of surrounding pixels, so that each pixel data has values ​​of the three components R, G, and B. The demosaic processing can be realized using a known method, for example, by generating the value of a missing color component by interpolating the values ​​of surrounding pixels having the value of that color component. The demosaic processing unit 303 outputs the processed image data to the color matrix processing unit 304.

[0031] The color matrix processing unit 304 applies color matrix processing to the image data to match the color gamut of the image data to the color gamut of the output device. 102 This can be achieved by a known method based on the spectral characteristics of the (imaging element) and the color gamut of the device to which the image data is output. The color matrix processing unit 304 outputs the processed image data to the gamma processing unit 305.

[0032] The gamma processing unit 305 applies an optical-electrical transfer function (OETF) to the image data. The OETF applied here is determined according to the electro-optical transfer function (EOTF) of the device (display device) to which the image data is output. The gamma processing converts the values ​​of the image data into values ​​that can be appropriately displayed on the output device. The image data to which the gamma processing has been applied is output to the synthesis processing unit 202. 2, the synthesis processing unit 202 performs alignment processing of the image data of the multiple frames generated by the development processing unit 201, and synthesis processing of the aligned image data of the multiple frames. The synthesis processing unit 202 generates synthetic image data, for example, by averaging the image data of the multiple frames for each corresponding pixel. The synthesis processing unit 202 outputs the generated synthetic image data to the local tone mapping processing unit 203.

[0033] The local tone mapping processor 203 applies local tone mapping to the composite image data. The local tone mapping process is a process for changing (here, brightening) the brightness of a local area from low brightness to medium brightness, for example.

[0034] Next, a series of operations performed by the imaging device 100 according to this embodiment when generating a composite image will be described. When the imaging device 100 is set to a mode for capturing low-illumination scenes, such as a night scene mode, or when the captured image is analyzed to determine that the captured scene is a low-illumination scene, the imaging device 100 can capture multiple frames and generate composite images on the premise of compositing. Hereinafter, the process of capturing multiple images and compositing the image data of the multiple frames to generate an image of a low-illumination scene will be referred to as low-illumination scene capture processing.

[0035] FIG. 4 is a flowchart showing the low-illumination scene shooting process performed by the image capture device 100 in this embodiment. In S401, when the system control unit 106 detects that a shooting start instruction has been input from the operation unit 107, it instructs the exposure control unit 105 to start still image shooting related to low illuminance scene shooting processing.

[0036] It is assumed that in a shooting standby state, the exposure control unit 105 sequentially determines shooting conditions for obtaining proper exposure from an evaluation value generated by the image processing unit 104 based on moving image data shot for live view display, for example. It is also assumed that the exposure control unit 105 continuously drives the focus lens based on an evaluation value generated by the image processing unit 104 based on moving image data shot for live view display, for example.

[0037] In response to an instruction from the system control unit 106, the exposure control unit 105 executes still image shooting processing with underexposure multiple times in succession. It is assumed that the degree of underexposure with respect to the proper exposure and the number of times of shooting are determined in advance. As an example, it is assumed that shooting is performed three times with 1 Ev underexposure from the proper exposure. It is also possible to dynamically determine the number of times of shooting depending on, for example, one or more of the shooting conditions for obtaining the proper exposure, the brightness of the scene, and the magnitude of the difference between the underexposure and the proper exposure.

[0038] The difference between the underexposure and the appropriate exposure may be dynamically determined based on the luminance information of the shooting scene, or may be determined by other methods. Also, the user may be able to specify the difference arbitrarily.

[0039] By shooting with underexposure, it is possible to shoot with a faster shutter speed than when shooting with proper exposure or overexposure, which reduces the time required to shoot multiple times and suppresses camera shake.

[0040] Every time one frame is captured, image data is supplied to an image processing unit 104 via an imaging unit 102 and an A / D conversion unit 103 .

[0041] In S402, the image processing unit 104 (developing unit 201) applies demosaic processing and the like to the image data as described above to generate full-color image data.

[0042] In S403 , the image processing unit 104 (the synthesis processing unit 202 ) applies a position adjustment process to the full-color image data generated by the development processing unit 201 .

[0043] In S404, the image processing unit 104 (the synthesis processing unit 202) synthesizes the image data for the three frames aligned in S403 to generate synthetic image data.

[0044] In S405, the image processing unit 104 (local tone mapping processing unit 203) applies local tone mapping processing to the composite image data generated in S404. The system control unit 106 stores the composite image data to which the local tone mapping processing has been applied and which has been output from the image processing unit 104 in a data file of a predetermined format, and records the data in the recording unit 109. Note that the system control unit 106 can apply necessary processing, such as encoding processing, to the composite image data before recording.

[0045] Furthermore, the image processing unit 104 generates image data for display from the generated composite image data, and writes the image data to a video memory area of ​​the memory 111, thereby displaying the composite image on the display unit 108. difference This may be allowed.

[0046] <Development process explanation> Next, the development process in S402 in Fig. 4 will be described in detail with reference to the flowchart shown in Fig. 5. Here, the process executed by the development processing unit 201 is called the development process. Each step in Fig. 5 corresponds to the process executed by each of the functional blocks 301 to 305 shown in Fig. 3.

[0047] In S501, the white balance processor 301 applies white balance processing to the input image data to adjust the color balance according to the color temperature of the ambient light at the time of shooting. The white balance processing is a process that adjusts the color balance in the image by multiplying each of the R, G, and B components by an individual gain, as shown in the following formula (1).

[0048] In formula (1), R in , G in , B in are the R, G, and B component values ​​before white balance processing. out , G out , B out are the R, G, and B component values ​​after white balance processing. Gain R , Gain G , Gain B are the gains multiplied to the R, G, and B components, respectively. R out =Gain R ×R in G out =Gain G ×G in ...Equation (1) B out =Gain B ×B in

[0049] The gain value can be determined, for example, by estimating the color temperature of the ambient light, extracting a white area corresponding to the color temperature from the image, and setting a coefficient value such that the pixels in the white area become achromatic. The white balance processing unit 301 outputs the processed image data to the noise reduction processing unit 302.

[0050] In S502, the noise reduction processing unit 302 applies noise reduction processing to the input image data to reduce dark current noise and optical shot noise. The noise reduction processing can be performed by a known method using a low-pass filter or a bilateral filter. The noise reduction processing unit 302 outputs the processed image data to the demosaic processing unit 303.

[0051] In S503, the demosaic processing unit 303 applies demosaic processing to the input image data. As described above, through demosaic processing, each pixel data constituting the image data has values ​​of the three components R, G, and B. The demosaic processing unit 303 outputs the processed image data to the color matrix processing unit 304.

[0052] In S504, the development processing section 201 The color matrix processing unit 304 applies color matrix processing to the input image data to match the color gamut of the output device. Specifically, the color matrix processing unit 304 calculates the color component value R in , G in , B in 3×3 coefficient k 11 ~k 33 This applies a matrix composed of the following: The color component values ​​of each pixel data are converted to color component values ​​R suitable for the color gamut handled by the output device. out , G out , B out Convert to.

number

[0053] In S505, the gamma processor 305 applies an optical-electrical transfer function (OETF) corresponding to an electro-optical transfer function (EOTF) that represents the input / output characteristics of the display device at the output destination to the input image data. The EOTF and OETF are also called gamma curves or gamma characteristics.

[0054] A specific example of gamma processing performed by the gamma processing unit 305 will be described. In this embodiment, the gamma processing unit 305 selectively uses one of a plurality of predetermined OETFs to apply gamma processing. For example, the gamma processing unit 305 uses either an OETF for a case where high luminance gradation is prioritized or an OETF for a case where high luminance gradation is not prioritized.

[0055] When high luminance tones are prioritized, an EOTF with a characteristic of raising the brightness of intermediate tones is applied in the output device. This makes it possible to suppress blown-out highlights in high-luminance areas caused by capturing images with underexposure while maintaining the overall brightness of the image finally displayed at the same level as with the correct exposure. Note that when high luminance tones are prioritized, an OETF that prioritizes high luminance tones may be used in the gamma processing unit 305, and an EOTF that does not prioritize high luminance tones may be used in the output device.

[0056] Therefore, when performing low-illuminance scene shooting processing, the gamma processor 305 applies gamma processing using an OETF that prioritizes high luminance tones to the image data.

[0057] FIG. 6 shows a specific example of an OETF (high luminance priority OETF) 602 when high luminance gradation is prioritized, and an OETF (normal OETF) 601 when high luminance gradation is not prioritized. Here, it is assumed that the image data output by the A / D conversion unit 103 has a 12-bit gradation value (0 to 4095) per color component. Also, it is assumed that the target gradation value (AE target value) in automatic exposure (AE) is 512. The AE target value is a gradation value obtained when an object of a specific brightness, specifically, an achromatic object with a reflectance of 18%, is photographed with proper exposure. That is, in the AE processing, a photographing condition is determined such that the gradation value obtained when an achromatic object with a reflectance of 18% is photographed becomes the AE target value.

[0058] Here, to record image data with 8-bit gradation values ​​(0 to 255) per color component, both OETFs 601 and 602 have input / output characteristics that convert 12-bit input values ​​to 8-bit output values. However, OETFs 601 and 602 do not convert the number of bits, and bit conversion may be performed separately.

[0059] Furthermore, the OETF 601 (first gamma curve) is usually designed so that the AE target value 512 is converted to a gradation value of 123, which is the median value of the output gradation range of 0 to 255. This is because an achromatic color with a reflectance of 18% is perceived by human vision as having a brightness halfway between black and white.

[0060] Here, the low light In the low light scene shooting process, the image is shot at 1 Ev underexposure from the correct exposure, so the gradation value of an achromatic subject with a reflectance of 18% is 256. light The high-luminance-priority OETF 602 (second gamma curve) used in the high-luminance scene shooting process has a gradation characteristic with a steeper rise in the low gradation value range than the normal OETF 601, and is designed to convert an input gradation value of 256 to an output gradation value of 123.

[0061] In other words, by applying gamma processing using the high-luminance-priority OETF 602, the brightness of an image captured with 1 Ev underexposure is converted to the same brightness as an image captured with proper exposure.

[0062] However, by shooting with underexposure, it is possible that some pixels that would be saturated in gradation values ​​if shot with proper exposure are not saturated. In other words, shooting with underexposure has the effect of suppressing blown-out highlights.

[0063] In this way, by combining underexposure shooting with gamma processing using the high-brightness-priority OETF 602, it is possible to obtain an image in which blown-out highlights in high-brightness areas are suppressed while maintaining the overall brightness of the image.

[0064] <Description of alignment and composition processing> Next, a description will be given of the details of the alignment process in S403 and the synthesis process in S404 in Fig. 4. First, a description will be given of the details of the alignment process in S403 in Fig. 4 with reference to the flowchart shown in Fig. 7.

[0065] In S701, the synthesis processing unit 202 acquires data of an image (reference image) that serves as a reference for alignment from among image data for a plurality of frames. The data of the reference image may be image data obtained in the first shooting in the low-illumination scene shooting process.

[0066] In S702, the synthesis processing unit 202 acquires data of an image (target image) to be aligned with the reference image. The target image is an image other than the reference image and has not been aligned. If the reference image is an image obtained by the first shooting, the synthesis processing unit 202 can sequentially acquire image data obtained by the second and subsequent shootings as data of the target image.

[0067] In S703, the synthesis processing unit 202 calculates the amount of positional deviation between the reference image and the target image. The synthesis processing unit 202 can calculate the amount of positional deviation as a motion vector of the target image relative to the reference image. For example, the synthesis processing unit 202 divides the reference image into a plurality of blocks of the same size, and performs template matching using each block as a template to search for an area in the target image that is most similar to the template. The similarity between images can be calculated as the sum of absolute differences (SAD), sum of squared differences (SSD), or normalized cross-correlation (NCC) of corresponding pixel values. The search range may be the entire target image, or a part of it.

[0068] The synthesis processor 202 then detects the difference between the position of the searched region and the position of the block used as the template in the reference image as a motion vector. The synthesis processor 202 calculates the motion vector of the entire target image relative to the reference image from the motion vector detected for each block, and regards this as the amount of positional deviation.

[0069] In S704, the synthesis processing unit 202 calculates a transformation coefficient from the amount of positional deviation between the reference image and the target image. The synthesis processing unit 202 calculates, for example, a projective transformation coefficient as the transformation coefficient. An affine transformation coefficient or a simpler horizontal / vertical shift amount may be calculated as the transformation coefficient.

[0070] In S705, the synthesis processing unit 202 transforms the target image using the transformation coefficient calculated in S704. For example, the synthesis processing unit 202 can transform the target image according to the following equation (3).

number

[0071] In equation (3), (x, y) are coordinates before transformation, and (x', y') are coordinates after transformation. Matrix A indicates the transformation coefficients calculated by the synthesis processing unit 202 in S704. The synthesis processing unit 202 stores the data of the target image after transformation in, for example, the memory 111 as image data that has been subjected to the alignment processing.

[0072] In S706, the synthesis processing unit 202 determines whether the alignment process has been applied to all images other than the reference image. If it is determined that there are images to which the alignment process has not been applied, the synthesis processing unit 202 repeats the process from S702. If it is determined that the alignment process has been applied to all images other than the reference image, the synthesis processing unit 202 ends the process shown in the flowchart of FIG.

[0073] Next, the synthesis process performed by the synthesis processor 202 in S404 of FIG. 4 will be described. The synthesis processing unit 202 sequentially acquires aligned image data from the memory 111 and synthesizes it with the data of the reference image. The synthesis processing unit 202 generates data of the synthetic image by, for example, averaging pixel data at corresponding coordinates. Random noise superimposed on the image can be reduced by averaging. Although synthetic image data may be generated by a method other than averaging, it is preferable to use a method that reduces random noise.

[0074] <Explanation of local tone mapping process> Next, the local tone mapping process performed by the local tone mapping processor 203 in S405 of Fig. 4 will be described. Note that by converting images captured underexposure to a brightness equivalent to the correct exposure using gamma processing and combining them, it is possible to obtain an image with an expanded dynamic range by suppressing blown-out (saturation) in high-luminance areas. Therefore, the local tone mapping process is not essential. However, by applying the local tone mapping process, it is possible to achieve a further expansion of the dynamic range.

[0075] The local tone mapping process in this embodiment is a process for locally brightening areas from low luminance to medium luminance. By applying such a local tone mapping process, it is possible to improve the gradation of dark areas while maintaining the gradation of high luminance areas.

[0076] Furthermore, in the imaging and development process described above, if imaging is performed at an underexposure of more than 1 Ev and a similar development process is performed on the input image, an underexposed image can be generated that suppresses deterioration of the gradation in high-brightness areas.

[0077] Local tone mapping processing can be achieved by a known method of generating a gain map that uses the results of discrimination of images or regions of different frequency bands to locally change the gradation characteristics, and applying gradation conversion processing while referring to the gain map.

[0078] 8A shows an example of a gain map used in local tone mapping. The gain map is similar to the gamma characteristic in that it represents the characteristics of tone conversion, and local tone mapping processing can also be said to be local gamma processing.

[0079] Gain map 803 is a straight line with a slope of 1 that passes through the origin. Gain map 803 has a characteristic of not changing the gradation value. On the other hand, gain maps 801 and 802 have gradation characteristics that brighten input gradation values ​​from low to medium luminance. Gain map 801 has a larger correction amount than gain map 802. In this embodiment, a gain map used in local tone mapping processing is generated by performing a weighted addition of gain maps 801 and 802, which have different correction amounts.

[0080] Specifically, the local tone mapping processor 203 generates a histogram of the input composite image data. Fig. 8(b) shows an example of the histogram. The local tone mapping processor 203 then calculates the ratio of low to medium luminance areas in the image by summing up the frequencies below a threshold. The threshold TH may be, for example, the middle tone value of the tone range of the input signal.

[0081] Next, the local tone mapping processing unit 203 uses the calculated ratio to obtain a coefficient α from a preset table as shown in FIG. 8(c). The coefficient α is a weight of the gain map 801, and 0≦α≦1. In the example shown in FIG. 8(c), the coefficient α is set so that the amount of correction of the brightness of the low- to medium-luminance regions becomes smaller as the ratio of the low- to medium-luminance regions in the image increases. This is to suppress the influence of the local tone mapping on the brightness of the entire image when the ratio of the region to be corrected in brightness in the image is large. The table for obtaining the coefficient α may be changed according to the purpose or intention of the low-illuminance scene shooting process. For example, a table of the coefficient α used when the effect of improving the gradation of the dark region is emphasized rather than suppressing the influence of the local tone mapping on the brightness of the entire image may be prepared. In this case, the coefficient α can be set so that the amount of correction of the brightness of the low- to medium-luminance regions becomes larger as the ratio of the low- to medium-luminance regions in the screen increases.

[0082] Then, the local tone mapping processor 203 calculates a gain map to be applied to the composite image data by weighting and adding the gain maps 801 and 802 using the coefficient α. If the gradation value of the input pixel is x, the gradation value after conversion is y, and the gain map 801 is y=tm_a(x) and the gain map 802 is y=tm_b(x), the gain map tm(x) used for local tone mapping is obtained by the following formula (4). y=tm(x)=α×tm_a(x)+(1-α)×tm_b(x)...Equation (4)

[0083] The local tone mapping processor 203 applies the gain map tm(x) obtained by equation (4) to the pixel values ​​of the synthetic image data. The gradation value of the input pixel is p, and the output gradation value p_out after applying the local tone mapping process is expressed by equation (5). p_out=tm(p)...Equation (5)

[0084] This type of local tone mapping process suppresses blackouts in low to medium luminance areas and improves gradation, further expanding the dynamic range in the low luminance direction.

[0085] According to this embodiment, when shooting a low-illumination scene, multiple shots are taken with underexposure, and the resulting images for multiple frames are synthesized. Therefore, shooting can be performed at a faster shutter speed than when shooting with proper exposure or overexposure, and camera shake and subject blur can be suppressed. In addition, the time required for multiple shots can be shortened. Furthermore, blown-out highlights in high-luminance areas can be suppressed.

[0086] In addition, gamma processing is performed on images taken with underexposure to correct the brightness of the entire image to the same brightness as an image taken with proper exposure before combining. This makes it possible to obtain a composite image with appropriate brightness while suppressing blown-out highlights in high-brightness areas. Furthermore, it is possible to reduce random noise during the composition process.

[0087] In addition, by applying local tone mapping processing to the low and medium luminance regions of the composite image, it is possible to suppress blackout and further expand the dynamic range of the composite image in the low luminance direction.

[0088] ●(Second embodiment) Next, a second embodiment of the present invention will be described. Fig. 9 is a flowchart of low-illumination scene shooting processing performed by the imaging device 100 in this embodiment. In Fig. 9, the same reference numerals as in Fig. 4 are used for steps in which the same processing as in the first embodiment is performed. In this embodiment, the timing of performing local tone mapping processing is different from that in the first embodiment.

[0089] Specifically, the development processing unit 201 outputs image data to the local tone mapping processing unit 203. Then, the local tone mapping processing unit 203 performs local tone mapping on the image data of each frame before the alignment processing is performed. toneThe local tone mapping process may be the same as that in the first embodiment, except that the pre-compositing image is used instead of the composite image.

[0090] The local tone mapping processor 203 outputs the processed image data to the synthesis processor 202. The synthesis processor 202 aligns and synthesizes the images that have been subjected to the local tone mapping process, and outputs the result from the image processor 104. In Fig. 2, the flow of image data within the image processor 104 in this embodiment is shown by dotted lines.

[0091] This embodiment can also provide the same effects as the first embodiment. Even if the movement between images is large and alignment and synthesis are not possible, it is possible to obtain image data having the effects achieved by processing up to the local tone mapping process.

[0092] (Other embodiments) The low-illuminance scene shooting process described in the above embodiment may be performed when shooting a scene other than a low-illuminance scene. For example, it may be performed to generate an image with an expanded dynamic range regardless of the shooting scene.

[0093] In addition, among the low-illuminance scene photographing processes described in the above embodiment, processes other than those related to photographing may be performed by an electronic device (image processing device) that does not have an image capturing function. That is, the image processing device acquires image data for a plurality of frames photographed underexposed from an external device, and performs the steps S402 onwards The above processing can be applied. With such an image processing device, it is possible to obtain a composite image having appropriate brightness while suppressing blown-out highlights in high-luminance areas. Furthermore, it is possible to reduce random noise during composition.

[0094] In the first embodiment, if the amount of deviation cannot be detected in S703 or is equal to or greater than a threshold, it may be determined that the alignment process cannot be performed normally, and local tone mapping may be applied to the image data of each frame, thereby achieving the same effect as in the second embodiment.

[0095] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0096] The disclosure of the present embodiment includes the following imaging device, image processing device, and control method and program for the imaging device. (Item 1) An imaging means for capturing images multiple times underexposed and acquiring image data for multiple frames; a correction means for correcting brightness of the image data for the plurality of frames; a generating means for generating data of a composite image by synthesizing the image data of the plurality of frames corrected by the correcting means; An imaging device having the above configuration. (Item 2) 2. The imaging device according to item 1, wherein the correction means corrects brightness of the image data by applying a gamma curve having gradation characteristics according to the magnitude of the difference between proper exposure and the underexposure to the image data. (Item 3) The imaging device described in item 1, wherein the correction means corrects the brightness of the image data obtained by shooting with underexposure by applying a second gamma curve having gradation characteristics with a steeper rise in the low gradation value range than a first gamma curve applied to image data obtained by shooting with proper exposure. (Item 4) 4. The imaging device according to item 3, wherein the first gamma curve and the second gamma curve have gradation characteristics that convert gradation values ​​corresponding to a specific subject into the same gradation value. (Item 5) 5. The imaging device according to item 4, wherein the specific subject is an achromatic subject having a reflectance of 18%. (Item 6) 6. The imaging device according to any one of items 1 to 5, further comprising a processing unit for applying a local tone mapping process to brighten data of the composite image whose gradation value is equal to or less than a threshold value. (Item 7) 6. The imaging device according to any one of items 1 to 5, further comprising a processing unit that applies a local tone mapping process to the image data for the plurality of frames corrected by the correction unit before the synthesis, in order to brighten data whose gradation values ​​are equal to or less than a threshold value. (Item 8) 8. The imaging device according to item 7, wherein the processing means applies the local tone mapping process when the synthesis is not possible. (Item 9) 9. The imaging device according to any one of items 6 to 8, wherein the threshold value is determined based on a histogram of tone values ​​of image data to which the local tone mapping process is applied. (Item 10) An acquisition means for acquiring image data for a plurality of frames obtained by photographing with underexposure; a correction means for correcting brightness of the image data for the plurality of frames; a generating means for generating data of a composite image by synthesizing the image data of the plurality of frames corrected by the correcting means; 13. An image processing device comprising: (Item 11) A control method executed by an imaging device, comprising: Taking multiple underexposed photographs to obtain multiple frames of image data; correcting brightness of the image data for the plurality of frames; synthesizing the corrected image data for the plurality of frames to generate data of a composite image; A control method for an imaging apparatus comprising the steps of: (Item 12) A program for causing a computer included in an imaging apparatus to function as each of the means included in the imaging apparatus according to any one of items 1 to 9. (Item 13) A program for causing a computer to function as each of the means possessed by the image processing device according to item 10.

[0097] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]

[0098] 100: imaging device, 102: imaging section, 104: image processing section, 106: system control section, 201: development processing section, 202: image synthesis section, 203: local tone mapping processing section

Claims

1. An imaging means for capturing images multiple times underexposed and acquiring image data for multiple frames; a correction means for correcting brightness of the image data of the plurality of frames obtained by shooting underexposure to the same brightness as that obtained by shooting underexposure by applying a second gamma curve having gradation characteristics with a steeper rise in a region of low gradation values ​​than a first gamma curve applied to image data obtained by shooting underexposure; and a processing means for applying a local tone mapping process to the image data for the plurality of frames corrected by the correction means by applying the second gamma curve, for brightening data having a gradation value equal to or less than a threshold value; a generating means for generating data of a composite image by averaging the image data of the plurality of frames to which the local tone mapping process has been applied; An imaging device having the above configuration.

2. 2. The imaging device according to claim 1, wherein the first gamma curve and the second gamma curve have gradation characteristics that convert gradation values ​​corresponding to a specific subject into the same gradation value.

3. 3. The imaging device according to claim 2, wherein the specific subject is an achromatic subject having a reflectance of 18%.

4. The imaging device according to claim 1 , wherein the processing means applies the local tone mapping process when the synthesis is not possible.

5. The imaging device according to claim 1 , wherein a gain map used in the local tone mapping process is determined based on a proportion of tone values ​​equal to or less than a threshold value in the image data to which the local tone mapping process is applied.

6. An imaging means for capturing images multiple times underexposed and acquiring image data for multiple frames; a correction means for correcting brightness of the image data for the plurality of frames; a generating means for generating data of a composite image by synthesizing the image data of the plurality of frames corrected by the correcting means; a processing means for applying a local tone mapping process to the image data for the plurality of frames corrected by the correction means before the synthesis, in order to brighten data having a gradation value equal to or less than a threshold value; An imaging device having the above configuration.

7. An acquisition means for acquiring image data for a plurality of frames obtained by photographing with underexposure; a correction means for correcting brightness of the image data of the plurality of frames obtained by shooting underexposure to the same brightness as that obtained by shooting underexposure by applying a second gamma curve having gradation characteristics with a steeper rise in a region of low gradation values ​​than a first gamma curve applied to image data obtained by shooting underexposure; and a processing means for applying a local tone mapping process to the image data for the plurality of frames corrected by the correction means by applying the second gamma curve, for brightening data having a gradation value equal to or less than a threshold value; a generating means for generating data of a composite image by averaging the image data of the plurality of frames to which the local tone mapping process has been applied; 13. An image processing device comprising:

8. A control method executed by an imaging device, comprising: Taking multiple underexposed photographs to obtain multiple frames of image data; applying a second gamma curve having gradation characteristics with a steeper rise in a region of low gradation values ​​than a first gamma curve applied to image data obtained by shooting with proper exposure, thereby correcting brightness of the image data for the plurality of frames obtained by shooting with the underexposure to brightness equivalent to that obtained by shooting with the proper exposure; applying a local tone mapping process to the image data for the plurality of frames corrected by applying the second gamma curve, the local tone mapping process brightening data whose gradation value is equal to or less than a threshold value; generating data of a composite image by averaging the image data of the plurality of frames to which the local tone mapping process has been applied; A control method for an imaging apparatus comprising the steps of:

9. A program for causing a computer included in an imaging apparatus to function as each of the means included in the imaging apparatus according to any one of claims 1 to 6.

10. A program for causing a computer to function as each of the means included in the image processing device according to claim 7.

Images