Imaging apparatus and method for controlling the same

Abstract

To provide an imaging apparatus capable of recording information for supporting the application of predetermined processing to a visible light image by properly using the information based on an invisible light image by an external device and to provide a method for controlling the imaging apparatus.SOLUTION: The imaging apparatus can acquire the visible light image and the invisible light image. The imaging apparatus determines a method for using the information based on the invisible light image, to apply the predetermined processing to the visible light image. The imaging apparatus generates the information based on the invisible light image according to the result of the determination. The imaging apparatus records a data file in which the visible light image, the information indicating the result of the determination, and the information based on the invisible light image generated by generation means are associated with each other.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an imaging apparatus and a control method thereof. [Background technology]

[0002] A method has been proposed for improving the visibility of a visible light image by synthesizing a visible light image of a scene where visibility is reduced by fog with an infrared light image of the same scene (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-157902 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, synthesizing an infrared image does not necessarily improve the visibility of a visible light image, and it is necessary to select an appropriate synthesis method. In addition, there are cases where it is better to use information from an infrared image by a method other than synthesis, and cases where improvement in visibility by synthesis cannot be expected.

[0005] Conventionally, imaging devices capable of recording visible light images and invisible light images have not provided information to assist other devices in executing processes to improve the visibility of the visible light images. Therefore, it has not been easy for other devices to appropriately use the information of the invisible light images to improve the visibility of the visible light images. Furthermore, imaging devices capable of recording visible light images with different exposure amounts have not provided information for other devices to appropriately execute processes using these images.

[0006] The present invention aims to solve one or more of the problems associated with the conventional techniques. In one aspect, the present invention provides an imaging device capable of recording information for assisting an external device in applying predetermined processing to a visible light image by appropriately utilizing information based on an invisible light image, and a control method for the imaging device. [Means for solving the problem]

[0007] The above-mentioned object can be achieved by an imaging device capable of acquiring visible light images and invisible light images, characterized in having a determination means for determining a method of using information based on the invisible light image to apply a predetermined processing to the visible light image, a generation means for generating information based on the invisible light image depending on the result of the determination, and a recording means for recording a data file that associates the visible light image, information indicating the result of the determination, and the information based on the invisible light image generated by the generation means. Effect of the Invention

[0008] According to the present invention, it is possible to provide an imaging device and a control method thereof that are capable of recording information to assist an external device in appropriately utilizing information based on an invisible light image to apply specified processing to a visible light image. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of a functional configuration of an imaging apparatus according to an embodiment; [Diagram 2] FIG. 1 is a block diagram illustrating a process executed by an image processing unit of an imaging device according to a first embodiment; [Diagram 3] FIG. 1 shows examples of visible light images and infrared light images. [Figure 4] 1 is a flowchart showing the operation of the auxiliary information generating unit in the first embodiment. [Diagram 5] Flowchart regarding the operation of the accessory image generating unit in the first embodiment [Figure 6] Flowchart regarding gain map creation operation in the first embodiment [Figure 7]FIG. 13 is a diagram showing an example of a gain amount determination method in the first embodiment. [Figure 8] FIG. 1 is a block diagram showing an example of a functional configuration of an image processing apparatus according to a first embodiment; [Figure 9] FIG. 1 is a block diagram illustrating a process executed by an image processing unit in a first embodiment; [Figure 10] Flowchart regarding visibility improvement processing in the first embodiment [Figure 11] FIG. 11 is a block diagram showing a schematic diagram of a process executed by an image processing unit in a second embodiment; [Figure 12] 11 is a flowchart of the operation of a quality adjustment unit in the second embodiment. [Figure 13] FIG. 13 is a block diagram illustrating a process executed by an image processing unit of an imaging device according to a third embodiment. [Figure 14] A flowchart of the operation of the auxiliary information generating unit in the third embodiment. [Figure 15] FIG. 13 is a block diagram illustrating a process executed by an image processing unit of an image processing apparatus according to a third embodiment. [Figure 16] Flowchart regarding dynamic range expansion processing in the third embodiment [Figure 17] FIG. 13 is a diagram showing an example of a synthesis ratio in the third embodiment. [Figure 18] FIG. 13 is a diagram showing an example of a gain amount in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] ●(First embodiment) The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0011] In the following, the present invention will be described in terms of an embodiment in which the present invention is implemented using a digital camera. However, the present invention can also be implemented in any electronic device having an imaging function. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0012] The configurations shown as blocks in the diagrams may be realized by integrated circuits (ICs) such as ASICs or FPGAs, by discrete circuits, or by a combination of a memory and a processor that executes a program stored in the memory. One block may be realized by multiple integrated circuit packages, or multiple blocks may be realized by one integrated circuit package. The same block may be implemented in different configurations depending on the operating environment, required capabilities, etc.

[0013] 1 is a block diagram showing an example of a functional configuration of an image capture device 100 as an example of an image processing device according to the present invention. The control unit 101 is a processor capable of executing a program, such as a CPU. The control unit 101 controls the operation of each functional block of the image capture device 100 by, for example, loading a program stored in a ROM 102 into a RAM 103 and executing the program, thereby achieving the functions of the image capture device 100. Note that, when the optical system 104 is an exchangeable lens unit, the control unit 101 controls the operation of the optical system 104 through communication with a controller included in the optical system 104.

[0014] The ROM 102 is a rewritable non-volatile memory. The ROM 102 stores programs executed by the control unit 101, various setting values ​​of the imaging device 100, GUI data, etc. The RAM 103 is the main memory of the control unit 101. The RAM 103 is used to load programs executed by the control unit 101, to hold parameters necessary for executing the programs, and as a working memory for the image processing unit 107. In addition, a part of the area of ​​the RAM 103 is used as a video memory for storing image data to be displayed on the display unit 109.

[0015] The optical system 104 has an imaging optical system made up of a lens group including a movable lens (zoom lens, focus lens, etc.), and a drive circuit for the movable lens. The optical system 104 may also have an aperture and a drive circuit for the aperture.

[0016] The imaging unit 105 may be, for example, a known CCD or CMOS color image sensor (imaging element) having a primary color Bayer array color filter. The imaging element has a pixel array in which a plurality of pixels are arranged two-dimensionally, and a peripheral circuit for reading out signals from the pixels. Each pixel has a photoelectric conversion element such as a photodiode, and accumulates electric charge according to the amount of incident light during the exposure period. By reading out a signal having a voltage according to the amount of electric charge accumulated during the exposure period from each pixel, a group of pixel signals (analog image signals) representing the subject image formed on the imaging plane by the imaging optical system can be obtained.

[0017] In this embodiment, the imaging unit 105 has an imaging element capable of capturing a visible light image and an invisible light image. For example, such an imaging element may be a pixel array in which some of the pixels are used as pixels for capturing an invisible light image. The pixels for capturing an invisible light image may be pixels having an optical filter that transmits the invisible light wavelength band and blocks the visible light wavelength range.

[0018] For example, one of the two green (G) filters included in a repeating unit (2×2 pixels) of a primary color Bayer array color filter is replaced with an optical bandpass filter that transmits invisible light (e.g., infrared light) wavelengths. This allows one of the green (G) pixels to be used as an imaging pixel for an invisible light image. When generating a visible light image, the value of the G pixel that should originally exist at the position of the imaging pixel for the invisible light image can be generated by, for example, interpolating using the values ​​of the surrounding G pixels. In addition, for the invisible light image, an image obtained based on a signal from the imaging pixel for the invisible light image can be enlarged to have the same resolution (number of pixels) as the visible light image.

[0019] There is no particular limitation on the method of acquiring the visible light image and the invisible light image, and they may be acquired by other methods. For example, an image sensor for capturing the visible light image and an image sensor for capturing the invisible light image may be provided separately. In addition, although the invisible light image is an infrared light image in this embodiment, it may be an image of another invisible wavelength band.

[0020] The A / D conversion unit 106 converts the analog image signal read out from the imaging unit 105 into a digital image signal. The A / D conversion unit 106 writes the digital image signal into the RAM 103.

[0021] Image processing unit 107 applies predetermined image processing to the digital image signal stored in RAM 103, generates signals and image data according to the application, and acquires and / or generates various information. Image processing unit 107 may be, for example, a dedicated hardware circuit such as an ASIC designed to realize a specific function, or may be configured to realize a specific function by a programmable processor such as a DSP executing software.

[0022] The image processing applied by the image processing unit 107 includes pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing includes signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation is a process that interpolates the values ​​of color components that cannot be obtained during shooting, and is also called demosaic processing.

[0023] The correction processing includes white balance adjustment, gradation correction, correction of image degradation caused by optical aberration of the optical system 104 (image restoration), correction of the effect of peripheral shading of the optical system 104, color correction, etc. The correction processing also includes a synthesis process of an infrared light image for the purpose of improving the visibility of the visible light image, which will be described later. The detection process includes detection of feature regions (for example, face regions and human body regions) and their movements, and person recognition processing. The data processing includes processes such as synthesis, scaling, encoding and decoding, and header information generation (data file generation).

[0024] The evaluation value calculation process includes processes such as generating signals and evaluation values ​​used in automatic focus detection (AF), generating evaluation values ​​used in automatic exposure control (AE), etc. Note that AE and AF are executed by the control unit 101 based on the evaluation values. This process also includes generating information on the infrared light image and information on how to use the infrared light image, which will be described later.

[0025] Special effects processing includes adding a blur effect, changing color tones, relighting, and the like. Note that these are merely examples of processes that the image processing unit 107 can apply, and do not limit the processes that the image processing unit 107 can apply.

[0026] The recording unit 108 records data on a recording medium such as a memory card, and reads data recorded on the recording medium. The recording medium does not need to be removable. The recording medium may also be a communicable external storage device. In this embodiment, in addition to the visible light image, information on the infrared light image and information on how to use the infrared light image can also be recorded.

[0027] The display unit 109 is, for example, a liquid crystal display, and displays captured images, images read by the recording unit 108, information about the imaging device 100, GUIs such as menu screens, etc. By continuously capturing video and displaying the captured video on the display unit 109, the display unit 109 can function as an electronic viewfinder (EVF). The display unit 109 may be a touch display.

[0028] The operation unit 110 is a general term for input devices (such as buttons, switches, and dials) that are provided for a user to input instructions to the imaging device 100. The input devices that make up the operation unit 110 have names according to the functions assigned to them. For example, the operation unit 110 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, direction keys, and a decision key. The release switch is a switch for recording still images, and the control unit 101 recognizes the half-pressed state of the release switch as a shooting preparation instruction and the full-pressed state as a shooting start instruction. In addition, the control unit 101 recognizes the video recording switch pressed in a shooting standby state as a video recording start instruction, and recognizes the video recording switch pressed during video recording as a recording stop instruction. Note that the functions assigned to the same input device may be variable. In addition, the input device may be a software button or key using a touch display.

[0029] 2 is a block diagram that uses functional blocks to diagrammatically represent a series of processes executed by the image processing unit 107 in this embodiment. The configuration described as a functional block may be implemented by individual hardware or may be implemented as a software module.

[0030] It is assumed that shooting is performed before the execution of the processing described below, and data of the visible light image and infrared light image obtained by shooting is stored in the RAM 103. However, infrared light image data does not have to be generated all the time. The infrared light image data may be generated, for example, according to the shooting mode or other settings.

[0031] The first image acquisition unit 201 acquires data of the infrared light image stored in the RAM 103. The second image acquisition unit 202 acquires data of the visible light image stored in the RAM 103. When an infrared light image and a visible light image are acquired in one shot by providing pixels for an infrared light image in the imaging element, the first image acquisition unit 201 matches the resolution of the infrared light image to the resolution of the visible light image. The second image acquisition unit 202 obtains the value of the visible light image at the position of the pixel for the infrared light image from the values ​​of the surrounding pixels.

[0032] The correction unit 203 applies a predetermined correction process to the infrared light image. The correction process includes, for example, a noise removal process. The correction process also includes a level adjustment process for matching the brightness of the infrared light image to the brightness of the visible light image. The correction process also includes a process for correcting image distortion caused by aberration of the optical system 104. These are merely examples, and other correction processes may be applied.

[0033] The basic signal processing unit 204 applies a predetermined signal processing to the visible light image. The signal processing applied here includes noise removal processing, gamma correction processing, color interpolation processing, conversion from RGB format to YCbCr format, optimization processing, etc. These are merely examples, and other signal processing may also be applied.

[0034] The auxiliary information generating unit 205 generates auxiliary information that is an indicator or flag indicating one of a plurality of predetermined usage methods of the infrared light image. The auxiliary information generating unit 205 generates, for example, auxiliary information that is an indicator or flag indicating one of a plurality of predetermined usage methods of the infrared light image. The auxiliary information can be generated based on one or more of the information obtained from the infrared light image and the visible light image, and the shooting conditions of the infrared light image. The operation of the auxiliary information generating unit 205 will be described in detail later.

[0035] The accessory image generating unit 206 generates an accessory image to be recorded in association with the visible light image according to the accessory information (or the method indicated by the accessory information). Note that, depending on the accessory information, there may be no accessory image, or the infrared light image may be the accessory image as it is. The accessory image is data in a format that can be used to improve the visibility of the corresponding visible light image. The operation of the accessory image generating unit 206 will be described in detail later.

[0036] The combination of collateral information and collateral image is information that assists an external device in appropriately utilizing information based on an invisible light image to apply a specified processing (here, processing to improve visibility) to a visible light image.

[0037] The encoding unit 207 generates one data file that contains the visible light image, the auxiliary information, and the auxiliary image. The encoding unit 207 can generate file data that complies with a known container format, for example. The encoding unit 207 stores the generated file data in the RAM 103. The recording unit 108 reads the file data from the RAM 103 and records it as a file on a recording medium.

[0038] Here, a specific example in which the visibility of a visible light image can be improved by using an infrared light image will be described with reference to Fig. 3. Visible light images 301 and 311 are schematic diagrams showing examples of visible light images in which the visibility can be improved by using an infrared light image. Visible light image 301 is an image with proper exposure but reduced contrast due to fog or haze. Visible light image 311 is an image that is dark due to insufficient exposure. Infrared light image 302 is an infrared light image captured of the same scene as visible light image 301. Infrared light image 312 is an infrared light image captured of the same scene as visible light image 311.

[0039] Visibility can be improved for a visible light image 301 that has sufficient brightness by combining an alternating current (AC) component of an infrared light image 302. On the other hand, visibility can be improved for a visible light image 311 that lacks brightness by directly combining an infrared light image 312. The infrared light image and the visible light image are combined on a pixel-by-pixel basis after being aligned. The pixel-by-pixel combination may be a process of adding the value of the infrared light image to the luminance component of the visible light image.

[0040] The state of a visible light image in which the visibility is expected to be improved by using an infrared light image is not limited to the example shown in Fig. 3. Similarly, the method of using an infrared light image is not limited to the method of use described here.

[0041] In this way, the method of using the infrared light image suitable for improving visibility varies depending on the characteristics of the visible light image. Also, depending on the shooting conditions of the visible light image and the infrared light image, there are cases where improvement in visibility by combining the infrared light image cannot be expected.

[0042] Therefore, the auxiliary information generating unit 205 determines whether or not the infrared image should be used, and if so, how it should be used, and generates auxiliary information based on the determination result.

[0043] The operation of the auxiliary information generating unit 205 will be described with reference to the flowchart shown in FIG. In S401, the auxiliary information generating unit 205 obtains an infrared light image from the correcting unit 203, a visible light image from the basic signal processing unit 204, and imaging conditions for the infrared light image from the control unit 101.

[0044] In S402, the auxiliary information generating unit 205 determines whether the infrared light image is not suitable for improving the visibility of the visible light image. For example, when the subject blur or image blur of the infrared light image is considered to be large, the auxiliary information generating unit 205 can determine that the infrared light image is not suitable for improving the visibility of the visible light image.

[0045] For example, if the shutter speed during capture of the infrared light image is slower than the first speed threshold, at least one of image blur and subject blur is considered to be large. Also, if the movement of the image capture device 100 during capture is greater than the first movement threshold, image blur is considered to be large. The movement of the image capture device 100 can be detected, for example, by a gyro sensor provided for image blur correction.

[0046] If it is determined that the infrared light image is not suitable for improving the visibility of the visible light image, the auxiliary information generating unit 205 executes S403, and if not, executes S404.

[0047] In S403, the additional information generating unit 205 generates additional information indicating the processing is inappropriate, and ends the additional information generating process.

[0048] In S404, the auxiliary information generating unit 205 determines whether the infrared image is not suitable for direct synthesis but can be indirectly used (whether it is at the indirect use level). Here, if it is considered that at least one of subject blur and image blur in the infrared image exists to such an extent that the condition of S402 is not satisfied, it can be determined that it is at the indirect use level. Therefore, the auxiliary information generating unit 205 can make a determination in the same manner as S402, using a second speed threshold faster than the first speed threshold and a second speed threshold smaller than the first motion threshold.

[0049] Furthermore, even if the capture timing of the infrared light image and the visible light image differs, the incidental information generating unit 205 can determine that the infrared light image is at the indirect use level. For example, this applies to the case where the infrared light image and the visible light image are acquired by continuous shooting. This is because, if the capture timing differs, the capture ranges of the visible light image and the infrared light image do not match, or the position of a moving subject changes. The incidental information generating unit 205 may determine that processing is inappropriate if the difference in the capture timing is equal to or greater than a threshold, and may determine that the indirect use level is at the indirect use level if the difference is less than the threshold (>0).

[0050] If it is determined that the infrared light image is at the indirect use level, the auxiliary information generating unit 205 executes S405, and if not, executes S406.

[0051] In S405, the additional information generating unit 205 generates additional information indicating indirect use, and ends the additional information generating process.

[0052] In S406, the auxiliary information generating unit 205 determines whether the brightness of the visible light image is equal to or greater than the brightness threshold. The brightness may be, for example, a brightness evaluation value used in automatic exposure control (AE), an average brightness value of a predetermined region in the image, or another value related to brightness. If the auxiliary information generating unit 205 determines that the brightness of the visible light image is equal to or greater than the brightness threshold, it executes S407, and if not, it executes S408.

[0053] In S407, the auxiliary information generating unit 205 generates auxiliary information indicating a combination of the AC components of the infrared light image, and ends the auxiliary information generating process. In S408, the auxiliary information generating unit 205 generates auxiliary information indicating a combination of all the components (the infrared light image itself), and ends the auxiliary information generating process.

[0054] Next, the operation of the accessory image generating unit 206 will be described with reference to the flowchart shown in FIG. In S501 , the auxiliary image generating unit 206 obtains an infrared light image from the correcting unit 203 , a visible light image from the basic signal processing unit 204 , and auxiliary information from the auxiliary information generating unit 205 .

[0055] In S502, the auxiliary image generating unit 206 determines whether the auxiliary information indicates processing inappropriateness, and if it is determined that the auxiliary information indicates processing inappropriateness, executes S503, and if not, executes S504.

[0056] In S503, the collateral image generating unit 206 ends the collateral image generating process without generating the collateral image. In S504, the accessory image generating unit 206 determines whether the accessory information indicates indirect use or not, and if it is determined that it indicates indirect use, executes S505, and if it is not determined that it indicates indirect use, executes S506.

[0057] In S505, the auxiliary image generating unit 206 generates a gain map as an auxiliary image. The gain map is data indicating a gain by which the luminance (Y) component of each pixel of the visible light image is multiplied in order to improve the visibility of the visible light image.

[0058] The method of generating the gain map in S505 will be described with reference to the flowchart shown in FIG. In S602, the auxiliary image generating unit 206 applies filter processing to each of the infrared light image and the visible light image acquired in S501. The filter applied here is a smoothing filter or a low-pass filter that reduces high-frequency components. The frequency characteristics of the filter can be appropriately determined through experiments, etc. By reducing the high-frequency components, it is possible to suppress a decrease in contrast due to the application of the gain amount.

[0059] In S603, the auxiliary image generating unit 206 determines the amount of gain to be applied to the visible light image for each pixel, and generates a gain map. For example, the auxiliary image generating unit 206 determines a gain amount exceeding 1 for pixels in the visible light image whose luminance value is equal to or less than a threshold value, and whose luminance value differs from the value of a pixel whose position corresponds to the infrared image by equal to or more than a threshold value. A corresponding pixel is a pixel whose position (coordinates) in the image is the same as that of the target pixel.

[0060] Fig. 7 is a diagram showing an example of a method for determining a gain amount. First, the auxiliary image generating unit 206 determines the maximum value of the gain amount for a target pixel according to the luminance value of the target pixel in the visible light image. Fig. 7(a) shows an example of the relationship between the luminance value and the maximum value of the gain amount.

[0061] As shown in Fig. 7(a), for target pixels whose luminance value is equal to or greater than threshold th1, the maximum gain amount is set to 1. Since the minimum gain amount is 1, there is no change in brightness for pixels whose luminance value is equal to or greater than threshold th1. On the other hand, for target pixels whose luminance value is less than threshold th1, the maximum gain amount is greater than 1, and the lower the luminance value, the greater the maximum gain amount. In this way, the maximum gain amount is determined for each pixel.

[0062] Next, the auxiliary image generating unit 206 determines a gain amount for the target pixel that is greater than or equal to 1 and less than or equal to the maximum value, depending on the difference (signal level difference) between the luminance value of the target pixel in the visible light image and the value of the pixel at the corresponding position in the infrared light image. Here, the signal level difference is determined as the value obtained by subtracting the luminance value of the visible light image from the value of the infrared light image. Fig. 7(b) shows an example of the relationship between the signal level difference and the determined gain amount.

[0063] As shown in FIG. 7(b), when the difference in signal levels is less than threshold th2, the gain amount for the target pixel is set to 1. Therefore, the brightness of pixels whose luminance values ​​are greater than the value of the infrared image and pixels whose luminance values ​​are smaller than the value of the infrared image but whose difference is less than threshold th2 remains unchanged. On the other hand, for target pixels whose difference in signal levels is greater than or equal to threshold th2, a larger gain amount is determined as the difference in signal levels increases until the previously determined maximum gain amount is reached. After the maximum gain amount is reached, the maximum gain amount is set to the final gain amount.

[0064] The auxiliary image generating unit 206 can hold in advance tables or formulas corresponding to Fig. 7(a) and Fig. 7(b). The auxiliary image generating unit 206 can then determine the amount of gain to be applied to the target pixel by using the luminance value and signal level difference of the target pixel and the table or formula.

[0065] After determining the gain amount for each pixel of the visible light image, the auxiliary image generating unit 206 outputs a gain map formed from the determined gain amounts to the encoding unit 207 as an auxiliary image.

[0066] Returning to FIG. 5, in S506, the auxiliary image generating unit 206 determines whether the auxiliary information indicates a combination of AC components, and executes S507 if it is determined that it indicates a combination of AC components, and executes S508 if it is not determined that it indicates a combination of AC components.

[0067] In S507, the auxiliary image generating unit 206 generates an AC image of the infrared light image as an auxiliary image. The auxiliary image generating unit 206 can generate the AC image of the infrared light image by applying high-pass filter processing to the infrared light image. The characteristics of the high-pass filter can be determined in advance by experiments or the like. The auxiliary image generating unit 206 outputs the generated AC image to the encoding unit 207 as an auxiliary image.

[0068] In S508, the collateral image generating unit 206 outputs the infrared light image itself to the encoding unit 207 as an collateral image.

[0069] Next, an image processing device as an external device that uses the data file recorded by the imaging device 100 will be described. The image processing device may be any electronic device. Such electronic devices include, but are not limited to, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and drive recorders.

[0070] FIG. 8 is a block diagram showing an example of the functional configuration of the image processing device 800. As shown in FIG. The control unit 801 is, for example, a CPU, which reads out an operation program for each block of the image processing device 800 from a ROM 802 described below, expands the program into a RAM 803 described below, and executes the program to control the operation of each block of the image processing device 800. The ROM 802 is an electrically erasable and recordable non-volatile memory, and stores parameters and the like required for the operation of each block in addition to the operation program for each block of the image processing device 800. The RAM 803 is a rewritable volatile memory, and is used as a temporary storage area for data output in the operation of each block of the image processing device 800.

[0071] The control unit 801 is, for example, a processor (CPU, MPU, microprocessor, etc.) capable of executing a program. The control unit 801 loads a program stored in a ROM 802 into a RAM 803 and executes the program, thereby controlling the operation of each unit of the image processing device 800 and achieving the functions described below.

[0072] The ROM 802 is a rewritable non-volatile memory, and stores the programs executed by the control unit 801, various setting values ​​of the image processing device 800, and the like. The RAM 803 is a main memory used when the control unit 801 executes a program. The RAM 803 is also used as a working memory for the image processing unit 804 and a video memory for the display unit 806.

[0073] The image processing unit 804 may be implemented as hardware for image processing such as a GPU, or may be realized by a processor executing a program. The image processing unit 804 executes a process for improving visibility of a visible light image using the visible light image, the auxiliary image, and the auxiliary information recorded by the imaging device 100 under the control of the control unit 801. The image processing unit 804 can also apply various other image processing.

[0074] The recording unit 805 is, for example, a recording device that uses a removable memory card. The recording unit 805 further includes other storage devices such as a hard disk drive (HDD) and a solid state drive (SSD). The other storage devices store operating systems (OS), application programs, user data, and the like.

[0075] Here, it is assumed that a memory card on which a data file is recorded by the imaging device 100 is attached to the recording unit 805. Note that the data file recorded by the imaging device 100 may be acquired from an external device (including the imaging device 100) through communication.

[0076] The display unit 806 has a display device such as a liquid crystal display (LCD) and displays a graphical user interface (GUI) provided by the OS and application programs. Visible light images and user data are displayed in the GUI (e.g., an application window) of the application program that handles them.

[0077] The operation unit 807 has one or more input devices such as a keyboard, a mouse, a touch panel, etc. The operation unit 807 is used by a user of the image processing device 800 to input instructions to the image processing device 800. The operation of the operation unit 807 is monitored by the control unit 801. When the control unit 801 detects an operation of the operation unit 807, it executes an operation according to the operation.

[0078] 9 is a block diagram that uses functional blocks to diagrammatically represent a series of processes related to improving the visibility of a visible light image executed by the image processing unit 804. The configurations described as functional blocks may be implemented by individual hardware or as software modules.

[0079] For example, while an image processing application is being executed, the control unit 801 receives an instruction from a user to read image data recorded on a memory card attached to the recording unit 805. The control unit 801 acquires image data designated by the user from the recording unit 805 and stores it in the RAM 803. The control unit 801 also displays the read image in a window of the image processing application. Here, it is assumed that the read image data is visible light image data recorded in association with incidental information and incidental image.

[0080] When a user instructs the execution of visibility improvement processing for a displayed image through, for example, a menu operation of an image processing application, the control unit 801 instructs the image processing unit 804 to execute visibility improvement processing. As a result, the image processing unit 804 executes the visibility improvement processing described below.

[0081] The decoding unit 901 extracts visible light image data, accessory information data, and accessory image data from a data file in a container format stored in the RAM 103 .

[0082] The visibility improvement processing unit 902 applies processing for improving visibility to the visible light image based on the auxiliary information and the auxiliary image. The visibility improvement processing unit 902 stores the processed visible light image data in the RAM 803. The operation of the visibility improvement processing unit 902 will be described in detail later.

[0083] The post-adjustment unit 903 applies a predetermined post-adjustment to the visible light image data to which the visibility improvement processing has been applied. The post-adjustment may be, for example, adjustment of color or brightness according to a user specification, adjustment of the gradation curve, etc. When the post-adjustment is completed, the control unit 801 records the post-adjusted visible light image data in the recording unit 805 as image data after the visibility improvement processing has been applied.

[0084] The operation of the visibility improvement processing unit 902 will be described in detail with reference to the flowchart shown in FIG. In S1001 , the visibility improvement processing unit 902 refers to the data of the additional information extracted by the decoding unit 901 .

[0085] In S1002, the visibility improvement processing unit 902 determines whether the incidental information indicates processing inappropriateness, and if it is determined that the incidental information indicates processing inappropriateness, executes S1003, and if not, executes S1004.

[0086] In S1003, the visibility improvement processing unit 902 ends the visibility improvement processing without performing any operations. At this time, the display unit 806 may display a message dialogue indicating that the visibility improvement processing cannot be applied to the image being displayed.

[0087] In S1004, the visibility improvement processing unit 902 determines whether the incidental information indicates indirect use or not, and if it is determined that it indicates indirect use, executes S1005, and if it is not determined that it indicates indirect use, executes S1006.

[0088] In S1005, the visibility improvement processing unit 902 recognizes that the auxiliary image is a gain map based on the auxiliary information. Then, the visibility improvement processing unit 902 applies (multiplies) the luminance value of each pixel of the visible light image by a gain amount based on the gain map. If the read visible light image is in RGB format, the visibility improvement processing unit 902 converts it to YCbCr format, and then multiplies the luminance (Y) component by the gain amount.

[0089] The visibility improvement processing unit 902 updates the image in the application window with the visible light image to which the gain amount has been applied, and ends the visibility improvement processing.

[0090] In S1006, the visibility improvement processing unit 902 determines whether the incidental information indicates a composition of AC images, and executes S1007 if it is determined that it indicates indirect use, and executes S1008 if it is not determined that it indicates indirect use.

[0091] In S1007, the visibility improvement processing unit 902 recognizes that the auxiliary image is an alternating current component (AC image) of an infrared light image based on the auxiliary information. Then, the visibility improvement processing unit 902 synthesizes (adds) the AC image to the luminance component of the visible light image. Specifically, the visibility improvement processing unit 902 adds the pixel value of the corresponding position of the AC image to the luminance value of each pixel of the visible light image.

[0092] The visibility improvement processing unit 902 updates the image in the application window with the visible light image synthesized with the AC image, and ends the visibility improvement processing.

[0093] In S1008, the visibility improvement processing unit 902 recognizes that the auxiliary image is the infrared light image itself based on the auxiliary information. Then, the visibility improvement processing unit 902 synthesizes (adds) the infrared light image to the luminance component of the visible light image. Specifically, the visibility improvement processing unit 902 adds the pixel value of the corresponding position of the infrared light image to the luminance value of each pixel of the visible light image.

[0094] The visibility improvement processing unit 902 updates the image in the application window with the visible light image synthesized with the infrared light image, and ends the visibility improvement processing.

[0095] As described above, the imaging device of this embodiment records information based on an infrared light image (accompanying image) that can be used to improve the visibility of a visible light image and information indicating a method of using the accompanying image in association with the visible light image. Therefore, in an external device, the visibility of a visualized image can be easily improved by using the accompanying image. In addition, since the imaging device generates an accompanying image in a form suitable for improving the visibility of a visible light image, the external device can improve the visibility of the visible light image in an optimal manner simply by using the accompanying image according to the indicated method of use. Furthermore, the storage capacity of the recording medium can be used more efficiently than when an infrared light image is always recorded in association with a visible light image.

[0096] In the visibility improvement process in the image processing device, the gain amount shown in the gain map may be set to 100%, and the strength (%) of the gain amount actually applied may be adjusted by the user. Similarly, when synthesizing an AC image or an infrared image, the actual addition rate (%) may be adjusted by the user, with the case where the images are added as is set to 100%.

[0097] In the present embodiment, the incidental information indicates one of four types of processing methods, but it may indicate three or less types or five or more types of processing methods. When incidental information indicating a new processing method is introduced, the incidental image will also correspond to the method.

[0098] In the present embodiment, the processing applied to the visible light image using the information based on the invisible light image is processing for improving visibility, but other processing may be used. Also, the invisible image is not limited to the infrared light image.

[0099] ●(Second embodiment) Next, a second embodiment of the present invention will be described, focusing on the differences from the first embodiment. 11 is a block diagram showing a series of processes performed by the image processing unit 107 in the second embodiment, which are expressed using functional blocks. The functional blocks that perform the same processes as in the first embodiment are given the same reference numerals as in FIG. 2, and the description thereof will be omitted.

[0100] The image processing unit 107 of this embodiment has a quality adjustment unit 1106 instead of the auxiliary image generation unit 206. Also, the data handled by the encoding unit 1107 is different from that in the first embodiment. Therefore, the operations of the quality adjustment unit 1106 and the encoding unit 1107 will be described below.

[0101] First, the operation of the quality adjustment unit 1106 will be described with reference to the flowchart shown in FIG. In S 1201 , the quality adjustment unit 1106 refers to the additional information acquired from the additional information generation unit 205 . In S1202, the quality adjustment unit 1106 judges whether the additional information indicates unsuitability for processing, and if it is judged that the additional information indicates unsuitability for processing, executes S1203, and if not, executes S1204.

[0102] In S1203, the quality adjustment unit 1106 adjusts the quality of the infrared light image acquired from the correction unit 203 to low quality. Here, it is assumed that parameters corresponding to four types of quality are predefined, each of which differs in one or more of known arbitrary parameters that affect image quality, such as resolution, number of bits per pixel, and compression rate during lossy compression encoding. Then, in S1203, the quality adjustment unit 1106 adjusts the quality of the infrared light image according to the parameter corresponding to the lowest quality of the four types of quality. The quality adjustment unit 1106 outputs the infrared light image after the quality adjustment to the encoding unit 1107 as an auxiliary image.

[0103] In S1204, the quality adjustment unit 1106 determines whether the incidental information indicates indirect use, and if it is determined that it indicates indirect use, executes S1205, and if not, executes S1206.

[0104] In S1205, the quality adjustment unit 1106 adjusts the quality of the infrared light image acquired from the correction unit 203 to medium quality. The medium quality is the second lowest quality among the four qualities. The quality adjustment unit 1106 outputs the infrared light image after the quality adjustment to the encoding unit 1107 as an auxiliary image.

[0105] In S1206, the quality adjustment unit 1106 determines whether the supplementary information indicates a composition of AC images. If it is determined that the supplementary information indicates a composition of AC images, the quality adjustment unit 1106 executes S1207, and if not, the quality adjustment unit 1106 executes S1208.

[0106] In S1207, the quality adjustment unit 1106 adjusts the quality of the infrared light image acquired from the correction unit 203 to high quality. High quality is the second highest quality among the four qualities. The quality adjustment unit 1106 outputs the infrared light image after the quality adjustment to the encoding unit 1107 as an auxiliary image.

[0107] In S1208, the quality adjustment unit 1106 adjusts the quality of the infrared light image acquired from the correction unit 203 to the highest quality. If lossy encoding is not performed, no adjustment needs to be made to the infrared light image in S1208. The quality adjustment unit 1106 outputs the infrared light image after the quality adjustment to the encoding unit 1107 as an auxiliary image.

[0108] The operation of the encoding unit 1107 is the same as in the first embodiment, except that the contents of the collateral image are different.

[0109] The visibility improvement processing unit 902 of the image processing device 800 using the data file recorded by the imaging device 100 of this embodiment recognizes the quality of the infrared light image recorded as the auxiliary image based on the auxiliary information. Then, when the auxiliary image is an infrared light image of low quality, the visibility improvement processing unit 902 does not execute processing for improving the visibility of the visible light image. Also, when the infrared light image of the auxiliary image is of medium quality, the visibility improvement processing unit 902 generates a gain map in the same manner as the auxiliary image generating unit 206 of the first embodiment and applies it to the visible light image. Also, when the infrared light image of the auxiliary image is of high quality, the visibility improvement processing unit 902 generates an AC image in the same manner as the auxiliary image generating unit 206 of the first embodiment and synthesizes it with the visible light image. Also, when the infrared light image of the auxiliary image is of the highest quality, the visibility improvement processing unit 902 synthesizes the infrared light image directly with the visible light image.

[0110] This embodiment can achieve the same effect as the first embodiment. Furthermore, instead of generating a gain map or generating an AC image, an auxiliary image is generated by adjusting the quality according to the auxiliary information. Therefore, the processing is simpler than that of the first embodiment, and the processing load is small. Furthermore, when the data amount is reduced even in the case of the highest image quality, the data amount of the auxiliary image can be reduced.

[0111] ●(Third embodiment) Next, a third embodiment of the present invention will be described. In this embodiment, unlike the first and second embodiments, visible light images with different exposure amounts are captured instead of invisible light images. Then, auxiliary information and auxiliary images that facilitate appropriate execution of dynamic range expansion processing of the visible light images by an external device are recorded.

[0112] 13 is a block diagram that uses functional blocks to diagrammatically represent a series of processes performed by image processing unit 107 in the third embodiment. Note that, before the processes described below are executed, it is assumed that image capture with different exposure amounts has been performed in succession, and data of two frames of visible light images obtained by the capture has been stored in RAM 103.

[0113] Here, it is assumed that visible light images are captured with proper exposure and underexposure. The exposure conditions during capture are the same aperture value, and in principle the shutter speed is different. In some cases, the capture sensitivity may also be different. The difference in exposure amount between proper exposure and underexposure is, for example, about 1EV to 3EV. Below, a visible light image captured with proper exposure is called a proper exposure image, and a visible light image captured with an exposure amount less than the proper exposure is called an underexposure image.

[0114] The first image acquisition unit 1301 acquires data of a properly exposed image stored in the RAM 103. The second image acquisition unit 1302 acquires data of an underexposed image stored in the RAM 103.

[0115] The first basic signal processing unit 1303 and the second basic signal processing unit 1304 apply the same processing as that of the basic signal processing unit 204 in the first and second embodiments.

[0116] The incidental information generating unit 1305 generates incidental information from the properly exposed image, the underexposed image, and the shooting conditions of these images. The incidental information is an indicator or flag that indicates one of a plurality of usage methods of the properly exposed image and the underexposed image.

[0117] The incidental information generating unit 1305 can generate incidental information based on, for example, information obtained from the properly exposed image and the underexposed image, and one or more of the shooting conditions of the properly exposed image. The operation of the incidental information generating unit 1305 will be described in detail later.

[0118] The encoding unit 1306 generates file data in the same manner as the encoding unit 207 of the first embodiment or the encoding unit 1107 of the second embodiment, except that the contents of the incidental information and incidental images to be recorded are different.

[0119] Here, the dynamic range expansion process will be described. In general dynamic range expansion process, a properly exposed image and an underexposed image are combined at a ratio according to the brightness of one of them.

[0120] However, as described above, since the aperture value is common under the shooting conditions, the shutter speed of the properly exposed image is slower than that of the underexposed image. A slower shutter speed makes it easier for moving subjects to blur or camera shake to occur. If subject blur or image blur occurs (or is highly likely to occur) in the properly exposed image, it is better to expand the dynamic range of the underexposed image. This is because the underexposed image is shot under exposure conditions with a faster shutter speed than the properly exposed image. The dynamic range of the underexposed image can be expanded by applying tone conversion that expands the gradation, especially in dark areas.

[0121] However, if there is a lot of noise in the dark areas of an underexposed image, expanding the gradation of the dark areas will emphasize the noise and reduce the quality of the image after the dynamic range is expanded. For this reason, it is better not to expand the dynamic range by gradation conversion.

[0122] It is easier to determine whether a properly exposed image and an underexposed image are suitable for dynamic range expansion based on the above-mentioned conditions at the time of shooting (recording). Therefore, by recording the determination result at the time of recording as supplementary information, it is possible to support the dynamic range expansion process in an external device.

[0123] The operation of the auxiliary information generating unit 1305 will be described with reference to the flowchart shown in FIG. In S1401, the auxiliary information generating unit 1305 obtains an appropriately exposed image from the first basic signal processing unit 1303, an underexposed image from the second basic signal processing unit 1304, and the imaging conditions of these images from the control unit 101, respectively.

[0124] In S1402, the incidental information generating unit 1305 determines whether or not the properly exposed image is suitable for the dynamic range expansion processing. For example, when the properly exposed image is considered to have a large subject blur or image blur, the incidental information generating unit 1305 can determine that the properly exposed image is not suitable for the dynamic range expansion processing.

[0125] For example, if the shutter speed when capturing a properly exposed image is slower than the first speed threshold, at least one of image blur and subject blur is considered to be large. Also, if the movement of the image capturing device 100 during capture is greater than the first movement threshold, image blur is considered to be large. The movement of the image capturing device 100 can be detected, for example, by a gyro sensor provided for image blur correction.

[0126] If the auxiliary information generating unit 1305 determines that the properly exposed image is not suitable for the dynamic range expansion processing, it executes S1404, and if not, it executes S1403.

[0127] In S1403, since the properly exposed image is suitable for the dynamic range expansion processing, the incidental information generating unit 1305 generates incidental information indicating the synthesis processing, and ends the incidental information generating processing.

[0128] In S1404, the auxiliary information generating unit 1305 determines whether the underexposed image is suitable for dynamic range expansion processing. If a predetermined condition for increasing the amount of noise is satisfied, the auxiliary information generating unit 1305 can determine that the underexposed image is not suitable for dynamic range expansion processing by tone conversion.

[0129] The condition that the amount of noise becomes large may be, for example, when the shooting sensitivity of the underexposed image is equal to or greater than the sensitivity threshold, or when the luminance evaluation value of the underexposed image is less than the luminance threshold. Alternatively, the auxiliary information generating unit 1305 may measure the integral value of the difference between the pixel value of the optical black area of ​​the image sensor and the black level when the underexposed image is shot as the amount of noise. If the amount of noise is equal to or greater than the noise threshold, the auxiliary information generating unit 1305 can determine that the underexposed image is not suitable for dynamic range expansion processing by tone conversion.

[0130] If the auxiliary information generating unit 1305 determines that the underexposed image is not suitable for the dynamic range expansion processing, it executes S1406, and if not, it executes S1405.

[0131] In S1405, the auxiliary information generating unit 1305 generates auxiliary information indicating that only the underexposed image is to be used since the underexposed image is suitable for the dynamic range expansion processing, and ends the auxiliary information generating processing.

[0132] In S1406, since neither the properly exposed image nor the underexposed image is suitable for the dynamic range expansion processing, the incidental information generating unit 1305 generates incidental information indicating that the processing is unsuitable, and ends the incidental information generating process.

[0133] When the incidental information indicates inappropriate processing, the encoding unit 1306 generates file data that does not include an incidental image and records the incidental information in association with a properly exposed image. When the incidental information does not indicate inappropriate processing, the encoding unit 1306 generates file data that records an underexposed image as an incidental image and the incidental information in association with a properly exposed image. The file data is recorded on a memory card attached to the recording unit 108.

[0134] Next, a description will be given of an image processing device serving as an external device that uses a data file recorded by the imaging device 100. The image processing device according to this embodiment may be similar to the image processing device 800 described in the first embodiment, except for the operation of the image processing unit 804. Therefore, overlapping descriptions will be omitted, and the following description will mainly focus on the operation of the image processing unit 804.

[0135] 15 is a block diagram that uses functional blocks to diagrammatically represent a series of processes related to dynamic range expansion executed by the image processing unit 804 in this embodiment. The configuration described as a functional block may be implemented by individual hardware or may be implemented as a software module.

[0136] For example, while an image processing application is being executed, the control unit 801 is instructed by the user to read image data recorded on a memory card attached to the recording unit 805. The control unit 801 acquires image data designated by the user from the recording unit 805 and stores it in the RAM 803. The control unit 801 also displays the read image in a window of the image processing application. Here, it is assumed that the read image data is data of a properly exposed image recorded in association with collateral information and a collateral image.

[0137] When a user instructs the execution of dynamic range expansion processing for a displayed image through, for example, a menu operation of an image processing application, the control unit 801 instructs the image processing unit 804 to execute dynamic range expansion processing. As a result, the image processing unit 804 executes the dynamic range expansion processing described below.

[0138] The decoding unit 1501 extracts the data of the properly exposed image, the data of the collateral information, and the data of the collateral image from the data file in the container format stored in the RAM 103 .

[0139] The dynamic range expansion processing unit 1502 applies processing for expanding the dynamic range to either the properly exposed image or the underexposed image based on the incidental information and the incidental image. The dynamic range expansion processing unit 1502 stores the processed image data in the RAM 803. The operation of the dynamic range expansion processing unit 1502 will be described in detail later.

[0140] The post-adjustment unit 1503 applies a predetermined post-adjustment to the image data to which the dynamic range expansion processing has been applied. The post-adjustment may be, for example, adjustment of color or brightness according to a user specification, or adjustment of a gradation curve. When the post-adjustment is completed, the control unit 801 records the post-adjusted visible light image data in the recording unit 805 as image data to which the dynamic range expansion processing has been applied.

[0141] The operation of the dynamic range expansion processor 1502 will be described in detail with reference to the flowchart shown in FIG. In S 1601 , the dynamic range expansion processing unit 1502 refers to the data of the additional information extracted by the decoding unit 1501 .

[0142] In S1602, the dynamic range expansion processing unit 1502 determines whether the accompanying information indicates a synthesis process or not, and if it is determined that the accompanying information indicates a synthesis process, executes S1603, and if not, executes S1604.

[0143] In S1603, the dynamic range expansion processing unit 1502 synthesizes the properly exposed image and the underexposed image, which is an auxiliary image, to generate an image with an expanded dynamic range. Fig. 17 is a diagram showing an example of the relationship between the brightness evaluation value of the properly exposed image and the synthesis ratio α of the properly exposed image in the synthesis process of the properly exposed image and the underexposed image.

[0144] The properly exposed image and the underexposed image are combined on a pixel-by-pixel basis. Therefore, the brightness evaluation value of the properly exposed image may be, for example, the luminance value of the target pixel to which the combining process is applied. When the dynamic range is expanded by combining the underexposed image, the combining ratio of the underexposed image is increased in bright parts of the properly exposed image (particularly parts close to the saturation level). Therefore, the combining ratio α shown in FIG. 17 has the characteristic that the underexposed image is not combined in low luminance parts, the properly exposed image is replaced with the underexposed image in high luminance parts, and the combining ratio α decreases as the luminance in intermediate luminance parts increases.

[0145] The dynamic range expansion processor 1502 determines a blending ratio α of the properly exposed image for a target pixel of the properly exposed image according to the characteristics shown in Fig. 17. Then, blending processing is applied to the target pixel according to the following formula (1). X = α × A + (1 - α) × B (1)

[0146] Here, X is the luminance value after synthesis, A is the luminance value of the target pixel, and B is the luminance value of the corresponding pixel in the synthetic image. The dynamic range expansion processing unit 1502 similarly applies synthesis processing to each pixel of the properly exposed image to generate a synthetic image in which the dynamic range of the properly exposed image is expanded.

[0147] The dynamic range expansion processor 1502 updates the image in the application window with a composite image whose dynamic range has been expanded, and ends the dynamic range expansion process.

[0148] In S1604, the dynamic range expansion processing unit 1502 determines whether or not the incidental information indicates the use of only underexposed images, and if it is determined that it indicates the use of only underexposed images, executes S1605, and if not, executes S1606.

[0149] In S1605, the dynamic range expansion processing unit 1502 applies tone conversion processing to the underexposed image, which is an auxiliary image, to expand the dynamic range of the underexposed image. Fig. 18 is a diagram showing an example of the characteristics of tone conversion applied to the underexposed image for dynamic range expansion. The characteristics of tone conversion are defined by the relationship between the brightness evaluation value of a pixel and the magnitude of the gain to be applied.

[0150] The tone conversion process of the underexposed image is performed on a pixel-by-pixel basis. Therefore, the brightness evaluation value of the underexposed image may be, for example, the luminance value of the target pixel to which the tone conversion process is applied. When the dynamic range is expanded by applying the tone conversion process to the underexposed image, the gradation of the dark part of the underexposed image is enhanced. Therefore, the gain amount shown in FIG. 18 does not change the brightness in the high-luminance luminance part, with the gain amount being 1. On the other hand, in the low-luminance part, the gain amount is set to a value greater than 1, and the lower the luminance, the higher the gain amount. However, the increase in the gain amount with respect to the decrease in luminance is not constant, and the rate of increase is greater in the section with low luminance. However, the increase in the gain amount is very small in the section with a luminance value close to 0.

[0151] The dynamic range expansion processing unit 1502 determines the gain amount for a target pixel of the underexposed image according to the characteristics shown in Fig. 18. Then, the dynamic range expansion processing unit 1502 applies tone conversion processing to the underexposed image by multiplying the luminance value of the target pixel by the determined gain amount.

[0152] The dynamic range expansion processing unit 1502 updates the image in the application window with an underexposed image whose dynamic range has been expanded, and ends the dynamic range expansion processing.

[0153] In S1606, the dynamic range expansion processing unit 1502 ends the dynamic range expansion processing without performing any operations. At this time, a message dialogue may be displayed on the display unit 806 to the effect that the dynamic range expansion processing cannot be applied to the properly exposed image being displayed.

[0154] When the imaging device of this embodiment captures multiple frames of visible light images with different exposure amounts, it generates additional information indicating the type of visible light image suitable for dynamic range expansion processing or an appropriate dynamic range expansion method. The imaging device then records the additional information in association with the multiple frames of visible light images. Therefore, by referring to the additional information, an external device can easily expand the dynamic range of a visualized image in an appropriate manner.

[0155] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0156] The disclosure of the present embodiment includes the following image processing device, imaging device, image processing method, and program. (Item 1) An imaging device capable of acquiring a visible light image and an invisible light image, a determining means for determining a method for using information based on the invisible light image to apply a predetermined processing to the visible light image; a generating means for generating information based on the invisible light image in accordance with a result of the determination; a recording means for recording a data file in which the visible light image, information indicating a result of the determination, and information based on the invisible light image generated by the generating means are associated with each other; An imaging device comprising: (Item 2) 2. The imaging device according to item 1, wherein the determination means determines the method based on a capturing condition of the invisible light image or a movement of the imaging device when the invisible light image was captured. (Item 3) The imaging device described in item 2, characterized in that the determination means determines that information based on the invisible light image is not suitable for use if the shutter speed when the invisible light image was captured is slower than a first speed threshold, or if the movement of the imaging device when the invisible light image was captured is greater than a first movement threshold. (Item 4) 4. The imaging device according to item 3, wherein if the determination means determines that the information based on the invisible light image is not suitable for use, the generation means does not generate the information based on the invisible light image. (Item 5) The imaging device described in item 3 or 4, characterized in that the determination means determines the first method to be used if the shutter speed when the invisible light image was captured is not slower than the first speed threshold and slower than the second speed threshold, or if the movement of the imaging device when the invisible light image was captured is smaller than the first motion threshold and larger than the second motion threshold. (Item 6) the predetermined processing is processing for improving visibility of the visible light image, 6. The imaging device according to item 5, characterized in that, when the determination means determines the first method, the generation means generates a gain map to be applied to the visible light image based on the invisible light image and the visible light image. (Item 7) 7. The imaging device according to claim 2, wherein the determining means determines the method based on brightness of the visible light image. (Item 8) 8. The imaging device according to item 7, wherein the determination means determines a second method if the brightness of the visible light image is less than a brightness threshold, and determines a third method if the brightness of the visible light image is equal to or greater than the brightness threshold. (Item 9) the predetermined processing is processing for improving visibility of the visible light image, 9. The imaging device according to item 8, wherein, when the determination means determines the second method, the generation means generates an AC image of the invisible light image. (Item 10) the predetermined processing is processing for improving visibility of the visible light image, 10. The imaging device according to item 8 or 9, wherein when the determination means determines the third method, the generation means generates the invisible light image as information based on the invisible light image. (Item 11) 4. The imaging device according to claim 1, wherein the generation means adjusts quality of the invisible light image in accordance with a result of the determination to generate information based on the invisible light image. (Item 12) 12. The imaging device according to any one of items 1 to 11, wherein the invisible light image is an infrared light image. (Item 13) An acquisition means for acquiring a data file recorded by the imaging device according to any one of items 1 to 12; an extraction means for extracting information indicating a result of the determination, the visible light image, and information based on the invisible light image from the data file; a processing means for applying a predetermined process to the visible light image using information based on the invisible light image in a manner indicated by the information indicating the result of the determination; and 13. An image processing device comprising: (Item 14) 1. A method for controlling an imaging device, the method being executed by an imaging device capable of acquiring a visible light image and an invisible light image, the method comprising: determining how to use information based on the invisible light image to apply a predetermined processing to the visible light image; a generating step of generating information based on the invisible light image in accordance with a result of the determination in the determining step; a recording step of recording a data file in which the visible light image, information indicating a result of the determination, and information based on the invisible light image generated in the generating step are associated with each other; 13. A method for controlling an imaging apparatus comprising: (Item 15) An imaging device capable of recording a first visible light image captured with a proper exposure and a second visible light image captured with an exposure amount less than the proper exposure, a determining means for determining a method for increasing the dynamic range of the first visible light image or the second visible light image; a recording means for recording a data file in which information indicating a result of the determination is associated with at least the second visible light image out of the first visible light image and the second visible light image; An imaging device comprising: (Item 16) Item 16. The imaging device described in item 15, wherein the determination means determines one of a plurality of methods including a first method of combining the first visible light image and the second visible light image and a second method of gradation converting the second visible light image. (Item 17) A program for causing a computer included in an imaging device to function as each of the means included in the imaging device according to any one of items 1 to 12, 15, and 16.

[0157] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]

[0158] 100: imaging device, 101: control unit, 102: ROM, 103: RAM, 104: optical system, 105: imaging unit, 106: A / D conversion unit, 107: image processing unit, 108: recording unit, 109: display unit

Claims

1. An imaging device capable of acquiring a visible light image and an invisible light image, a determining means for determining a method for using information based on the invisible light image to apply a predetermined process to the visible light image; a generating means for generating information based on the invisible light image in accordance with the result of the determination; a recording means for recording, on a recording medium, a data file that associates the visible light image, information indicating the result of the determination, and information based on the invisible light image generated by the generating means; An imaging device comprising:

2. 2. The imaging device according to claim 1, wherein the determining unit determines the method based on the conditions under which the invisible light image is captured or the movement of the imaging device when the invisible light image is captured.

3. The imaging device described in claim 2, characterized in that the determination means determines that information based on the invisible light image is not suitable for use if the shutter speed when the invisible light image was captured is slower than a first speed threshold, or if the movement of the imaging device when the invisible light image was captured is greater than a first movement threshold.

4. 4. The imaging device according to claim 3, wherein, when the determining means determines that the information based on the invisible light image is not suitable for use, the generating means does not generate the information based on the invisible light image.

5. The imaging device described in claim 3, characterized in that the determination means determines the first method to be used if the shutter speed when capturing the invisible light image is not slower than the first speed threshold and slower than the second speed threshold, or if the movement of the imaging device when capturing the invisible light image is smaller than the first motion threshold and greater than the second motion threshold.

6. the predetermined processing is processing for improving visibility of the visible light image, 6. The imaging device according to claim 5, wherein, when the determination means determines the first method, the generation means generates a gain map to be applied to the visible light image based on the invisible light image and the visible light image.

7. 3. The imaging device according to claim 2, wherein the determining means determines the method based on brightness of the visible light image.

8. 8. The imaging device according to claim 7, wherein the determining means determines a second method if the brightness of the visible light image is less than a brightness threshold, and a third method if the brightness of the visible light image is equal to or greater than the brightness threshold.

9. the predetermined processing is processing for improving visibility of the visible light image, 9. The imaging device according to claim 8, wherein when the determining means determines that the second method is used, the generating means generates an AC image of the invisible light image.

10. the predetermined processing is processing for improving visibility of the visible light image, 9. The imaging device according to claim 8, wherein, when the determining means determines that the third method is used, the generating means generates the invisible light image as information based on the invisible light image.

11. The imaging device according to claim 1 , wherein the generating means adjusts the quality of the invisible light image in accordance with the result of the determination, and generates information based on the invisible light image.

12. 2. The imaging device according to claim 1, wherein the invisible light image is an infrared light image.

13. an acquisition unit for acquiring a data file recorded by the imaging device according to any one of claims 1 to 12; an extracting means for extracting, from the data file, information indicating the determination result, the visible light image, and information based on the invisible light image; a processing means for applying a predetermined process to the visible light image using information based on the invisible light image in a manner indicated by the information indicating the result of the determination; and 1. An image processing device comprising:

14. 1. A control method for an imaging device, executed by an imaging device capable of acquiring visible light images and invisible light images, comprising: determining how to use information based on the invisible light image to apply a predetermined process to the visible light image; generating information based on the invisible light image according to a result of the determining; recording a data file that associates the visible light image, information indicating the result of the determination, and information based on the invisible light image generated by the generation, on a recording medium; 10. A method for controlling an imaging device, comprising:

15. An imaging device capable of recording a first visible light image captured with a proper exposure and a second visible light image captured with an exposure amount less than the proper exposure, a determining means for determining a method for expanding the dynamic range of the first visible light image or the second visible light image; a recording means for recording, on a recording medium, a data file that associates information indicating the determination result with at least the second visible light image of the first visible light image and the second visible light image; An imaging device comprising:

16. 16. The imaging device according to claim 15, wherein the determining means determines one of a plurality of methods, including a first method of combining the first visible light image and the second visible light image, or a second method of gradation-converting the second visible light image.

17. A control method for an imaging device, executed by the imaging device capable of recording a first visible light image captured with a proper exposure and a second visible light image captured with an exposure amount less than the proper exposure, comprising: determining a method for increasing the dynamic range of the first visible light image or the second visible light image; recording, on a recording medium, a data file that associates information indicating the determination result with at least the second visible light image of the first visible light image and the second visible light image; 10. A method for controlling an imaging device, comprising:

18. A program for causing a computer included in an imaging apparatus to function as each of the means included in the imaging apparatus according to any one of claims 1 to 12, 15, and 16.

Images