Imaging device and its control method, program, and storage medium

Abstract

To provide an imaging apparatus capable of properly correcting visible light images by using non-visible light images.SOLUTION: An imaging apparatus includes a determining unit for determining an exposure amount for capturing a visible light image and a non-visible light image, an imaging unit for capturing a visible light image and a plurality of non-visible light images at the exposure amount determined by the determining unit, a synthesizing unit for synthesizing the plurality of non-visible light images to generate a synthesized non-visible light image, and a correcting unit for correcting the visible light image using the synthesized non-visible light image.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a technique for correcting a visible light image using an invisible light image.

Background Art

[0002] Conventionally, in images taken with visible light, there are cases where the contrast is low or the visibility is poor. For example, when photographing a distant mountain range with visible light, it may not be possible to sufficiently capture the information on the unevenness of the mountain range due to fog or haze, resulting in a flat impression with low contrast. Also, in photography under low illuminance conditions such as in dark places, the visibility may be poor.

[0003] On the other hand, in an invisible light (infrared light) image, it is possible to capture information on the unevenness of the mountain range compared to a visible light image and to take an image with high visibility under low illuminance conditions. Therefore, a technique has been proposed to correct a visible light image using an infrared light image to improve the visibility.

[0004] Patent Document 1 discloses a technique for detecting a region that is difficult to visually recognize in a visible light image and improving the visibility by synthesizing an infrared light image for that region.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, visible light and infrared light have different spectral characteristics, and when infrared images are captured using exposure conditions determined based on visible light, the brightness of the visible light image and the infrared image may differ. In such cases, attempting to match the brightness of the infrared image to that of the visible light image may result in the infrared image becoming highly noisy, or it may be saturated, making it impossible to obtain the information necessary for correction.

[0007] The present invention has been made in view of the above-mentioned problems, and its objective is to provide an imaging device that can appropriately correct a visible light image using a non-visible light image. [Means for solving the problem]

[0008] The imaging device according to the present invention includes a determination means for determining the exposure for capturing a visible light image and a non-visible light image, and capturing a visible light image with the exposure determined by the determination means. multiple An imaging means for capturing several invisible light images, a synthesis means for combining the multiple invisible light images to generate a composite invisible light image, and a correction means for correcting the visible light image using the composite invisible light image. A number determination means for determining the number of non-visible light images to be captured so that the noise level of the composite non-visible light image and the noise level of the visible light image are approximately the same, It is characterized by having the following features. [Effects of the Invention]

[0009] According to the present invention, it is possible to appropriately correct a visible light image using a non-visible light image. [Brief explanation of the drawing]

[0010] [Figure 1] A figure showing examples of visible light images and infrared light images in the first embodiment. [Figure 2] A diagram showing the filter used in the imaging unit in the first embodiment. [Figure 3] A diagram showing the imaging device in the first embodiment. [Figure 4] A diagram showing the configuration of the image processing unit in the first embodiment. [Figure 5]A flowchart showing the processing flow of the imaging device in the first embodiment. [Figure 6] A diagram showing an example of block division in the first embodiment. [Figure 7] A diagram showing the degree of enhancement in the first embodiment. [Figure 8] A diagram showing the configuration of the image processing unit in the second embodiment. [Figure 9] A flowchart showing the processing flow of the imaging device in the second embodiment. [Figure 10] A figure showing examples of visible light images and infrared light images in the second embodiment. [Modes for carrying out the invention]

[0011] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0012] (First Embodiment) The present invention relates to a technique for correcting a visible light image using a non-visible light image. In this embodiment, the case in which the brightness of the infrared light image, which is the non-visible light image, is dimmer than the brightness of the visible light image will be described.

[0013] For example, as shown in Figure 1(a), visible light images may have low contrast due to the effects of fog, haze, etc. On the other hand, as shown in Figure 1(b), infrared light images, when taken under the same exposure conditions as visible light images, may have higher contrast but be darker than visible light images. Furthermore, in this embodiment, we will describe a case where the imaging unit that acquires visible light images and infrared light images has a filter configuration in which R (red), G (green), and B (blue) color filters for visible light image capture and an IR filter for infrared light image capture are regularly arranged, as shown in Figure 2(a).

[0014] FIG. 3 is a block diagram showing the configuration of an imaging device according to the first embodiment of the present invention.

[0015] In FIG. 3, the control unit 101 is, for example, a CPU, reads a control program for each block included in the imaging device 100 from the ROM 102 described later, expands it in the RAM 103 described later, and executes it. Thereby, the control unit 101 controls the operations of each block included in the imaging device 100.

[0016] The ROM 102 is an electrically erasable and recordable non-volatile memory, and stores operation programs for each block included in the imaging device 100, as well as parameters and the like necessary for the operations of each block.

[0017] The RAM 103 is a rewritable volatile memory, and is used for expanding programs executed by the control unit 101 and temporarily storing data generated by the operations of each block included in the imaging device 100.

[0018] The optical system 104 is composed of a lens group including a zoom lens and a focus lens, and forms an object image on the imaging surface of an imaging element arranged in the imaging unit 105. [[ID=2I]]

[0019] The imaging unit 105 includes an imaging element such as a CCD or a CMOS sensor, photoelectrically converts the optical image formed on the imaging surface of the imaging element by the optical system 104, and outputs the obtained analog image signal to the A / D conversion unit 106.

[0020] The A / D conversion unit 106 converts the input analog image signal into digital image data. The digital image data output from the A / D conversion unit 106 is temporarily stored in the RAM 103.

[0021] The image processing unit 107 performs various image processing operations on the image data stored in the RAM 103. Specifically, it performs various image processing operations for developing, displaying, and recording digital image data, such as correcting pixel defects caused by the optical system 104 or image sensor, demosaicing, white balance correction, color interpolation, and gamma processing.

[0022] The recording unit 108 records data, including image data, on its built-in recording medium. The display unit 109 includes a display device such as an LCD and displays images stored in the RAM 103 and images recorded in the recording unit 108 on the display device. The display unit 109 also displays a user interface for receiving instructions from the user.

[0023] The instruction input unit 110 is an input interface that includes various physical operating elements such as a touch panel and a shutter button, and accepts instruction input from the user. The communication unit 111 connects to an external server wirelessly and exchanges data including image data and placement information, which will be described later. An example of an external server is a server that manages a service such as a social networking service (hereinafter referred to as SNS) that transmits captured images and makes them available to SNS viewers.

[0024] Furthermore, the imaging device 100, under the control of the control unit 101, performs live view display, which sequentially displays the analog image signals output from the imaging unit 105 on a display device via the A / D conversion unit 106, RAM 103, image processing unit 107, and display unit 109. When using live view display, it is possible to prepare for actual shooting, such as determining the composition for recording image data to a recording medium, changing exposure conditions such as aperture and shutter speed, and image processing conditions to change image quality such as brightness, gradation, and color.

[0025] Next, Figure 4 is a block diagram showing the configuration of the image processing unit 107.

[0026] In Figure 4, the image processing unit 107 includes a brightness adjustment unit 201 that adjusts the brightness of an infrared light image, a synthesis processing unit 202 that synthesizes multiple infrared light images with adjusted brightness to generate a composite infrared light image (composite invisible light image), and an enhancement processing unit 203 that performs enhancement processing on a visible light image using the composite infrared light image.

[0027] Figure 5 is a flowchart showing the processing flow of the imaging device 100 in this embodiment. Each step in this flowchart is executed by the control unit 101 or by each block of the imaging device 100 at the instruction of the control unit 101.

[0028] When the user starts up the imaging device 100, in step S301, the control unit 101 controls the optical system 104, imaging unit 105, A / D conversion unit 106, RAM 103, image processing unit 107, and display unit 109 to start live view display. During the live view display period, the imaging device 100 captures and acquires live images sequentially, and the acquired live images are displayed on the display device of the display unit 109. The user can determine the composition and change exposure conditions and image processing conditions while checking the live images that are displayed sequentially.

[0029] In step S301, the control unit 101 determines the exposure conditions for the visible light image to be captured, evaluates the brightness of the infrared image under the same exposure conditions, and determines the number of infrared images to be captured and the brightness adjustment coefficient required in subsequent steps. In this embodiment, it is assumed that the infrared image is darker than the visible light image, and in order to match the brightness of the infrared image (composite infrared image) to that of the visible light image, a brightness adjustment coefficient is multiplied in subsequent steps. However, since the noise also increases by the amount of multiplication by the coefficient, noise is suppressed by capturing and combining multiple infrared images. For example, if the brightness of the infrared image is evaluated and compared with the brightness of the visible light image and it is one stop darker, the number of infrared images to be captured is set to 2 and the brightness adjustment coefficient to 2. If it is two stops darker, the number of infrared images to be captured is set to 4 and the brightness adjustment coefficient to 4, and so on. In other words, the number of infrared images to be captured is determined so that the amount of noise in the composite infrared image and the amount of noise in the visible light image are approximately the same.

[0030] In step S302, the control unit 101 uses the imaging unit 105 to acquire a visible light image based on the exposure conditions determined in step S301, and multiple infrared light images based on the exposure conditions and number of images determined in step S301.

[0031] In step S303, the control unit 101 uses the brightness adjustment unit 201 to adjust the brightness of the infrared light image based on the brightness adjustment coefficient determined in step S301. For example, if the brightness adjustment coefficient is 2, the brightness of each of the multiple infrared light images acquired in step S302 is adjusted by multiplying them by the coefficient of 2.

[0032] In step S304, the control unit 101 uses the synthesis processing unit 202 to synthesize multiple infrared images whose brightness was adjusted in step S303 to generate a composite infrared image. For example, if there are two infrared images whose brightness was adjusted in step S303, the composite infrared image is generated by adding the two images together and averaging them. This makes it possible to generate a composite infrared image with high contrast while suppressing an increase in noise.

[0033] Although a detailed explanation is omitted here, it is possible to perform alignment on multiple infrared images using known techniques before performing averaging. In this embodiment, since a filter like the one shown in Figure 2(a) is used, the visible light image and the first infrared image are captured at the same time. On the other hand, the second and subsequent infrared images are captured at different times, so the position of the subject may be shifted. Therefore, it is possible to perform alignment using the first infrared image as a reference to align the position of the subject.

[0034] In step S305, the control unit 101 uses the enhancement processing unit 203 to perform enhancement processing (correction processing) on ​​the captured visible light image based on the composite infrared light image generated in step S304.

[0035] In this step, we first calculate tonal information for the visible light image and the composite infrared light image. As tonal information, we obtain the edge intensity characteristics within the subject. This is obtained by detecting edge signals by applying a bandpass filter to the image and calculating the integral value of the edge signals of an arbitrary block size. The calculated evaluation value increases when the integral value of the edge signals is large, that is, when there is a lot of texture. Figure 601 is the result of dividing into M × N blocks of an arbitrary block size, and the evaluation value is calculated using the pixel value of one block, 602, using the following formula (1).

[0036] E = ΣiΣje(i,j) …(1) e(i,j): Edge strength value at each coordinate within any block E: Integral value of edge strength in any block For the visible light image and the composite infrared light image, an evaluation value E, which represents the grayscale information, is calculated for each, and these evaluation values ​​are designated as Ecolor and Eir.

[0037] Next, we will explain the process for determining the areas to be enhanced in the visible light image. The areas to be enhanced are determined using the grayscale information calculated earlier. First, the grayscale information of the visible light image and the composite infrared image are compared. To improve the visibility of the visible light image, it is necessary to detect areas where there are few grayscales in the visible light image and many grayscales in the composite infrared image. Therefore, the difference Diff is calculated from the evaluation values ​​Eir and Ecolor of blocks at the same position for which grayscale information was calculated, as shown in equation (2).

[0038] Diff = Eir - Ecolor …(2) A set of blocks in a visible light image where the evaluation value Ecolor is small and the difference Diff between evaluation values ​​is larger than a predetermined value is determined to be an enhancement region.

[0039] Next, we will explain the process of enhancing the visible light image. In the enhancement region determined earlier, the enhancement process is performed using Figure 7 and the following equation (3). Figure 7(a) shows the ratio of mixing the visible light image and the composite infrared image with respect to the evaluation value Ecolor of the visible light image. ratio1 shows the mixing ratio of the composite infrared image based on Ecolor, and the mixing ratio of the composite infrared image is increased for regions where Ecolor is small, i.e., regions with fewer gradations in the visible light image.

[0040] Figure 7(b) shows the ratio of mixing the visible light image and the composite infrared image with respect to the difference in evaluation values ​​(Diff) between the visible light image and the composite infrared image. Ratio 2 represents the mixing ratio of the composite infrared image based on Diff, and the proportion of the composite infrared image is increased in regions where Diff is large, i.e., regions where the composite infrared image has more tonal range than the visible light image. Enhancement processing is performed using the values ​​of ratio 1 and ratio 2 and equation (3) to generate an enhanced visible light image.

[0041] I'color=(1-ratio1×ratio2)×Icolor + ratio1×ratio2×Iir …(3) ratio1: Mixing ratio calculated from Ecolor ratio2: Mixing ratio calculated from Diff Icolor: Visible light image Iir: Synthetic infrared image I'color: Enhanced visible light image As described above, according to this embodiment, even when the infrared light image is darker than the visible light image, it is possible to obtain information from the infrared light image that allows for appropriate enhancement processing to be applied to the visible light image, thereby improving the contrast of the visible light image.

[0042] In this embodiment, the brightness evaluation of the visible light image and infrared light image in step S301 may be performed on the entire image or on a portion of the image. As an example of using a portion of the image, the enhancement region extracted in step S305 may be used. In this case, the gradation information calculation and enhancement region determination that were performed in step S305 are performed in the preceding step S301, and the results are used for brightness evaluation.

[0043] Furthermore, although step S305 of this embodiment describes a process of mixing a composite infrared light image with regions of the visible light image that have few gradations, the invention is not limited to this, and AC components extracted from the composite infrared light image may also be mixed.

[0044] Furthermore, although this embodiment describes the case in which the filter shown in Figure 2(a) is used as the filter for the imaging unit, it is not limited to this, and as shown in Figures 2(b) and (c), a filter for visible light images and a filter for infrared light images may be prepared and switched between to perform imaging.

[0045] Furthermore, in this embodiment, grayscale information from visible light images and infrared light images was used to determine the area to be enhanced, but the user may select the area to be enhanced.

[0046] Furthermore, in this embodiment, multiple infrared images captured and a composite infrared image created by combining them were used for enhancement processing. However, both of these may be recorded together with the enhanced visible light image, or only the composite infrared image may be recorded together with the enhanced visible light image.

[0047] (Second embodiment) The first embodiment described a case where the brightness of the infrared image is dimmer than that of the visible light image. The second embodiment describes a case where the infrared image is partially saturated under exposure conditions based on the visible light image.

[0048] Note that the same reference numerals as in the first embodiment represent the same operations and processes as in the first embodiment, and their explanations are omitted. Also, the configuration of the imaging device in the second embodiment is the same as in the first embodiment, so its explanation is omitted.

[0049] Figure 8 is a block diagram showing the configuration of the image processing unit 107.

[0050] In Figure 8, the image processing unit 107 includes a synthesis processing unit 801 that synthesizes multiple infrared light images with different exposures, and an enhancement processing unit 203 that performs enhancement processing on a visible light image using this synthesized infrared light image.

[0051] Figure 9 is a flowchart showing the processing flow of the imaging device 100 in this embodiment. Each step in this flowchart is executed by the control unit 101 or by each block of the imaging device 100 at the instruction of the control unit 101.

[0052] In step S901, the procedure is the same as in step S301 in Figure 5, up to determining the exposure conditions for the visible light image to be captured and evaluating the brightness of the infrared light image under those exposure conditions. As shown in Figure 10(a), the visible light image has low contrast, similar to the first embodiment. On the other hand, as shown in Figure 10(b), the infrared light image is partially saturated (saturation is above a predetermined level), and in this state, it is difficult to obtain information for appropriate enhancement processing to the visible light image. In such cases, the control unit 101 controls the system in the subsequent step S902 to shorten the Tv (exposure time) and capture multiple infrared light images.

[0053] In step S902, the control unit 101 uses the imaging unit 105 to capture a visible light image based on the exposure conditions determined in step S901, and to capture multiple infrared light images with the same exposure conditions as determined in step S901 and with a shortened Tv. As an example of shortening Tv (exposure time), as shown in Figures 10(c) and (d), the Tv is changed to -1 stop and -2 stops respectively with respect to the exposure conditions determined in step S901, and infrared light images are captured.

[0054] In step S903, the control unit 101 uses the synthesis processing unit 801 to synthesize multiple infrared images with different exposures taken in step S902. A detailed explanation of the synthesis method is omitted, but for example, a known HDR synthesis method is used. This prevents saturated areas in the infrared image taken under the same exposure conditions as the visible light image from becoming saturated, making it possible to generate a high-contrast synthesized infrared image.

[0055] The enhancement process for visible light images is the same as in the first embodiment, so we will omit the explanation.

[0056] The second embodiment described above concerns the case where the infrared light image is partially saturated. As described above, according to the above embodiment, it is possible to appropriately correct the visible light image using the invisible light image.

[0057] (Other embodiments) Furthermore, the present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0058] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0059] 100: Imaging device, 101: Control unit, 102: ROM, 103: RAM, 104: Optical system, 105: Imaging unit, 106: A / D conversion unit, 107: Image processing unit, 108: Recording unit, 109: Display unit, 110: Instruction input unit, 111: Communication unit

Claims

1. A determination means for determining the exposure for capturing visible light images and invisible light images, An imaging means that captures a visible light image and multiple invisible light images with the exposure determined by the aforementioned determination means, A synthesis means for generating a composite invisible light image by combining the aforementioned multiple invisible light images, Correction means for correcting the visible light image using the composite invisible light image, A number determination means for determining the number of non-visible light images to be captured so that the noise level of the composite non-visible light image and the noise level of the visible light image are approximately the same, An imaging device characterized by comprising:

2. The imaging apparatus according to claim 1, further comprising brightness adjustment means for adjusting the brightness of the plurality of non-visible light images.

3. The imaging apparatus according to claim 2, characterized in that the synthesis means synthesizes the plurality of invisible light images whose brightness has been adjusted by the brightness adjustment means to generate the synthesized invisible light image.

4. The imaging apparatus according to claim 3, characterized in that the brightness adjustment means adjusts the brightness of the plurality of invisible light images so that the brightness of the composite invisible light image matches the brightness of the visible light image.

5. The imaging apparatus according to claim 1, characterized in that the determination means causes the exposure of the plurality of invisible light images to differ when the saturation of the invisible light image is above a predetermined level.

6. The imaging apparatus according to claim 1, characterized in that the determination means shortens the exposure time of the plurality of invisible light images when the saturation of the invisible light image is above a predetermined level.

7. The imaging apparatus according to any one of claims 1 to 6, further comprising a region determination means for determining a region in the visible light image to be corrected using the composite non-visible light image.

8. The imaging apparatus according to claim 7, characterized in that the region determination means determines a region to be corrected where the gradation of the visible light image is less than the gradation of the composite non-visible light image, and the difference between the gradation of the visible light image and the gradation of the composite non-visible light image is greater than a predetermined value.

9. The imaging apparatus according to claim 7, characterized in that the region determination means determines the region to be corrected according to the user's instructions.

10. The imaging apparatus according to any one of claims 1 to 9, further comprising recording means for recording the visible light image and recording at least one of the plurality of non-visible light images and the composite non-visible light image.

11. The imaging apparatus according to any one of claims 1 to 10, characterized in that the non-visible light image is an infrared light image.

12. The imaging apparatus according to any one of claims 1 to 11, characterized in that the correction means performs an enhancement process as a process for correcting the visible light image.

13. A determination step for determining the exposure for capturing visible light images and invisible light images, An imaging step is performed in which a visible light image is captured and multiple invisible light images are captured with the exposure determined in the above determination step, A synthesis step of generating a composite invisible light image by combining the aforementioned multiple invisible light images, A correction step of correcting the visible light image using the composite non-visible light image, A step to determine the number of images to be captured such that the noise level of the composite non-visible light image and the noise level of the visible light image are approximately the same, A control method for an imaging device, characterized by having the following features.

14. A program for causing a computer to function as one of the means of an imaging apparatus according to any one of claims 1 to 12.

15. A computer-readable storage medium storing a program for causing the computer to function as each of the means of the imaging apparatus described in any one of claims 1 to 12.

Images