Image processing device, control method for the same, and program

Abstract

To calculate an appropriate invisible light irradiation condition.SOLUTION: An image processing device 100 includes an irradiation unit 107. The image processing device 100 calculates the feature amount of a visible light image obtained by imaging a subject irradiated with visible light by the irradiation unit 107, and calculates the feature amount of an invisible light image obtained by imaging the subject irradiated with invisible light by the irradiation unit 107. The image processing device 100 calculates the invisible light irradiation information, which is an irradiation condition of the invisible light irradiated by the irradiation unit 107 on the basis of the feature amount of the visible light image and the feature amount of the invisible light image.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a control method for an image processing apparatus, and a program.

Background Art

[0002] A method for improving the image quality of a visible light image with poor visibility due to haze or darkness using a non-visible light image such as an infrared light image is known. In order to acquire a visible light image and a non-visible light image, it is necessary to acquire light in the visible light wavelength band and light in the non-visible light wavelength band with appropriate exposure. However, generally, the intensity of light is different between visible light and non-visible light, and since the exposure time and aperture value cannot be set individually for the visible light light receiving element and the non-visible light light receiving element, it is difficult to acquire the visible light image and the non-visible light image with appropriate exposure respectively.

[0003] In Patent Document 1, in order to acquire a visible light image and a non-visible light image with appropriate exposure respectively, a technique for evaluating the brightness of two images with different wavelengths and controlling the irradiation amounts of two light sources with different wavelengths based on the evaluation results is disclosed.

Prior Art Documents

[0007] To achieve the above objective, the present invention provides an image processing apparatus comprising an illumination means, characterized by comprising: a first feature calculation means for calculating feature quantities of a visible light image obtained by photographing a subject irradiated with visible light by the illumination means; a second feature calculation means for calculating feature quantities of a non-visible light image obtained by photographing the subject irradiated with non-visible light by the illumination means; and a calculation means for calculating the irradiation conditions of the non-visible light irradiated by the illumination means based on the feature quantities of the visible light image and the feature quantities of the non-visible light image. [Effects of the Invention]

[0008] According to the present invention, it is possible to calculate appropriate irradiation conditions for invisible light. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram schematically showing the configuration of the image processing apparatus according to this embodiment. [Figure 2] This is a block diagram schematically showing the configuration of the non-visible light irradiation condition calculation unit included in the image processing unit shown in Figure 1. [Figure 3] This flowchart shows the procedure for the invisible light control processing performed by the image processing device shown in Figure 1. [Figure 4] This figure shows an example of the grayscale information for visible light and invisible light images acquired by the image processing device shown in Figure 1. [Figure 5] Figure 3 is a flowchart showing the procedure for calculating non-visible light irradiation information in step S306. [Figure 6] This figure illustrates the determination in step S503 of Figure 5. [Figure 7]Figure 5 is a diagram illustrating the calculation of non-visible light irradiation information in step S504. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described in detail below with reference to the drawings. In this embodiment, the image processing device enhances the gradation, contrast, and resolution of visible light images by adjusting the illumination conditions for invisible light based on the characteristic quantities of visible light and invisible light images obtained by photographing the same subject at the same angle of view, and then irradiates with invisible light. The illumination conditions for invisible light include the amount of invisible light, the direction of illumination, and the illumination range of the invisible light.

[0011] Figure 1 is a schematic block diagram showing the configuration of the image processing apparatus 100 according to this embodiment. In Figure 1, the image processing apparatus 100 comprises a control unit 101, a ROM 102, a RAM 103, a UI unit 104, an image processing unit 105, a display unit 106, and an illumination unit 107. These are interconnected by a bus 108.

[0012] The control unit 101 is, for example, a CPU. The control unit 101 reads the control programs for each block of the image processing device 100 from the ROM 102, and then loads the read control programs into the RAM 103 and executes them. In this way, the control unit 101 controls the operation of each block of the image processing device 100. The ROM 102 is an electrically erasable and recordable non-volatile memory. The ROM 102 stores the control programs for each block of the image processing device 100, as well as parameters necessary for the operation of each block. The RAM 103 is a rewritable volatile memory. The RAM 103 is used as an area to load programs executed by the control unit 101, etc., and as an area to temporarily store data generated by the operation of each block of the image processing device 100.

[0013] The UI unit 104 receives user input for the image processing device 100. For example, a pointing device or keyboard can be used as the UI unit 104. A pointing device could be a touch panel or a mouse. The image processing unit 105 performs image processing such as white balance adjustment, color interpolation, and gamma processing on the image data stored in the RAM 103. The image processing unit 105 also includes a non-visible light irradiation condition calculation unit 200, as shown in Figure 2 (described later). The display unit 106 displays images stored in the ROM 102. The display unit 106 is, for example, a display device such as a smartphone or television. The display unit 106 also provides UI displays for receiving instructions from the user. The illumination unit 107 is, for example, a strobe light, which illuminates the subject with visible and non-visible light. The illumination unit 107 adjusts the illumination amount, direction, and range of non-visible light on the subject based on the non-visible light irradiation information calculated by the image processing unit 105 (described later).

[0014] Figure 2 is a schematic block diagram showing the configuration of the invisible light irradiation condition calculation unit 200 included in the image processing unit 105 of Figure 1. The invisible light irradiation condition calculation unit 200 consists of a visible light image feature quantity calculation unit 201, an invisible light image feature quantity calculation unit 202, an enhancement effect calculation unit 203, and an invisible light irradiation condition adjustment unit 204.

[0015] The visible light image feature calculation unit 201 calculates the feature quantities of a visible light image obtained by photographing a subject illuminated with visible light by the illumination unit 107. The feature quantities of a visible light image are, for example, the gradation information of the brightness component of the visible light image. The visible light image feature calculation unit 201 extracts information on the AC component (gradation information) of the brightness component (I component) of the visible light image by performing a bandpass filter process that extracts a predetermined frequency band on the brightness component (I component) of the visible light image. The visible light image feature calculation unit 201 outputs this gradation information as a feature quantity of the visible light image to the enhancement effect calculation unit 203.

[0016] The invisible light image feature calculation unit 202 calculates the feature quantities of an invisible light image obtained by photographing a subject illuminated with invisible light such as infrared light by the illumination unit 107. The feature quantities of an invisible light image are, for example, the grayscale information of the invisible light image. The invisible light image feature calculation unit 202 extracts AC component information (grayscale information) of the invisible light image by performing a bandpass filter process on the invisible light image to extract a predetermined frequency band. The invisible light image feature calculation unit 202 outputs this grayscale information as a feature quantity of the invisible light image to the enhancement effect calculation unit 203.

[0017] The enhancement effect calculation unit 203 calculates the enhancement effect based on the visible light image feature quantities obtained from the visible light image feature quantity calculation unit 201 and the invisible light image feature quantities obtained from the invisible light image feature quantity calculation unit 202. The enhancement effect refers to the effect of improving gradation, contrast, resolution, etc. The invisible light irradiation condition adjustment unit 204 calculates invisible light irradiation information, which is the irradiation condition for the invisible light irradiated by the irradiation unit 107, based on the enhancement effect calculated by the enhancement effect calculation unit 203.

[0018] Figure 3 is a flowchart showing the procedure for invisible light control processing performed by the image processing device 100 in Figure 1. The processing in Figure 3 is realized by the control unit 101 reading a control program from the ROM 102, loading it into the RAM 103, and executing it. In the invisible light control processing in Figure 3, the image processing device 100 assumes that it has already acquired visible light images and invisible light images obtained by switching the light emitted from the illumination unit 107 and photographing the same subject at the same angle of view, from the imaging device that performed these photographs. The visible light image is an image obtained by photographing a subject illuminated with visible light by the illumination unit 107, and the invisible light image is an image obtained by photographing a subject illuminated with invisible light by the illumination unit 107. In this embodiment, a configuration in which visible light images and invisible light images are acquired from the imaging device that photographed the subject is described, but the configuration is not limited to this. For example, the image processing device 100 may have an imaging unit, and this imaging unit may photograph the subject and generate visible light images and invisible light images.

[0019] In FIG. 3, first, the control unit 101 stores the acquired visible light image in the RAM 103 (step S301). Next, the control unit 101 stores the acquired non-visible light image in the RAM 103 (step S302). Next, the control unit 101 calculates the feature amount of the visible light image stored in the RAM 103 by the visible light image feature amount calculation unit 201 of the image processing unit 105 (step S303). Next, the control unit 101 calculates the feature amount of the non-visible light image stored in the RAM 103 by the non-visible light image feature amount calculation unit 202 of the image processing unit 105 (step S304).

[0020] Next, the control unit 101 calculates an enhancement effect based on the feature amount of the visible light image and the feature amount of the non-visible light image by the enhancement effect calculation unit 203 of the image processing unit 105 (step S305). Here, the calculation of the enhancement effect will be described using FIG. 4. FIG. 4(a) is a diagram showing the gradation information of the visible light image, and FIG. 4(b) is a diagram showing the gradation information of the non-visible light image. In step S305, the difference in gradation information of corresponding regions in the visible light image and the non-visible light image is calculated. Specifically, the difference between the gradation information of a predetermined region L401 in the visible light image and the gradation information of the predetermined region L401 in the non-visible light image is calculated. The predetermined region L401 is a region designated by the user or a low-contrast region in the visible light image where the gradation information is below a predetermined value. The region L401 in FIG. 4(a) shows an example where the gradation property is insufficient due to haze or the like. On the other hand, the region L401 in FIG. 4(b) shows an example where sufficient gradation property is obtained as an image signal due to the wavelength characteristics of the non-visible light. The enhancement effect calculation unit 203 outputs the calculated result as the enhancement effect to the non-visible light irradiation condition adjustment unit 204.

[0021] Next, the control unit 101, using the image processing unit 105, performs the invisible light irradiation information calculation process shown in Figure 5 (step S306), which will be described later, and calculates invisible light irradiation information. Then, the control unit 101 adjusts the irradiation amount, irradiation direction, and irradiation range based on the invisible light irradiation information and irradiates the subject with invisible light from the irradiation unit 107 (step S307), and terminates this process.

[0022] Figure 5 is a flowchart showing the procedure for calculating invisible light irradiation information in step S306 of Figure 3. The invisible light irradiation information calculation process in Figure 5 is performed by the image processing unit 105.

[0023] In Figure 5, the enhancement effect calculation unit 203 of the image processing unit 105 inputs the calculated enhancement effect to the invisible light illumination condition adjustment unit 204 (step S501). Next, the invisible light illumination condition adjustment unit 204 of the image processing unit 105 determines the enhancement effect region for determining the direction and range of invisible light illumination (step S502). In step S502, the invisible light illumination condition adjustment unit 204 determines, for example, a region specified by the user as the enhancement effect region. This allows the user's intentions to be reflected in the determination of the direction and range of invisible light illumination. Alternatively, the invisible light illumination condition adjustment unit 204 determines a low-contrast region in the visible light image where the gradation information is below a predetermined value as the enhancement effect region. This allows the region where image quality improvement is expected to be necessary to be determined as the enhancement effect region without forcing the user to specify a region.

[0024] Next, the non-visible light irradiation condition adjustment unit 204 determines whether the enhancement effect in the enhancement effect region determined in step S502 is below a threshold (step S503). Figure 6(a) shows an example where the enhancement effect in the enhancement effect region is below a threshold, and Figure 6(b) shows an example where the enhancement effect in the enhancement effect region exceeds a threshold. In Figures 6(a) and 6(b), region L601 indicates the enhancement effect region.

[0025] In step S503, if the enhancement effect in the enhancement effect region determined in step S502 is below a threshold (see, for example, Figure 6(a)), the invisible light irradiation condition adjustment unit 204 calculates invisible light irradiation information so that the enhancement effect exceeds the threshold in the enhancement effect region (step S504). Here, the calculation of invisible light irradiation information will be explained. Figure 7(a) is a diagram showing the gradation information of a visible light image, and Figure 7(b) is a diagram showing the gradation information of an invisible light image. Region L701 in Figures 7(a) and 7(b) shows an example where gradation is insufficient due to haze, etc. Region L701 in Figures 7(a) and 7(b) indicates the enhancement effect region. In other words, region L701 is the same region as region L601.

[0026] For example, in a non-visible light image with improper exposure, effects such as haze remain, resulting in insufficient tonal information as shown in Figure 7(b). The enhancement effect, which is the difference between the tonal information of such a non-visible light image and the tonal information of a visible light image, falls below a threshold. In such cases, in step S504, the non-visible light irradiation condition adjustment unit 204 determines the irradiation direction and irradiation range of the non-visible light so as to irradiate the area corresponding to the enhancement effect region with non-visible light, and also determines the irradiation amount of non-visible light from the enhancement effect. The non-visible light irradiation condition adjustment unit 204 outputs the determined irradiation direction, irradiation range, and irradiation amount as non-visible light irradiation information. After that, this process ends.

[0027] In step S503, if the enhancement effect in the enhancement effect region determined in step S502 exceeds a threshold (see, for example, Figure 6(b)), this process terminates.

[0028] According to the embodiment described above, invisible light irradiation information, which is the irradiation condition for invisible light irradiated by the irradiation unit 107, is calculated based on the feature quantities of the visible light image and the feature quantities of the invisible light image. As a result, when capturing invisible light images to improve the image quality of visible light images, optimized invisible light can be irradiated onto the subject, thereby improving the image quality of visible light images.

[0029] Alternatively, based on the invisible light irradiation information calculated in step S504, invisible light with adjusted irradiation amount, direction, and range may be irradiated onto the subject, an invisible light image obtained by photographing the subject may be acquired from the imaging device, and the invisible light control processing described above may be executed again based on the invisible light image. This allows for readjustment of the irradiation amount, direction, and range of the invisible light if the intended enhancement effect is not obtained even after irradiating the subject with invisible light with adjusted irradiation amount, direction, and range based on the above invisible light irradiation information.

[0030] In the above-described embodiment, the shooting conditions for capturing the subject may be adjusted based on the features of the visible light image, the features of the non-visible light image, and the non-visible light illumination information. Examples of shooting conditions include ISO sensitivity, F-number, and shutter speed. This allows for optimization of not only the amount, direction, and range of non-visible light illumination, but also the shooting conditions to improve the image quality of the visible light image.

[0031] Furthermore, in the above-described embodiment, a warning may be issued if the non-visible light irradiation information includes information that exceeds the specifications of the irradiation unit 107. For example, if the non-visible light irradiation information includes at least one of the following: an irradiation amount exceeding the upper limit of the irradiation unit 107's irradiability limit, an irradiation direction other than the irradiation direction the irradiation unit 107 can irradiate, or an irradiation range other than the irradiation range the irradiation unit 107 can irradiate, a warning notification is displayed on the display unit 106. This allows the user to be informed of the limits of the adjustment of non-visible light.

[0032] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of symbols]

[0033] 100 Image Processing Devices 101 Control Unit 105 Image Processing Unit 106 Display section 201 Visible Light Image Feature Calculation Unit 202 Non-Invisible Light Image Feature Calculation Unit 204 Invisible light irradiation condition adjustment section

Claims

1. An image processing device comprising an illumination means composed of one or more light sources that illuminate one or both of visible light and non-visible light, a first feature amount calculation means for calculating a feature amount including spatial gradation information of a visible light image obtained by photographing a subject illuminated with visible light by the illumination means; a second feature amount calculation means for calculating a feature amount including spatial gradation information of an invisible light image obtained by photographing the subject illuminated with invisible light by the illumination means; a calculation means for calculating an enhancement effect based on a feature amount including spatial gradation information of the visible light image and a feature amount including spatial gradation information of the invisible light image, and for calculating an illumination condition of the invisible light illuminated by the illumination means based on the enhancement effect.

2. 2. The image processing device according to claim 1, further comprising an adjustment unit that adjusts shooting conditions for shooting the subject based on a feature amount including spatial gradation information of the visible light image, a feature amount including spatial gradation information of the invisible light image, and an illumination condition of the invisible light.

3. 3 . The image processing apparatus according to claim 1 , wherein the irradiation conditions of the invisible light include an irradiation amount of the invisible light, an irradiation direction of the invisible light, and an irradiation range of the invisible light.

4. 4. The image processing apparatus according to claim 3, further comprising a determination unit that determines an area for determining an irradiation direction of the invisible light and an irradiation range of the invisible light.

5. 5. The image processing apparatus according to claim 4, wherein the area is designated by a user.

6. The image processing device described in Claim 4, characterized in that the area is an area in which spatial gradation information in the visible light image is below a predetermined value.

7. 7. The image processing apparatus according to claim 1, wherein if the irradiation conditions of the invisible light include a condition that exceeds a specification of the irradiation means, a warning is issued.

8. A method for controlling an image processing device having irradiation means composed of one or more light sources that irradiate one or both of visible light and invisible light, comprising: a first feature amount calculation step of calculating a feature amount including spatial gradation information of a visible light image obtained by photographing a subject irradiated with visible light by the irradiating means; a second feature amount calculation step of calculating a feature amount including spatial gradation information of an invisible light image obtained by photographing the subject irradiated with invisible light by the irradiation means; a calculation step of calculating an enhancement effect based on a feature amount including spatial gradation information of the visible light image and a feature amount including spatial gradation information of the invisible light image, and calculating an illumination condition of the invisible light illuminated by the illumination means based on the enhancement effect.

9. A program for causing a computer to execute a control method for an image processing device having irradiation means composed of one or more light sources that irradiate one or both of visible light and invisible light, comprising: The method for controlling the image processing device includes: a first feature amount calculation step of calculating a feature amount including spatial gradation information of a visible light image obtained by photographing a subject irradiated with visible light by the irradiating means; a second feature amount calculation step of calculating a feature amount including spatial gradation information of an invisible light image obtained by photographing the subject irradiated with invisible light by the irradiation means; a calculation step of calculating an enhancement effect based on a feature amount including spatial gradation information of the visible light image and a feature amount including spatial gradation information of the invisible light image, and calculating an irradiation condition of the invisible light irradiated by the irradiation means based on the enhancement effect.

10. The spatial gradation information of the visible light image is obtained by subjecting the visible light image to a band-pass filter process that transmits components of a predetermined frequency band, and extracting information on AC components within the predetermined frequency band from data obtained from the visible light image, 8. The image processing device according to claim 1, wherein the spatial gradation information of the non-visible light image is obtained by performing a band-pass filter process on the non-visible light image that transmits components of a predetermined frequency band, and extracting information on AC components within the predetermined frequency band from data obtained from the non-visible light image.

11. The spatial gradation information of the visible light image is obtained by subjecting the visible light image to a band-pass filter process that transmits components of a predetermined frequency band, and extracting information on AC components within the predetermined frequency band from data obtained from the visible light image, The control method for an image processing device according to claim 8, characterized in that the spatial gradation information of the non-visible light image is obtained by subjecting the non-visible light image to band-pass filtering processing that transmits components of a predetermined frequency band, and extracting information on AC components within the predetermined frequency band from data obtained from the non-visible light image.

12. The spatial gradation information of the visible light image is obtained by subjecting the visible light image to a band-pass filter process that transmits components of a predetermined frequency band, and extracting information on AC components within the predetermined frequency band from data obtained from the visible light image, The program according to claim 9, characterized in that the spatial gradation information of the non-visible light image is obtained by subjecting the non-visible light image to band-pass filtering that transmits components of a predetermined frequency band, and extracting information on AC components within the predetermined frequency band from data obtained from the non-visible light image.

Images