Image processing device, method for controlling the image processing device, and program

Abstract

To calculate an appropriate invisible light irradiation condition.SOLUTION: An image processing device 100 includes an irradiation unit 107. The image processing device 100 calculates the feature amount of a visible light image obtained by imaging a subject irradiated with visible light by the irradiation unit 107, and calculates the feature amount of an invisible light image obtained by imaging the subject irradiated with invisible light by the irradiation unit 107. The image processing device 100 calculates the invisible light irradiation information, which is an irradiation condition of the invisible light irradiated by the irradiation unit 107 on the basis of the feature amount of the visible light image and the feature amount of the invisible light image.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a control method for an image processing apparatus, and a program.

Background Art

[0002] There is known a method of improving the image quality of a visible light image with poor visibility due to haze or darkness by using a non-visible light image such as an infrared light image. In order to acquire a visible light image and a non-visible light image, it is necessary to acquire light in the visible light wavelength band and light in the non-visible light wavelength band with appropriate exposure. However, generally, the intensity of light is different between visible light and non-visible light, and the exposure time and aperture value cannot be individually set for the visible light receiving element and the non-visible light receiving element. Therefore, it is difficult to acquire a visible light image and a non-visible light image with appropriate exposure, respectively.

[0003] In Patent Document 1, in order to acquire a visible light image and a non-visible light image with appropriate exposure, a technique is disclosed in which the brightness of two images with different wavelengths is evaluated, and the irradiation amounts of two light sources with different wavelengths are controlled based on the evaluation result.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technique of Patent Document 1 described above, since the irradiation amounts of two light sources with different wavelengths are controlled based on the brightness evaluation of two images with different wavelengths, there is a problem that a sufficient improvement effect on contrast and resolution cannot be obtained for the visible light image.

[0006] The present invention aims to provide an image processing apparatus, a control method for the image processing apparatus, and a program that can calculate appropriate irradiation conditions for invisible light. [Means for solving the problem]

[0007] To achieve the above objective, the present invention provides an image processing apparatus comprising an illumination means composed of one or more light sources that emit both visible light and invisible light, the apparatus comprising: a first feature quantity calculation means for calculating a feature quantity including spatial gradation information of a visible light image obtained by photographing a subject illuminated with visible light by the illumination means; a second feature quantity calculation means for calculating a feature quantity including spatial gradation information of an invisible light image obtained by photographing the subject illuminated with invisible light by the illumination means; and a calculation means for calculating an enhancement effect, which is a value indicating the degree of improvement in the gradation, contrast, and resolution of the visible light image, based on the feature quantity including spatial gradation information of the visible light image and the feature quantity including spatial gradation information of the invisible light image, and calculating the illumination conditions for the invisible light emitted by the illumination means based on the enhancement effect. The spatial gradation information of the visible light image is obtained by applying a bandpass filter to the visible light image to transmit components in a predetermined frequency band, and extracting information on the AC component within that predetermined frequency band from the data obtained from the visible light image. The spatial gradation information of the non-visible light image is obtained by applying a bandpass filter to the non-visible light image to transmit components in a predetermined frequency band, and extracting information on the AC component within that predetermined frequency band from the data obtained from the non-visible light image. It is characterized by the following. [Effects of the Invention]

[0008] According to the present invention, it is possible to calculate appropriate irradiation conditions for invisible light. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram schematically showing the configuration of the image processing apparatus according to this embodiment. [Figure 2] This is a block diagram schematically showing the configuration of the non-visible light irradiation condition calculation unit included in the image processing unit shown in Figure 1. [Figure 3] This flowchart shows the procedure for the invisible light control processing performed by the image processing device shown in Figure 1. [Figure 4] This figure shows an example of the grayscale information for visible light and invisible light images acquired by the image processing device shown in Figure 1. [Figure 5]Figure 3 is a flowchart showing the procedure for calculating non-visible light irradiation information in step S306. [Figure 6] This figure illustrates the determination in step S503 of Figure 5. [Figure 7] Figure 5 is a diagram illustrating the calculation of non-visible light irradiation information in step S504. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described in detail below with reference to the drawings. In this embodiment, the image processing device enhances the gradation, contrast, and resolution of visible light images by adjusting the illumination conditions for invisible light based on the characteristic quantities of visible light and invisible light images obtained by photographing the same subject at the same angle of view, and then irradiates with invisible light. The illumination conditions for invisible light include the amount of invisible light, the direction of illumination, and the illumination range of the invisible light.

[0011] Figure 1 is a schematic block diagram showing the configuration of the image processing apparatus 100 according to this embodiment. In Figure 1, the image processing apparatus 100 comprises a control unit 101, a ROM 102, a RAM 103, a UI unit 104, an image processing unit 105, a display unit 106, and an illumination unit 107. These are interconnected by a bus 108.

[0012] The control unit 101 is, for example, a CPU. The control unit 101 reads the control programs for each block of the image processing device 100 from the ROM 102, and then loads the read control programs into the RAM 103 and executes them. In this way, the control unit 101 controls the operation of each block of the image processing device 100. The ROM 102 is an electrically erasable and recordable non-volatile memory. The ROM 102 stores the control programs for each block of the image processing device 100, as well as parameters necessary for the operation of each block. The RAM 103 is a rewritable volatile memory. The RAM 103 is used as an area to load programs executed by the control unit 101, etc., and as an area to temporarily store data generated by the operation of each block of the image processing device 100.

[0013] The UI unit 104 receives user input for the image processing device 100. For example, a pointing device or keyboard can be used as the UI unit 104. A pointing device could be a touch panel or a mouse. The image processing unit 105 performs image processing such as white balance adjustment, color interpolation, and gamma processing on the image data stored in the RAM 103. The image processing unit 105 also includes a non-visible light irradiation condition calculation unit 200, as shown in Figure 2 (described later). The display unit 106 displays images stored in the ROM 102. The display unit 106 is, for example, a display device such as a smartphone or television. The display unit 106 also provides UI displays for receiving instructions from the user. The illumination unit 107 is, for example, a strobe light, which illuminates the subject with visible and non-visible light. The illumination unit 107 adjusts the illumination amount, direction, and range of non-visible light on the subject based on the non-visible light irradiation information calculated by the image processing unit 105 (described later).

[0014] Figure 2 is a schematic block diagram showing the configuration of the invisible light irradiation condition calculation unit 200 included in the image processing unit 105 of Figure 1. The invisible light irradiation condition calculation unit 200 consists of a visible light image feature quantity calculation unit 201, an invisible light image feature quantity calculation unit 202, an enhancement effect calculation unit 203, and an invisible light irradiation condition adjustment unit 204.

[0015] The visible light image feature calculation unit 201 calculates the feature quantities of a visible light image obtained by photographing a subject illuminated with visible light by the illumination unit 107. The feature quantities of a visible light image are, for example, the gradation information of the brightness component of the visible light image. The visible light image feature calculation unit 201 extracts information on the AC component (gradation information) of the brightness component (I component) of the visible light image by performing a bandpass filter process that extracts a predetermined frequency band on the brightness component (I component) of the visible light image. The visible light image feature calculation unit 201 outputs this gradation information as a feature quantity of the visible light image to the enhancement effect calculation unit 203.

[0016] The non-visible light image feature amount calculation unit 202 calculates the feature amount of a non-visible light image obtained by photographing a subject irradiated with non-visible light such as infrared light by the irradiation unit 107. The feature amount of the non-visible light image is, for example, the gradation information of the non-visible light image. The non-visible light image feature amount calculation unit 202 extracts the information (gradation information) of the AC component of the non-visible light image by performing a band-pass filter process for extracting a predetermined frequency band from the non-visible light image. The non-visible light image feature amount calculation unit 202 outputs this gradation information to the enhancement effect calculation unit 203 as the feature amount of the non-visible light image.

[0017] The enhancement effect calculation unit 203 calculates an enhancement effect based on the feature amount of the visible light image acquired from the visible light image feature amount calculation unit 201 and the feature amount of the non-visible light image acquired from the non-visible light image feature amount calculation unit 202. The enhancement effect indicates an effect of improving gradation, contrast, resolution, etc. The non-visible light irradiation condition adjustment unit 204 calculates non-visible light irradiation information that is the irradiation condition of the non-visible light irradiated by the irradiation unit 107 based on the enhancement effect calculated by the enhancement effect calculation unit 203.

[0018] FIG. 3 is a flowchart showing the procedure of the non-visible light control process executed by the image processing apparatus 100 of FIG. 1. The process of FIG. 3 is realized by the control unit 101 expanding and executing the control program read from the ROM 102 in the RAM 103. In the non-visible light control process of FIG. 3, the image processing apparatus 100 is assumed to have already acquired a visible light image and a non-visible light image obtained by switching the light irradiated from the irradiation unit 107 and photographing the same subject at the same angle of view. This visible light image is an image obtained by photographing a subject irradiated with visible light by the irradiation unit 107, and this non-visible light image is an image obtained by photographing a subject irradiated with non-visible light by the irradiation unit 107. In the present embodiment, a configuration for acquiring a visible light image and a non-visible light image from the imaging device that has photographed the subject will be described, but the configuration is not limited to this. For example, the image processing apparatus 100 may include an imaging unit, and this imaging unit may be configured to image a subject and generate a visible light image and a non-visible light image.

[0019] In FIG. 3, first, the control unit 101 stores the acquired visible light image in the RAM 103 (step S301). Next, the control unit 101 stores the acquired non-visible light image in the RAM 103 (step S302). Next, the control unit 101 calculates the feature amount of the visible light image stored in the RAM 103 by the visible light image feature amount calculation unit 201 of the image processing unit 105 (step S303). Next, the control unit 101 calculates the feature amount of the non-visible light image stored in the RAM 103 by the non-visible light image feature amount calculation unit 202 of the image processing unit 105 (step S304).

[0020] Next, the control unit 101 calculates an enhancement effect based on the feature amount of the visible light image and the feature amount of the non-visible light image by the enhancement effect calculation unit 203 of the image processing unit 105 (step S305). Here, the calculation of the enhancement effect will be described using FIG. 4. FIG. 4(a) is a diagram showing the gradation information of the visible light image, and FIG. 4(b) is a diagram showing the gradation information of the non-visible light image. In step S305, the difference in gradation information of the corresponding regions in the visible light image and the non-visible light image is calculated. Specifically, the difference between the gradation information of a predetermined region L401 in the visible light image and the gradation information of the predetermined region L401 in the non-visible light image is calculated. The predetermined region L401 is a region designated by the user or a low-contrast region in the visible light image where the gradation information is below a predetermined value. The region L401 in FIG. 4(a) shows an example where the gradation property is insufficient due to haze or the like. On the other hand, the region L401 in FIG. 4(b) shows an example where sufficient gradation property is obtained as an image signal due to the wavelength characteristics of the non-visible light. The enhancement effect calculation unit 203 outputs the calculated result as the enhancement effect to the non-visible light irradiation condition adjustment unit 204.

[0021] Next, the control unit 101, using the image processing unit 105, performs the invisible light irradiation information calculation process shown in Figure 5 (step S306), which will be described later, and calculates invisible light irradiation information. Then, the control unit 101 adjusts the irradiation amount, irradiation direction, and irradiation range based on the invisible light irradiation information and irradiates the subject with invisible light from the irradiation unit 107 (step S307), and terminates this process.

[0022] Figure 5 is a flowchart showing the procedure for calculating invisible light irradiation information in step S306 of Figure 3. The invisible light irradiation information calculation process in Figure 5 is performed by the image processing unit 105.

[0023] In Figure 5, the enhancement effect calculation unit 203 of the image processing unit 105 inputs the calculated enhancement effect to the invisible light illumination condition adjustment unit 204 (step S501). Next, the invisible light illumination condition adjustment unit 204 of the image processing unit 105 determines the enhancement effect region for determining the direction and range of invisible light illumination (step S502). In step S502, the invisible light illumination condition adjustment unit 204 determines, for example, a region specified by the user as the enhancement effect region. This allows the user's intentions to be reflected in the determination of the direction and range of invisible light illumination. Alternatively, the invisible light illumination condition adjustment unit 204 determines a low-contrast region in the visible light image where the gradation information is below a predetermined value as the enhancement effect region. This allows the region where image quality improvement is expected to be necessary to be determined as the enhancement effect region without forcing the user to specify a region.

[0024] Next, the non-visible light irradiation condition adjustment unit 204 determines whether the enhancement effect in the enhancement effect region determined in step S502 is below a threshold (step S503). Figure 6(a) shows an example where the enhancement effect in the enhancement effect region is below a threshold, and Figure 6(b) shows an example where the enhancement effect in the enhancement effect region exceeds a threshold. In Figures 6(a) and 6(b), region L601 indicates the enhancement effect region.

[0025] In step S503, if the enhancement effect in the enhancement effect region determined in step S502 is below a threshold (see, for example, Figure 6(a)), the invisible light irradiation condition adjustment unit 204 calculates invisible light irradiation information so that the enhancement effect exceeds the threshold in the enhancement effect region (step S504). Here, the calculation of invisible light irradiation information will be explained. Figure 7(a) is a diagram showing the gradation information of a visible light image, and Figure 7(b) is a diagram showing the gradation information of an invisible light image. Region L701 in Figures 7(a) and 7(b) shows an example where gradation is insufficient due to haze, etc. Region L701 in Figures 7(a) and 7(b) indicates the enhancement effect region. In other words, region L701 is the same region as region L601.

[0026] For example, in a non-visible light image with improper exposure, effects such as haze remain, resulting in insufficient tonal information as shown in Figure 7(b). The enhancement effect, which is the difference between the tonal information of such a non-visible light image and the tonal information of a visible light image, falls below a threshold. In such cases, in step S504, the non-visible light irradiation condition adjustment unit 204 determines the irradiation direction and irradiation range of the non-visible light so as to irradiate the area corresponding to the enhancement effect region with non-visible light, and also determines the irradiation amount of non-visible light from the enhancement effect. The non-visible light irradiation condition adjustment unit 204 outputs the determined irradiation direction, irradiation range, and irradiation amount as non-visible light irradiation information. After that, this process ends.

[0027] In step S503, if the enhancement effect in the enhancement effect region determined in step S502 exceeds a threshold (see, for example, Figure 6(b)), this process terminates.

[0028] According to the embodiment described above, invisible light irradiation information, which is the irradiation condition for invisible light irradiated by the irradiation unit 107, is calculated based on the feature quantities of the visible light image and the feature quantities of the invisible light image. As a result, when capturing invisible light images to improve the image quality of visible light images, optimized invisible light can be irradiated onto the subject, thereby improving the image quality of visible light images.

[0029] Alternatively, based on the invisible light irradiation information calculated in step S504, invisible light with adjusted irradiation amount, direction, and range may be irradiated onto the subject, an invisible light image obtained by photographing the subject may be acquired from the imaging device, and the invisible light control processing described above may be executed again based on the invisible light image. This allows for readjustment of the irradiation amount, direction, and range of the invisible light if the intended enhancement effect is not obtained even after irradiating the subject with invisible light with adjusted irradiation amount, direction, and range based on the above invisible light irradiation information.

[0030] In the above-described embodiment, the shooting conditions for capturing the subject may be adjusted based on the features of the visible light image, the features of the non-visible light image, and the non-visible light illumination information. Examples of shooting conditions include ISO sensitivity, F-number, and shutter speed. This allows for optimization of not only the amount, direction, and range of non-visible light illumination, but also the shooting conditions to improve the image quality of the visible light image.

[0031] Furthermore, in the above-described embodiment, a warning may be issued if the non-visible light irradiation information includes information that exceeds the specifications of the irradiation unit 107. For example, if the non-visible light irradiation information includes at least one of the following: an irradiation amount exceeding the upper limit of the irradiation unit 107's irradiability limit, an irradiation direction other than the irradiation direction the irradiation unit 107 can irradiate, or an irradiation range other than the irradiation range the irradiation unit 107 can irradiate, a warning notification is displayed on the display unit 106. This allows the user to be informed of the limits of the adjustment of non-visible light.

[0032] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of Symbols]

[0033] 100 Image Processing Devices 101 Control Unit 105 Image Processing Unit 106 Display section 201 Visible Light Image Feature Calculation Unit 202 Non-Invisible Light Image Feature Calculation Unit 204 Invisible light irradiation condition adjustment section

Claims

1. An image processing apparatus comprising an illumination means composed of one or more light sources that emit both visible light and non-visible light, A first feature quantity calculation means calculates a feature quantity including spatial grayscale information of a visible light image obtained by photographing a subject irradiated with visible light by the aforementioned irradiation means, A second feature quantity calculation means calculates a feature quantity including spatial grayscale information of a non-visible light image obtained by photographing the subject illuminated with non-visible light by the illumination means, The system includes a calculation means that calculates an enhancement effect, which is a value indicating the degree of improvement in the gradation, contrast, and resolution of the visible light image, based on a feature quantity including spatial gradation information of the visible light image and a feature quantity including spatial gradation information of the non-visible light image, and calculates the irradiation conditions for the non-visible light irradiated by the irradiation means based on the enhancement effect, The spatial gradation information of the visible light image is obtained by applying a bandpass filter to the visible light image to transmit components in a predetermined frequency band, and extracting information on the AC component within that predetermined frequency band from the data obtained from the visible light image. The spatial grayscale information of the invisible light image is obtained by applying a bandpass filter to the invisible light image to transmit components of a predetermined frequency band, and by extracting information of the AC component within the predetermined frequency band from the data obtained from the invisible light image.

2. The image processing apparatus according to claim 1, further comprising a feature quantity including spatial gradation information of the visible light image, a feature quantity including spatial gradation information of the non-visible light image, and an adjustment means for adjusting the shooting conditions for photographing the subject based on the irradiation conditions of the non-visible light.

3. The image processing apparatus according to claim 1 or 2, characterized in that the irradiation conditions for invisible light include the amount of invisible light, the direction of irradiation of the invisible light, and the irradiation range of the invisible light.

4. The image processing apparatus according to claim 3, further comprising determination means for determining the irradiation direction of the non-visible light and the region for determining the irradiation range of the non-visible light.

5. The image processing apparatus according to claim 4, characterized in that the region is specified by the user.

6. The image processing apparatus according to claim 4, characterized in that the region is a region in the visible light image where the spatial grayscale information is less than or equal to a predetermined value.

7. The image processing apparatus according to any one of claims 1 to 6, characterized in that it issues a warning when the irradiation conditions for the non-visible light include conditions that exceed the specifications of the irradiation means.

8. A control method for an image processing apparatus comprising an illumination means composed of one or more light sources that emit both visible light and non-visible light, A first feature calculation step involves calculating a feature quantity that includes spatial grayscale information of a visible light image obtained by photographing a subject irradiated with visible light by the aforementioned irradiation means, A second feature calculation step involves calculating a feature quantity that includes spatial grayscale information of a non-visible light image obtained by photographing the subject illuminated with non-visible light by the illumination means, The system includes a calculation step of calculating an enhancement effect, which is a value indicating the degree of improvement in the gradation, contrast, and resolution of the visible light image, based on a feature quantity including spatial gradation information of the visible light image and a feature quantity including spatial gradation information of the non-visible light image, and calculating the irradiation conditions for the non-visible light irradiated by the irradiation means based on the enhancement effect, The spatial gradation information of the visible light image is obtained by applying a bandpass filter to the visible light image to transmit components in a predetermined frequency band, and extracting information on the AC component within that predetermined frequency band from the data obtained from the visible light image. A control method for an image processing apparatus, characterized in that the spatial grayscale information of the invisible light image is obtained by applying a bandpass filter process to the invisible light image to transmit components of a predetermined frequency band, and extracting information of the AC component within the predetermined frequency band from the data obtained from the invisible light image.

9. A program that causes a computer to execute a control method for an image processing apparatus comprising an illumination means composed of one or more light sources that emit both visible light and non-visible light, The control method for the image processing device is: A first feature calculation step involves calculating a feature quantity that includes spatial grayscale information of a visible light image obtained by photographing a subject irradiated with visible light by the aforementioned irradiation means, A second feature calculation step involves calculating a feature quantity that includes spatial grayscale information of a non-visible light image obtained by photographing the subject illuminated with non-visible light by the illumination means, The system includes a calculation step of calculating an enhancement effect, which is a value indicating the degree of improvement in the gradation, contrast, and resolution of the visible light image, based on a feature quantity including spatial gradation information of the visible light image and a feature quantity including spatial gradation information of the non-visible light image, and calculating the irradiation conditions for the non-visible light irradiated by the irradiation means based on the enhancement effect, The spatial gradation information of the visible light image is obtained by applying a bandpass filter to the visible light image to transmit components in a predetermined frequency band, and extracting information on the AC component within that predetermined frequency band from the data obtained from the visible light image. The program is characterized in that the spatial gradation information of the invisible light image is obtained by applying a bandpass filter process to the invisible light image to transmit components in a predetermined frequency band, and by extracting information on the AC component within the predetermined frequency band from the data obtained from the invisible light image.

Images