Image processing apparatus and image processing method, imaging apparatus

Abstract

To provide an image processing apparatus and an image processing method that improve the gradation properties of a visible light image, while reducing the unnaturalness of the relationship in gradation properties between an area composed with an invisible light image and an area not composed with the invisible light image.SOLUTION: An image processing apparatus composes an invisible light image into a determined area of a visible light image to be composed with an invisible light image, thereby creating a composite image. The image processing apparatus composes the invisible light image with the visible light image after adjusting the invisible light image so that the difference in gradation information between an area composed with the invisible light image and an area not composed with the invisible light image in the composite image becomes equal to or less than a predetermined first threshold.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus, an image processing method, and an imaging apparatus. [Background technology]

[0002] A method has been proposed to improve the visibility of a visible light image by combining a visible light image taken of a scene with reduced visibility due to fog or haze with an infrared light image taken of the same scene (Patent Document 1). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2017-157902 [Overview of the project] [Problems that the invention aims to solve]

[0004] Patent Document 1 improves visibility by combining an infrared light image with a region of a visible light image where the transmittance of fog is low, thereby improving the gradation of the region where visibility is reduced due to fog.

[0005] However, when an infrared image is superimposed on a specific region, the tonal relationship between the region with the superimposed infrared image and the region without it may appear unnatural.

[0006] In one aspect, the present invention provides an image processing apparatus and an image processing method capable of improving the gradation of a visible light image while suppressing the unnatural relationship in gradation between regions where a non-visible light image is synthesized and regions where it is not. [Means for solving the problem]

[0007] The above objective is achieved by an image processing apparatus comprising: a determination means for determining a region in a visible light image to which a non-visible light image is to be synthesized; and a synthesis means for generating a composite image by synthesizing a non-visible light image with a visible light image based on the determination, wherein the synthesis means adjusts the non-visible light image so that the difference in gradation information between the region in the composite image where the non-visible light image is synthesized and the region where it is not synthesized is less than or equal to a predetermined first threshold, and then synthesizes it with the visible light image. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an image processing apparatus and an image processing method that can improve the gradation of a visible light image while suppressing the unnaturalness in the relationship of gradation between regions in which a non-visible light image is synthesized and regions in which it is not synthesized. [Brief explanation of the drawing]

[0009] [Figure 1] Block diagram showing an example of the functional configuration of an imaging device as an image processing apparatus according to the embodiment. [Figure 2] A diagram showing an example of pixel arrangement in an image sensor. [Figure 3] Block diagram showing an example of the functional configuration of the tone correction unit in the first embodiment. [Figure 4] Flowchart relating to the operation of the grayscale correction unit in the first embodiment [Figure 5] Block diagram showing an example of the functional configuration of the luminance enhancement unit in the first embodiment. [Figure 6] Flowchart relating to the operation of the brightness enhancement unit in the first embodiment [Figure 7] This figure shows the brightness component of a visible light image and an example of an infrared light image taken of the same scene. [Figure 8] Block diagram showing an example of the functional configuration of the color correction unit in the first embodiment. [Figure 9] Flowchart relating to the operation of the color correction unit in the first embodiment [Figure 10] A figure showing an example of saturation correction gain in the first embodiment. [Figure 11] Block diagram showing a functional configuration example of the gradation correction unit in the second embodiment [Figure 12] Block diagram showing a functional configuration example of the luminance enhancement unit in the second embodiment [Figure 13] Flowchart regarding the operation of the luminance enhancement unit in the second embodiment [Figure 14] Block diagram showing a functional configuration example of the color correction unit in the second embodiment [Figure 15] Flowchart regarding the operation of the color correction unit in the second embodiment [Figure 16] Diagram showing an example of an adjustment gain for considering spectral characteristics in the second embodiment [Figure 17] Block diagram showing a functional configuration example of the gradation correction unit in the third embodiment [Figure 18] Flowchart regarding the operation of the gradation correction unit in the third embodiment [Figure 19] Block diagram showing a functional configuration example of the haze thickening unit in the third embodiment [Figure 20] Flowchart regarding the operation of the haze thickening unit in the third embodiment [Figure 21] Diagram showing an example of the thickening gain characteristics in the third embodiment [Figure 22] Diagram schematically showing the change in pixel values due to the haze thickening process in the third embodiment [Figure 23] Block diagram showing a functional configuration example of the color correction unit in the third embodiment [Figure 24] Flowchart regarding the operation of the color correction unit in the third embodiment [Figure 25] Diagram showing an example of the saturation gain characteristics in the third embodiment

Mode for Carrying Out the Invention

[0010] ●(First Embodiment) The present invention will be described in detail below with reference to the attached drawings, based on exemplary embodiments thereof. Note that the following embodiments do not limit the invention to the claims. Furthermore, while multiple features are described in the embodiments, not all of them are essential to the invention, and the multiple features may be combined arbitrarily. In addition, in the attached drawings, the same or similar configurations are given the same reference numeral, and redundant descriptions are omitted.

[0011] In the following embodiments, the present invention will be described in relation to cases where it is implemented using an imaging device such as a digital camera. However, an imaging function is not essential to the present invention, and it can be implemented with any electronic device capable of handling image data. Such electronic devices include video cameras, computer equipment (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and dashcams. These are examples, and the present invention can be implemented with other electronic devices as well.

[0012] The configurations represented as blocks in the diagram can be implemented using integrated circuits (ICs) such as ASICs and FPGAs, discrete circuits, or a combination of memory and a processor that executes the program stored in memory. Furthermore, one block may be implemented using multiple integrated circuit packages, or multiple blocks may be implemented using a single integrated circuit package. Additionally, the same block may be implemented in different configurations depending on the operating environment and required capabilities.

[0013] Figure 1 is a block diagram showing an example of the functional configuration of an imaging device 100, which is an example of an image processing apparatus according to the present invention. The control unit 101 is a processor capable of executing programs, such as a CPU. The control unit 101 controls the operation of each functional block of the imaging device 100 and realizes the functions of the imaging device 100 by, for example, reading a program stored in ROM 102 into RAM 103 and executing it. In the case of an optical system 104 being a replaceable lens unit, the control unit 101 controls the operation of the optical system 104 through communication with the controller of the optical system 104.

[0014] ROM 102 is a rewritable non-volatile memory. ROM 102 stores programs executed by the control unit 101, various settings for the imaging device 100, GUI data, and the like. RAM 103 is the main memory of the control unit 101. RAM 103 is used to load programs executed by the control unit 101, to hold parameters necessary for program execution, and as working memory for the image processing unit 107. In addition, a portion of RAM 103 is used as video memory to store image data to be displayed on the display unit 109.

[0015] The optical system 104 includes an imaging optical system consisting of a lens group including a movable lens (zoom lens, focus lens, etc.) and a drive circuit for the movable lens. The optical system 104 may also include an aperture and its drive circuit.

[0016] The imaging unit 105 may be, for example, a known CCD or CMOS color image sensor (image sensor) having a primary color Bayer array color filter. The image sensor has a pixel array in which multiple pixels are arranged in two dimensions, and peripheral circuits for reading signals from the pixels. Each pixel has a photoelectric conversion element such as a photodiode and accumulates charge according to the amount of incident light during the exposure period. By reading signals having a voltage corresponding to the amount of charge accumulated during the exposure period from each pixel, a group of pixel signals (analog image signals) representing the subject image formed on the imaging surface by the imaging optical system is obtained.

[0017] In this embodiment, the imaging unit 105 is assumed to have an image sensor capable of capturing both visible light images and invisible light images. Such an image sensor may, for example, have some of the pixels in a pixel array designated as pixels for capturing invisible light images. The pixels for capturing invisible light images transmit the wavelength band of invisible light and the wavelength band of visible light band The pixel may have an optical filter with properties that block a certain range.

[0018] For example, as shown in Figure 2(a), in an image sensor equipped with a primary color Bayer array color filter, one of the two green (G) filters included in the repeating unit of the color filter (G pixel) can be replaced with a pixel used to capture a non-visible light image. In this case, the value of the G pixel that would originally be present at the position of the non-visible light image capturing pixel can be interpolated using the values ​​of other pixels, similar to the value of a defective pixel, to generate a visible light image. Furthermore, the non-visible light image can be enlarged based on the signal of the non-visible light image capturing pixel to achieve the same resolution (number of pixels) as the visible light image.

[0019] There are no particular restrictions on the method of acquiring visible light images and invisible light images, and they may be acquired by other methods. For example, as shown in Figure 2(b), an image sensor for capturing visible light images (left) and an image sensor for capturing invisible light images (right) may be provided separately. In this case, an independent optical system may be provided for each image sensor, or the optical image formed by one optical system may be distributed to the two image sensors by a prism or the like. In this embodiment, the invisible light image is assumed to be an infrared light image, but it may be an image of another invisible wavelength band.

[0020] The A / D conversion unit 106 converts the analog image signal read from the imaging unit 105 into a digital image signal. The A / D conversion unit 106 writes the digital image signal to the RAM 103.

[0021] The image processing unit 107 applies predetermined image processing to the digital image signal stored in the RAM 103 to generate signals and image data according to the application, and to acquire and / or generate various types of information. The image processing unit 107 may be a dedicated hardware circuit such as an ASIC designed to realize a specific function, or it may be a configuration in which a programmable processor such as a DSP executes software to realize a specific function.

[0022] The image processing applied by the image processing unit 107 includes preprocessing, color interpolation, correction, detection, data processing, evaluation value calculation, and special effects processing. Preprocessing includes signal amplification, reference level adjustment, and defective pixel correction. Color interpolation is a process that interpolates the values ​​of color components that could not be obtained during shooting, and is also called demosaicing.

[0023] The correction process includes white balance adjustment, tone correction, correction of image degradation caused by optical aberrations in the optical system 104 (image recovery), correction of the effects of vignetting in the optical system 104, and color correction. In addition, the infrared light image synthesis process (enhancement process) for the purpose of tone correction of the visible light image, which will be described later, is also included in the correction process. Detection processes include detecting feature regions (such as face regions or human body regions) and their movements, as well as recognizing people. Data processing includes processes such as synthesis, scaling, encoding and decoding, and header information generation (data file generation).

[0024] The evaluation value calculation process includes generating signals and evaluation values ​​used for autofocus detection (AF), and generating evaluation values ​​used for automatic exposure control (AE). Furthermore, the generation of evaluation values ​​for determining the imaging conditions for infrared light images, as described later, is also included in this process. Special effects processing includes adding blur effects, changing color tones, and relighting. These are merely examples of processes that the image processing unit 107 can apply, and do not limit the processes that the image processing unit 107 can apply.

[0025] The recording unit 108 records data to a recording medium such as a memory card, or reads data recorded on the recording medium. The recording medium does not have to be removable. The recording medium may also be a communication-enabled external storage device.

[0026] The display unit 109 is, for example, a liquid crystal display and displays captured images, images read by the recording unit 108, information from the imaging device 100, a GUI such as a menu screen, etc. By continuously performing video recording and displaying the recorded video on the display unit 109, the display unit 109 can be made to function as an electronic viewfinder (EVF). The display unit 109 may also be a touch display.

[0027] The operation unit 110 is a general term for input devices (buttons, switches, dials, etc.) provided for the user to input instructions to the imaging device 100. The input devices constituting the operation unit 110 have names according to the function they are assigned to. For example, the operation unit 110 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, directional keys, and a select key. The release switch is a switch for recording still images, and the control unit 101 recognizes a half-pressed state of the release switch as a shooting preparation instruction and a fully pressed state as a shooting start instruction. Also, the control unit 101 recognizes the video recording switch as a video recording start instruction when pressed in shooting standby mode, and as a recording stop instruction when pressed during video recording. The functions assigned to the same input device may be variable. Furthermore, the input devices may be software buttons or keys using a touch display.

[0028] Figure 3 is a functional block diagram schematically representing the image processing unit 107 as the tone correction unit 300 during the execution of the enhancement process, for the purpose of explaining the enhancement process implemented in this embodiment. Therefore, the functional blocks of the tone correction unit 300 are actually only a part of the functions realized by the image processing unit 107. The tone correction unit 300 includes an ICtCp conversion unit 301, a luminance enhancement unit 302, a color correction unit 303, and a YUV conversion unit 304.

[0029] Next, the enhancement processing steps applied by the tone correction unit 300 to the visible light image will be explained using the flowchart shown in Figure 4. Here, the visible light image data and infrared light image data input to the tone correction unit 300 may be images captured by the imaging unit 105 and supplied through the A / D conversion unit 106, or they may be read from the recording unit 108. The visible light image data and infrared light image data used for the enhancement processing are image data of the same scene captured simultaneously or at sufficiently close timings.

[0030] In S401, the ICtCp conversion unit 301 converts the color space of the visible light image data (e.g., the YUV color space) to the ICtCp color space. The ICtCp color space is a uniform color space defined by ITU-R BT.2100 that takes into account the characteristics of human vision. The ICtCp conversion unit 301 further separates the visible light image data converted to ICtCp format into an I component representing the brightness (luminance) component and a CtCp component representing the color component.

[0031] Since methods for converting YUV or RGB image data to ICtCp image data are publicly known, a detailed explanation will be omitted. Note that conversion to a uniform color space other than the ICtCp color space, such as the L*a*b* color space, is also permitted. When converted to the L*a*b* format, the L* component and the a*b* component are separated. The ICtCp conversion unit 301 outputs the I component to the luminance enhancement unit 302 and the CtCp component to the color correction unit 303.

[0032] In S402, the luminance enhancement unit 302 improves the gradation (contrast) of the visible light image data by synthesizing infrared light image data with the I component (luminance component) of the visible light image data. The luminance enhancement unit 302 outputs the gradation-corrected I component of the visible light image data to the YUV conversion unit 304. The luminance enhancement unit 302 also outputs adjustment enhancement information used for correcting the color component to the color correction unit 303. The gradation-corrected I component of the visible light image data is output to the YUV conversion unit 304. Details of the enhancement processing by the luminance enhancement unit 302 will be described later.

[0033] In S403, the color correction unit 303 applies color correction processing to the CtCp component (color component) of the visible light image data based on the adjustment enhancement information. Details of the color correction processing by the color correction unit 303 will be described later.

[0034] In S404, the YUV conversion unit 304 integrates the luminance component to which enhancement processing was applied in S402 and the color component to which color correction processing was applied in S403 to obtain image data in ICtCp format. The YUV conversion unit 304 then converts the image data from ICtCp format to YUV format. Since the method for converting image data in ICtCp format to image data in YUV format is publicly known, a detailed explanation thereof is omitted.

[0035] In this example, visible light image data in YUV format is input to the tone correction unit 300, and the visible light image data output after tone correction is also in YUV format. However, image data in other color spaces, such as RGB format, may also be used.

[0036] Next, the details of the luminance enhancement unit 302 will be explained with reference to Figures 5 and 6. Figure 5 is a block diagram showing an example of the functional configuration of the luminance enhancement unit 302. The luminance enhancement unit 302 includes a grayscale information extraction unit 501, an enhancement information generation unit 502, an enhancement information adjustment unit 503, and an enhancement processing unit 504. These functional blocks also schematically represent the functions realized by the image processing unit 107.

[0037] Figure 6 is a flowchart showing the operation of the brightness enhancement unit 302. In S601, the gradation information extraction unit 501 extracts gradation information of the luminance component (I component) of the visible light image data. Specifically, the gradation information extraction unit 501 extracts the AC component of the luminance component of the visible light image data as gradation information by applying a bandpass filter process that extracts a predetermined frequency band. The gradation information extraction unit 501 outputs the gradation information to the enhancement information generation unit 502.

[0038] In step S602, the tone information extraction unit 501 extracts tone information from the infrared light image data. Specifically, the tone information extraction unit 501 extracts the AC component of the infrared light image data as tone information by applying a bandpass filter process to the infrared light image data to extract a predetermined frequency band. Since the infrared light image data only has a luminance component, there is no need to separate the luminance component. The tone information extraction unit 501 outputs the tone information to the enhancement information generation unit 502. The tone information extraction unit 501 also outputs the tone information of the visible light image to the enhancement information adjustment unit 503.

[0039] Figure 7(a) schematically shows an example of an image represented by the luminance component of visible light image data before the enhancement process is applied. Figure 7(b) schematically shows an example of an image represented by infrared light image data used for the enhancement process of the visible light image data shown in Figure 7(a).

[0040] The visible light image (luminance component) shown in Figure 7(a) shows reduced gradation in the upper region L701 due to fog or haze. On the other hand, the infrared light image shown in Figure 7(b) is less affected by fog or haze due to the wavelength characteristics of infrared light, and therefore has better gradation in region L701 than the visible light image. Here, good gradation is synonymous with having sufficient contrast or dynamic range visually. The gradation information extraction unit 501 applies spatial bandpass filtering to extract gradation information (AC component) from these two images.

[0041] The frequency characteristics of the bandpass filters applied in S601 and S602 may be dynamically set according to the spatial frequency characteristics contained in the visible light image (luminance component) and the infrared light image. For example, suppose the pixel size of the image sensor that captures the infrared light image is larger than the pixel size of the image sensor that captures the visible light image. In this case, in order to match the frequency bands extracted from the visible light image data and the infrared light image data, the passband of the bandpass filter applied to the visible light image data can be set to a lower frequency than the passband of the bandpass filter applied to the infrared light image data.

[0042] In S603, the enhancement information generation unit 502 generates enhancement information based on the visible light image data (luminance component) and the gradation information extracted from the infrared light image data. The enhancement information has significant values ​​in the region of the visible light image data (luminance component) where the AC component of the infrared light image data is combined (enhancement processing is applied), and does not have significant values ​​in the region where it is not combined (enhancement processing is not applied).

[0043] Specifically, the enhancement information generation unit 502 outputs enhancement information extracted from infrared light image data for regions (low contrast regions) where the gradation information of the visible light image data (luminance component) is below a predetermined threshold. Furthermore, for regions where the gradation information of the visible light image data (luminance component) exceeds a predetermined threshold, the enhancement information generation unit 502 outputs a predetermined fixed value (e.g., "0") as enhancement information. The threshold can be determined experimentally in advance, for example.

[0044] For example, in the visible light image (luminance component) shown in Figure 7(a), for the low-contrast region L701, the gradation information (AC component) of region L701 extracted from the infrared light image data is output as enhancement information. For region L702, 0 is output as enhancement information. The enhancement information is information that indicates the region of the visible light image that should be combined with the infrared light image, and it is also a correction value that corrects the gradation of the visible light image. The enhancement information is output to the enhancement information adjustment unit 503.

[0045] Only in areas where the tonal range of the visible light image is insufficient due to haze or other factors, the AC component extracted from the infrared light image data is output as enhancement information. Therefore, the tonal range of the visible light image is maintained in areas where improvement of tonal range is not necessary, and unnecessary emphasis on contrast can be suppressed.

[0046] In S604, the enhancement information adjustment unit 503 adjusts the enhancement information based on the gradation information extracted from the visible light image data (luminance component). The enhancement information adjustment unit 503 outputs the adjusted enhancement information to the enhancement processing unit 504 and the color correction unit 303. Details of the method for adjusting the enhancement information will be described later.

[0047] In step S605, the enhancement processing unit 504 applies enhancement processing to the luminance component of the visible light image data by adding the adjusted enhancement information (the AC component of the adjusted infrared light image data). The enhancement processing unit 504 outputs the enhanced luminance component to the YUV conversion unit 304.

[0048] The enhancement process can be expressed by the following equation 1. In this specification, the coordinates within the image represented by the image data are expressed as coordinates (x,y) in the XY Cartesian coordinate system. I out (x,y) = EINFadj (x,y) + I in (x,y) ···Equation 1 I in (x,y) represents the luminance component of the visible light image data before enhancement at image coordinates (x,y), and EINFadj(x,y) represents the adjusted enhancement information at coordinates (x,y). out (x,y) represents the luminance component of the enhanced visible light image data at coordinate (x,y).

[0049] Furthermore, if the enhancement information is 0, Equation 1 may or may not be applied. In this specification, the region where the enhancement information is 0 is referred to as the region to which no enhancement processing is applied, regardless of whether Equation 1 is applied or not.

[0050] The operation of the enhancement information adjustment unit 503 will now be explained in detail. As explained with respect to Figure 7, the enhancement information is gradation information extracted from infrared light image data for low-contrast regions L701 in the visible light image, and a fixed value of 0 for non-low-contrast regions L702.

[0051] In this case, the Enhance Information Adjustment Unit 503 is (1) The sum of absolute values ​​of grayscale information extracted from the region L701 of the visible light image data (luminance component) where the enhancement information is not 0 (the sum of absolute values ​​of grayscale information of the region of the visible light image that is enhanced), (2) The sum of absolute values ​​of grayscale information extracted from the region L702 of the visible light image data (luminance component) where the enhancement information is 0 (the sum of absolute values ​​of grayscale information from the region of the visible light image that is not enhanced), and (3) Calculate the absolute sum of the region corresponding to region L701 of the enhancement information (the absolute sum of the grayscale information of the region used for enhancement processing in the infrared light image).

[0052] The enhancement information adjustment unit 503 then adjusts the enhancement information according to the relationship between the sum of the absolute values ​​of the grayscale information for regions L701 and L702. Specifically, the enhancement information adjustment unit 503 finds an adjustment gain such that the sum of (1) and (3) described above is the same as (2), or the difference between the two is less than or equal to a threshold. In the latter case, the threshold can be determined in advance, for example, experimentally.

[0053] The sum of (1) and (3) is the tonal information of the enhanced region in the enhanced visible light image (composite image), and (2) is the tonal information of the unenhanced region in the enhanced visible light image (composite image). By using the enhancement information corrected with the adjustment gain, it is possible to improve the visibility or tonality (contrast) of low-contrast regions while suppressing the unnaturalness of the tonality between the enhanced and unenhanced regions in the enhanced visible light image.

[0054] For example, the adjustment gain is the ratio of the absolute sum of the tonal information. Adjustment gain = ((1) + (3)) / (2) It can be calculated as follows. Furthermore, since the region of the visible light image that is enhanced is a region with low contrast, the contribution of (1) shown in the sum of (1) and (3) is small. Therefore, Adjustment gain = (3) / (2) You may also request it as follows:

[0055] Furthermore, if the adjustment gain exceeds 1, not only the tonal information of the infrared image but also the noise component of the infrared image is amplified. Therefore, an upper limit (>1) may be set for the adjustment gain. If the adjustment gain obtained by the above formula exceeds the upper limit, the adjustment gain is set to the upper limit. Note that whether the noise component added by the enhancement process is noticeable depends on the shooting conditions of the visible light image. For this reason, the upper limit when the ISO sensitivity during shooting of the visible light image is the second sensitivity (>first sensitivity) may be set lower than the upper limit when the ISO sensitivity is the first sensitivity. Also, the upper limit when the representative luminance value of the visible light image is the second value (<first value) may be set lower than the upper limit when the representative luminance value is the first value. The representative luminance value may be, for example, the average luminance value.

[0056] The enhancement information adjustment unit 503 adjusts the enhancement information by multiplying it by an adjustment gain, as shown in Equation 2 below. EINFadj (x,y) = EINF (x,y) × I_GAIN (x,y) ···Equation 2 Here, EINF(x,y) is the enhancement information at coordinate (x,y), and I_GAIN(x,y) is the adjustment gain at coordinate (x,y). Then, EINFadj(x,y) is the adjusted enhancement information at coordinate (x,y).

[0057] Here, as an example, we have explained how to determine the adjustment gain so that the absolute sum of the gradation information in region L702 is the same as the absolute sum of the gradation information in region L701. However, the adjustment gain can also be determined by other methods, such as determining the adjustment gain so that the difference between the absolute sum of the gradation information in region L702 and the absolute sum of the gradation information in region L701 is less than or equal to a threshold.

[0058] For example, the adjustment gain may be determined such that the sum of the absolute values ​​of the tonal information in the subject regions is equal or the difference is less than or equal to a threshold for a specific subject (e.g., a person) in region L701 to which enhancement processing is applied, and for a similar or identical subject in region L702. The specific subject can be detected using known methods such as template matching or feature detection. Alternatively, regions with similar color and brightness to the specific subject may be detected as regions of similar subjects.

[0059] Furthermore, the enhancement information adjustment unit 503 may change the adjustment gain according to the magnitude of the enhancement information value. For example, if the enhancement information value is small (for example, greater than 0 and below the threshold), it indicates that the contrast of the visible light image in the processing area is very low. Therefore, the enhancement information adjustment unit 503 may reduce the adjustment gain applied to small enhancement information values. This prevents the contrast from being increased unnecessarily in areas of the subject that are originally low contrast.

[0060] Furthermore, the enhancement information adjustment unit 503 may change the adjustment gain according to the subject distance. The further away the subject, the greater the reduction in visibility and contrast due to haze. Therefore, the adjustment gain may be changed so that it decreases as the subject distance in the area to which the enhancement processing is applied increases. This makes it possible to achieve a natural improvement in visibility or contrast, as if the haze had been reduced.

[0061] Next, the details of the color correction unit 303 will be described. Figure 8 is a block diagram showing an example of the functional configuration of the color correction unit 303. The color correction unit 303 includes a saturation correction gain calculation unit 801 and a saturation correction unit 802. These functional blocks also schematically represent the functions realized by the image processing unit 107.

[0062] The details of how the color correction unit 303 corrects the color components of visible light image data using the adjusted enhancement information will be explained using the flowchart shown in Figure 9. In S901, the saturation correction gain calculation unit 801 calculates the saturation correction gain S_gain using the adjusted enhancement information. Details of the operation of the saturation correction gain calculation unit 801 will be described later.

[0063] In S902, the saturation correction unit 802 applies saturation correction processing using the saturation correction gain S_gain obtained in S901 to the color components of the visible light image data separated in S401. This makes it possible to apply color correction processing corresponding to the strength of the enhancement processing applied to the luminance component.

[0064] The saturation correction process may be, for example, as shown in Equation 3 below. Ct' (x,y) = Ct (x,y) × S_gain (x,y) Cp' (x,y) = Cp (x,y) × S_gain(x,y)...Equation 3 Here, Ct(x,y) and Cp(x,y) are the signal values ​​of the color components at the image coordinates (x,y) before correction. Also, S_gain(x,y) is the saturation correction gain at the image coordinates (x,y). And Ct' (x,y) and Cp' (x,y) are the signal values ​​of the color components at the image coordinates (x,y) after saturation correction processing.

[0065] Next, the operation of the saturation correction gain calculation unit 801 will be described in detail. The saturation correction gain calculation unit 801 can determine the saturation correction gain corresponding to the adjusted enhancement information, for example, by referring to the saturation correction table.

[0066] Figure 10 is a schematic graph showing the relationship between the adjusted enhancement information registered in the saturation correction table and the saturation correction gain. The horizontal axis represents the adjusted enhancement information obtained with S604, and the vertical axis represents the saturation correction gain.

[0067] For intervals where the adjusted enhancement information is below a predetermined threshold A, the saturation correction gain is 1x (×1.0). Furthermore, in intervals where the adjusted enhancement information is greater than threshold A, the saturation correction gain increases linearly with increasing adjusted enhancement information. The saturation correction table stores associated saturation correction gains that satisfy the relationship shown in Figure 10 for discrete values ​​of the adjusted enhancement information. The saturation correction gain calculation unit 801 reads the saturation correction gains corresponding to the two values ​​closest to the adjusted enhancement information from the table and can determine the saturation correction gain corresponding to the adjusted enhancement information value through linear interpolation. Alternatively, a function may be stored instead of a table.

[0068] Alternatively, instead of using the adjusted enhancement information, a table may be used that registers the saturation correction gain corresponding to the absolute difference or ratio of the I component before and after applying the enhancement process.

[0069] Furthermore, the saturation correction gain calculation unit 801 may adjust the saturation correction gain according to the exposure conditions when capturing a visible light image. For example, when the shooting sensitivity (ISO sensitivity) is high, the amount of noise in the image increases, which increases the value of the adjusted enhancement information. Therefore, if the shooting sensitivity is above a predetermined threshold, the saturation correction gain calculation unit 801 may adjust the saturation correction gain to a value smaller than the normal value. This can suppress the emphasis on color noise.

[0070] Furthermore, the saturation correction gain may be adjusted considering the presence or degree of infrared auxiliary light illumination during shooting. When infrared auxiliary light is illuminated during infrared image capture, the pixel values ​​(luminance component values) of the infrared image become larger than when illumination is not performed, and therefore the adjusted enhancement information values ​​also become larger. By preparing saturation correction tables for both cases—with and without infrared light illumination—and calculating the saturation correction gain as shown in Equation 4, a saturation correction gain that takes into account the effect of infrared light illumination can be obtained. This enables appropriate saturation correction.

[0071] S_gain (x,y) = α1 × S_gain_on (x,y) + (1-α1) × S_gain_off (x,y) ···Equation 4 Here, S_gain_on(x,y) is the saturation correction gain for infrared light illumination at coordinate (x,y). S_gain_off(x,y) is the saturation correction gain for non-infrared light illumination at coordinate (x,y). α1 is the degree of infrared light illumination, with values ​​ranging from 0.0 to 1.0, where 1.0 represents the highest illumination. For example, the relationship between subject distance and α1 can be pre-registered, allowing the use of α1 according to the subject distance for each coordinate. S_gain(x,y) represents the saturation correction gain at coordinate (x,y).

[0072] In addition to infrared light illumination, the saturation correction gain may be adjusted according to the subject distance. By preparing saturation correction tables for both distant and close-up photography, the saturation correction gain can be determined as shown in Equation 5. This makes it possible to perform saturation correction using a larger saturation correction gain for distant subjects and a smaller saturation correction gain for closer subjects.

[0073] S_gain (x,y) = α2 × S_gain_far (x,y) + (1-α2) × S_gain_near (x,y) ···Equation 5 Specifically, S_gain_far(x,y) is the saturation correction gain for distant objects. S_gain_near(x,y) is the saturation correction gain for close objects. α2 indicates the subject distance and has values ​​from 0.0 to 1.0, for example, 1.0 represents infinity. And S_gain(x,y) is the saturation correction gain.

[0074] Furthermore, to prevent tone jumps within subject areas where the colors after saturation correction are the same or similar, the saturation correction gain may be adjusted so that the difference in correction amount for areas consisting of similar colors is 0 or less than a threshold. Specifically, the gradation correction unit 300 calculates filter weights such that the weight increases as the signal values ​​become more similar to the point of interest in the input visible light image, and applies the filter to the saturation correction gain obtained in S901 to weight and smooth the saturation correction gain. This smooths the difference in correction amount for areas consisting of similar colors. Note that known filters such as bilateral filters can be used as filters. This smoothing process can suppress tone jumps.

[0075] According to this embodiment, the components of the invisible light image to be combined with the visible light image are corrected so that the difference in gradation between the areas in the combined image where the invisible light image components are combined and the areas where they are not is reduced before the images are combined. As a result, it is possible to partially improve visibility and gradation (contrast) while suppressing the unnatural impression that arises from the difference in gradation between the areas in the combined image where the invisible light image components are combined and the areas where they are not.

[0076] ●(Second Embodiment) Next, a second embodiment of the present invention will be described. This embodiment differs from the first embodiment in that it performs contrast correction including the luminance component of the visible light image region in which the non-visible light image is not synthesized, and performs saturation correction using a saturation correction adjustment gain. Note that this embodiment is the same as the first embodiment except for the functional configuration or operation of the image processing unit 107, so the explanation of configurations and operations common to the first embodiment will be simplified.

[0077] Figure 11 is a functional block diagram schematically representing the image processing unit 107 as a tone correction unit 300' during enhancement processing, for the purpose of explaining the enhancement processing performed in this embodiment. Therefore, the functional blocks of the tone correction unit 300' are actually only a part of the functions realized by the image processing unit 107. The tone correction unit 300' includes an ICtCp conversion unit 301, a luminance enhancement unit 1101, a color correction unit 1102, and a YUV conversion unit 304. In Figure 11, functional blocks similar to those of the tone correction unit 300 described in the first embodiment are given the same reference numerals as in Figure 3.

[0078] In this embodiment, the operation of the luminance enhancement unit 1101 and the color correction unit 1102 differs from that of the first embodiment. Therefore, the operation of the luminance enhancement unit 1101 and the color correction unit 1102 will be described in detail.

[0079] Figure 12 is a block diagram showing an example of the functional configuration of the luminance enhancement unit 1101. The luminance enhancement unit 1101 includes a gradation information extraction unit 501, an enhancement information generation unit 502, an enhancement processing unit 1201, an enhanced image gradation information extraction unit 1202, a contrast correction information generation unit 1203, and a contrast correction unit 1204. These functional blocks also schematically represent the functions realized by the image processing unit 107. Furthermore, in Figure 12, functional blocks similar to those of the luminance enhancement unit 302 described in the first embodiment are given the same reference numerals as in Figure 5.

[0080] Next, the operation of the brightness enhancement unit 1101 will be explained in detail with reference to the flowchart in Figure 13. In Figure 13, the same reference numerals as in Figure 6 are used for the steps that perform the same operations as the brightness enhancement unit 302 in the first embodiment. S601, S602, and S603 are as described in the first embodiment, so their explanation will be omitted. In S603, the enhancement information generation unit 502 outputs the generated enhancement information to the color correction unit 1102, the contrast correction information generation unit 1203, and the enhancement processing unit 1201.

[0081] In S1301, the enhancement processing unit 1201 applies enhancement processing to the luminance component of the visible light image data. The enhancement information used by the enhancement processing unit 1201 in this embodiment is enhancement information generated by the enhancement information generation unit 502, and has not undergone the adjustments described in the first embodiment. However, the operation of applying the enhancement processing may be the same as in the first embodiment. The luminance component to which the enhancement processing has been applied (enhanced image data) is output to the enhanced image gradation information extraction unit 1202 and the contrast correction unit 1204.

[0082] In S1302, the enhanced image gradation information extraction unit 1202 extracts gradation information from the enhanced image data. Specifically, by applying a bandpass filter process that extracts a predetermined frequency band to the enhanced image data, the AC component of the enhanced image data is extracted as gradation information.

[0083] In S1303, the contrast correction information generation unit 1203 generates contrast correction information based on the gradation information of the enhanced image data and the enhancement information (gradation information of the infrared light image). The contrast correction information generation unit 1203 outputs the generated contrast correction information to the contrast correction unit 1204. Details regarding the operation of the contrast correction information generation unit 1203 will be described later.

[0084] In S1304, the contrast correction unit 1204 corrects the contrast of the enhanced image data using the contrast correction information. The contrast correction unit 1204 can correct the contrast of the enhanced image by adding the contrast correction information to the enhanced image, for example, as shown in Equation 6. P out (x,y) = CNTinf (x,y) + P in (x,y) ... Equation 6

[0085] Here, P in (x,y) is the pixel value of the enhanced image data at image coordinates (x,y), and CNTinf(x,y) is the contrast correction information at image coordinates (x,y). Then, P out (x,y) is the pixel value of the enhanced image data (contrast-corrected image data) after contrast correction at image coordinates (x,y). The contrast-corrected image data is output to the YUV conversion unit 304. The YUV conversion unit 304 generates image data in YUV format from the color components from the color correction unit 1102 and the contrast-corrected image data (luminance components), and outputs it as image data with gradation correction.

[0086] The operation of the contrast correction information generation unit 1203 will be described in detail. The contrast correction information generation unit 1203 processes the gradation information output by the enhanced image gradation information extraction unit 1202. (4) The sum of the absolute values ​​of the grayscale information corresponding to the region where the enhancement information is not zero, (5) The sum of the absolute values ​​of the grayscale information corresponding to the region where the enhancement information is 0 and Each of these is calculated. For the image shown in Figure 7, this means calculating the sum of the absolute values ​​of the grayscale information of the enhanced image data for regions L701 and L702. (4) corresponds to (1) + (3) in the first embodiment, and (5) corresponds to (2) in the first embodiment.

[0087] Then, the contrast correction information generation unit 1203 determines an adjustment gain such that (4) and (5) are the same value, or the difference between them is below a threshold.

[0088] For example, the adjustment gain is the ratio of (4) and (5). Adjustment gain = (4) / (5) It can be calculated as follows.

[0089] Thus, the adjustment gain in this embodiment can be determined in the same way as the adjustment gain in the first embodiment, except that it uses the gradation information of the composite image after enhancement processing. In this embodiment as well, an upper limit value for the adjustment gain may be set, similar to the first embodiment.

[0090] Then, the contrast correction information generation unit 1203 generates contrast correction information by multiplying the adjustment gain by the gradation information of the enhanced image data, as shown in Equation 7 below. CNTinf (x,y) = TONEinf (x,y) × CNT_GAIN (x,y) ···Equation 7

[0091] Here, TONEinf(x,y) is the grayscale information of the enhanced image data at image coordinates (x,y), and CNT_GAIN(x,y) is the contrast correction gain at image coordinates (x,y). Also, CNTinf(x,y) is the contrast correction information at image coordinates (x,y).

[0092] By correcting the tonal information of enhanced image data using contrast correction gain, it is possible to generate contrast correction information that balances the tonality between areas where enhancement processing has been applied and areas where it has not. Therefore, with enhanced image data to which contrast correction information has been applied, it becomes possible to partially improve visibility or tonality (contrast) while suppressing visual inconsistencies.

[0093] Here, we have explained an example of determining the adjustment gain for enhanced image data so that the sum of the absolute values ​​of the tonal information in the region where enhancement processing is not applied (L702) is the same as the sum of the absolute values ​​of the tonal information in the region where enhancement processing is applied (L701). However, the adjustment gain may also be determined by other methods, such as determining the adjustment gain so that the difference between the sum of the absolute values ​​of the tonal information in the region where enhancement processing is not applied (L702) and the sum of the absolute values ​​of the tonal information in the region where enhancement processing is applied (L701) is less than or equal to a threshold.

[0094] For example, in contrast image data, adjustment gains may be determined such that the sum of absolute values ​​of the tonal information of a specific subject (e.g., a person) in region L701 and a similar or identical subject in region L702 are equal, or the difference is less than or equal to a threshold. The specific subject can be detected using known methods such as template matching or feature detection. Alternatively, regions with similar color and brightness to the specific subject may be detected as regions of similar subjects.

[0095] Next, the details of the color correction unit 1102 will be described. Figure 14 is a block diagram showing an example of the functional configuration of the color correction unit 1102. The color correction unit 1102 includes a saturation correction gain calculation unit 801, a saturation correction adjustment gain calculation unit 1401, and a saturation correction unit 1402. These functional blocks also schematically represent the functions realized by the image processing unit 107. In Figure 14, functional blocks similar to those of the color correction unit 303 described in the first embodiment are given the same reference numerals as in Figure 8.

[0096] The operation of the color correction unit 1102 will be explained in detail using the flowchart shown in Figure 15. In Figure 15, steps that perform the same operations as the color correction unit 303 of the first embodiment are denoted by the same reference numerals as in Figure 9.

[0097] In S901, the saturation correction gain calculation unit 801 calculates the saturation correction gain S_gain using the enhancement information. The operation of the saturation correction gain calculation unit 801 is the same as in the first embodiment except for the use of the enhancement information, so a description is omitted.

[0098] In S1501, the saturation correction adjustment gain calculation unit 1401 determines whether each pixel coordinate of the input visible light image data is a pixel of a red subject. This determination can be performed using a known method based on the hue obtained from the pixel value.

[0099] The saturation correction adjustment gain calculation unit 1401 decides not to perform saturation correction adjustment on pixel coordinates that are determined not to be red subjects, and sets the adjustment gain to 1x to take spectral characteristics into consideration. On the other hand, for pixel coordinates that are determined to be red subjects, the saturation correction adjustment gain calculation unit 1401 further determines whether the value of the G component is greater than or equal to threshold B.

[0100] If the signal value G(x,y) at image coordinates (x,y) is determined to be greater than or equal to threshold B, the saturation correction adjustment gain calculation unit 1401 determines that the subject is red, with wavelengths close to G, and therefore decides not to perform saturation correction adjustment, setting the adjustment gain to 1x. The adjustment gain determined by the saturation correction adjustment gain calculation unit 1401 is an adjustment gain that takes into account the spectral characteristics of the image sensor, more specifically, the case where the R pixel is sensitive to infrared light wavelengths.

[0101] On the other hand, if it is determined that the signal value G(x,y) at image coordinates (x,y) is less than the threshold B, the saturation correction adjustment gain calculation unit 1401 calculates an adjustment gain that takes spectral characteristics into account. Here, the saturation correction adjustment gain calculation unit 1401 calculates an adjustment gain that has a value corresponding to the magnitude of the difference (IR-R) between the pixel value of the infrared image and the R component of the pixel value of the visible image at image coordinates (x,y).

[0102] Figure 16 shows an example of the relationship between the adjustment gain for considering spectral characteristics and IR-R. The saturation correction adjustment gain calculation unit 1401 can determine the adjustment gain for each target pixel using a pre-registered table that associates multiple discrete values ​​of IR-R with the adjustment gain, as shown in Figure 16. Alternatively, the adjustment gain may be determined by other methods, such as using a function that shows the relationship between IR-R and the adjustment gain, without using a table.

[0103] As shown in Figure 16, the adjustment gain is set to 1 (effectively no adjustment) when the difference between the pixel value of the infrared image and the R component value of the visible light image (IR-R) is within a predetermined threshold C. This is because pixels in this range are subjects that do not contain much IR signal. On the other hand, in the range exceeding threshold C, the adjustment gain is linearly decreased in response to the increase in IR-R. This is because, in the range where the value of IR-R increases, it is assumed that the amount of IR component included in the R component value of the visible light image data also increases.

[0104] The saturation correction adjustment gain calculation unit 1401 determines the saturation correction adjustment gain by adjusting the saturation correction gain calculated in S901 with an adjustment gain that takes spectral characteristics into consideration. The saturation correction adjustment gain calculation unit 1401 can determine the saturation correction adjustment gain as shown in Equation 8 below, for example. S_gain_adj_gain (x,y) = S_gain (x,y) × gain (x,y) ···Equation 8

[0105] Here, S_adj_gain(x,y) is the saturation correction adjustment gain at image coordinates (x,y), and S_gain(x,y) is the saturation correction gain before adjustment at image coordinates (x,y). Also, gain(x,y) is an adjustment gain to take into account the spectral characteristics at image coordinates (x,y).

[0106] The adjustment gain for considering spectral characteristics can also be smoothed in the same way as the saturation correction gain in the first embodiment. Specifically, first, spatial filter weights are calculated for the image coordinates (x,y) of the visible light image such that the weight increases as the signal values ​​become more similar. Then, by applying the spatial filter to the adjustment gain for considering spectral characteristics obtained for the pixel coordinates (x,y), the adjustment gain for considering spectral characteristics is weighted and smoothed. This makes it possible to suppress tone jumps within the same subject area after saturation correction.

[0107] In S1502, the saturation correction unit 1402 performs saturation correction by applying the saturation correction adjustment gain obtained in S1501 to the color components of the visible light image data separated in S401. The saturation correction unit 1402 can perform saturation correction using S_gain_adj_gain (x,y) instead of S_gain (x,y) in Equation 3 described in the first embodiment.

[0108] By adjusting the saturation correction gain based on the AC component of an infrared light image, taking into account the difference between the R component value and the IR component value, appropriate saturation correction can be performed for red subjects even when the R pixels of the image sensor are sensitive to infrared light.

[0109] In the first embodiment, the component of the invisible light image to be synthesized with the visible light image was adjusted considering the tonality of the region where the invisible light image component is not synthesized, thereby suppressing the occurrence of an unnatural difference in tonality between the region where the invisible light image component is synthesized and the region where it is not synthesized. On the other hand, in this embodiment, after synthesizing the component of the invisible light image, a tonal correction that considers the tonality of the region where the invisible light image component is not synthesized is applied, thereby suppressing the occurrence of an unnatural difference in tonality between the region where the invisible light image component is synthesized and the region where it is not synthesized.

[0110] ●(Third embodiment) Next, a third embodiment of the present invention will be described. This embodiment is similar to the first and second embodiments in that it suppresses the unnaturalness in the relationship of gradation that may occur between regions with different intensity of applied processing when processing that affects gradation is applied to a visible light image using information extracted based on an infrared light image. Since this embodiment is similar to the first embodiment except for the functional configuration or operation of the image processing unit 107, the explanation of configurations and operations common to the first embodiment will be omitted or simplified.

[0111] The following describes a case where the luminance component extracted based on an infrared light image is a haze component, and the processing is the enhancement (concentration) of the haze component, but is not limited to this case. Figure 17 is a functional block diagram schematically representing the image processing unit 107 as the tone correction unit 1700 when the haze concentration processing is performed, for the purpose of explaining the haze concentration processing (haze concentration processing). Therefore, the functional blocks of the tone correction unit 1700 are actually part of the functions realized by the image processing unit 107. The tone correction unit 1700 includes an ICtCp conversion unit 301, a haze concentration unit 1701, a color correction unit 1702, and a YUV conversion unit 304. In Figure 17, functional blocks similar to those of the tone correction unit 300 described in the first embodiment are given the same reference numerals as in Figure 3.

[0112] Next, the process of haze enhancement processing applied by the tone correction unit 1700 to the visible light image will be explained using the flowchart shown in Figure 18. Here, the visible light image data and infrared light image data input to the tone correction unit 1700 may be images captured by the imaging unit 105 and supplied through the A / D conversion unit 106, or they may be read from the recording unit 108. The visible light image data and infrared light image data used for haze enhancement processing are image data of the same scene captured simultaneously or at sufficiently close timings. In Figure 18, the same reference numerals as in Figure 4 are used for the process that performs the same operation as the tone correction unit 300 described in the first embodiment.

[0113] In S401, the ICtCp conversion unit 301 converts the color space of the visible light image data (e.g., YUV color space) to the ICtCp color space. The ICtCp conversion unit 301 further separates the visible light image data converted to ICtCp format into an I component representing the brightness (luminance) component and a CtCp component representing the color component. The ICtCp conversion unit 301 outputs the I component to the haze enhancement unit 1701 and the CtCp component to the color correction unit 1702.

[0114] In S1801, the haze enhancement unit 1701 enhances the haze in the I component (luminance component) of the visible light image data based on the infrared light image data, thereby reducing the gradation (contrast) of the visible light image data and increasing its brightness. The haze enhancement unit 1701 outputs the gradation-corrected I component of the visible light image data to the YUV conversion unit 304. The haze enhancement unit 1701 also outputs an enhancement gain used for color component correction to the color correction unit 1702. Details of the haze enhancement process by the haze enhancement unit 1701 will be described later.

[0115] In S1802, the color correction unit 1702 applies color correction processing to the CtCp component (color component) of the visible light image data based on the enrichment gain. Details of the color correction processing by the color correction unit 1702 will be described later.

[0116] Next, the operation of the haze thickening unit 1701 will be explained in detail. Figure 19 is a block diagram showing an example of the functional configuration of the haze enhancement unit 1701. The haze enhancement unit 1701 includes a tone information extraction unit 501, a haze amount calculation unit 1901, an enhancement gain calculation unit 1902, a haze enhancement processing unit 1903, and a tone information extraction unit 1904. These functional blocks also schematically represent the functions realized by the image processing unit 107. The tone information extraction unit 501 is the same as the one described using Figure 3.

[0117] Next, the operation of the haze-enhancing unit 1701 will be explained in detail with reference to the flowchart in Figure 20. In Figure 20, steps that perform the same operation as the luminance enhancement unit 302 in the first embodiment are denoted by the same reference numerals as in Figure 6.

[0118] Since steps S601 and S602 were described in the first embodiment, their explanation will be omitted. The gradation information extraction unit 501 outputs the gradation information extracted in S601 and S602 to the haze amount calculation unit 1901. The gradation information extraction unit 501 also outputs the gradation information of the visible light image extracted in S601 to the enrichment gain calculation unit 1902.

[0119] Here, the image shown in Figure 7(a) is an example of an image represented by the luminance component of visible light image data before the application of the haze enhancement process. The image shown in Figure 7(b) is an example of an image represented by the infrared light image data used for the haze enhancement process of the visible light image data shown in Figure 7(a).

[0120] The gradation information extraction unit 501 extracts gradation information (AC component) from these two images by applying spatial bandpass filtering, similar to the first embodiment. In this embodiment, the haze enhancement process emphasizes the decrease in gradation due to fog or haze, further reducing the gradation of region L701 in the visible light image shown in Figure 7(a). The haze enhancement process can be used, for example, to create a fantastical effect by representing dense fog or haze.

[0121] In S2001, the haze amount calculation unit 1901 calculates the haze amount based on the grayscale information extracted by the grayscale information extraction unit 501 from the visible light image data (luminance component) and infrared light image data. The haze amount has a significant value in the areas where haze occurs and a significant value in the areas where haze does not occur.

[0122] Specifically, the haze amount calculation unit 1901 outputs the haze amount as the absolute difference between the gradation information of the visible light image data (luminance component) and the gradation information of the infrared light image data. Therefore, the haze amount is calculated for each pixel. In the region where the absolute difference value is below a predetermined threshold, the haze amount may be output as a predetermined fixed value (for example, "0"). The threshold can be determined experimentally in advance, for example.

[0123] For example, in region L701 of Figure 7, haze occurs, so the absolute difference between the gradation information of the visible light image data (luminance component) and the infrared light image data is large. On the other hand, in region L702 of Figure 7, there is no haze, so the absolute difference between the gradation information of the visible light image data (luminance component) and the infrared light image data is small. Therefore, the haze amount calculation unit 1901 outputs a haze amount with a large value for region L701 and a haze amount with a small value (for example, "0") for region L702.

[0124] In S2002, the enrichment gain calculation unit 1902 sets the haze amount enrichment gain characteristic. The enrichment gain characteristic represents the relationship between the haze amount and the enrichment gain. An example of the enrichment gain characteristic is shown in Figure 21. The enrichment gain characteristic shown in Figure 21(a) is such that the enrichment gain is equal when the haze amount is 0 or less than or equal to a predetermined threshold TH, and when the haze amount exceeds the threshold TH, the enrichment gain decreases linearly as the haze amount increases. The enrichment gain indicates the intensity of the haze amount enrichment treatment, with an intensity of 0 (no effect on the haze amount) when it is equal to the haze amount, and an intensity > 0 (haze amount is emphasized) when it is less than equal to the haze amount. Therefore, in the enrichment gain characteristic shown in Figure 21(a), the intensity of the enrichment treatment is also maximum when the haze amount is maximum.

[0125] The enrichment gain characteristics shown in Figure 21(b) are obtained by increasing the lower limit of the enrichment gain compared to the enrichment gain characteristics shown in Figure 21(a). When the enrichment gain characteristics in Figure 21(b) are set, the maximum intensity of the haze enrichment treatment can be reduced compared to when the enrichment gain characteristics in Figure 21(a) are set.

[0126] Note that the gain enhancement characteristics shown in FIG. 21 are exemplary, and other characteristics may be used. For example, non-linear gain enhancement characteristics may be used. Also, the gain enhancement set by the gain enhancement calculation unit 1902 may be selectable by the user or automatically set. For example, a plurality of gain enhancement characteristics with different maximum intensities may be prepared, and the user may select the gain enhancement characteristic according to the desired maximum intensity. Also, different gain enhancement characteristics may be prepared according to the shooting scene, and when a shooting mode for a specific scene is set, the gain enhancement calculation unit 1902 may set the gain enhancement characteristic corresponding to the scene.

[0127] In S2003, the gain enhancement calculation unit 1902 applies the amount of blurring calculated for each pixel to the gain enhancement characteristic to obtain the gain enhancement for each pixel. The gain enhancement calculation unit 1902 can obtain the gain enhancement corresponding to the amount of blurring, for example, by referring to a table corresponding to the gain enhancement characteristic.

[0128] In the table, correction gains corresponding to the gain enhancement characteristic are registered in association with discrete values of the amount of blurring. transformation The gain enhancement calculation unit 1902 can read out the gain enhancements corresponding to the two values closest to the value of the amount of blurring from the table and obtain the gain enhancement corresponding to the value of the amount of blurring by linear interpolation. Note that a function may be stored instead of the table. The gain enhancement calculation unit 1902 outputs the obtained gain enhancement to the blurring enhancement processing unit 1903 and the color correction unit 1702.

[0129] In S2004, the blurring enhancement processing unit 1903 applies blurring enhancement processing to the luminance component of the visible light image data based on the gain enhancement to generate a blurring-enhanced image. The blurring enhancement processing unit 1903 outputs the luminance component after the blurring enhancement processing to the gradation information extraction unit 1904 and the YUV conversion unit 304.

[0130] The blurring enhancement processing can be expressed by Equation 9 below. I out(x,y) = F_TGT - F_GAIN(x,y)×(F_TGT - I in (x,y))···Formula 9 I in (x,y) is the luminance component of the visible light image data before haze enhancement processing at coordinate (x,y) of the image, F_GAIN(x,y) is the enhancement gain at coordinate (x,y), and F_TGT is a predetermined target value for the amount of haze. out (x,y) represents the luminance component of the visible light image data after haze enhancement processing at coordinate (x,y).

[0131] Figure 22(a) shows an example of pixel values ​​before haze enhancement processing, and Figure 22(b) shows an example of pixel values ​​after haze enhancement processing. The haze enhancement processing unit 1903 calculates the pixel value after haze enhancement processing by subtracting the value obtained by applying an enhancement gain to the difference between the pixel value of the visible light image data and the target value of haze amount (shaded area in Figure 22(a)) from the target value of haze amount. This haze enhancement processing makes the pixel values ​​in the coordinate interval 1 to 5 low contrast and performs gradation correction that makes them brighter, thus giving the effect of increased haze. The target value of haze amount can be determined in advance, for example, experimentally.

[0132] In S2005, the tone information extraction unit 1904 extracts tone information of the luminance component of the haze-enhanced image. The tone information extraction unit 1904 outputs the extracted tone information to the enlargement gain calculation unit 1902. The method for extracting tone information is the same as that of the tone information extraction unit 501 described in the first embodiment, so the explanation is omitted.

[0133] In S2006, the enrichment gain calculation unit 1902 is: (6) Calculate the sum of absolute values ​​of the tonal information extracted from the region L701, which is the region where the haze amount information is not 0, among the tonal information extracted from the haze-enhanced image data (luminance component). This corresponds to the sum of absolute values ​​of the tonal information extracted from the region of the haze-enhanced image where the intensity of the haze enhancement process is not 0.

[0134] In S2007, the enrichment gain calculation unit 1902 is: (7) Calculate the sum of the absolute values ​​of the grayscale information extracted from the region L702, where the haze amount information is 0, among the grayscale information extracted from the visible light image data (luminance component). This corresponds to the sum of the absolute values ​​of the grayscale information extracted from the region where the haze enhancement intensity is 0 in the haze enhancement image.

[0135] In S2008, the enrichment gain calculation unit 1902 determines whether the difference between (6) and (7) is less than or equal to a predetermined value. If it is determined to be less than or equal to the predetermined value, the haze enrichment process is terminated; otherwise, S2009 is executed.

[0136] S2009 is executed when the difference between (6) and (7) exceeds a predetermined value. In this case, the tonality of the area where haze enhancement processing was performed and the area where it was not performed are significantly different, resulting in an unbalanced and unnatural appearance when viewed as a whole image. Therefore, in S2009, the enhancement gain calculation unit 1902 readjusts the enhancement gain characteristics to weaken the intensity of the haze enhancement processing. Subsequently, by re-executing the processes of S2003 to S2007, a haze-enhanced image is regenerated that suppresses the difference in tonality between the area where haze enhancement processing was applied and the area where it was not applied.

[0137] The enrichment gain characteristics to be reset in S2009 may be enrichment gain characteristics such as those shown in Figure 21(b). For example, multiple enrichment gain characteristics with different maximum intensities can be prepared, and the enrichment gain characteristics can be set to gradually decrease the maximum intensity each time S2009 is executed.

[0138] The haze enhancement unit 1701 outputs the enhancement gain and the luminance component of the haze-enhancing image when the difference between (6) and (7) ultimately falls below a predetermined value through the above processing.

[0139] Furthermore, the intensity gain calculation unit 1902 may change the adjustment gain according to the subject distance. The further the subject is, the greater the reduction in visibility and contrast due to haze. Therefore, the intensity gain may be changed so that it decreases as the subject distance in the area to which haze intensity processing is applied increases. This makes it possible to achieve a natural reduction in visibility or contrast where the haze becomes denser as the subject is farther away.

[0140] Next, the details of the color correction unit 1702 will be described. Figure 23 is a block diagram showing an example of the functional configuration of the color correction unit 1702. The color correction unit 1702 includes a saturation correction gain calculation unit 2301 and a saturation correction unit 2302.

[0141] The details of how the color correction unit 1702 corrects the color components of visible light image data using the enrichment gain will be explained using the flowchart shown in Figure 24. In S2401, the saturation correction gain calculation unit 2301 calculates the saturation correction gain S_gain using the enrichment gain. Details of the operation of the saturation correction gain calculation unit 2301 will be described later.

[0142] In S2402, the saturation correction unit 2302 applies a saturation correction process using the saturation correction gain S_gain obtained in S2401 to the color component (CtCp component) of the visible light image data output by the ICtCp conversion unit 301 in S401. This makes it possible to apply a color correction process corresponding to the strength of the haze enhancement process applied to the luminance component. The saturation correction process can be performed according to Equation 3, as in the first embodiment. The saturation correction unit 2302 outputs the signal values ​​Ct' and Cp' of the color components after the saturation correction process to the YUV conversion unit 304.

[0143] Next, the operation of the saturation correction gain calculation unit 2301 will be described in detail. The saturation correction gain calculation unit 2301 can determine the saturation correction gain corresponding to the concentration gain by, for example, referring to a saturation correction table corresponding to the saturation gain characteristics.

[0144] Figure 25 shows an example of saturation gain characteristics. The saturation gain characteristics represent the relationship between the enrichment gain and the saturation correction gain. The horizontal axis shows the enrichment gain obtained with S2003, and the vertical axis shows the saturation correction gain.

[0145] The saturation gain characteristics shown in Figure 25 indicate that in the interval where the enrichment gain is below a predetermined threshold A, the saturation correction gain is 1x (×1.0), and the color components are not substantially corrected. Furthermore, in the interval where the enrichment gain is greater than threshold A, the value of the saturation correction gain decreases linearly as the enrichment gain increases. When the saturation correction gain is less than 1x, the saturation correction unit 2302 performs a correction process to reduce saturation.

[0146] The saturation correction table stores saturation correction gains that satisfy the relationship shown in Figure 25 for discrete values ​​of the enrichment gain. The saturation correction gain calculation unit 2301 reads the saturation correction gains corresponding to the two values ​​closest to the enrichment gain from the table and can determine the saturation correction gain corresponding to the enrichment gain value by linear interpolation. Alternatively, a function may be stored instead of a table.

[0147] Alternatively, instead of using a concentration gain, a table may be used that registers a saturation correction gain corresponding to the absolute difference or ratio of the I component before and after applying the haze concentration processing.

[0148] According to this embodiment, when processing a visible light image using information obtained using a non-visible light image, it is possible to suppress the unnaturalness in the gradation relationship that may occur between regions where the intensity of the applied processing differs.

[0149] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0150] This embodiment includes the following image processing apparatus, imaging apparatus, image processing method, and program. (Item 1) A determination means for determining the region in a visible light image from which a non-visible light image is synthesized, The system includes a synthesis means that generates a composite image by combining the invisible light image with the visible light image based on the aforementioned determination, The image processing apparatus is characterized in that the synthesis means adjusts the invisible light image so that the difference in gradation information between the region in the synthesized image where the invisible light image is synthesized and the region where it is not synthesized is less than or equal to a predetermined first threshold, and then synthesizes it with the visible light image. (Item 2) The image processing apparatus according to item 1, characterized in that the synthesis means adjusts the invisible light image so that the difference in gradation information between the regions of the same type of subject included in the region where the invisible light image is synthesized and the region where it is not synthesized in the synthesized image is less than or equal to a predetermined threshold, and then synthesizes it with the visible light image. (Item 3) The image processing apparatus according to item 1 or 2, further characterized in that the synthesis means adjusts the non-visible light image according to the subject distance and then synthesizes it with the visible light image. (Item 4) The aforementioned synthesis means synthesizes the invisible light image with the visible light image without adjusting it. The aforementioned image processing device The image processing apparatus according to item 1, further comprising correction means for correcting the contrast of the composite image such that the difference in gradation information between the region in the composite image in which the non-visible light image is composited and the region in which it is not composited is less than or equal to the first threshold. (Item 5) The image processing apparatus according to any one of items 1 to 4, characterized in that the determination means determines a region in the visible light image where the grayscale information is below a second threshold as a region for synthesizing the non-visible light image. (Item 6) The image processing apparatus according to any one of items 1 to 5, characterized in that the synthesis means synthesizes the invisible light image with respect to the luminance component of the visible light image. (Item 7) The image processing apparatus according to item 6, characterized in that the luminance component is the luminance component when the visible light image is represented in a uniform color space. (Item 8) The image processing apparatus according to item 7, characterized in that the uniform color space is the ICtCp color space or the L*a*b* color space. (Item 9) Furthermore, the image processing apparatus according to any one of items 1 to 8 is characterized by having a correction means for correcting the color components of the visible light image. (Item 10) The image processing apparatus according to item 9, characterized in that the correction means corrects the color components based on the gradation information of the non-visible light image. (Item 11) The image processing apparatus according to item 9, characterized in that the correction means adjusts the amount of correction for the color component according to the pixel values ​​before and after the synthesis of the non-visible light image. (Item 12) The image processing apparatus according to item 9, characterized in that the correction means adjusts the amount of correction for the color component according to the exposure conditions when capturing the visible light image. (Item 13) The image processing apparatus according to item 9, characterized in that the correction means adjusts the amount of correction for the color component according to whether or not infrared auxiliary light is irradiated when capturing the visible light image. (Item 14) The image processing apparatus according to item 9, characterized in that the correction means adjusts the amount of correction for the color component according to the distance to the subject. (Item 15) The image processing apparatus according to item 9, characterized in that the correction means adjusts the difference in the amount of correction for regions consisting of similar colors included in the visible light image so that it is less than or equal to a third threshold. (Item 16) The image processing apparatus according to item 15, characterized in that the correction means performs the adjustment by smoothing the amount of correction for regions consisting of similar colors included in the visible light image. (Item 17) The image processing apparatus according to item 9, characterized in that the correction means adjusts the amount of correction for pixels of a red subject among the pixels of the visible light image based on the difference with the corresponding pixel value of the non-visible light image. (Item 18) An imaging means having an image sensor and acquiring visible light images and invisible light images, An image processing apparatus according to any one of items 1 to 17 that uses the visible light image and the non-visible light image acquired by the imaging means, An imaging device characterized by having the following features. (Item 19) An image processing method performed by an image processing device, Determining the region of the visible light image from which the invisible light image will be synthesized, Based on the above determination, the invisible light image is combined with the visible light image to generate a composite image, The aforementioned synthesis is The invisible light image is adjusted such that the difference in gradation information between the region in the composite image where the invisible light image is composited and the region where it is not composited is less than or equal to a predetermined first threshold. An image processing method characterized by comprising: synthesizing the adjusted non-visible light image with the visible light image. (Item 20) A program for causing a computer to function as one of the means of an image processing device described in any one of items 1 through 17. (Item 21) An acquisition means for acquiring information for processing a visible light image corresponding to the invisible light image using a non-visible light image, The system includes a correction means for applying the processing to the visible light image with an intensity based on the aforementioned information, The correction means is characterized in that, in a visible light image to which the processing has been applied, if the difference in gradation information between a region where the intensity of the processing is 0 and a region where the intensity of the processing is not 0 is greater than a predetermined fourth threshold, the processing intensity is corrected to be less than or equal to the fourth threshold. (Item 22) The image processing apparatus according to item 21, characterized in that the acquisition means acquires the information from the luminance component of the non-visible light image and the luminance component of the visible light image. (Item 23) The image processing apparatus according to item 21 or 22, characterized in that the correction means corrects the intensity so that the intensity increases in accordance with an increase in the subject distance. (Item 24) The image processing apparatus according to any one of items 21 to 23, characterized in that the correction means applies processing to the luminance component of the visible light image. (Item 25) The image processing apparatus according to item 24, characterized in that the luminance component is the luminance component when the visible light image is represented in a uniform color space. (Item 26) The image processing apparatus according to item 25, characterized in that the uniform color space is the ICtCp color space or the L*a*b* color space. (Item 27) The image processing apparatus according to any one of items 24 to 26, further characterized in that the correction means corrects the color components of the visible light image. (Item 28) The image processing apparatus according to item 27, characterized in that the correction means corrects the color components based on the intensity of the processing. (Item 29) An imaging apparatus characterized by comprising: an imaging means having an image sensor and acquiring a visible light image and a non-visible light image; and an image processing apparatus according to any one of items 21 to 28 that uses the visible light image and the non-visible light image acquired by the imaging means. (Item 30) An image processing method performed by an image processing device, Using a non-visible light image, information is obtained for processing the non-visible light image and the corresponding visible light image. The process is applied to the visible light image with an intensity based on the aforementioned information, Applying the processing includes correcting the intensity of the processing so that, in a visible light image to which the processing has been applied, the difference in gradation information between a region where the intensity of the processing is 0 and a region where the intensity of the processing is not 0 is greater than a predetermined fourth threshold, the difference becomes less than or equal to the fourth threshold. An image processing method characterized by the following: (Item 31) A program for causing a computer to function as one of the means of an image processing apparatus described in any one of items 21 to 28.

[0151] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0152] 100...Imaging device, 101...Control unit, 102...ROM, 103...RAM, 104...Optical system, 105...Imaging unit, 106...A / D conversion unit, 107...Image processing unit, 108...Recording unit, 109...Display unit

Claims

1. A determination means for determining the region in a visible light image from which a non-visible light image is synthesized, The system includes a synthesis means that generates a composite image by combining the invisible light image with the visible light image based on the aforementioned determination, The image processing apparatus is characterized in that the synthesis means adjusts the invisible light image so that the difference in gradation information between the region in the synthesized image where the invisible light image is synthesized and the region where it is not synthesized is less than or equal to a predetermined first threshold, and then synthesizes it with the visible light image.

2. The image processing apparatus according to claim 1, wherein the synthesis means adjusts the invisible light image so that the difference in gradation information between the regions of the same type of subject included in the region where the invisible light image is synthesized and the region where it is not synthesized in the synthesized image is less than or equal to a predetermined threshold, and then synthesizes it with the visible light image.

3. The image processing apparatus according to claim 1, further characterized in that the synthesis means adjusts the non-visible light image according to the subject distance and then synthesizes it with the visible light image.

4. The aforementioned synthesis means synthesizes the invisible light image with the visible light image without adjusting it. The aforementioned image processing device The image processing apparatus according to claim 1, further comprising correction means for correcting the contrast of the composite image such that the difference in gradation information between the region in the composite image in which the invisible light image is composited and the region in which it is not composited is less than or equal to the first threshold.

5. The image processing apparatus according to claim 1, characterized in that the determination means determines a region in the visible light image in which the grayscale information is below a second threshold as a region for synthesizing the invisible light image.

6. The image processing apparatus according to claim 1, characterized in that the synthesis means synthesizes the invisible light image with respect to the luminance component of the visible light image.

7. The image processing apparatus according to claim 6, characterized in that the luminance component is the luminance component when the visible light image is represented in a uniform color space.

8. The image processing apparatus according to claim 7, characterized in that the uniform color space is the ICtCp color space or the L*a*b* color space.

9. Furthermore, the image processing apparatus according to claim 1 is characterized by having a correction means for correcting the color components of the visible light image.

10. The image processing apparatus according to claim 9, characterized in that the correction means corrects the color components based on the gradation information of the non-visible light image.

11. The image processing apparatus according to claim 9, characterized in that the correction means adjusts the amount of correction for the color component according to the pixel values ​​before and after the synthesis of the non-visible light image.

12. The image processing apparatus according to claim 9, characterized in that the correction means adjusts the amount of correction for the color component according to the exposure conditions when capturing the visible light image.

13. The image processing apparatus according to claim 9, characterized in that the correction means adjusts the amount of correction for the color component according to whether or not infrared auxiliary light is irradiated when capturing the visible light image.

14. The image processing apparatus according to claim 9, characterized in that the correction means adjusts the amount of correction of the color component according to the subject distance.

15. The image processing apparatus according to claim 9, characterized in that the correction means adjusts the difference in the amount of correction for regions consisting of similar colors included in the visible light image so that it is less than or equal to a third threshold.

16. The image processing apparatus according to claim 15, characterized in that the correction means performs the adjustment by smoothing the amount of correction for regions consisting of similar colors included in the visible light image.

17. The image processing apparatus according to claim 9, characterized in that the correction means adjusts the amount of correction for pixels of a red subject among the pixels of the visible light image based on the difference with the corresponding pixel value of the non-visible light image.

18. An imaging means having an image sensor and acquiring visible light images and invisible light images, An image processing apparatus according to any one of claims 1 to 17, which uses the visible light image and the non-visible light image acquired by the imaging means, An imaging device characterized by having the following features.

19. An image processing method performed by an image processing device, Determining the region of the visible light image from which the invisible light image will be synthesized, Based on the above determination, the invisible light image is combined with the visible light image to generate a composite image, The aforementioned synthesis is The invisible light image is adjusted such that the difference in gradation information between the region in the composite image where the invisible light image is composited and the region where it is not composited is less than or equal to a predetermined first threshold. An image processing method characterized by comprising: synthesizing the adjusted non-visible light image with the visible light image.

20. A program for causing a computer to function as one of the means of the image processing apparatus described in any one of claims 1 to 17.

21. An acquisition means for acquiring information for processing a visible light image corresponding to the invisible light image using a non-visible light image, The system includes a correction means for applying the processing to the visible light image with an intensity based on the aforementioned information, The correction means is characterized in that, in a visible light image to which the processing has been applied, if the difference in gradation information between a region where the intensity of the processing is 0 and a region where the intensity of the processing is not 0 is greater than a predetermined fourth threshold, the processing intensity is corrected to be less than or equal to the fourth threshold.

22. The image processing apparatus according to claim 21, characterized in that the acquisition means acquires the information from the luminance component of the non-visible light image and the luminance component of the visible light image.

23. The image processing apparatus according to claim 21, characterized in that the correction means corrects the intensity so that the intensity increases in accordance with the increase in subject distance.

24. The image processing apparatus according to claim 21, characterized in that the correction means applies processing to the luminance component of the visible light image.

25. The image processing apparatus according to claim 24, characterized in that the luminance component is the luminance component when the visible light image is represented in a uniform color space.

26. The image processing apparatus according to claim 25, characterized in that the uniform color space is the ICtCp color space or the L*a*b* color space.

27. The image processing apparatus according to claim 24, wherein the correction means further corrects the color components of the visible light image.

28. The image processing apparatus according to claim 27, characterized in that the correction means corrects the color components based on the intensity of the processing.

29. An imaging apparatus comprising: an imaging means having an image sensor for acquiring a visible light image and a non-visible light image; and an image processing apparatus according to any one of claims 21 to 28 that uses the visible light image and the non-visible light image acquired by the imaging means.

30. An image processing method performed by an image processing device, Using a non-visible light image, information is obtained for processing the non-visible light image and the corresponding visible light image. The process is applied to the visible light image with an intensity based on the aforementioned information, Applying the processing includes correcting the intensity of the processing so that, in a visible light image to which the processing has been applied, the difference in gradation information between a region where the intensity of the processing is 0 and a region where the intensity of the processing is not 0 is greater than a predetermined fourth threshold, the difference becomes less than or equal to the fourth threshold. An image processing method characterized by the following:

31. A program for causing a computer to function as one of the means of the image processing apparatus described in any one of claims 21 to 28.

Images