Image processing apparatus and image processing method, imaging apparatus

Abstract

To provide an image processing device capable of setting imaging conditions for non-visible light images that effectively improve the image quality of visible light images by combining them with the visible light images.SOLUTION: The image processing device determines imaging conditions for non-visible light images to be combined with visible light images. The image processing device determines a first target value for a signal level of the non-visible light images suitable for adjusting the brightness of the visible light images and a second target value for a signal level of the non-visible light images suitable for adjusting the contrast of the visible light images. The image processing device further determines a final target value using the first target value and the second target value and determines imaging conditions for the non-visible light images on the basis of the final target value.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus, an image processing method, and an imaging apparatus. [Background technology]

[0002] A method has been proposed to improve the visibility of a visible light image by combining a visible light image taken of a scene with reduced visibility due to fog with an infrared light image taken of the same scene (Patent Document 1). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2017-157902 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] Patent Document 1 improves the visibility of the visible light image in each region by increasing the proportion of invisible light images used in areas where visibility is poor in the visible light image. Furthermore, the imaging conditions (exposure conditions) for the invisible light image are set so that the brightness of the invisible light image and the visible light image become similar in areas where the proportion of invisible light images used is high.

[0005] However, in the method described in Patent Document 1, if the area with a high proportion of invisible light images is very dark, the imaging conditions are set such that the invisible light images also become very dark, and therefore the effect of improving visibility is not sufficiently obtained. The same is true if the area with a high proportion of invisible light images is very bright.

[0006] In view of the problems of the prior art, the present invention provides, in one embodiment, an image processing apparatus that can set imaging conditions for a non-visible light image that is synthesized with a visible light image to effectively improve the image quality of the visible light image. [Means for solving the problem]

[0007] The above objective is an image processing device that determines the imaging conditions for a non-visible light image to be synthesized with a visible light image, wherein the brightness of the visible light image A first target value for the signal level of an invisible light image, which is determined based on the relationship between the two values ​​and a predetermined threshold, and a second target value for the signal level of an invisible light image, which is determined based on information regarding the contrast of the invisible light image. Using The signal level of the invisible light image that determines the imaging conditions The final target value is determined by 1 The determination means and the method for determining imaging conditions based on the final target value. 2 Means of decision, and do Image processing apparatus characterized by [Effects of the Invention]

[0008] According to the present invention, it is possible to provide an image processing device that can set imaging conditions for a non-visible light image that is synthesized with a visible light image to effectively improve the image quality of the visible light image. [Brief explanation of the drawing]

[0009] [Figure 1] Block diagram showing an example of the functional configuration of an imaging device as an image processing apparatus according to the embodiment. [Figure 2] Flowchart relating to the operation of the imaging device in the first embodiment [Figure 3] Flowchart relating to the operation of the imaging device in the first embodiment [Figure 4] Schematic diagrams showing examples of visible light and infrared light images. [Figure 5] Figure showing an example of block division in the first embodiment. [Figure 6] Diagram illustrating the method for determining the first target value in the first embodiment. [Figure 7] Diagram illustrating the method for determining the second target value in the first embodiment. [Figure 8] Diagram relating to the weights used to determine the third target value in the first embodiment. [Figure 9] Flowchart relating to the operation of the imaging device in the second embodiment [Figure 10] Flowchart relating to the operation of the imaging device in the second embodiment [Figure 11] Diagram showing the method for determining the gain amount in the second embodiment. [Figure 12] Figure related to the method for determining the gain amount in the second embodiment [Figure 13] Flowchart related to the operation of the imaging device in the third embodiment [Figure 14] Figure related to the method for determining the imaging conditions and gain in the third embodiment

Modes for Carrying Out the Invention

[0010] Hereinafter, the present invention will be described in detail based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Also, although a plurality of features are described in the embodiments, not all of them are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0011] In the following embodiments, the case where the present invention is implemented in an imaging device such as a digital camera will be described. However, the present invention can also be implemented in an electronic device having an imaging function and a control device capable of controlling such an electronic device. Such electronic devices and control devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game machines, robots, drones, and drive recorders. These are examples, and the present invention can also be implemented in other electronic devices and control devices.

[0012] Note that the configurations represented as blocks in the figures can be realized by an integrated circuit (IC) such as an ASIC or FPGA, by discrete circuits, or by a combination of a memory and a processor that executes a program stored in the memory. Also, one block may be realized by a plurality of integrated circuit packages, or a plurality of blocks may be realized by one integrated circuit package. Further, the same block may be implemented in different configurations according to the operating environment, required capabilities, etc.

[0013] Figure 1 is a block diagram showing an example of the functional configuration of an imaging device 100 as an example of an image processing apparatus according to the present invention. The control unit 101 is a processor capable of executing programs, such as a CPU. The control unit 101 controls the operation of each functional block of the imaging device 100 and realizes the functions of the imaging device 100 by, for example, reading a program stored in ROM 102 into RAM 103 and executing it. Note that if the optical system 104 is a replaceable lens unit, the control unit 101 controls the optical system 1 04 The operation of the optical system 1 is controlled through communication with the controller it possesses.

[0014] ROM102 is a rewritable non-volatile memory. ROM102 stores programs executed by the control unit 101, various settings for the imaging device 100, GUI data, etc. RAM103 is the main memory of the control unit 101. RAM103 loads programs executed by the control unit 101, holds parameters necessary for program execution, and is used as a working memory for the image processing unit 107. Rito It is used in this way. In addition, a portion of the RAM 103 is used as video memory to store image data to be displayed on the display unit 109.

[0015] The optical system 104 includes an imaging optical system consisting of a lens group including a movable lens (zoom lens, focus lens, etc.) and a drive circuit for the movable lens. The optical system 104 may also include an aperture and its drive circuit.

[0016] The imaging unit 105 may be, for example, a known CCD or CMOS color image sensor (image sensor) having a primary color Bayer array color filter. The image sensor has a pixel array in which multiple pixels are arranged in two dimensions, and peripheral circuits for reading signals from the pixels. Each pixel has a photoelectric conversion element such as a photodiode and accumulates charge according to the amount of incident light during the exposure period. By reading signals having a voltage corresponding to the amount of charge accumulated during the exposure period from each pixel, a group of pixel signals (analog image signals) representing the subject image formed on the imaging surface by the imaging optical system is obtained.

[0017] In this embodiment, the imaging unit 105 is assumed to have an image sensor capable of capturing both visible light images and invisible light images. Such an image sensor may, for example, have some of the pixels in a pixel array designated as pixels for capturing invisible light images. The pixels for capturing invisible light images may have an optical filter that transmits the wavelength band of invisible light and blocks the wavelength band of visible light.

[0018] For example, in an image sensor equipped with a primary color Bayer array color filter, one of the two green (G) filters in the repeating unit of the color filter (G pixel) can be replaced with a pixel used to capture a non-visible light image. In this case, the value of the G pixel that would normally be present at the position of the non-visible light image capturing pixel can be interpolated using the values ​​of other pixels, similar to the value of a defective pixel, to generate a visible light image. Furthermore, the non-visible light image can be enlarged based on the signal of the non-visible light image capturing pixel to achieve the same resolution (number of pixels) as the visible light image.

[0019] There are no particular restrictions on the method of acquiring visible light images and invisible light images, and they may be acquired by other methods. For example, a separate image sensor may be provided for capturing visible light images and a separate image sensor for capturing invisible light images. In this embodiment, the invisible light image is assumed to be an infrared light image, but other methods may be used. Non Images in the visible wavelength range are also acceptable.

[0020] The A / D conversion unit 106 converts the analog image signal read from the imaging unit 105 into a digital image signal. The A / D conversion unit 106 writes the digital image signal to the RAM 103.

[0021] The image processing unit 107 applies predetermined image processing to the digital image signal stored in the RAM 103 to generate signals and image data according to the application, and to acquire and / or generate various types of information. The image processing unit 107 may be a dedicated hardware circuit such as an ASIC designed to realize a specific function, or it may be a configuration in which a programmable processor such as a DSP executes software to realize a specific function.

[0022] The image processing applied by the image processing unit 107 includes preprocessing, color interpolation, correction, detection, data processing, evaluation value calculation, and special effects processing. Preprocessing includes signal amplification, reference level adjustment, and defective pixel correction. Color interpolation is a process that interpolates the values ​​of color components that could not be obtained during shooting, and is also called demosaicing.

[0023] The correction process includes white balance adjustment, gradation correction, correction of image degradation caused by optical aberrations in the imaging optical system 101 (image recovery), correction of the effect of vignetting in the imaging optical system 101, and color correction. In addition, the infrared light image synthesis process, which is intended to improve the visibility of the visible light image and will be described later, is also included in the correction process. Detection processes include detecting feature regions (such as face regions or human body regions) and their movements, as well as recognizing people. Data processing includes processes such as synthesis, scaling, encoding and decoding, and header information generation (data file generation).

[0024] The evaluation value calculation process includes generating signals and evaluation values ​​used for autofocus detection (AF), and generating evaluation values ​​used for automatic exposure control (AE). Furthermore, the generation of evaluation values ​​for determining the imaging conditions for infrared light images, as described later, is also included in this process. Special effects processing includes adding blur effects, changing color tones, and relighting. These are merely examples of processes that the image processing unit 107 can apply, and do not limit the processes that the image processing unit 107 can apply.

[0025] The recording unit 108 records data to a recording medium such as a memory card, and reads data recorded on the recording medium. The recording medium does not have to be removable. The recording medium may also be a communication-enabled external storage device.

[0026] The display unit 109 is, for example, a liquid crystal display and displays captured images, images read by the recording unit 108, information from the imaging device 100, a GUI such as a menu screen, etc. By continuously performing video recording and displaying the recorded video on the display unit 109, the display unit 109 can be made to function as an electronic viewfinder (EVF). The display unit 109 may also be a touch display.

[0027] The operation unit 110 is a general term for input devices (buttons, switches, dials, etc.) provided for the user to input instructions to the imaging device 100. The input devices constituting the operation unit 110 have names according to the function they are assigned to. For example, the operation unit 110 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, directional keys, and a select key. The release switch is a switch for recording still images, and the control unit 101 recognizes a half-pressed state of the release switch as a shooting preparation instruction and a fully pressed state as a shooting start instruction. Also, the control unit 101 recognizes the video recording switch as a video recording start instruction when pressed in shooting standby mode, and as a recording stop instruction when pressed during video recording. The functions assigned to the same input device may be variable. Furthermore, the input devices may be software buttons or keys using a touch display.

[0028] Figure 2 is a flowchart of the enhancement process performed by the imaging device 100 to improve the image quality of visible light images by combining infrared light images. Here, it is assumed that the enhancement process shown in Figure 2 is performed in the shooting standby state. Generally, the shooting standby state is a state in which the display unit 109 functions as an EVF and the device waits for input of a command to shoot a still image or video.

[0029] In step S203, the control unit 101 controls the imaging unit 105 to sequentially capture visible light images and infrared light images. The initial imaging conditions are predetermined. The imaging unit 105 sequentially captures visible light images and infrared light images. The analog image signals of the visible light and infrared light images obtained through imaging are converted into digital image signals via the A / D conversion unit 106 and stored in the RAM 103.

[0030] The image processing unit 107 applies interpolation and scaling processes to the digital image signal as needed to supplement missing pixel data. Then, the RAM 103 stores the digital signals of the visible light image (hereinafter referred to as visible light image data) and the digital signals of the infrared light image (hereinafter referred to as infrared light image data) of the same resolution.

[0031] Furthermore, the imaging in S203 may also serve as the imaging for the video captured for live view display, which is performed in the shooting standby state. For example, video may be captured at 60 frames / second, and visible light images and infrared light images may be read alternately. In this case, the visible light image data is also used to generate image data for live view display. In addition, the imaging conditions may differ between the frames in which visible light images are acquired and the frames in which infrared light images are acquired. When acquiring visible light images and infrared light images as video frames, it is sufficient to acquire one frame at a time when executing S203, and it is not necessary to execute S203 in accordance with the video imaging frame rate.

[0032] In step S205, the control unit 101 updates the imaging conditions for the infrared image, which are used to improve the image quality or visibility of the visible light image, based on the visible light image data and the infrared image data. Details of the processing in step S205 will be described later. The imaging conditions are parameters related to the amount of exposure during shooting, such as shutter speed (exposure period), aperture value, and ISO sensitivity.

[0033] Furthermore, the control unit 101 updates the imaging conditions for the visible light image based, for example, on a known automatic exposure control (AE) method.

[0034] In S207, the control unit 101 determines whether or not a shooting start instruction has been input through the operation unit 110. Here, the determination is made regarding a still image shooting start instruction. If the control unit 101 determines that a shooting start instruction has been input, it executes S209. If the control unit 101 does not determine that a shooting start instruction has been input, it repeatedly executes S203 and S205.

[0035] In S209, the control unit 101 controls the imaging unit 105 to capture a visible light image and an infrared light image for recording, according to the imaging conditions updated in the most recent execution in S205. Generally, the image for recording has a higher resolution than the image for live view display.

[0036] In S211, the control unit 101 controls the image processing unit 107 to apply an enhancement process to the visible light image data obtained in S209, which involves combining infrared light image data. The enhancement process is a process that improves the image quality or visibility of the visible light image. The region to which the enhancement process is applied and the method of combining the infrared light image can be determined, for example, in S205.

[0037] Furthermore, infrared images can be combined with visible light images by combining all or part of their frequency components. For example, in areas where the visibility of a subject is reduced due to fog or haze, combining only the high-frequency (AC) components of the infrared image can enhance the edges and improve the visibility of the subject. When combining only the AC components, the AC components can be simply added together, but the level (gain) of the AC components may also be adjusted before adding them to minimize the noticeable change in brightness caused by the combination.

[0038] On the other hand, in areas that are too dark or too bright, reducing the visibility of the subject, edge enhancement has little effect in improving the visibility of the subject within that area. Therefore, by synthesizing all frequency components of the infrared light image, including the low-frequency (DC) component, the visibility of the subject within that area can be further improved. In this way, by synthesizing infrared light images according to the state of the visible light image region into which the infrared light image is synthesized, the visibility of the subject can be improved more appropriately.

[0039] The image processing unit 107, in accordance with the control from the control unit 101, combines infrared light image data with visible light image data to generate composite image data.

[0040] In S213, the image processing unit 107 applies image processing to the composite image data to generate image data for recording. The image processing applied here may vary depending on the recording format. For example, when recording in JPEG or HIEF format, the image processing includes color interpolation and encoding.

[0041] Image data for recording is recorded as an image data file on the recording medium by the recording unit 108.

[0042] Next, we will further explain the imaging condition update process in S205 using the flowchart shown in Figure 3. In S302, the control unit 101 instructs the image processing unit 107 to generate gradation information for visible light image data and infrared light image data. At this stage, the visible light image data is in a state where each individual pixel data constituting it has a value for one color component corresponding to the color of the color filter provided on the pixel. Such image data is referred to as RAW data in this specification.

[0043] An example of a method for generating grayscale information is described below. Here, we assume that an evaluation value of edge intensity is generated as grayscale information. As shown in Figure 5, the image processing unit 107 divides the image 501 into M horizontal divisions and N vertical divisions to generate M × N blocks 502 (M and N are predetermined integers of 2 or more). The image processing unit 107 then generates an evaluation value for each block.

[0044] Specifically, the image processing unit 107 extracts edge components for each block using known methods, such as applying a spatial filter to one of the RGB components (in this case, the G component). For pixels where the G component is not available, the G component may be obtained by interpolation before the edge components are extracted. The image processing unit 107 then generates an evaluation value for the block based on the integral value of the extracted edge components. A large evaluation value indicates that the image within the block contains many edge components.

[0045] For example, let e(i,j) be the edge strength at coordinates (i,j) within a block (where i,j are integers and 0 ≤ i,j ≤ the maximum value determined by the block size). In this case, the evaluation value E of the block can be generated by the following equation (1). E = Σ i Σ j e (i, j) ···(1) Note that the edge strength evaluation value E may be determined by other methods.

[0046] In S303, the control unit 101 (region determination means) determines, based on the evaluation values ​​generated by the image processing unit 107, the regions in the visible light image where visibility should be improved (regions to be enhanced), using the aforementioned blocks as units.

[0047] For example, the visible light image schematically shown in 401 of Figure 4 shows a reduction in contrast of distant subjects due to fog or haze. The infrared light image of the same scene is schematically shown in 402 of Figure 4. Since infrared images are less affected by the reduction in contrast due to fog or haze, the visibility of distant subjects can be improved by combining the distant portion of the infrared light image with the visible light image. Therefore, in S303, the control unit 101 controls the visible light image of Figure 4. light Areas like the distant view in image 401 are designated as the areas to be enhanced.

[0048] Blocks with low image contrast have fewer edge components, resulting in a lower edge strength evaluation value. Similarly, areas that are completely black (shown schematically as 411 and 412 in Figure 4) and areas that are overexposed (not shown) also have low contrast, leading to a lower edge strength evaluation value.

[0049] The control unit 101 compares the evaluation values ​​generated for the visible light image data and the infrared light image data for each corresponding block. The control unit 101 then determines blocks in which the edge evaluation value of the visible light image data is small (e.g., below the first threshold) and the difference between it and the edge evaluation value of the infrared light image data is large (e.g., above the second threshold) as the target area for enhancement processing. The first and second thresholds can be predetermined, for example, experimentally.

[0050] Note that the area to be enhanced may be determined by other methods. For example, reduced visibility of subjects due to fog or haze is more likely to occur in distant views. Therefore, the area where the subject distance is greater than or equal to a distance threshold may be determined as the area to be enhanced.

[0051] Alternatively, the user may be presented with a visible light image and allowed to select the area to be enhanced. Furthermore, two or more of the methods exemplified here may be combined to determine the area to be enhanced.

[0052] The control unit 101 stores information identifying the target area for the determined enhancement process in the RAM 103 so that it can be used during the synthesis process. If the enhancement process is performed after recording the visible light image data, the infrared light image data used for the enhancement process and the information identifying the target area for the enhancement process may be recorded in association with the visible light image data in the recording unit 108.

[0053] In S304, the control unit 101 determines a signal level target value for determining the imaging conditions of the infrared image used for enhancement processing. In this embodiment, the final target value T is determined from a target value L (first target value) based on the brightness of the visible light image and a target value C (second target value) based on the contrast of the infrared image. The signal level of the infrared image may be, for example, the average value of pixels included in the region used for enhancement processing.

[0054] Figure 6 shows an example of a method for determining the target value L based on the brightness of a visible light image. The target value L (first target value) is the target value for obtaining an infrared light image suitable for adjusting the brightness (signal level) in the area to be enhanced. The horizontal axis shows the brightness evaluation value of the area to be enhanced in the visible light image, for example, the average luminance value of the area to be enhanced. The average luminance value can be obtained, for example, as the value of the Y component when the average values ​​of the R pixels, B pixels, and G pixels in the area to be enhanced are converted to YCbCr format.

[0055] As shown in Figure 6, the control unit 101 (first determination means) linearly increases the target signal level L of the region used for enhancement processing in the infrared light image as the brightness evaluation value of the visible light image increases, in the interval where the brightness evaluation value is above a first threshold and below a second threshold. Figure 6 shows an example in which, in the interval where the brightness evaluation value is above a first threshold Th1 and below a second threshold Th2, the target signal level L of the region used for enhancement processing in the infrared light image is determined to be equal to the brightness evaluation value of the visible light image. Here, the signal level of the region used for enhancement processing in the infrared light image may be the average value of the pixels in that region, and corresponds to the brightness evaluation value of the visible light image.

[0056] On the other hand, for intervals where the brightness evaluation value is less than the first threshold Th1 (less than the first threshold), the control unit 101 fixes the signal level target value L of the region of the infrared light image used for enhancement processing to the signal level target value L1 when the brightness evaluation value is the first threshold Th1. In this way, by preventing the signal level target value L of the infrared light image from falling below a certain value, it becomes possible to set imaging conditions that enable the acquisition of an infrared light image in which the visibility of the subject in areas that would be blacked out in the visible light image is improved.

[0057] Similarly, for intervals where the brightness evaluation value is greater than the second threshold Th2, the control unit 101 fixes the signal level target value L of the region of the infrared light image used for enhancement processing to the signal level target value L2 when the brightness evaluation value is the second threshold Th2. In this way, by preventing the signal level target value L of the infrared light image from rising above a certain value, it becomes possible to set imaging conditions that enable the acquisition of an infrared light image that improves the visibility of subjects in areas that would be overexposed in the visible light image. The first threshold Th1 and the second threshold Th2 of the brightness evaluation value are thresholds for distinguishing between underexposed and overexposed areas, and can be determined in advance, for example, experimentally.

[0058] Next, we will explain how to determine the target value C (second target value) based on the contrast of the infrared image. Figure 7, 701, is an example of a histogram of pixel values ​​(signal levels) included in the region of the infrared image used for enhancement processing.

[0059] As mentioned above, in the area targeted for enhancement processing, the edge intensity evaluation value of the infrared image is higher than that of the visible light image. This corresponds to the infrared image having higher contrast than the visible light image. Therefore, further increasing the contrast of the infrared image is not essential. However, since the difference in evaluation values ​​is a difference in relative contrast, the infrared image may not necessarily have sufficient contrast in terms of improving the image quality or visibility of the subject in the visible light image. It is also possible that the contrast of the infrared image is too high.

[0060] Therefore, in this embodiment, an appropriate contrast value, in terms of improving the image quality of the visible light image or the visibility of the subject, is experimentally determined in advance as the contrast target value grad. The control unit 101 (second determination means) then determines the infrared signal level target value C such that the contrast of the infrared light image becomes the target value grad. The signal level target value C (second target value) is a target value for obtaining an infrared light image suitable for adjusting the contrast in the area to be enhanced.

[0061] The control unit 101 first determines the correction amount k using the following equation (2). k = grad / (H max - H min )···(2) Here, grad: Target contrast value H max : The maximum signal level within the region used for enhancement processing of the infrared image obtained under the current imaging conditions. H min : The minimum signal level within the region used for enhancement processing of the infrared image obtained under the current imaging conditions. Therefore, the correction amount k corresponds to the magnification required to bring the current contrast to the target value.

[0062] Next, the control unit 101 uses the correction amount k to determine the signal level target value C based on the contrast of the infrared light image using the following equation (3). C = k * H ave ...(3) Here, H ave : The average signal level or median signal level within the region used for enhancement processing of the infrared image obtained under the current imaging conditions. That is the case.

[0063] The control unit 101 (third determination means) weights and adds a first target value L based on the brightness evaluation value of the visible light image and a second target value C based on the contrast of the infrared light image to determine the final signal level target value T of the infrared light image using the following equation (4). T = α × L + (1 - α) × C ... (4) α: Weight of the target value L (0 ≤ α ≤ 1)

[0064] Figure 8 shows an example of the relationship between the weight α of the target value L used in equation (4) and the brightness evaluation value of the area targeted for enhancement processing of the visible light image. When enhancement processing is applied to blocks where the brightness evaluation value of the visible light image is less than the first threshold Th1 or greater than the second threshold Th2, improvement of brightness is prioritized over contrast of the visible light image. Therefore, the weight of the signal level target value L should be greater than 0.5.

[0065] On the other hand, when applying enhancement processing to blocks where the brightness evaluation value of the visible light image is between the first threshold Th1 and the second threshold Th2, contrast improvement takes precedence over brightness improvement of the visible light image. Therefore, the weight α of the signal level target value L is made smaller than the weight (1-α) of the signal level target value C (α<0.5). Note that the weight α of the signal level target value L for blocks where contrast improvement is prioritized only needs to be smaller than the weight α of the signal level target value L for blocks where signal level improvement is prioritized, and does not necessarily have to be less than 0.5.

[0066] Finally, in S305, the control unit 101 (fourth determination means) determines (updates) the imaging conditions for the infrared image based on the signal level target value T determined in S304. The imaging conditions that achieve the signal level target value T can be determined by referring to a predetermined program diagram and using a combination of shutter speed Tv and aperture value Av when the signal level target value T is the exposure amount Ev value. Since it is desirable for the depth of field of the infrared image and the visible light image to match, the control unit 101 can prioritize using the same aperture value as the imaging conditions for the visible light image when determining the imaging conditions for the infrared image.

[0067] The imaging device may have different sensitivities to visible light and infrared light. Therefore, when using a program diagram to determine the imaging conditions for an infrared light image, by using the program diagram for the infrared light image, the signal level of the infrared light image can be accurately controlled.

[0068] Alternatively, without using a program diagram, the correction amount (exposure adjustment amount) of the current imaging conditions may be obtained. The control unit 101 obtains the exposure adjustment amount EV for realizing the signal level target value T, for example, by the following formula (5). EV = log2(T / H ave )···(5) As described above, H ave is the average signal level or the median value of the signal level within the region used for enhancement processing of the infrared light image obtained under the current imaging conditions.

[0069] The exposure adjustment amount EV is a value indicating the difference between the exposure amount for realizing the signal level target value T and the exposure amount under the current imaging conditions in terms of steps. For example, when EV = 1, it indicates that it is necessary to adjust the exposure to be one step brighter from the current imaging conditions. Therefore, the control unit 101 can update the current imaging conditions of the infrared light image by any of the following: setting the shutter speed to 1 / 2 (doubling the exposure time), setting the aperture value to 1 / 2, or doubling the ISO sensitivity. In this case as well, the control unit 101 can preferentially update the imaging conditions of the infrared light image using the same aperture value as the imaging conditions of the visible light image. If the light amount is insufficient, an auxiliary light source such as an infrared light flash may be used.

[0070] For the sake of clarity and ease of understanding, this explanation describes the case where there is only one target region for enhancement processing (which may include multiple adjacent blocks). However, it is also possible to have multiple target regions for enhancement processing. In this case, multiple imaging conditions may be determined based on the target value T for each target region, and an infrared image may be captured for one frame at a time under each imaging condition. In this case, pixel data of the region used for enhancement processing can be extracted from each frame and used for the synthesis process. Alternatively, one imaging condition may be determined based on a representative or average value of multiple target values ​​T, and an infrared image of one frame may be captured.

[0071] According to this embodiment, the imaging conditions for the invisible light image to be synthesized into the visible light image are determined by considering both the viewpoint of improving the signal level and the viewpoint of improving contrast. As a result, it becomes possible to capture an invisible light image suitable for the region of the visible light image into which the invisible light image is synthesized, thereby effectively improving the visibility of the subject and the image quality in the visible light image.

[0072] ●(Second Embodiment) Next, a second embodiment of the present invention will be described. In the first embodiment, a method for determining the imaging conditions for an infrared image was described when it is possible to capture both a visible light image and an infrared image under separate imaging conditions. This embodiment will describe the operation when a visible light image and an infrared image are captured under the same imaging conditions.

[0073] Since this embodiment can also be implemented using the imaging device 100, the explanation will be given using the components of the imaging device 100. Figure 9 is a flowchart of the process performed by the imaging device 100 in the second embodiment to improve the image quality of a visible light image by synthesizing infrared light images. Steps that perform the same operations as in the first embodiment are denoted by the same reference numerals as in Figure 2.

[0074] In S903, the control unit 101 controls the imaging unit 105 to capture a visible light image and an infrared light image. The initial imaging conditions are predetermined. The imaging unit 105 takes one image using imaging conditions suitable for a visible light image and reads out the analog image signal of the visible light image and the analog image signal of the infrared light image from the image sensor.

[0075] Subsequently, in the same manner as in the first embodiment, the RAM 103 stores a digital signal of a visible light image with the same resolution (hereinafter referred to as visible light image data) and a digital signal of an infrared light image (hereinafter referred to as infrared light image data).

[0076] In step S905, the control unit 101 determines the target signal level value for the infrared light image data. Details of the processing in S905 will be described later. The control unit 101 also updates the imaging conditions for the visible light image, for example, based on a known automatic exposure control (AE) method.

[0077] In S207, the control unit 101 determines whether or not a shooting start instruction has been input through the operation unit 110. Here, the determination is made regarding a still image shooting start instruction. If the control unit 101 determines that a shooting start instruction has been input, it executes S909. If the control unit 101 does not determine that a shooting start instruction has been input, it repeatedly executes S903 and S905.

[0078] In S909, the control unit 101 controls the imaging unit 105 to capture a visible light image and an infrared light image for recording, according to the visible light image acquisition conditions updated in the most recent execution, S905. In the same manner as in S903, the visible light image data and infrared light image data of the same resolution are stored in the RAM 103.

[0079] In S911, the control unit 101 adjusts the signal level of the infrared light image data obtained in S909 based on the target value determined in S905. Details of the processing in S911 will be described later. Since the process from S211 onwards is the same as in the first embodiment, the explanation will be omitted.

[0080] Figure 10 is a flowchart detailing the operation for determining the signal level target value in S905. The same reference numerals as in Figure 3, which describes the operation in the first embodiment, are used for the steps that perform the same operation as in Figure 3. Thus, in this embodiment as well, the operation for determining the signal level target value T of the infrared light image is the same as in the first embodiment. Unlike the first embodiment, in this embodiment, the imaging conditions for the infrared light image cannot be set individually, so the imaging conditions for the infrared light image are not updated.

[0081] Next, we will explain the signal level adjustment operation of infrared optical image data in S911. The control unit 101 determines the gain amount required to set the signal level of the infrared light image data to the target value T determined in S905 using the following equation (6). Loss = T / H ave ...(6) H ave This is the average signal level or median signal level within the region used for enhancement processing of the infrared image obtained under the current imaging conditions.

[0082] Next, the control unit 101 controls the image processing unit 107 and adjusts the signal level of the infrared light image data by applying the obtained gain amount to the individual pixel data included in the region of the infrared light image data used for enhancement processing.

[0083] Here, if we let input(x, y) be the pixel data to which the gain amount gain is applied, and output(x, y) be the pixel data after the gain amount has been applied, then the signal level adjustment is: output(x, y) = gain × input(x, y) (7) It can be expressed as follows. Note that (x, y) are the pixel coordinates in the image coordinate system.

[0084] When there are multiple areas to be enhanced, and these areas have different target values ​​T, a step in the signal level occurs at the boundary between the areas due to the difference in target values ​​T. Therefore, the gain amount should be adjusted so that it changes smoothly in the blocks at the boundary of areas with different target values ​​T.

[0085] Figure 11 schematically shows the infrared image 1101, the six target regions determined for the infrared image 1101, and the target values ​​T1 to T6 determined for each target region. Each target region contains one or more adjacent blocks 1102.

[0086] Here, we focus on region 1111, which contains two blocks located at the boundary of target regions with different target values. The upper block in region 1111 has a target value T2, and the lower block... This is the target value T1.

[0087] In this case, to prevent the gain amount gain from changing discontinuously at the block boundary, the control unit 101 sets the target value T of the adjacent point 1112 of the block. α This can be calculated using the following equation (8). T α = L2 / (L1+L2) × α × T1+ L1 / (L1+ L2) × T2 (8) Here, L1 and L2 are the distances from the center of the adjacent block to adjacent point 1112, as shown in Figure 11. Since adjacent point 1112 is the midpoint of the line connecting the centers of the adjacent blocks, L1 = L2, and T α = T1 / 2 + T2 / 2.

[0088] For pixels within region 1113, which lies between the center of the upper block and the center of the lower block, the target value is determined according to equation (8) based on the y-coordinate of the pixel. This causes the target value to change continuously from T1 to T2 in the y-direction, and the gain amount gain based on the target value also changes continuously.

[0089] In this way, in regions where target regions with different target values ​​are adjacent, the target value can be weighted and added according to the distance from the center of the adjacent block, thereby avoiding discontinuous changes in the gain amount.

[0090] Furthermore, if target regions with different target values ​​are adjacent, the weight of the target value of one of the target regions may be increased. For example, the weight of the target value of a target region with a large area (number of blocks) and / or a large difference in grayscale information can be increased. Figure 12 shows the information of the target region related to region 1111 shown in Figure 11.

[0091] In this case, the area of ​​the target region with target value T1 is larger in area (number of blocks) than the area of ​​the target region with target value T2, and the difference in grayscale information is also larger. In this case, in equation (8), the weight of target value T1 is made larger than that of target value T2. α We seek.

[0092] According to this embodiment, the signal level of a non-visible light image captured under the same imaging conditions as a visible light image is adjusted according to a target value determined considering both the improvement of the signal level and the improvement of contrast. As a result, it becomes possible to obtain a non-visible light image suitable for the region of the visible light image into which the non-visible light image is synthesized, thereby effectively improving the visibility and image quality of the subject in the visible light image.

[0093] ●(Third embodiment) Next, the present invention 3 An embodiment will now be described. This embodiment relates to operation when there are limitations on the imaging conditions for the infrared light image to be synthesized. Since this embodiment can also be implemented with the imaging device 100, it will be described using the components of the imaging device 100.

[0094] Figure 13 is a flowchart of the process performed by the imaging device 100 in the third embodiment to improve the image quality of a visible light image by synthesizing infrared light images. Steps that perform the same operations as in the first and second embodiments are denoted by the same reference numerals as in Figure 2 or Figure 9.

[0095] Here, a lower limit is set on the shutter speed as a restriction on the imaging conditions used to capture infrared light images for synthesis, which are performed in response to the instruction to start shooting. This restriction may be, for example, intended to prevent image blur. In addition, if a minimum aperture value or a maximum ISO sensitivity is set, the same operation can be performed as when a lower limit is set on the shutter speed.

[0096] In this embodiment, similar to the second embodiment, the infrared light image signal level target value T is determined before determining in S207 whether or not a still image capture instruction has been input, but the imaging conditions are not determined.

[0097] In S207, the control unit 101 determines whether or not a shooting start instruction has been input through the operation unit 110. Here, the determination is made regarding a still image shooting start instruction. If the control unit 101 determines that a shooting start instruction has been input, it executes S1309. If the control unit 101 does not determine that a shooting start instruction has been input, it executes S1313.

[0098] In S1309, the control unit 101 determines the imaging conditions used for capturing the infrared light image for synthesis, taking limitations into consideration. Furthermore, if necessary, the control unit 101 determines the gain amount for adjusting the signal level of the infrared light image. The details of the operation in S1309 are described below.

[0099] Figure 14 schematically shows an example of the operation for determining imaging conditions in S1309. Here, it is assumed that the lower limit of the shutter speed (the slowest shutter speed) among the imaging condition parameters is limited to 1 / 30 second. It is also assumed that the shutter speed of the imaging condition in the pre-shoot performed in the most recent execution S203 was 1 / 125 second.

[0100] First, the control unit 101 determines the exposure adjustment amount EV based on the signal level target value T determined in the most recent execution S905, as described in the first embodiment. Here, let's assume EV = 3. To achieve EV = 3 solely by adjusting the shutter speed, it needs to be 1 / 15 second. However, the lower limit of the shutter speed is restricted to 1 / 30 second.

[0101] Therefore, the control unit 101 sets the shutter speed to 1 / 30 second in the imaging conditions. The control unit 101 also determines the gain amount (×2) to compensate for the one-stop exposure deficit after imaging. If the imaging conditions corresponding to the target amount cannot be achieved by shutter speed alone, one or more other parameters of the imaging conditions (aperture value and ISO sensitivity) may be changed, or an auxiliary light source emitting infrared light may be used. Here again, priority is given to making the aperture value as close as possible to the imaging conditions for visible light images.

[0102] Meanwhile, in S1313, the control unit 101 determines (updates) the imaging conditions for the infrared light image in the same manner as described in S305 using Figure 3 in the first embodiment. The imaging conditions determined here are not used for imaging the infrared light image for synthesis, so there are no particular restrictions. For example, in the example shown in Figure 14, the control unit 101 changes the shutter speed to 1 / 15 second. After that, the control unit 101 executes again from S203.

[0103] In S209, the control unit 101 controls the imaging unit 105 to capture a visible light image and an infrared light image for recording, according to the imaging conditions for the visible light image determined in the most recent execution, S905, and the imaging conditions determined in S1309.

[0104] In S1311, the control unit 101 instructs the image processing unit 107 to apply the gain amount to the infrared light image data if the gain amount has been determined in S1309. The image processing unit 107 applies the gain amount to the infrared light image data stored in RAM 103. Note that if the gain amount has not been determined in S1309, S1311 does not need to be executed. From S211 onward, the operation is the same as in the first embodiment, except that the infrared light image data to which the gain amount has been applied is used, so the explanation is omitted.

[0105] This section has described cases where there are restrictions on the imaging conditions for infrared light images. However, from the perspective of preventing image blur, the imaging conditions for visible light images may also be restricted. In this case, when determining the imaging conditions for visible light images, the gain amount can be determined in addition to the imaging conditions, and the gain amount can be applied to the visible light image data in S1311.

[0106] In this embodiment, for synthesis Non If there are limitations on the imaging conditions for visible light images, and it is not possible to determine imaging conditions that correspond to the target signal level within the limits, the gain to compensate for the insufficient exposure is applied after imaging. Non It was made applicable to visible light image data. Therefore, in addition to the same effects as in the first embodiment, for example, Non This makes it possible to suppress image blur in visible light images.

[0107] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0108] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0109] 100...Imaging device, 101...Control unit, 102...ROM, 103...RAM, 104...Optical system, 105...Imaging unit, 106...A / D conversion unit, 107...Image processing unit, 108...Recording unit, 109...Display unit

Claims

1. An image processing device for determining the imaging conditions of a non-visible light image to be combined with a visible light image, A first determination means for determining the final target value of the signal level of the invisible light image for determining the imaging conditions, using a first target value of the signal level of the invisible light image, which is determined based on the relationship between the brightness of the visible light image and a predetermined threshold, and a second target value of the signal level of the invisible light image, which is determined based on information regarding the contrast of the invisible light image. An image processing apparatus characterized by having a second determination means for determining the imaging conditions based on the final target value.

2. The image processing apparatus according to claim 1, further comprising a synthesis means for synthesizing a non-visible light image captured according to the imaging conditions determined by the second determination means with the visible light image.

3. The image processing apparatus according to claim 1 or 2, further comprising a region determination means for determining a target region from the visible light image to which a non-visible light image captured according to the imaging conditions is to be synthesized.

4. The image processing apparatus according to claim 3, characterized in that the region determination means determines the target region based on the grayscale information of the visible light image and the grayscale information of a non-visible light image captured together with the visible light image.

5. The region determination means determines the target region using blocks obtained by dividing the visible light image as units, The image processing apparatus according to claim 4, characterized in that the grayscale information is an evaluation value of the edge strength for each block.

6. The image processing apparatus according to claim 5, characterized in that the region determination means determines as the target region a block among the blocks obtained by dividing the visible light image, the block whose evaluation value is less than or equal to a first threshold, and the difference between the evaluation value of the corresponding block in the non-visible light image and the evaluation value of the corresponding block is greater than or equal to a second threshold.

7. The image processing apparatus according to claim 5 or 6, characterized in that the region determination means determines, among the blocks obtained by dividing the visible light image, a block in which the subject distance is equal to or greater than a distance threshold as the target region.

8. The image processing apparatus according to any one of claims 5 to 7, characterized in that the region determination means determines a block selected by the user from among the blocks obtained by dividing the visible light image as the target region.

9. The image processing apparatus according to any one of claims 1 to 6, characterized in that the first target value is fixed to a first value when the brightness of the visible light image is less than a third threshold, and fixed to a second value greater than the first value when it is greater than a fourth threshold.

10. The image processing apparatus according to claim 9, characterized in that the first target value has a value corresponding to the brightness of the visible light image when the brightness of the visible light image is greater than or equal to the third threshold and less than or equal to the fourth threshold.

11. The image processing apparatus according to any one of claims 1 to 10, characterized in that the second target value is determined based on the contrast of a non-visible light image captured together with the visible light image.

12. The image processing apparatus according to claim 11, characterized in that the second target value is determined as a target value of the signal level for obtaining an invisible light image having a predetermined contrast.

13. The image processing apparatus according to any one of claims 1 to 12, characterized in that the first determination means determines the final target value by weighting and adding the first target value and the second target value.

14. The image processing apparatus according to claim 13, wherein the first determination means determines the weights of the first target value and the second target value based on the brightness of the region in the visible light image where the invisible light image captured according to the imaging conditions is synthesized.

15. The image processing apparatus according to claim 1, characterized in that, when a non-visible light image used for synthesis is captured under the imaging conditions for a visible light image, the second determination means, instead of determining the imaging conditions, determines the amount of gain to be applied to the non-visible light image captured under the imaging conditions for a visible light image based on the final target value.

16. The image processing apparatus according to claim 15, characterized in that the second determination means determines the gain amount such that the gain amount does not change discontinuously at the boundary between adjacent blocks among the blocks into which the visible light image has been divided.

17. The image processing apparatus according to claim 1, wherein, if there are restrictions on the imaging conditions for the non-visible light image, and the imaging conditions according to the final target value cannot be determined within the restricted range, the second determination means determines the imaging conditions within the restricted range and further determines a gain amount to compensate for the insufficient exposure amount.

18. The system further includes an application means for applying the gain amount to a non-visible light image captured under the aforementioned imaging conditions, The image processing apparatus according to claim 17, characterized in that the non-visible light image to which the gain amount is applied is used for the synthesis.

19. When the first target value is changed, the change in brightness of the visible light image due to the synthesis of the invisible light images is greater than when the second target value is changed. The image processing apparatus according to claim 1, characterized in that when the second target value is changed, the change in the contrast of the visible light image due to the synthesis of the invisible light images is greater than when the first target value is changed.

20. An image processing apparatus for determining the imaging conditions of a non-visible light image to be synthesized with a visible light image, A first determination means is configured to determine a final target value for the signal level of the invisible light image that determines the imaging conditions by weighting and adding a first target value determined based on the brightness of the visible light image and a second target value determined based on the contrast of the invisible light image. The system includes a second determination means configured to determine the imaging conditions based on the final target value, The image processing apparatus is characterized in that the weights of the first target value and the second target value in the weighted addition are determined based on the brightness of the visible light image.

21. An image sensor capable of capturing visible light images and invisible light images, An imaging device comprising an image processing device according to any one of claims 1 to 20, which uses an image captured using the image sensor.

22. The imaging apparatus according to claim 21, further comprising recording means for recording, in association with the visible light image, a non-visible light image to be synthesized with the visible light image and information regarding the region in the visible light image on which the non-visible light image should be synthesized.

23. An image processing method for determining the imaging conditions of a non-visible light image to be combined with a visible light image, which is performed by an image processing device, A first determination step in which the final target value of the signal level of the invisible light image that determines the imaging conditions is determined using a first target value of the signal level of the invisible light image, which is determined based on the relationship between the brightness of the visible light image and a predetermined threshold, and a second target value of the signal level of the invisible light image, which is determined based on information regarding the contrast of the invisible light image. A second determination step in which the imaging conditions are determined based on the final target value, An image processing method characterized by having the following features.

24. A program for causing a computer to function as one of the means of the image processing apparatus described in any one of claims 1 to 20.

Images