Image processing device, image processing method and image acquisition device

Abstract

Image processing device (100, 200) for generating a synthetic image from a first image (DIN2) acquired by receiving visible light, which is light in a first wavelength range, and a second image (DIN1) acquired by receiving invisible light, which is light in a second wavelength range different from the first wavelength range, wherein the first image (DIN2) and the second image (DIN1) have the same field of view, which image processing device (100, 200) comprises: a first image separation device (102, 202) for separating, from the first image (DIN2), a first brightness base component (D3A, D9A) which is a brightness component separated from a first base component which is a base component of the first image (DIN2), a first color base component (D3B, D10) which is a color component separated from the first base component, and a first brightness detail component which is a brightness component separated from a first detail component (D4, D9B) which is a detail component of the first image (DIN2); a second image separation device (101, 201) for separating a second detail component (D1), which is a detail component of the second image (DIN1), from the second image (DIN1); a detail synthesizer (103, 203) for generating a synthetic brightness detail component (D5) which is a value obtained by taking a summation average of a second gain setting component obtained by performing a gain setting of the second detail component and a first gain setting component obtained by performing a gain setting on the first brightness detail component using a gain that is less than a gain used for the gain setting on the second detail component; a generating device for a synthetic brightness component (104, 204) for generating a synthetic brightness component (D6) by combining the synthetic brightness detail component and the first brightness base component; and a brightness-color synthesizer (105, 205) for generating the synthetic image (DOUT) by combining the synthetic brightness component and the first color basis component.

Description

TECHNICAL FIELD

[0001] The present invention relates to an image processing apparatus, an image processing method, and an image pickup apparatus for generating a synthetic image with high visibility from a first image acquired by receiving light in a first wavelength range from an image pickup object and a second image acquired by receiving light in a second wavelength range from the same image pickup object. STATE OF THE ART

[0002] There is an image integration technology for generating a high-visibility image by selecting multiple types of images that differ in image characteristics, such as a visible image and an infrared image, an RGB image and a narrowband image, or the like, depending on the scene (the state of the image-taking object, the state around the image-taking object, or the like), and integrating (combining) the selected images. For example, near-infrared images achieve high visibility when haze has occurred or in a high-dynamic-range environment, while narrowband images obtained by image acquisition from a satellite or drone are suitable for use in monitoring water and vegetation areas.Furthermore, monitoring devices are widely available on the market, including an irradiation device for irradiating the object with invisible light, such as a near-infrared light source, and a camera for acquiring an image with a high signal level and a low noise level, suitable for monitoring scenes during nighttime. By combining one of such images, which are different in image characteristics, with a visible image, an object or target requiring attention can be easily distinguished even when a scene in which discrimination is difficult with only a visible image is captured.

[0003] Patent Document 1 describes an image processing method for accurately enhancing edges included in a first image for display from the first image and a second image obtained by imaging the same object under different conditions by using edge components of the second image.

[0004] US 2014 / 0 340 515 A1 describes methods and devices for synthesizing two images of an object created by recording images at different wavelengths. To improve the visibility of the resulting image, edge, detail, and brightness information obtained from the original images is combined. PRIOR ART DOCUMENT PATENT DOCUMENT

[0005] Patent Document 1: Japanese Patent Application Publication JP 2014 - 241 584 A (claim 1, Fig. 2, paragraph 0127). SUMMARY OF THE INVENTION PROBLEM TO BE SOLVED BY THE INVENTION

[0006] However, the image processing method described in Patent Document 1 has a problem in that noise is included in a visible image under low-illumination conditions. Specifically, while the technology of Patent Document 1 is capable of suppressing noise by adjusting an edge strength with respect to noise contained in a brightness component of visible light, noise contained in a color component of the visible image (hereinafter also referred to as "color noise") is directly used for synthesis, and thus, the color noise is not sufficiently suppressed, making it impossible to generate an image (image data) with high visibility.

[0007] It is therefore an object of the present invention to provide an image processing apparatus, an image processing method and an image pickup apparatus capable of generating a synthetic image of high visibility with reduced influence of color noise even when a scene in which color noise is included in an image has been picked up. MEANS TO SOLVE THE PROBLEM

[0008] An image processing device according to one aspect of the present invention is an image processing device for generating a synthetic image from a first image acquired by receiving light in a first wavelength range and a second image acquired by receiving light in a second wavelength range different from the first wavelength range, wherein the first image and the second image have the same field of view. The image processing device includes: a first image separator for separating, from the first image, a first brightness base component, which is a brightness component separated from a first base component, which is a base component of the first image, a first color base component, which is a color component separated from the first base component, and a first brightness detail component, which is a brightness component.which has been separated from a first detail component, which is a detail component of the first image; a second image separator for separating a second detail component, which is a detail component of the second image, from the second image; a detail synthesizer for generating a synthetic brightness detail component, which is a value obtained by taking a summation average of a first gain adjustment component, which was obtained by performing a gain adjustment of the first brightness detail component, and a second gain adjustment component,obtained by performing a gain adjustment on the second detail component; a synthetic brightness component generating device for generating a synthetic brightness component by combining the synthetic brightness detail component and the first brightness base component; and a brightness-color synthesizer for generating the synthetic image by combining the synthetic brightness component and the first color base component.

[0009] An image pickup apparatus according to another aspect of the present invention includes the above-described image processing apparatus and an image sensor including infrared pixels for detecting infrared light, red pixels for detecting red light, green pixels for detecting green light, and blue pixels for detecting blue light, and acquiring the invisible image and the visible image.

[0010] An image processing method according to another aspect of the present invention is an image processing method for generating a synthetic image from a first image acquired by receiving light in a first wavelength range and a second image acquired by receiving light in a second wavelength range different from the first wavelength range, wherein the first image and the second image have the same field of view. The image processing method includes: a first image separation step of separating, from the first image, a first brightness base component, which is a brightness component separated from a first base component, which is a base component of the first image, a first color base component, which is a color component separated from the first base component, and a first brightness detail component, which is a detail component separated from a first detail component.which is a detail component of the first image; a second image separation step of separating a second detail component, which is a detail component of the second image, from the second image; a detail synthesis step of generating a synthetic brightness detail component, which is a value obtained by taking a summation average of a first gain adjustment component, which was obtained by performing gain adjustment of the first brightness detail component, and a second gain adjustment component,obtained by performing a gain adjustment of the second detail component; a synthetic brightness component generation step of generating a synthetic brightness component by combining the synthetic brightness detail component and the first brightness base component; and a brightness-color synthesis step of generating the synthetic image by combining the synthetic brightness component and the first color base component. EFFECT OF THE INVENTION

[0011] According to the present invention, a synthetic image of high visibility with reduced influence of color noise can be generated even when a scene in which a visible image contains color noise has been captured. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to a first embodiment of the present invention. Fig. 2(a) is a block diagram showing an example of a schematic configuration of an invisible image filter separating device according to the first embodiment, and Fig. 2(b) is a block diagram showing an example of a schematic configuration of a visible image filter separating apparatus according to the first embodiment. Fig. 3 is a block diagram showing a schematic configuration for a detail synthesizer according to the first embodiment. Fig. 4 is a flowchart showing the operation of the image processing apparatus according to the first embodiment. Fig. 5 is a block diagram showing a schematic configuration of an image processing apparatus according to a second embodiment of the present invention. Fig. 6 is a flowchart showing the operation of the image processing apparatus according to the second embodiment. Fig. 7 is a block diagram showing a schematic configuration of an image pickup apparatus according to a third embodiment of the present invention. Fig. 8 is a hardware configuration diagram showing a configuration of a modification. MODE FOR CARRYING OUT THE INVENTION

[0012] Image processing devices, image processing methods, and image pickup devices according to embodiments will be described below with reference to drawings.

[0013] In the image processing apparatuses and image processing methods according to the embodiments, a synthetic image having excellent visibility is generated from a first image (e.g., visible image, RGB image, or the like) acquired by an image pickup device (photoreceptor elements) by receiving light in a first wavelength range, and a second image (e.g., invisible image, white image, or the like) acquired by the image pickup device (photoreceptor elements) by receiving light in a second wavelength range different from the first wavelength range.

[0014] In the following description, a visible image means an image (image data) created by capturing images by an image capturing device capable of imaging light rays visible to the human eye (visible light). An invisible image means an image that is not a visible image, that is, an image (image data) created by capturing images by an image processing device capable of imaging light rays invisible to the human eye (invisible light). Generally, visible light means light in a wavelength range of approximately 400 nm to 700 nm, including, for example, light in the red (R), green (G), and blue (B) wavelength ranges.Visible light means light in a range other than the range of visible light, which includes, for example, ultraviolet light (generally light in a wavelength range below or equal to approximately 400 nm) and infrared light (generally light in a wavelength range above or equal to approximately 700 nm).

[0015] Furthermore, in the following embodiments, image data is processed after being separated into a detail component and a basic component, which is a component different from the detail component. The detail component is a component in which the pixel value (signal value) of each pixel changes rapidly by small amounts compared to the neighboring pixels, which is also called a high-frequency component. The basic component is a component in which the pixel value of each pixel does not change rapidly by small amounts compared to the neighboring pixels, which is also called a low-frequency component. Thus, the detail component represents rapid small changes of pixel values ​​in the image, and the basic component represents smooth changes of pixel values ​​in a certain area in the image.The detail component is, for example, a light component reflected from the subject, and the base component is, for example, an illumination light component.

[0016] Furthermore, the inventions of the image processing devices described below can also be regarded as inventions of image processing methods. (1) First embodiment (1-1) Configuration

[0017] First, an overview of the components of an image processing apparatus 100 according to a first embodiment will be described. Fig. 1 is a block diagram showing a schematic configuration of the image processing apparatus 100 according to the first embodiment of the present invention. As shown in Fig. 1, the image processing apparatus 100 according to the first embodiment includes an invisible image filter separator 101 as a second image separator (invisible image separator), a visible image separator 102 as a first image separator, a detail synthesizer 103, a synthetic brightness component generator 104, and a brightness-color synthesizer 105. The visible light separator 102 includes a visible light filter separator 102a, a basic brightness color separator 102b, and a detail brightness color separator 102c.

[0018] As in Fig. As shown in Figure 1, an invisible image (second image) DIN1 and a visible image (first image) DIN2 are input as image data to the image processing device 100. It is desirable that the invisible image DIN1 and the visible image DIN2 be acquired by image pickup means having the same field of view. The same field of view means that the image pickup means that picks up the invisible image DIN1 and the image pickup means that picks up the visible image DIN2 have the same image pickup region (image pickup area), which does not require that the image pickup region of the image pickup means that picks up the invisible image DIN1 and the image pickup region of the image pickup means that picks up the visible image DIN2 perfectly match each other.

[0019] For example, images acquired by an invisible image camera and a visible image camera located adjacent to each other can be directly used as the invisible image DIN1 and the visible image DIN2. It is also possible to perform position alignment adjustment of images acquired by an invisible image camera and a visible image camera located adjacent to each other, and use the images after adjusting the position alignment as the invisible image DIN1 and the visible image DIN2.

[0020] For example, the invisible image DIN1 and the visible image DIN2 can be acquired by using an image sensor with infrared pixels that capture infrared light for the invisible image DIN1 and RGB pixels that capture visible light for the visible image DIN2 (red pixels that capture red light, green pixels that capture green light, and blue pixels that capture blue light). It is also possible to acquire two types of images instead of the invisible image DIN1 and the visible image DIN2 by using an image sensor with, for example, white pixels (W pixels) that capture white light and RGB pixels.

[0021] As in Fig. As shown in Figure 1, the invisible image filter separator 101 performs filtering on the invisible image DIN1, thereby generating an invisible detail component D1 that is a detail component of the invisible image DIN1 (second detail component). The visible image filter separator 102a performs filtering on the visible image DIN2, thereby generating a visible basic component D2A that is a basic component of the visible image (first basic component) and a visible detail component D2B that is a detail component of visible light (first detail component).

[0022] As in Fig. As shown in Figure 1, the base brightness-color separator 102b separates the visible base component D2A into a visible brightness base component (brightness component in the visible base component) D3A as a first brightness base component and a visible color base component (color component in the visible base component) D3G as a first color base component. The detail brightness-color separator 102c extracts a visible brightness detail component (brightness component in the visible detail component) D4 from the visible detail component D2B.

[0023] As in Fig. As shown in Figure 1, the detail synthesizer 103 generates a synthetic brightness detail component D5 by combining the invisible detail component D1 and the visible brightness detail component D4. The synthetic brightness component generator 104 generates a synthetic brightness component D6 by combining the synthetic brightness detail component D5 and the visible brightness base component D3A. The brightness-color synthesizer 105 generates and outputs a synthetic image DOUT by combining the synthetic brightness component D6 and the visible color base component D3B.

[0024] Next, details of the components of the image processing apparatus 100 according to the first embodiment will be described. Fig. 2(a) is a block diagram showing an example of a schematic configuration of the invisible image filter separating device 101 according to the first embodiment, and Fig. 2(b) is a block diagram showing an example of a schematic configuration of the visible image filter separator 102a according to the first embodiment. As shown in Fig. As shown in Fig. 2(a), the invisible image filter separating device 101 includes an invisible image filter processing unit 111 and an invisible image difference processing unit 112.

[0025] The invisible image filter processing unit 111 receives the invisible image DIN1, performs filter processing to smooth the received invisible image DIN1, and thereby generates a smoothed invisible base component D11. The invisible image difference processing unit 112 receives the smoothed invisible base component D11 and the invisible image DIN1, and generates the invisible detail component D1, which is the difference between the invisible base component D11 and the invisible image DIN1.

[0026] The filter performing the filter processing in the invisible image filter processing unit 111 is, for example, a smoothing filter such as a box filter, a Gaussian filter, or the like, or an edge-preserving smoothing filter such as a bilateral filter, a guided filter, or the like.

[0027] As in Fig. As shown in Fig. 2(b), the visible image filter separator 102a includes a visible image filter processing unit 121 and a visible image difference processing unit 122. The visible image filter processing unit 121 receives the visible image DIN2, performs filter processing to smooth the received visible image DIN2, and thereby generates a smoothed visible basic component D2A. The visible image difference processing unit 122 receives the smoothed visible basic component D2A and the visible image DIN2 and generates the visible detail component D2B, which is the difference between the visible basic component D2A and the visible image DIN2. It is desirable that the filter processing in the visible image filter separator 102a be performed on each channel (e.g., on each of an R signal, a G signal, and a B signal).

[0028] As in Fig. As shown in FIG. 1, the base luminance-color separator 102b receives the visible base component D2A from the visible image filter separator 102a and separates the received visible base component D2A into the visible luminance base component D3A, which is a luminance component, and the visible color base component D2B, which is a chroma component. In a case where the visible image is an RGB image, the base luminance-color separator 102b converts the RGB image into one of the HSV (Hue, Saturation, Luminosity) format consisting of hue, saturation, and luminance, the YUV format consisting of luminance and chroma components, and the YCbCr format consisting of luminance and chroma components.The basic brightness color separator 102b outputs an H (hue) channel or a Y (brightness) channel as the visible brightness basic component D3A and outputs the remaining channels of color, hue and saturation or the like as the visible color basic component D3B.

[0029] The detail brightness color separator 102c receives the visible detail component D2B from the visible image filter separator 102a and extracts the visible brightness detail component D4, which is a brightness component, from the visible detail component D2B. The detail brightness color separator 102c outputs an H channel, a Y channel, or the like as the visible brightness detail component D4 in a manner similar to the basic brightness color separator 102b.

[0030] In contrast, the detail brightness color separator 102c does not extract or separate a visible color detail component from the visible detail component D2B. Since the visible color detail component is a component that causes color noise under low illumination conditions, avoiding the use of the visible color detail component enables the generation of a high-visibility synthetic image DOUT with a reduced influence of color noise under low illumination conditions.

[0031] Fig. 3 is a block diagram showing a schematic configuration of the detail synthesizer 103 according to the first embodiment. As shown in Fig. As shown in Fig. 3, the detail synthesizer 103 according to the first embodiment includes gain adjustment devices 103a and 103b and an addition processing unit 103c. The detail synthesizer 103 generates the synthetic brightness detail component D5 by combining the invisible detail component D1 received from the invisible image filter separator 101 and the visible brightness detail component D4 received from the detail brightness color separator 102c.

[0032] The gain adjustment device 103a receives the invisible detail component D1 from the invisible image filter separator 101, performs gain adjustment (boost or attenuation) on the invisible detail component D1, and thereby outputs a gain adjustment component D41 as a second gain adjustment component. The gain adjustment device 103b receives the visible brightness detail component D4 from the detail brightness color separator 102c, performs gain adjustment (boost or attenuation) on the visible brightness detail component D4, and thereby outputs a gain adjustment component D42 as a first gain adjustment component.

[0033] The addition processing unit 103c receives the gain adjustment component D41 and the gain adjustment component D42 from the gain adjusters 103a and 103b and outputs a value D5 as a weighted average (summation average) of the components. The gains acquired by the gain adjusters 103a and 103b can be set to either fixed values ​​depending on the scene, application, or image characteristics.

[0034] In the detail synthesizer 103, the gain adjustment is performed separately for the invisible detail component D1 and the visible brightness detail component D4, thereby enabling gain adjustment appropriate for the application type and easy multi-application processing. Specifically, when it is desired to perform, for example, shadow correction using infrared light irradiation and the visible image DIN2 is considered to contain a lot of noise, the gain for the visible brightness detail component D4 is set to a low value, and the gain for the invisible detail component D1 is set to a value greater than or equal to 1.

[0035] Further, when it is desired to perform fog correction by utilizing atmospheric penetrability of infrared light, the gain of the visible brightness detail component D4 is set to a value close to 1, and the gain for the invisible detail component D1 is set to a value greater than or equal to 1. When the gain is set high to increase visibility, an enhancement process operates, and thus, it is desirable to set each gain separately depending on the type of application, characteristics of the image pickup equipment, or the like.

[0036] The gains in these cases can be obtained by simple addition of scalar values ​​or can be obtained by energy calculation such as gamma conversion.

[0037] The synthetic brightness component generator 104 outputs the synthetic brightness component D6 by combining the synthetic brightness detail component D5 received from the detail synthesizer 103 and the visible brightness base component D3A received from the base brightness color separation device 102b. Methods usable for combining by the synthetic brightness component generator 104 include a method that takes a simple summation average and a method that selects a high-visibility component with respect to, for example, each area.

[0038] The brightness-color synthesizer 105 outputs the synthetic image DOUT by combining the synthetic brightness component D6 received from the synthetic brightness component generating device 104 and the visible color base component D3B received from the basic brightness-color separator 102b. The method of combining the brightness component and the color component used by the brightness-color synthesizer 105 is assumed to be a method corresponding to the conversion method performed by the basic brightness-color separator 102b and the detailed brightness-color separator 102c. For example, when the basic brightness color separator 102b and the detail brightness color separator 102c have performed the RGB to YUV conversion, the brightness color synthesizer 105 performs the YUV to RGB conversion.On the other hand, when the basic brightness color separator 102b and the detail brightness color separator 102c have performed the YUV to RGB conversion, the brightness color synthesizer 105 performs the RGB to YUV conversion. (1-2) Working method

[0039] Next, filter processing using the guided filter is explained. Using the following linear regression coefficients: a and b, the base component q as the output signal of the guided filter and the input image I can be expressed in a linear relationship according to the following expression (1): q=al+b

[0040] Here, by using Ω, which is defined as a local area centered at a particular pixel x of the input image I, the linear regression coefficients a(x) and b(x) can be represented by expression (2). a¯(x)=∑y∈Ω(x)a(y)2,b¯(x)=∑y∈Ω(x)b(y)2

[0041] Next, a method for deriving the linear regression coefficients is explained. A pixel value of the variance varl of the input image I corresponding to the pixel position x is represented by Expression (3). varl(x)=∑y∈Ω(x)l(y)2−(∑y∈Ω(x)l(y))2

[0042] Subsequently, a smoothing process is performed only on an area where the variance value is extremely small, and in order to preserve the texture of the other areas, a grayscale conversion is performed on the variance varl(x) according to the following expression (4), thereby obtaining the linear regression coefficient a(x): a(x)=varl(x)varl(x)+eps where eps is a constant parameter that determines the degree of edge preservation.

[0043] Furthermore, the coefficient b is derived according to expression (5). b(x)=∑y∈Ω(x)l(y)−a(x)∑y∈Ω(x)l(y)

[0044] Based on the linear regression coefficients a(x) and b(x) obtained from expressions (4) and (5), the output value q of the guided filter is derived. The value q corresponds to the invisible base component D11, as shown in Fig. 2(a).

[0045] As in Fig. As shown in Figure 2(a), the invisible detail component D1 is obtained from the difference between the invisible image DIN1 and the invisible base component D11.

[0046] While an example using a smoothing filter as the processing performed by the invisible image filter separator 101 has been described above, the processing by the invisible image filter separator 101 is not limited to this example; it is also possible to directly output the detail component by using a high-pass filter such as a Laplacian filter.

[0047] Next, the operation of the image processing apparatus 100 according to the first embodiment will be described. Fig. 4 is a flowchart showing the operation of the image processing apparatus 100 according to the first embodiment. As shown in Fig. 4, when the operation of the image processing apparatus 100 is started, the invisible image DIN1 and the visible image DIN2 are input to the image processing apparatus 100 in step S1.

[0048] In the next step S2, the visible image DIN2 is input to the visible image filter separator 102a, and the filter processing is performed on the visible image DIN2.

[0049] In the next step S4, the brightness-color separation processing is performed on the visible base component D2A and the visible detail component D2B.

[0050] In parallel with the processing of steps S2 and S4, in step S3, the invisible image DIN1 is input to the invisible image filter separator 101, and the filter processing is performed on the invisible image DIN1.

[0051] In the next step S5, the synthetic brightness detail component D5 is obtained by combining the invisible detail component D1 and the visible brightness detail component D4 (detail synthesis).

[0052] In the next step S6, the synthetic brightness component D6 is obtained by combining the synthetic brightness detail component D5 and the visible brightness base component D3A (component synthesis).

[0053] Finally, in step S7, the synthetic image DOUT is obtained by combining the synthetic brightness component D6 and the visible color base component D3B (brightness-color synthesis). (1-3) Effect

[0054] With the image processing apparatus 100 according to the first embodiment, filtering and brightness-color separation processing are performed on the visible image, and the visible color detail component causing color noise is not used for synthesis. Accordingly, even in a case where a scene in which the visible image contains color noise (e.g., a scene under low illumination conditions) is captured, a high-visibility synthetic image DOUT with reduced influence of color noise can be acquired.

[0055] In the image processing apparatus 100 according to the first embodiment, the detail synthesizer 103 includes the gain adjusters 103a and 103b and the addition processing unit 103c, and generates the synthetic brightness detail component D5 by taking the weighted average (summation average) of the invisible detail component D1 and the visible brightness detail component D4 after gain adjustment. Accordingly, the acquisition of high-visibility images in a variety of scenes can be realized.

[0056] In the image processing apparatus 100 according to the first embodiment, the filter processing in the invisible image filter separation apparatus 101 and the visible image filter separation apparatus 102a is constituted by only simple filter operation processing and addition and subtraction processing. Accordingly, image processing can be implemented with low computational costs.

[0057] In the image processing apparatus 100 of the first embodiment, high-speed image processing suitable for real-time processing is performed with a simple configuration and low computational cost. Accordingly, an object or target requiring attention can be easily distinguished even when a scene that is conventionally difficult to distinguish using only a visible image is captured.

[0058] While a case where the inputs to the image processing device 100 are a visible image and an invisible image was taken as an example in this embodiment, equivalent effects can be obtained without the need to limit the inputs to the above-described combination, as long as the inputs are a combination of a high-visibility image and a low-visibility image. For example, the high-visibility image may be a long-exposure image, and the low-visibility image may be a short-exposure image. Alternatively, the high-visibility image may be an image of a bright environment taken during the day, and the low-visibility image may be an image of a dark environment taken at night. (2) Second embodiment (2-1) Configuration

[0059] Fig. 5 is a block diagram showing a schematic configuration of an image processing apparatus 200 according to a second embodiment of the present invention. Fig. 5 is any component that is connected to a Fig. 1 shown component (first embodiment) is identical or corresponds to it, the same reference numeral as in Fig. 1. As in Fig. As shown in FIG. 5, the image processing apparatus 200 according to the second embodiment differs from the first embodiment in the configuration of a visible image separator 202. Specifically, the visible image separator 202 in the second embodiment includes a brightness-color separator 202a as a first image brightness-color separator, a filter separator 202b as a first brightness component separator, and a filter processing unit 202c as a first color component separator.

[0060] In the image processing apparatus 100 according to the first embodiment, filtering processing is performed on the visible image DIN2, and then brightness-color separation processing is performed on the separated components (the visible base component D2A and the visible detail component D2B). In contrast, in the image processing apparatus 200 according to the second embodiment, brightness-color separation processing is first performed on the visible image DIN2, and filtering processing is performed after brightness-color separation processing on the separated brightness and color components.

[0061] The brightness-color separator 202a receives the visible image DIN2 and separates the received image DIN2 into a visible brightness component D8A, which is a brightness component, and a visible color component D8B, which is a color component. The separation processing in the brightness-color separator 202a is similar to that in the basic brightness-color separator 102b and the detailed brightness-color separator 102c in the first embodiment.

[0062] The filter separation device 202b receives the visible brightness component D8A from the brightness-color separation device 202a, separates the received visible brightness component D8A into a visible brightness base component D9A, which is a base component, and a visible brightness detail component D9B, which is a detail component, and outputs the visible brightness base component D9A and the visible brightness detail component D9B. The separation processing in the filter separation device 202b is processing similar to that in the invisible image filter separation device 101 and the visible image filter separation device 102a according to the first embodiment.

[0063] The filter processing unit 202c receives the visible color component D8B from the luminance color separation device 202a, performs smoothing by filter processing on the received visible color component D8B, and outputs the smoothed visible color base component D10. For the filter processing, it is desirable to use an edge-preserving smoothing filter such as a bilateral filter or a guided filter in addition to a smoothing filter such as a box filter or a Gaussian filter.

[0064] The configurations of the detail synthesizer 103, the synthetic luminance component generating device 104, and the luminance-color synthesizer 105 in the second embodiment are the same as those in the first embodiment. (2-2) Working method

[0065] Next, the operation of the image processing apparatus 200 according to the second embodiment will be described. Fig. 6 is a flowchart showing the operation of the image processing apparatus 200 according to the second embodiment. As shown in Fig. 6, when the operation of the image processing apparatus 200 is started, the invisible image DIN1 and the visible image DIN2 are input to the image processing apparatus 200 in step S11.

[0066] In the next step S12, the brightness color separation processing is performed on the visible image DIN2.

[0067] In the next step S14, the filter processing is performed on the visible brightness component D8A and the visible color component D8B.

[0068] In parallel with the processing of steps S12 and S14, in the next step S13, the invisible image DIN1 is input to the invisible image filter separating device 101, by which the filter processing is performed on the invisible image DIN1.

[0069] In the next step S15, the synthetic brightness detail component D5 is obtained by combining the invisible detail component D1 and the visible brightness detail component D4 (detail synthesis)

[0070] In the next step S16, the synthetic brightness component D6 is obtained by combining the synthetic brightness detail component D5 and the visible brightness base component D3A (component synthesis).

[0071] In the last step S17, the synthetic image DOUT is obtained by combining the synthetic brightness component D6 and the visible color base component D3B (brightness-color synthesis). (2-3) Effect

[0072] With the image processing apparatus 200 according to the second embodiment, effects similar to those of the image processing apparatus 100 according to the first embodiment can be obtained.

[0073] With the image processing apparatus 200 according to the second embodiment, the filter processing is performed after performing the brightness-color separation processing on the visible image DIN2, and thus the number of times of performing the brightness-color separation processing can be reduced to 1, and the number of calculations can be further reduced compared with the image processing apparatus 100 according to the first embodiment. (3) Third embodiment

[0074] Fig. 7 is a block diagram showing a schematic configuration of an image pickup device 300 according to a third embodiment of the present invention. As shown in Fig. As shown in Fig. 7, the image pickup device 300 includes the image processing device 100 according to the first embodiment (or the image processing device 200 according to the second embodiment), a first image sensor 301, and a second image sensor 302.

[0075] As the first image sensor 301 and the second image sensor 302, CCD (charge coupled device) image sensors, CMOS (complementary metal oxide semiconductor) image sensors, or the like can be used as an example.

[0076] The first image sensor 301 acquires the invisible image DIN1 by imaging an image-capturing object and inputs the invisible image DIN1 to the image processing device 100. The second image sensor 302 acquires the visible image DIN2 by imaging the image-capturing object and inputs the visible image DIN2 to the image processing device 100. The first image sensor 301 and the second image sensor 302 are arranged adjacent and close to each other as desired to have the same field of view.

[0077] In a case where the invisible image DIN1 acquired by the first image sensor 301 and the visible image DIN2 acquired by the second image sensor 302 have the same field of view, the invisible image DIN1 and the visible image DIN2 can be directly used for synthesis. In a case where the range of the invisible image DIN1 and the range of the visible image DIN2 are different from each other, these images can be used for synthesis processing after performing a process of aligning the positions of these two images with each other, whereby a captured image with high visibility can be acquired. (4) Modifications

[0078] While the present invention has been described above based on specific embodiments, these embodiments are merely examples for illustration, and the present invention is not limited to these embodiments.

[0079] Fig. 8 is a hardware configuration diagram showing a configuration of a modification of the image processing apparatus 100 and 200 according to the above-described first and second embodiments, respectively. While the image processing apparatus 100, 200 can be implemented by a semiconductor integrated circuit, the image processing apparatus 100, 200 can also be implemented by using a memory 91 as a storage device for storing a program as software and a processor 92 as an information processing unit for executing the program stored in the memory 91 (e.g., by a computer), as shown in Fig. 8. It is also possible to replace a part of the image processing device 100, 200 by the Fig. 8 and the processor 92 for executing a program.

[0080] In addition to an infrared image acquired by using infrared light, it is also possible to use an ultraviolet light image acquired by using ultraviolet light, a radar image, etc. as the invisible image in the above description of the embodiments.

[0081] While a visible image and an invisible image are combined in the above description of the embodiments, a combination of an RGB image and a narrowband image (including a visible band) is also usable in addition to the combination of a visible image and an invisible image.

[0082] The present invention is particularly effective when a low-noise specific spectrum image and a visible image containing color information are combined by light source irradiation in a case where a surveillance camera or a camera installed in a vehicle captures images while the object is irradiated with light from a specific spectrum light source such as a near-infrared light source. DESCRIPTION OF REFERENCE SYMBOLS

[0083] 100, 200: Image processing device, 101: Invisible image separation device (invisible image filter separation device, second image separation device), 102, 202: Visible image separation device (first image separation device), 102a: Visible image filter separation device (first image filter separation device), 102b: Basic brightness-color separation device, 102c: Detail brightness-color separation device, 103: Detail synthesizer, 103a, 103b: Gain adjustment device, 103c: Addition processing unit, 104: Synthetic brightness component generation device, 105: Brightness-color synthesizer, 111: Invisible image filter processing unit (second image filter processing unit), 112: Invisible image difference processing unit (second Image difference processing unit), 121: Visible image filter processing unit (first image filter processing unit),122: Visible image difference processing unit (first image difference processing unit), 202a: Brightness-color separation device, 202b: Filter separation device, 202c: Filter processing unit, 300: Image pickup device, 301: First image sensor, 302: Second image sensor, DIN1: Invisible image (second image), DIN2: Visible image (first image), DOUT: Synthetic image.

Claims

[1] An image processing device (100, 200) for generating a synthetic image from a first image (DIN2) acquired by receiving visible light, which is light in a first wavelength range, and a second image (DIN1) acquired by receiving invisible light, which is light in a second wavelength range different from the first wavelength range, wherein the first image (DIN2) and the second image (DIN1) have the same field of view, which image processing device (100, 200) comprises: a first image separating device (102, 202) for separating, from the first image (DIN2), a first brightness base component (D3A, D9A), which is a brightness component separated from a first base component, which is a base component of the first image (DIN2), a first color base component (D3B, D10), which is a color component separated from the first base component, and a first brightness detail component, which is a brightness component separated from a first detail component (D4, D9B), which is a detail component of the first image (DIN2); a second image separating device (101, 201) for separating a second detail component (D1), which is a detail component of the second image (DIN1), from the second image (DIN1); a detail synthesizer (103, 203) for generating a synthetic brightness detail component (D5) which is a value obtained by taking a summation average of a second gain adjustment component obtained by performing gain adjustment of the second detail component and a first gain adjustment component obtained by performing gain adjustment on the first brightness detail component using a gain lower than a gain used for the gain adjustment on the second detail component; a synthetic brightness component generating device (104, 204) for generating a synthetic brightness component (D6) by combining the synthetic brightness detail component and the first brightness base component; and a luminance-color synthesizer (105, 205) for generating the synthetic image (DOUT) by combining the synthetic luminance component and the first color base component. [2] An image processing apparatus (100) according to claim 1, wherein the first image separating device (102) includes: a first image filter separator (102a) for performing edge-preserving smoothing filter processing on the first image (DIN2), wherein the smoothed first basic component is generated and the first detail component is generated from a difference between the first basic component and the first image (DIN2); a basic brightness-color separator (102b) for separating the first brightness basic component, which is the brightness component of the first basic component, and the first color basic component, which is the color component of the first basic component, from the first basic component; and a detail brightness color separator (102c) for separating the first brightness detail component, which is the brightness component of the first detail component, from the first detail component. [3] The image processing apparatus (200) according to claim 1, wherein the first image separating device (202) includes: a first image brightness-color separation device (202a) for separating a first brightness component, which is a brightness component of the first image (DIN2), and a first color component, which is a color component of the first image (DIN2), from the first image (DIN2); a first brightness component separating device (202b) for separating the first brightness basic component, which is a basic component of the first brightness component, and the first brightness detail component, which is a detail component of the first brightness component, from the first brightness component; and a first color component separating device (202c) for separating the first color base component, which is a base component of the first color component, from the first color component. [4] An image processing apparatus (100) according to any one of claims 1 to 3, wherein the second image separating device (101) includes: a second image filter processing unit (111) for performing filter processing on the second image (DIN1), thereby generating a smoothed second basis component; and a second image difference processing unit (112) for generating the second detail component from a difference between the second base component and the second image (DIN1). [5] Image recording device (300) comprising: the image processing device (100, 200) according to one of claims 1 to 4; a first image sensor (301) for acquiring the second image (DIN1); and a second image sensor (302) arranged adjacent to the first image sensor for acquiring the first image (DIN2). [6] The image pickup device (300) according to claim 5, wherein the image pickup device performs adjustment of aligning the positions of the second image (DIN1) acquired by the first image sensor and the first image (DIN2) acquired by the second image sensor. [7] An image processing method for generating a synthetic image from a first image (DIN2) acquired by receiving visible light which is light in a first wavelength range and a second image (DIN1) acquired by receiving invisible light which is light in a second wavelength range different from the first wavelength range, wherein the first image (DIN2) and the second image (DIN1) have the same field of view, which image processing method comprises: a first image separation step (S2, S4, S12, S14) of separating, from the first image (DIN2), a first brightness base component (D3A, D9A) which is a brightness component separated from a first base component which is a base component of the first image (DIN2), a first color base component (D3B, D10) which is a color component separated from the first base component, and a first brightness detail component which is a brightness component separated from the first detail component (D4, D9B) which is a detail component of the first image (DIN2); a second image separation step (S3, S13) of separating a second detail component (D1), which is a detail component of the second image (DIN1), from the second image (DIN1); a detail synthesis step (S5, S15) of generating a synthetic brightness detail component (D5) which is a value obtained by taking a summation average of a second gain adjustment component obtained by performing gain adjustment on the second detail component and a first gain adjustment component obtained by performing gain adjustment on the first brightness detail component using a gain lower than a gain used for the gain adjustment on the second detail component; a synthetic brightness component generation step (S6, S16) of generating a synthetic brightness component (D6) by combining the synthetic brightness detail component and the first brightness base component; and a brightness-color synthesis step (S7, S17) of generating the synthetic image (DOUT) by combining the synthetic brightness component and the first color base component.

Images