Image processing apparatus, imaging apparatus, and image processing method

Abstract

The present invention discloses an image processing apparatus, an imaging apparatus, and an image processing method. The image processing apparatus includes: an input unit that receives an image signal of visible light pixels and an image signal of near-infrared pixels output from a sensor including visible light pixels and near-infrared pixels; a determination unit that determines whether an output signal of the near-infrared pixels is higher than a threshold; a detector that detects saturated visible light pixels; and a switching unit that switches a saturation process to be applied to the saturated visible light pixels based on a determination result of the determination unit.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an imaging apparatus, and an image processing method. Background Art

[0002] In recent years, the use of near-infrared images captured by imagers sensitive to the near-infrared region has increased. For example, irradiating a fluorescent substance injected intravenously, such as indocyanine green, with light having a near-infrared wavelength makes blood vessels or lymphatic vessels visible, and this is useful in medical applications and in vivo observations. In addition, determining the NDVI (Normalized Difference Vegetation Index), which is an index of the presence / absence or activity of vegetation, based on the ratio of the near-infrared wavelength to the infrared wavelength of the captured image is useful for crop growth observation or remote sensing.

[0003] Imaging apparatuses that can simultaneously acquire near-infrared images and visible light images have been developed. For example, an imaging apparatus is known that can simultaneously acquire a near-infrared image and a visible light image by an imager having pixels provided with a visible light color filter and pixels provided with a near-infrared color filter, where these pixels are arranged on the same plane.

[0004] Japanese Patent Application Laid-Open No. 2014-165528 discloses an imaging apparatus that can separate a near-infrared component from the output signal of visible light pixels by an imager having an array of pixels provided with an RGB color filter and a transparent filter having an equal transmittance for near-infrared light.

[0005] Japanese Patent Application Laid-Open No. 2014-165528 employs different color signal correction processes depending on whether the pixels are saturated. Summary of the Invention

[0006] According to one aspect of the present disclosure, there is provided an image processing apparatus including: an input unit that receives an image signal of visible light pixels and an image signal of near-infrared pixels output from a sensor including visible light pixels and near-infrared pixels; a determination unit that determines whether an output signal of the near-infrared pixels is higher than a threshold; a detector that detects saturated visible light pixels; and a switching unit that switches a saturation process to be applied to the saturated visible light pixels based on a determination result of the determination unit.

[0007] According to another aspect of the present disclosure, there is provided an image processing method including: an input step of inputting an image signal of visible light pixels and an image signal of near-infrared pixels output from a sensor including visible light pixels and near-infrared pixels; a determination step of determining whether an output signal of the near-infrared pixels is higher than a threshold; a detection step of detecting saturated visible light pixels; and a switching step of switching a saturation process to be applied to the saturated visible light pixels based on a determination result at the determination step.

[0008] Other features of the present invention will become apparent from the following description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram of an image processing apparatus according to a first embodiment;

[0010] Figure 2A and Figure 2B is an exemplary diagram of a pixel assembly in an imaging unit;

[0011] Figure 3 is an exemplary diagram of characteristics of an imaging unit;

[0012] Figure 4A is an exemplary diagram of an image acquired in visible light, and Figure 4B is an exemplary diagram of an image acquired in near infrared;

[0013] Figures 5A to 5C is an exemplary diagram of white balance processing according to the prior art;

[0014] Figures 6A to 6C is an exemplary diagram illustrating problems in saturation processing according to the prior art;

[0015] Figure 7 is an exemplary diagram of a processing flow according to a first embodiment;

[0016] Figures 8A to 8C is an exemplary diagram of signal processing according to a first embodiment;

[0017] Figures 9A to 9C is an exemplary diagram of signal processing according to a first embodiment;

[0018] Figure 10 is a block diagram of an image processing apparatus according to a second embodiment; and

[0019] Figure 11 is an exemplary diagram of signal processing according to a second embodiment. DETAILED DESCRIPTION

[0020] The imaging device of Japanese Patent Application Publication No. 2014-165528 determines whether a pixel is saturated based on a signal including an overlapping portion of visible light components and near infrared components. When saturation occurs, the device cannot determine which of visible light and near infrared causes the saturation. Therefore, the problem caused by the device is that false coloring appears because the correct color cannot be reproduced by the corrected color signal.

[0021] An object of the present disclosure is to provide an image processing technique capable of obtaining a high-quality image with less pseudo-color from an image including saturated pixels.

[0022] Hereinafter, an image processing apparatus according to each embodiment will be described with reference to the accompanying drawings.

[0023] First Embodiment

[0024] Overall Configuration

[0025] Figure 1 FIG. is a diagram illustrating a configuration example of an image processing apparatus according to an embodiment of the present disclosure, in which the image processing apparatus is provided with an imaging unit. The image processing apparatus of this embodiment can also be understood as an imaging apparatus. In other embodiments of the present disclosure, the image processing apparatus may be composed only of an image processor, and signals are input from a separate imaging unit to the image processor.

[0026] Figure 1 The image processing apparatus 100 illustrated in FIG. is composed of an imaging unit 101 and an image processor 104.

[0027] The imaging unit 101 includes a plurality of pixels. In this embodiment, the imaging unit 101 includes a near-infrared pixel assembly 102 provided with a color filter most sensitive to near-infrared light and a visible-light pixel assembly 103 provided with a color filter most sensitive to visible light. As will be described later, the imaging unit 101 has a plurality of pixel groups of red pixels, green pixels, blue pixels, and near-infrared pixels arranged in two columns and two rows. Although it is assumed here that the imager is a CMOS image sensor, it may also be a CCD image sensor. Here, the sensitivity of the visible-light pixel assembly 103 to the near-infrared spectral range is equal to the sensitivity of the near-infrared pixel assembly 102 to the near-infrared spectral range.

[0028] The image processor 104 includes a near-infrared image signal input unit 105, a visible-light image signal input unit 106, a near-infrared component subtraction unit 109, and a white balance processing unit 110. The image processor 104 also includes a near-infrared level determination unit 107, a saturated pixel detection unit 108, a first saturation processing unit 111, a second saturation processing unit 112, a saturation processing switching unit 113, and a saturation processing application unit 114 as functional units for saturation processing. These units can be implemented by a dedicated hardware circuit such as an ASIC, or can be implemented by a general-purpose processor such as a CPU that executes a program.

[0029] The near-infrared image signal input unit 105 receives the input of the image signal from the near-infrared pixel assembly 102 of the imaging unit 101. The visible light image signal input unit 106 receives the input of the image signal from the visible light pixel assembly 103 of the imaging unit 101. Hereinafter, the image signals input to the input units 105 and 106 shall also be referred to as the near-infrared signal and the visible light signal, respectively.

[0030] The near-infrared level determination unit 107 determines the signal level of the near-infrared image. Specifically, the near-infrared level determination unit 107 determines whether the level of the output signal of each near-infrared pixel is higher than the level determination threshold. The determination result of the near-infrared level determination unit 107 is input to the saturation processing switching unit 113. The level determination threshold may be set such that if the near-infrared level is lower than the threshold, the saturation occurring in the visible light pixels can be attributed to the visible light component. For example, the level determination threshold may be a value that is half of the maximum output level (saturation level) of the imaging unit 101.

[0031] The saturated pixel detection unit 108 detects saturated visible light pixels (saturated pixels) from the visible light image signal. The detection result of the saturated pixel detection unit 108 is input to the saturation processing application unit 114.

[0032] The near-infrared component subtraction unit 109 subtracts the output value of the near-infrared pixel near the visible light pixel from the output signal of the visible light pixel. Through the subtraction process, the visible light components of each color - namely, red (R), green (G), and blue (B) - from which the near-infrared component has been removed are obtained. The near-infrared pixel near the visible light pixel may be, for example, a near-infrared pixel within the same pixel group as the visible light pixel. The near-infrared component subtraction unit 109 may subtract from the output signal of the visible light pixel a value (e.g., an average value) based on the output values of a plurality of adjacent near-infrared pixels of the visible light pixel.

[0033] The white balance processing unit 110 processes the visible light signal after subtracting the near-infrared signal, thereby adjusting the values of the respective color components to achieve a color ratio that allows a white object to appear as natural white according to the light source. Specifically, the white balance processing unit 110 performs white balance processing of multiplying the pixel value of each color included in the image signal of the visible light pixel by a gain corresponding to the color. The gain for each color is predetermined.

[0034] The first saturation processing unit 111 performs a first saturation processing on the visible light image signal after white balance processing. The first saturation processing is, for example, a clipping operation. Specifically, the first saturation processing unit 111 replaces the pixel values of saturated visible light pixels of each color with a predetermined value. The predetermined value is a white level value and can be determined based on the pixel values of visible light pixels of one color (e.g., green) within the same pixel group as the pixel being processed, or can be a preset value.

[0035] The second saturation processing unit 112 performs a second saturation processing on the visible light image signal after white balance processing. The second saturation processing is a process of determining the pixel value of a saturated visible light pixel based on the pixel values of visible light pixels around the saturated visible light pixel. A specific example of the second saturation processing is a process of replacing the pixel value of a saturated pixel with a pixel value obtained by interpolating a plurality of visible light pixels around the saturated visible light pixel. The replacement pixel value can be determined by referring only to the unsaturated pixels among the visible light pixels around the saturated pixel.

[0036] The saturation processing switching unit 113 selects the first saturation processing or the second saturation processing to be applied to the saturated visible light pixels based on the determination result of the near-infrared level determination unit 107. Specifically, if the output signal of the near-infrared pixel near the saturated pixel is lower than the level determination threshold, the saturation processing switching unit 113 selects the first saturation processing, and if the output signal is higher than the level determination threshold, the saturation processing switching unit 113 selects the second saturation processing.

[0037] The saturation processing application unit 114 applies the saturation processing selected by the saturation processing switching unit 113 to the saturated visible light pixels, while outputting the image signal of the unsaturated visible light pixels after white balance processing as it is. The output signal from the saturation processing application unit 114 is output to other image processors, image output units, image recording units, display units, etc. (not shown) for subsequent operations.

[0038] Imager

[0039] Figure 2A and Figure 2B is an exemplary diagram of the image information input from the imaging unit 101 of the first embodiment.

[0040] Figure 2B is a schematic diagram of a conventional (common) visible light color imager 211. The color imager 211 has pixels 212R, 212G, and 212B that are respectively provided with red (R), green (G), and blue (B) filters and arranged in a Bayer pattern.

[0041] Figure 2AFIG. 0 is a schematic diagram of an imager 203 that can acquire both an image signal of visible light and an image signal of near-infrared light and can be used in this embodiment. In addition to pixels 204R, 204G, and 204B each provided with a red (R), green (G), and blue (B) filter, respectively, the imager 203 further includes a pixel 204IR provided with a near-infrared filter.

[0042] The image signal of near-infrared light is separated from the image signal obtained by the imager 203 and subtracted from the image signal of the visible-light pixels, so that images 205, 206, and 207 composed only of red (R), green (G), and blue (B) are generated. An image 208 composed only of near-infrared components is obtained from the near-infrared pixel 204IR.

[0043] The imager 203 includes a plurality of pixel groups of a first visible-light pixel 204R, a second visible-light pixel 204G, a third visible-light pixel 204B, and a near-infrared pixel 204IR arranged in two columns and two rows. The arrangement of each pixel in the pixel group may be different from Figure 2A the configuration shown in. One pixel group may include a plurality of pixels corresponding to the same color.

[0044] Figure 3 FIG. 12 is a schematic diagram showing the quantum efficiency (sensitivity) of each pixel provided with a visible-light color filter and a near-infrared filter. Curves 301, 302, 303, and 304 represent the quantum efficiency (sensitivity) of the blue (B) pixel 204B, the green (G) pixel 204G, the red (R) pixel 204R, and the near-infrared (IR) pixel 204IR, respectively.

[0045] Reference numeral 305 denotes a wavelength band removed by a band-stop filter inserted between the imager 203 and the lens. The band-stop filter removes the wavelength band from red to near-infrared. As Figure 3 shown in, the quantum efficiency of the red (R), green (G), and blue (B) pixels is matched in the near-infrared at least at a predetermined threshold value λ2. The band-stop filter removes wavelengths in the near-infrared range where the quantum efficiency of each color pixel is different, so that these wavelengths do not enter the visible-light pixels and the near-infrared pixels. Specifically, the band-stop filter removes wavelengths in the range from λ1 to the above-mentioned wavelength λ2, where λ1 is the shortest wavelength in the sensitivity spectrum of the near-infrared pixel. By using such a band-stop filter, the near-infrared components included in the visible-light image signals of each color can be made equal to the near-infrared component 304 included in the near-infrared image signal. Therefore, each visible-light component can be obtained only by subtracting the near-infrared component 304 from the output signal from each visible-light pixel.

[0046] Effect of pixel saturation

[0047] Reference Figures 4A to 6CDescribe the problems that occur when saturation occurs in visible light images. An example of using the imager 203 to capture an image of an apple is given for illustration. The imager 203 includes the reference Figure 2A visible light pixels and near-infrared pixels described.

[0048] Figure 4A Fig. shows the image obtained by the red (R) pixel 204R of the visible light pixels. Since the fruit surface 401a of the red apple contains many red components, a high output signal is obtained, making the image bright. On the other hand, the green leaf part 402a contains few red components, resulting in a low output signal and a dark image. In Figure 4A and Figure 4B In, the light and dark of the drawings represent the magnitude of the output signal, that is, the whiter, the higher the pixel value (brighter), and the blacker, the lower the pixel value (darker).

[0049] Figure 4B Fig. shows the image obtained by the near-infrared pixel 204IR. Since both the fruit surface 401b and the green leaf part 402b reflect a large amount of near-infrared light, a high output signal is obtained from both parts, making the image bright.

[0050] When there is a visible light illumination source (not shown) on the upper right side of the object (apple), this illumination can create a light reflection point 403a on the upper right of the apple surface in the visible light image in Figure 4A , where the pixels become saturated. However, if the illumination light is from an LED source or the like and does not contain near-infrared light, the influence of the saturation caused by the visible light illumination does not appear at the point 403b in the image obtained by the near-infrared pixels in Figure 4B .

[0051] On the other hand, when there is a near-infrared illumination source (not shown) on the lower left side of the object (apple) for obtaining a near-infrared image, bright pixels caused by the influence of the near-infrared illumination appear at the point 404b on the lower left of the apple surface in the image obtained by the near-infrared pixels in Figure 4B . In addition, since the visible light pixels are also sensitive to the near-infrared wavelength, the influence 404a caused by the near-infrared component also appears on the lower left of the apple surface in the visible light image. As described above, there are cases where the pixels become saturated not by only near-infrared light or visible light, but by a combination of both the near-infrared component and the visible light component entering the visible light pixels.

[0052] Next, refer to Figures 5A to 6C to describe the saturation processing in this case.

[0053] Figure 5A is a diagram showing the relationship with Figure 4A and Figure 4B obtained from the pixels shown in Fig. 2.Schematic diagram of the output level of the pixel (unsaturated pixel) corresponding to point A in. The horizontal axis represents red (R), green (G), blue (B), and near-infrared (IR) pixels, and the vertical axis represents the output signal level of the pixel components. Reference numeral 501 in the drawing represents the full-scale level (65535 LSB) of 16-bit signal processing, 502 represents the pixel saturation level of the imager, and 503 represents the output signal level of the near-infrared pixel near point A. Reference numerals 504, 505, and 506 represent the outputs of red (R), green (G), and blue (B) pixels respectively. Since point A is on the fruit surface of the apple, the level of the red component of visible light is high. As already referred to Figure 3 As described, the visible light image signals of red (R), green (G), and blue (B) each include a near-infrared component overlapping with each visible light component, and the near-infrared component of each color is generally close to the amount of the near-infrared component obtained by the adjacent near-infrared pixel.

[0054] Figure 5B Shows a schematic diagram of the signal level obtained by subtracting the near-infrared component of the adjacent near-infrared pixel from each signal component of the pixel in Figure 5A Reference numerals 604, 605, and 606 represent the values obtained by subtracting the near-infrared component 503 of the adjacent near-infrared pixel from the respective visible light image signals 504, 505, and 506 of red (R), green (G), and blue (B).

[0055] Figure 5C Shows a schematic diagram of the signal level after white balance processing. In the example shown here, Figure 5B The signals of each color in are multiplied by white balance coefficients such that red (R), green (G), and blue (B) are multiplied by 1.5 times, 1 time, and 2 times respectively, so that white appears natural according to the illumination used when capturing the image. Reference numerals 704, 705, and 706 represent the values obtained by multiplying the visible light image signals 604, 605, and 606 of red (R), green (G), and blue (B) by the white balance coefficients after subtracting the near-infrared component.

[0056] As Figure 5C Shown in, the color information is correctly retained for unsaturated pixels. However, problems occur when the same processing is performed on saturated pixels. Hereinafter, refer to Figures 6A to 6C To explain this problem.

[0057] Figure 6A Is a diagram showing the relationship with obtained by the pixels shown in Figure 2 Figure 4A And Figure 4BSchematic diagram of the output level of the pixel (saturated pixel) corresponding to point B in []. The horizontal axis represents red (R), green (G), blue (B), and near-infrared (IR) pixels, and the vertical axis represents the output signal level of the pixel components. Reference numeral 501 in the drawing represents the full-scale level (65535 LSB) of 16-bit signal processing, 502 represents the pixel saturation level of the imager, and 503 represents the output signal level of the near-infrared pixel.

[0058] Reference numerals 804 and 805 respectively represent the expected outputs of the red (R) and green (G) pixels if the pixels are not saturated, and 806 represents the output of the blue (B) pixel. Since the outputs 804 and 805 exceed the pixel saturation level 502, the actual output levels of the red (R) and green (G) pixels become equal to the saturation level 502.

[0059] Figure 6B is a signal level obtained by subtracting the near-infrared component of the adjacent near-infrared pixel from each signal component of the pixel in [] Figure 6A and is a schematic diagram similar to []. Reference numerals 904, 905, and 906 represent the values obtained by subtracting the near-infrared component 503 of the adjacent near-infrared pixel from each visible light image signal of red (R), green (G), and blue (B). Figure 5B

[0060] Similar to [] Figure 5C Similarly, Figure 6C shows a schematic diagram of the signal level after white balance processing. The content of the white balance processing is the same as that described above. In the example shown here, Figure 6B the signals of each color in [] are multiplied by white balance coefficients such that red (R), green (G), and blue (B) are multiplied by 1.5 times, 1 time, and 2 times respectively. By multiplying the color signals 904, 905, and 906 by 1.5 times, 1 time, and 2 times respectively, the signals after white balance processing represented as 1004, 1005, and 1006 are obtained.

[0061] The dotted lines represented as 1007 and 1008 represent the expected outputs after white balance processing corresponding to 804 and 805 in [] when the pixels are not saturated. Figure 6A

[0062] As shown, because of pixel saturation, the signal output that should have the color balance indicated by the outputs 1007, 1008, and 1006 actually results in a different color balance indicated by the outputs 1004, 1005, and 1006. In the case of the example shown in [] Figure 6C saturation results in a color close to purple instead of white that should appear when not saturated. As confirmed above, saturation may cause false colors and color noise and deteriorate the image quality.

[0063] The object of this embodiment is to suppress such false colors and color noises caused by pixel saturation, and to implement an image processing apparatus that allows for obtaining good visible light images and near-infrared images.

[0064] Image processing operations

[0065] Now, refer to Figure 1 and use Figure 7 FIGs. 8 to 9 to describe the operations in the first embodiment. Figure 7 FIG. 8 is a flowchart for generating a color image signal from the output signal of visible light pixels in the first embodiment. FIGS. 8 and 9 are illustrative diagrams showing signal levels to be used for explaining the operations.

[0066] The process of generating a color image signal from the output signal of visible light pixels starts at step S1101. This process can start at any time, for example, immediately after the imager 203 has obtained the image signals of visible light pixels and near-infrared pixels.

[0067] At step S1102, the near-infrared image signal input unit 105 acquires the pixel signal values of near-infrared pixels, and the visible light image signal input unit 106 acquires the pixel signal values of visible light pixels.

[0068] At step S1103, the near-infrared component subtraction unit 109 subtracts the pixel values of adjacent near-infrared pixels from each pixel value of the visible light image signal. Thereafter, at step S1104, the white balance processing unit 110 performs white balance processing of multiplying the signals of each color by white balance coefficients.

[0069] Meanwhile, at step S1105, the saturated pixel detection unit 108 compares the output signal of the visible light pixel before subtracting the near-infrared light with a saturation determination threshold to determine whether the pixel is saturated.

[0070] At step S1106, the process is selected according to whether the output value of the visible light pixel is greater than the saturation determination threshold. This process selection is determined for each pixel. If the output value of the visible light pixel is not greater than the saturation determination threshold, that is, if the pixel is not saturated (S1106: No), the process proceeds to step S1107, where the saturation processing application unit 114 outputs the output signal of the visible light pixel after white balance processing as it is without applying saturation processing.

[0071] On the other hand, if at step S1106, the output value of the visible light pixel is greater than the saturation determination threshold, that is, if the pixel is saturated (S1106: Yes), the process proceeds to step S1110.

[0072] In step S1109, the near-infrared level determination unit 107 determines the level of the adjacent near-infrared image signal of the visible light pixels, that is, compares it with a level determination threshold. A control signal based on the level determination result is input to the saturation processing switching unit 113.

[0073] In step S1110, the saturation processing switching unit 113 selects a saturation processing according to the determination result in step S1109. If the IR pixel signal is at a low level - that is, not greater than the level determination threshold, the saturation processing switching unit 113 selects the first saturation processing (clipping) performed by the first saturation processing unit 111. On the other hand, if the IR pixel signal is at a high level - that is, greater than the level determination threshold, the second saturation processing (interpolation) performed by the second saturation processing unit 112 is selected.

[0074] The first saturation processing (clipping) performed by the first saturation processing unit 111 at step S1111 is described below. At step S1111, the first saturation processing unit 111 clips the R and B levels to the same level to match the pixel value of G in the visible light pixel signal, thereby converting the color to white. When the level of the near-infrared light is low, it is possible to cause saturation due to visible light. In this case, it is preferable to clip to the white level to eliminate the color.

[0075] Figure 8A It is a schematic diagram showing the level of the image signal obtained at step S1102 when the near-infrared level is not greater than the threshold when saturation occurs. The horizontal axis represents the red (R), green (G), blue (B), and near-infrared (IR) pixels, and the vertical axis represents the output signal level of the pixel components.

[0076] Reference numeral 1208 represents the level determination threshold for determining the near-infrared level. In Figure 8A the example, the output signal level 1207 of the near-infrared pixel is lower than the level determination threshold level 1208.

[0077] Reference numerals 1204 and 1205 respectively represent the expected outputs of the red (R) and green (G) pixels if the pixels are not saturated, and 1206 represents the output of the blue (B) pixel. Since the outputs 1204 and 1205 exceed the pixel saturation level 502, the actual output levels of the red (R) and green (G) pixels become equal to the saturation level 502.

[0078] Figure 8B is obtained by from Figure 8AThe signal components of each pixel in are subtracted from the near-infrared components of adjacent near-infrared pixels - that is, a schematic diagram of the signal level after the processing in step S1103. Reference numerals 1304, 1305, and 1306 represent values obtained by subtracting the near-infrared component 1207 of adjacent near-infrared pixels from the visible light image signals of each color of red (R), green (G), and blue (B).

[0079] Figure 8C shows a schematic diagram of the signal level after white balance processing and the signal level after clipping processing applied similarly to Figure 6C . In the white balance processing in step S1104, Figure 8B the signals of each color - red (R), green (G), and blue (B) in are multiplied by white balance coefficients, for example, multiplied by 1.5 times, 1 time, and 2 times respectively. The dotted lines represented as 1404, 1405, and 1406 indicate the signal levels after white balance processing. In this example, the white balance coefficient of green (G) is 1, so that the output 1305 is equal to the output 1405.

[0080] The image after white balance processing has different colors due to the varying levels of color pixels. Since the output from the pixels saturates, the original color information is lost, making it possible for the color after white balance processing to be a false color. Since the near-infrared component of the pixel is below the threshold, it is unlikely that saturation is caused by high-intensity near-infrared light, that is, it is likely that saturation is caused by visible light. Therefore, it can be determined that it is more appropriate to saturate the pixel through clipping processing to show white rather than to assign a color to the pixel. In the clipping operation (first saturation processing) in step S1111, as Figure 8C shown in, the green color signal with the lowest signal level among the color signals of the pixel after white balance processing is determined as the clipping level 1309, and the signal levels of red and blue are clipped to this level.

[0081] As described above, in this embodiment, when there is pixel saturation (S1106: yes) and the level of near-infrared light is low (S1110: no), the clipping operation performed by the first saturation processing unit 111 is selected and applied. Therefore, the target pixel is replaced with white image information without color.

[0082] Next, the second saturation process (interpolation) performed by the second saturation processing unit 112 at step S1112 will be described. The second saturation processing unit 112 generates an interpolation signal from the pixels surrounding the saturated pixel and replaces the signal of the saturated pixel therewith. This is because saturation is likely to occur due to the near-infrared component, and most of the pixel signals of the original visible light are lost due to saturation. The pixels surrounding the saturated pixel may be defined as the pixels within a predetermined distance from the saturated pixel (e.g., within three pixels), or may be defined as a predetermined number of pixels starting from the saturated pixel.

[0083] Figure 9A is a schematic diagram showing the levels of the pixel signals obtained at step S1102 when saturation occurs and the near-infrared level is greater than the threshold. The horizontal axis represents the red (R), green (G), blue (B), and near-infrared (IR) pixels, and the vertical axis represents the output signal levels of the pixel components.

[0084] Reference numeral 1208 represents the level determination threshold for determining the near-infrared level. At Figure 9A in the example, the output signal level 1507 of the near-infrared pixel is higher than the determined threshold level 1208.

[0085] Reference numerals 1504 and 1505 respectively represent the expected outputs of the red (R) and green (G) pixels if the pixels are not saturated, and 1506 represents the output of the blue (B) pixel. Since the outputs 1504 and 1505 exceed the pixel saturation level 502 of the imager, the actual output levels of the red (R) and green (G) pixels become equal to the saturation level 502.

[0086] Figure 9B shows a schematic diagram of the signal levels obtained by subtracting the near-infrared component of the neighboring near-infrared pixel from the signal components of the respective pixels of Figure 9A - that is, after the processing at step S1103. Reference numerals 1604, 1605, and 1606 represent the values obtained by subtracting the near-infrared component 1507 of the neighboring near-infrared pixel from the visible light image signals of the respective colors of red (R), green (G), and blue (B).

[0087] Figure 9C shows a schematic diagram of the signal levels after the white balance processing and the signal levels after the interpolation applied similarly to Figure 6C In the white balance processing at step S1104, Figure 9BThe signals of each color - red (R), green (G), and blue (B) in are multiplied by white balance coefficients. For example, they are multiplied by 1.5 times, 1 time, and 2 times respectively. The dashed lines represented as 1704, 1705, and 1706 indicate the signal levels after white balance processing. In this example, the white balance coefficient of green (G) is 1, such that output 1605 is equal to output 1705.

[0088] The image after white balance processing has different colors due to the varying levels of color pixels. Since the output from the pixels saturates, the original color information is lost, making it possible for the colors after white balance processing to be pseudo - colors. Since the near - infrared component of the pixels is higher than the threshold, it is possible that the saturation is caused by high - intensity near - infrared light and most of the visible light information is lost. Therefore, it can be determined that the color information of these pixel components should not be adopted. Since the saturation is not caused by high - intensity visible light, it is also inappropriate to make the pixels saturated through clipping processing to display white. In the interpolation processing (second saturation processing) at step S1112, the color information of the pixels is replaced with the color information obtained by interpolating the information of multiple adjacent unsaturated pixels of the same color. Reference numerals 1707, 1708, and 1709 represent the corresponding image signals of the red (R), green (G), and blue (B) pixels replaced by the interpolation processing at step S1112.

[0089] As described above, in this embodiment, when there is pixel saturation (S1106: yes) and the level of near - infrared light is high (S1110: yes), the interpolation processing performed by the second saturation processing unit 112 is selected and applied. When the near - infrared component is higher than the threshold, it is possible that visible light does not cause pixel saturation. Therefore, a more appropriate image signal can be obtained by using a color that matches the surrounding colors instead of eliminating the color information through clipping processing and replacing the color with white.

[0090] As described above, the saturation processing is selected according to the level of near - infrared light to suppress pseudo - colors and achieve simultaneous acquisition of high - quality visible light images and near - infrared images.

[0091] Second Embodiment

[0092] Overall Configuration

[0093] Figure 10 The configuration example of the image processing apparatus 1800 according to the second embodiment is illustrated. Similar to the first embodiment, the image processing apparatus 1800 is configured with an imaging unit 101 and an image processor 1804. In the following description, elements similar to those of the first embodiment are given the same reference numerals and will not be described again.

[0094] The image processor 1804 is different from the image processor of the first embodiment in that it includes a saturated pixel flag setting unit 1808 instead of the saturated pixel detection unit 108, and it further includes a surrounding pixel information acquisition unit 1816 and a saturated pixel flag detection unit 1815. The processing contents of the second saturation processing unit 1812 and the saturation processing application unit 1817 are also different from those of the second saturation processing unit 112 and the saturation processing application unit 114 in the first embodiment.

[0095] The saturated pixel flag setting unit 1808 performs processing to detect saturated pixels from the visible light image signal input to the visible light image signal input unit 106, and when it detects a saturated pixel, sets a saturation flag indicating that the pixel is saturated for the visible light image signal. The surrounding pixel information acquisition unit 1816 acquires color information from the pixels surrounding the target pixel (saturated pixel) and outputs it to the second saturation processing unit 1812. The second saturation processing unit 1812 will be described in detail later. The saturated pixel flag detection unit 1815 determines whether the target pixel is saturated based on the saturated pixel flag and outputs the detection result to the saturation processing application unit 1817. If the pixel is saturated, the saturation processing application unit 1817 applies the saturation processing selected by the saturation processing switching unit 113, and if the pixel is not saturated, outputs the pixel value of the visible light image after white balance processing as it is.

[0096] Image processing operations

[0097] Reference Figure 10 and Figure 11 Describe the operation according to the second embodiment. Figure 11 is a flowchart for generating a color image signal from the output signal of visible light pixels in the second embodiment.

[0098] The process for generating a color image signal from the output signal of visible light pixels starts at step S1901. This processing can start at any time, for example, immediately after the imager 203 obtains the image signals of visible light pixels and near-infrared pixels.

[0099] At step S1902, the near-infrared image signal input unit 105 acquires the pixel signal value of the near-infrared pixels, and the visible light image signal input unit 106 acquires the pixel signal value of the visible light pixels.

[0100] At step S1903, the saturated pixel flag setting unit 1808 compares the output signal of the visible light pixel before subtracting the near-infrared light with a saturation determination threshold to determine whether the pixel is saturated. If the output signal of the visible light pixel is higher than the saturation determination threshold, that is, if it is determined that the pixel is saturated (S1904: Yes), the process proceeds to step S1905, where the saturated pixel flag setting unit 1808 sets 1 as the saturation flag of the pixel. On the other hand, if the output value of the visible light pixel is not greater than the saturation determination threshold, that is, if it is determined that the pixel is not saturated (S1904: No), the process proceeds to step S1906, where the saturated pixel flag setting unit 1808 sets 0 as the saturation flag of the pixel. Here, although 1 and 0 of the saturation flag indicate saturation and non-saturation respectively, any value can be set as the flag.

[0101] At step S1907, the near-infrared component subtraction unit 109 subtracts the near-infrared image signal obtained from adjacent near-infrared pixels from each pixel value of the visible light image signal. Thereafter, at step S1908, the white balance processing unit 110 performs white balance processing of multiplying the signal of each color by a white balance coefficient.

[0102] At step S1909, processing is selected according to whether the saturation flag of the visible light pixel is 1 - that is, whether the visible light pixel is saturated. This processing selection is determined for each pixel. If the saturation flag is 0, that is, if the pixel is not saturated (S1909: No), the process proceeds to step S1910, where the saturation processing application unit 114 outputs, as it is, the output signal of the visible light pixel after white balance processing without applying saturation processing.

[0103] On the other hand, if the saturation flag is 1 at step S1909, that is, if the pixel is saturated (S1909: Yes), the process proceeds to step S1912. At step S1912, the near-infrared level determination unit 107 determines the level of the near-infrared image signal of adjacent near-infrared pixels of the visible light pixel, that is, compares it with a level determination threshold. A control signal based on the level determination result is input to the saturation processing switching unit 113. At step S1913, the saturation processing switching unit 113 selects a saturation processing according to the determination result at step S1912. If the IR pixel signal is at a low level - that is, not greater than the determination threshold, the saturation processing switching unit 113 selects the first saturation processing (clipping) performed by the first saturation processing unit 111. On the other hand, if the IR pixel signal is at a high level - that is, greater than the determination threshold, the second saturation processing (color interpolation) performed by the second saturation processing unit 1812 is selected.

[0104] The first saturation process (clipping) performed by the first saturation processing unit 111 at step S1914 is the same as the first saturation process of the first embodiment. That is, the first saturation processing unit 111 clips the R and B levels to the same level to match the pixel value of G in the visible light pixel signal, thereby converting the color to white.

[0105] Now, the second saturation process (color interpolation) performed by the second saturation processing unit 1812 at step S1915 will be described. The second saturation processing unit 1812 detects the chromaticity of unsaturated pixels around a target pixel (saturated pixel) acquired by the surrounding pixel information acquisition unit 1816 to generate correction information and corrects the pixel value of the saturated pixel. The correction process may be a process of correcting the pixel value of the saturated pixel to achieve the same chromaticity as the chromaticity of the surrounding unsaturated pixels. When the near-infrared component near the saturated pixel is at a high level, it is possible that the saturation is caused by the near-infrared component. In this case, it is more appropriate to correct the color based on the color information of the surrounding unsaturated pixels rather than making the pixel saturated by the clipping process to make it appear white.

[0106] As described above, the saturation process is selected according to the level of the near-infrared light to suppress false colors and achieve simultaneous acquisition of high-quality visible light images and near-infrared images.

[0107] The main differences between this embodiment and the first embodiment lie in the content of the second saturation process and the method of sending the detection result of the saturated pixel to the saturation process application unit 114. These modifications do not need to be applied to the first embodiment in this combination. One of these processes may be the same as the process of the first embodiment.

[0108] Other Embodiments

[0109] The content of the saturation process to be applied is not limited to the processes described in the first embodiment and the second embodiment. The second saturation process applied when the level of the near-infrared light is high may be different from the above process as long as the process determines the pixel value of the saturated pixel based on the pixel values of the neighboring pixels of the saturated pixel. For example, the color information of the saturated pixel may be estimated from the pixel values of the pixels around the saturated pixel using a pre-configured database and replaced with the estimated color information. Alternatively, the color information of the saturated pixel may be replaced with the color information obtained by inputting the pixel values of the pixels around the saturated pixel into a machine learning model designed to estimate the color information of the central pixel from the pixel values of the surrounding pixels.

[0110] Although the near-infrared component subtraction unit 109 is used in the above-described embodiments to subtract the output signal of the infrared pixel from the output signal of the visible light pixel, this process may be omitted. In this case, false colors can also be suppressed by switching the saturation process according to which of the near-infrared component and the visible light component causes saturation.

[0111] One or more embodiments of the present invention can also be implemented by a computer of a system or apparatus that reads and executes computer-executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be more fully referred to as a "non-transitory computer-readable storage medium") to perform the functions of one or more of the above-described embodiments and / or includes one or more circuits (e.g., an application specific integrated circuit (ASIC)) for performing the functions of one or more of the above-described embodiments, and by a method performed by a computer of the system or apparatus that reads and executes computer-executable instructions from a storage medium, for example, to perform the functions of one or more of the above-described embodiments and / or control one or more circuits to perform the functions of one or more of the above-described embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of individual computers or individual processors to read and execute the computer-executable instructions. The computer-executable instructions may be provided to the computer from a network or a storage medium, for example. The storage medium may include, for example, one or more of a hard disk, a random access memory (RAM), a read only memory (ROM), a storage device of a distributed computing system, an optical disc (such as a compact disc (CD), a digital versatile disc (DVD), or a Blu-ray disc (BD) TM ), a flash device, a memory card, etc.

[0112] Other embodiments

[0113] Embodiments of the present invention can also be implemented by a method in which software (a program) that performs the functions of the above-described embodiments is provided to a system or apparatus via a network or various storage media, and the computer or the central processing unit (CPU) or the microprocessing unit (MPU) of the system or apparatus reads and executes the program.

[0114] The present disclosure allows for obtaining a high-quality image with fewer false colors from an image including saturated pixels.

[0115] Although the present invention has been described with reference to exemplary embodiments, it is to be understood that the present invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be given the broadest interpretation to encompass all such modifications as well as equivalent structures and functions.

Claims

1. An image processing apparatus, characterized in that, Comprising: An input unit that receives an image signal of visible light pixels and an image signal of near-infrared pixels output from a sensor including visible light pixels and near-infrared pixels; A determination unit that determines whether the output signal of the near-infrared pixels is higher than a threshold; A detector that detects saturated visible light pixels, the output value of the saturated visible light pixels being higher than a saturation threshold; and A switching unit that switches the saturation processing to be applied to the saturated visible light pixels based on the determination result of the determination unit, wherein, when the output signal of the near-infrared pixels is lower than the threshold, the switching unit selects a first saturation processing in which the pixel value of the saturated visible light pixels of each color is replaced with a preset value or a value determined based on the pixel value of a visible light pixel of one color within the same pixel group as the saturated visible light pixels, and wherein, when the output signal of the near-infrared pixels is higher than the threshold, the switching unit selects a second saturation processing in which the pixel value of the saturated visible light pixels is determined based on the pixel value of the unsaturated visible light pixels near the saturated visible light pixels for color matching between the saturated visible light pixels and the unsaturated visible light pixels.

2. The image processing apparatus according to claim 1, wherein, the sensor includes a plurality of groups, the plurality of groups including first visible light pixels, second visible light pixels, and third visible light pixels, and the first saturation processing is a processing of replacing the pixel value of the saturated visible light pixels with the pixel value of one of the visible light pixels within the same group, where the pixel values are the same.

3. The image processing apparatus according to claim 1, wherein, the sensor includes a plurality of groups, the plurality of groups including first visible light pixels, second visible light pixels, and third visible light pixels, and wherein, the second saturation processing is a processing in which, based on the pixel value of each unsaturated visible light pixel in another group near the group including the saturated visible light pixels among the plurality of groups, the pixel value of each visible light pixel included in the one group is determined.

4. The image processing apparatus according to claim 1, wherein The second saturation processing is a processing of correcting the pixel value of the saturated visible light pixels to obtain the same chromaticity as the chromaticity of the unsaturated visible light pixels near them.

5. The image processing apparatus according to claim 1, further comprising: A white balance processing unit that performs white balance processing of multiplying the pixel value of the image signal of the visible light pixels by a gain according to color, and A saturation processing application unit that applies the saturation processing selected by the switching unit to the image signal after white balance processing when the visible light pixels are saturated.

6. The image processing apparatus according to claim 5, wherein, When the visible light pixels are not saturated, the saturation processing application unit outputs the image signal after white balance processing without modification.

7. The image processing apparatus according to claim 6, wherein, the detection result of the detector is input to the saturation processing application unit, and the saturation processing application unit determines whether the saturation processing is to be applied to the visible light pixels based on the detection result.

8. The image processing apparatus according to claim 7, wherein, The detector adds information indicating whether a visible light pixel is saturated to the image signal of the visible light pixel, and the saturation processing application unit determines whether saturation processing is to be applied to the visible light pixel based on the information added to the image signal of the visible light pixel.

9. The image processing apparatus according to claim 5, further comprising a subtraction unit that subtracts the output value of a near-infrared pixel near the visible light pixel from the output signal of the visible light pixel, wherein white balance processing is performed on the output signal of the visible light pixel from which the output value of the near-infrared pixel has been subtracted.

10. An imaging device, characterized in that, Comprising: a sensor including visible light pixels and near-infrared pixels, and the image processing apparatus according to any one of claims 1 to 9.

11. The imaging device according to claim 10, wherein, In the sensor, the visible light pixels and the near-infrared pixels are arranged on the same plane.

12. The imaging apparatus according to claim 10, further comprising a band-stop filter that removes, from incident light on the sensor, a wavelength band in the near-infrared range to which the visible light pixels and the near-infrared pixels have different sensitivities.

13. An image processing method, characterized in that, Comprising: an input step of inputting an image signal of visible light pixels and an image signal of near-infrared pixels output from a sensor including visible light pixels and near-infrared pixels; a determination step of determining whether the output signal of the near-infrared pixel is higher than a threshold; a detection step of detecting saturated visible light pixels, the output values of which are higher than a saturation threshold; and a switching step of switching the saturation processing to be applied to the saturated visible light pixels based on the determination result of the determination step, wherein, when the output signal of the near-infrared pixel is lower than the threshold, the switching step selects a first saturation processing in which the pixel value of each color of the saturated visible light pixel is replaced with a preset value or a value determined based on the pixel value of a visible light pixel of one color within the same pixel group as the saturated visible light pixel, and wherein, when the output signal of the near-infrared pixel is higher than the threshold, the switching step selects a second saturation processing in which the pixel value of the saturated visible light pixel is determined based on the pixel value of an unsaturated visible light pixel near the saturated visible light pixel for color matching between the saturated visible light pixel and the unsaturated visible light pixel.

14. A computer-readable medium storing a program, characterized in that, Wherein, the program causes a computer to execute an input step of inputting an image signal of visible light pixels and an image signal of near-infrared pixels output from a sensor including visible light pixels and near-infrared pixels; a determination step of determining whether the output signal of the near-infrared pixel is higher than a threshold; a detection step of detecting saturated visible light pixels, the output values of which are higher than a saturation threshold; and a switching step of switching the saturation processing to be applied to the saturated visible light pixels based on the determination result of the determination step, wherein, when the output signal of the near-infrared pixel is lower than the threshold, the switching step selects a first saturation processing in which the pixel value of each color of the saturated visible light pixel is replaced with a preset value or a value determined based on the pixel value of a visible light pixel of one color within the same pixel group as the saturated visible light pixel, and Wherein, when the output signal of the near-infrared pixel is higher than the threshold, the switching step selects a second saturation process, in which the pixel value of the saturated visible light pixel is determined based on the pixel values of the unsaturated visible light pixels near the saturated visible light pixel for color matching between the saturated visible light pixel and the unsaturated visible light pixel.

Images