IR cut filter switch control

Abstract

The present invention relates to IR cut filter switching control. A method of performing better controlled switching between day mode imaging and night mode imaging in a camera, wherein those illumination sources that contribute to ambient light in day mode are taken into account when determining visible light during night mode. Characteristic values ​​of these illumination sources are mixed with certain levels of IR light to simulate the presence of IR illuminators, and these characteristic values ​​are compared with corresponding values ​​taken from the color components of the ambient light in night mode to determine the IR ratio and, thereby, the amount of visible light.

Description

Technical Field

[0001] The present invention relates to a method for controlling an infrared cut-off (IR-cut) filter to switch between a night mode and a day mode, which is performed in a camera. Background Art

[0002] Today, digital cameras are used in a variety of ways, from more traditional purposes such as video surveillance to deter crime to other areas such as monitoring traffic conditions or improving the operation of logistics centers.

[0003] Image sensors used in such cameras typically have a spectral response with a non-negligible component in the infrared (IR) spectrum. To illustrate this, Figure 2 Shown are the relative sensitivities of three different color channels of a typical image sensor used in surveillance cameras.

[0004] In low light conditions, the IR component of the ambient light can be used to provide useful additional information about the imaged scene. On the other hand, this characteristic also makes color imaging more difficult, since each color channel of the camera will also have a spectral response in the IR, meaning that the blue, green, and red channels will all include IR radiation, even though it is actually longer wavelength than red. As a result, during daytime imaging, any IR component reaching the image sensor will distort the color balance in the image and may also saturate the image sensor. Furthermore, since the camera will detect all incident radiation, parameters such as exposure settings may be affected, which is not conducive to good image quality.

[0005] A known method of maintaining the advantages of improved low-light imaging while suppressing the less favorable effects of distorted colors and possible saturation is to add a removable infrared cut-off (IR-cut) filter in the beam path in front of the image sensor during day mode operation. The IR-cut filter blocks the IR portion of the incident radiation from reaching the sensor. The IR-cut filter (sometimes referred to as an IR filter) is used in daylight conditions to enable the acquisition of color images.

[0006] In day mode, with the IR cut filter in place, the pixels of the image sensor will be able to detect incident light as charges on individual photodetectors, each of which is provided with a color filter array to receive primarily red, primarily green, or primarily blue radiation, thereby achieving color separation.

[0007] In night mode, i.e. in low light conditions, the IR cut filter is removed. This means that the intensity originating from the IR part of the spectrum will be increased, and this increase serves to improve the image quality. The camera may also be provided with an IR illumination unit, providing IR light to the scene and improving the image quality of images captured in low light conditions.

[0008] It should be noted again that when the IR cut filter is removed and no longer blocks IR light from reaching the sensor, it will no longer be possible to provide a color image. As mentioned, without the IR cut filter, the IR radiation will increase the intensity of each color channel because each color channel has a response in the IR portion of the spectrum. This will distort the color information by adding an unknown factor in each color channel. Therefore, instead of performing color separation during night mode operation, for each pixel, only the total intensity of the incident radiation is recorded, and the resulting image will generally appear as a monochrome intensity image.

[0009] The camera will decide whether night mode or day mode is appropriate based on the amount of ambient light received by the sensor. However, when switching from night mode to day mode, the camera will often find that after the switch, when the IR cut filter blocks the IR portion of the ambient light, the ambient light level is actually too low, prompting a switch back to night mode. This can result in inappropriate switching back and forth between night mode and day mode, resulting in noticeable image flicker and unnecessary wear on the mechanism that moves the IR filter.

[0010] Since this repeated switching back and forth is obviously disadvantageous, an additional estimation can be performed by the camera before switching, in which the camera attempts to determine the amount of visible light, rather than just the total amount of ambient light in the scene, to determine that switching to day mode is desirable. An alternative approach in this paper is to use an auxiliary light sensor (with an IR cut filter) to measure the visible light in the scene. The disadvantage of this option is that it adds additional components to the camera, thereby increasing the cost and complexity of the final product.

[0011] US9386230 discloses a method to solve this problem, wherein the color temperature of various known light sources is compared with the color temperature measured from the image captured by the camera. Switching to day mode is performed only if the total lux level is high enough and using different lux thresholds (depending on whether the identified light source is determined to be daylight or incandescent light with a high IR component, or the measured color temperature indicates that it is a non-IR light source).

[0012] When an IR lighting unit is used to illuminate the scene, this will further increase the complexity of estimating when to perform switching of the IR cut filter. In US9386230, this situation is handled by refraining from performing measurements until the added IR lighting level is significantly reduced and therefore does not contribute to the lux level in any major way.

[0013] The invention relates to a method for determining whether a switch from night mode to day mode is desirable based on an estimation of the IR ratio in an acquired image or video, in particular in the presence of an IR lighting unit. Summary of the invention

[0014] It is an object of the present invention to facilitate switching between day mode and night mode by providing a better estimate of the amount of visible light in the monitored scene during night mode.

[0015] According to a first aspect, there is provided a method of controlling an infrared cut-off (IR-cut) filter of a camera to switch between a day mode and a night mode, performed in a camera, wherein the camera comprises an image sensor having a color filter array, the color filter array being configured to allow the sensor to sense different color components at different areas of the sensor, the method comprising the following steps:

[0016] The image sensor measures the ambient light intensity and the color components of the ambient light.

[0017] In day mode,

[0018] determining a set of illumination sources present in the scene and contributing to the ambient light intensity by comparing the color components to a first set of characteristic values ​​associated with different illumination sources,

[0019] When the ambient light intensity is lower than the first threshold, switch to night mode.

[0020] In night mode,

[0021] determining the IR light proportion of the ambient light intensity by comparing the color component to a second set of characteristic values ​​associated with different IR light proportions,

[0022] determining the visible light intensity based on the determined IR light proportion and the ambient light intensity,

[0023] When the visible light intensity is higher than the second threshold, switch to day mode.

[0024] Wherein, the second set of eigenvalues ​​is obtained by mixing the wavelength spectrum of each illumination source in the determined set of illumination sources with a plurality of predefined intensity levels of added IR light, so that the second set of eigenvalues ​​contains a plurality of eigenvalues ​​for each illumination source, wherein the proportion of IR light for each eigenvalue in the second set includes both the IR light from the illumination source and the intensity level of the added IR light.

[0025] Using this approach, it will be possible to make more informed decisions about when to switch the camera to day mode, and reduce the switching back and forth of the IR filter, which is caused by misjudging the amount of visible light in the scene and switching to day mode at an inappropriate time due to strong IR light in the scene. Since the second set of feature values ​​contains simulated values ​​of the light sources in the scene with added IR light, it will be possible to estimate whether the measured ambient light intensity is mainly from IR, mainly visible light, or somewhere in between.

[0026] By limiting the second set of eigenvalues ​​to only those values ​​associated with the set of light sources, and only those values ​​that contribute to the light in the scene, the number of eigenvalues ​​in the second set of eigenvalues ​​will be reduced to a reasonable amount that will, with a high probability, give usable and unambiguous results when compared to the measured values.

[0027] Conversely, in the case where all possible available light sources had been used mixed with some different IR light intensity levels, a graph presenting these values ​​would be too dense to allow for any kind of practical usability, as multiple results would overlap with each other, making the determination of which values ​​were the relevant values ​​to consider very uncertain. Instead, by reducing the light sources to the set of light sources that were determined to contribute to the light in the scene in day mode, a much less dense graph of characteristic values ​​is provided in night mode to decide which light sources would be possible, and more importantly, what levels of IR light are present in the scene.

[0028] It may be noted that throughout the specification, both the terms IR and NIR are used, as this is standard practice in imaging. However, in practice, due to the spectral response of the sensor combined with the characteristics of the color filter array, only the NIR portion of the IR light is relevant to the different aspects of the present invention. In the case where a camera is available that performs imaging both in the visible part of the spectrum and in other parts of the infrared region (i.e. outside the NIR region), the method described herein will be equally applicable. However, with current technology, imaging outside the visible and NIR spectral regions is accomplished by other types of imaging devices, that is, thermal imaging or thermal cameras using, for example, microbolometers, and in such cameras, IR cut-off filters are generally irrelevant to use, and therefore the present invention is not used in these types of cameras. Therefore, when the term "infrared" or "IR" is used herein, it will be interpreted by a person skilled in the art as radiation in the near infrared (NIR) region of the spectrum.

[0029] The IR cut filter is arranged to block IR light from reaching the image sensor of the camera in day mode and to allow IR light to reach the image sensor in night mode.

[0030] The color filter array may be configured to allow the sensor to measure three different color components, which may be red, blue, and green, a common setup where standard components are available.

[0031] The comparison of the color components with the characteristic values ​​may include obtaining a first ratio of the first color component to one or more of the other color components, and a second ratio of the second color component to one or more of the other color components, and comparing the obtained ratios with corresponding characteristic values ​​of the ratios.

[0032] This would give the possibility to consider eg three color components using only two eigenvalues, which means that the comparison would be easier to perform.

[0033] The step of determining a set of illumination sources may include looking up in a first matrix table, or accessing a first chart containing characteristic values ​​associated with different illumination sources, and the step of determining the IR light ratio may include looking up in a second matrix table, or accessing a second chart containing characteristic values ​​associated with different IR light ratios.

[0034] Both options will provide efficient ways to store and access corresponding feature values ​​for comparison.

[0035] The method can be repeatedly executed at predefined time intervals. In this way, changes in the lighting environment will be taken into account and the latest information is always available.

[0036] The first threshold may be lower than the second threshold. In this way, switching back and forth is avoided by introducing a safety margin so that when a switch between modes has been performed, the camera will not switch back immediately even if there is a small oscillation in the ambient light intensity. This in turn will improve image quality and user experience by avoiding flickering images.

[0037] The plurality of predefined levels of IR intensity light preferably comprises at least three different intensity levels. In this way it will be possible to more accurately determine which IR ratio corresponds to the ambient light in the described scene.

[0038] The plurality of predefined levels of IR light intensity preferably contain IR light of a wavelength spectrum corresponding to the wavelength spectrum of an IR LED lighting unit mounted near the camera. In this way, the characteristic values ​​used for comparison in night mode will more closely match the measured values.

[0039] According to a second aspect of the present invention, there is provided a camera, comprising

[0040] an image sensor having a color filter array configured to allow the sensor to sense different color components at different areas of the sensor, wherein the image sensor is configured to measure both ambient light intensity and color components of the ambient light,

[0041] IR-cut filter, configured to be controlled by the camera to switch between day mode and night mode,

[0042] The camera is configured as follows:

[0043] In day mode,

[0044] determining a set of illumination sources present in the scene and contributing to the ambient light intensity by comparing the color components to a first set of characteristic values ​​associated with different illumination sources,

[0045] When the ambient light intensity is lower than the first threshold, switch to night mode.

[0046] In night mode,

[0047] determining the IR light proportion of the ambient light intensity by comparing the color component to a second set of characteristic values ​​associated with different IR light proportions,

[0048] determining the visible light intensity based on the determined IR light proportion and the ambient light intensity,

[0049] When the visible light intensity is higher than the second threshold, switch to day mode.

[0050] It is characterized in that the camera is configured to obtain a second set of characteristic values ​​by mixing the wavelength spectrum of each illumination source in a determined set of illumination sources with a plurality of predefined intensity levels of added IR light, so that the second set of characteristic values ​​contains a plurality of characteristic values ​​for each illumination source, wherein the IR light ratio of each characteristic value in the second set includes both the IR light from the illumination source and the intensity level of the added IR light.

[0051] The camera may include an IR LED lighting unit, wherein the plurality of predefined levels of IR light intensity contain IR light of a wavelength spectrum corresponding to the wavelength spectrum of the IR LED lighting unit. The IR LED lighting unit provides better image quality during dark hours, and by matching the predefined intensity levels of IR light to the wavelength spectrum of the LED, the determination of the amount of visible light contained in the ambient light will be more accurate.

[0052] According to a third aspect of the present invention, there is provided a non-transitory computer-readable storage medium having stored thereon instructions for implementing the method when the instructions are executed on a device having processing capabilities.

[0053] Where applicable, the above features of the method according to the first aspect also apply to the second and third aspects. To avoid unnecessary repetitions, reference is made to the above.

[0054] Further scope of application of the present invention will become apparent from the detailed description given below. However, it should be understood that the detailed description and specific examples, although indicating preferred embodiments of the present invention, are given by way of illustration only, as various changes and modifications within the scope of the present invention will become apparent to those skilled in the art from the detailed description.

[0055] Therefore, it should be understood that the present invention is not limited to the specific components of the described devices or the actions of the described methods, because such devices and methods can vary. It should also be understood that the terms used in this article are only for the purpose of describing specific embodiments and are not intended to be limiting. It must be noted that the words "one", "the" and "said" as used in the specification and the appended claims are intended to indicate the presence of one or more elements, unless the context clearly indicates otherwise. Therefore, for example, references to "a unit" or "the unit" may include several devices, etc. In addition, the words "comprise", "include" and similar expressions do not exclude other elements or steps. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The present invention will now be described more fully below with reference to the accompanying drawings, in which currently preferred embodiments of the present invention are shown. However, the present invention may be implemented in many different forms and should not be construed as limited to the embodiments set forth herein; on the contrary, these embodiments are provided for completeness and completeness and to fully convey the scope of the present invention to those skilled in the art.

[0057] Figure 1 A video camera is illustrated.

[0058] Figure 2 The spectral response curve is shown.

[0059] Figure 3 An example of a monitoring scenario is illustrated.

[0060] Figure 4 Different light source spectra are shown.

[0061] Figure 5 Different light source spectra with added IR are illustrated.

[0062] Figure 6 is a graph showing characteristic values ​​of a plurality of light sources.

[0063] Figure 7 Shown are the spectral response curves of the camera with the IR cut filter turned on (in place) and off (removed).

[0064] Figure 8a Shows coverage in Figure 7 Spectral response curve of a D75 light source with the IR cut filter turned off.

[0065] Figure 8b Shows Figure 8a The D75 spectrum is multiplied by Figure 8a Spectral response curve.

[0066] Figure 9a Shown are spectra of a D65 light source with different amounts of added IR.

[0067] Figure 9b is included Figure 9a A plot of several eigenvalues ​​of the spectrum.

[0068] Fig.10 A method for day mode operation is shown.

[0069] Fig.11 A method for night mode operation is shown. DETAILED DESCRIPTION

[0070] Figure 1A surveillance camera 100 is illustrated. The camera has an image sensor 102 and imaging optics 104 which images the monitored scene on the image sensor 102.

[0071] Image sensor 102 includes a plurality of imaging pixels, each of which is sensitive to radiation in a spectral region. Each pixel is equally sensitive in a spectral region ranging from near ultraviolet to near infrared (NIR), and possibly further to the infrared spectral region.

[0072] In order to provide color imaging, a color filter array is arranged in front of the sensor 102, providing each pixel of the sensor with a color filter located in front of it. A common type of color filter array is a Bayer filter, which has one blue filter element, one red filter element, and two green filter elements for each group of four pixels. The Bayer filter is transparent in at least part of the IR region of the spectrum, namely the near infrared (NIR) region. The effect of this fact, combined with the spectral response of the image sensor 102, is that NIR radiation will reach and affect every pixel of the image sensor. As mentioned previously, this Figure 2 is shown in the figure.

[0073] To achieve color imaging, an IR cut filter 106 is therefore placed in front of the sensor. The IR cut filter is inserted during day mode imaging and retracted during night mode imaging.

[0074] The captured images are processed in the image processing pipeline IPP 108 for generating a video stream, and the video stream is then encoded in the image encoding unit 110 according to a coding standard such as H.264 or H.265, and finally the encoded video stream is transmitted from the camera via the network interface 112. The video stream may also be stored locally in a memory provided for this purpose in the camera.

[0075] A central processing unit CPU 114 controls the operation of the camera 100. In reality, the functionality of the CPU may be distributed over several different units (including, for example, a graphics processing unit GPU), but for descriptive purposes only a single unit is illustrated.

[0076] An illumination unit is provided in the form of an IR illuminator 116 to further improve the imaging capabilities of the camera when in night mode. In the illustrated embodiment, the IR illuminator 116 (e.g., in the form of one or more IR LEDs) is integrated with the camera 100, but it can also be provided as a separate illuminator controlled by the camera 100, or as a separate unit. The IR illuminator 116 typically provides illumination using a wavelength interval centered around 850 nanometers or 940 nanometers. The latter interval is invisible to the human eye, while the former interval may be perceived as red light to a bystander.

[0077] Figure 3 The scene 300 imaged by the camera 100 is illustrated, wherein a person 302 is walking toward a row of houses 304. For the present invention, it is of interest to note the lighting situation. Potential light sources in the scene are provided by lamp posts 306, lamps 308 above the entrance of one of the houses 304, and light sources present on the inside of windows 310 of the house 304, such as candles 312, lamps 314, and fairy lights 316 on a Christmas tree. In addition to such artificial light sources, the sun is also an important light source during dawn, daytime, and dusk, and serves as an indirect light source when reflected by the moon 318. Other intermittent light sources, such as fireworks 320 and the headlights of cars 322, may also appear in the scene from time to time.

[0078] In addition to their contribution to increasing the light in the visible spectrum, some light sources in the scene also have spectral components in the IR or NIR spectrum. This is the case with incandescent lamps (ordinary light bulbs with tungsten filaments) present in lamp 314, as well as burning flames such as in lit candles 312, and sunlight.

[0079] As already described, the camera 100 is equipped with an IR illuminator 116, which is typically activated in night mode when the IR cut filter 106 is removed and the camera also uses at least a portion of the near infrared to infrared spectrum to perform imaging. Thus, during night mode, the illumination available to the camera 100 will consist of light from any light source in the scene 300 combined with illumination from the IR illuminator 116, and the light from the light sources in the scene can be in both the visible and IR portions of the spectrum.

[0080] Returning to the key objective of the present invention, namely determining when it is desirable to perform a switch from night mode (without an IR cut filter) to day mode (with an IR cut filter), the IR component present during night mode imaging makes this a difficult task. The high intensity of IR illumination in night mode can easily make the scene appear to be well illuminated, when in reality, the insertion of the IR cut filter would make the acquired image too dark.

[0081] A simple but somewhat impractical approach is to simply insert an IR cut filter and estimate the situation, and then if the light level is too low, retract the filter again, or temporarily turn off any IR illuminators under the camera's control and perform the estimate. However, both of these options are difficult to perform without visible effects in the video stream, and often result in the disturbing flickering effects mentioned in the Background section.

[0082] In order to provide a better and more efficient way of controlling the switching of the IR filter, information about which light sources are active in the scene would be of interest. Light sources can usually be characterized by their spectral composition or color temperature. The spectral composition of the image is in turn linked to the spectral composition of the light source via the spectral response of the image sensor and the effects of optical elements such as lenses and filters used in the camera. In other words, the spectral composition of the image will depend on both the lighting present in the scene and the characteristics of the camera, in particular the spectral response of the image sensor.

[0083] As mentioned earlier, the latter Figure 2 For example, in Figure 2 The typical spectral response of an image sensor is shown in FIG. Figure 2 The graph in shows the spectral response (or spectral sensitivity) of each color (blue 202, green 204, and red 206) channel as a function of wavelength. It is interesting to note in the figure that when the IR cut filter is removed ("off"), that is, in night mode, there is a spectral response in the NIR region regardless of the color channel, and above 850 nanometers, the responses are essentially equal. It is also apparent from the magnitude of the response how the inclusion of IR or NIR can significantly increase the imaging of a scene.

[0084] on the other hand, Figure 4 A diagram 400 showing the spectral composition of a number of different common light sources, where A is a black body radiator, F12 is a standard fluorescent lamp, and D65 is a simulated daylight spectrum, and Figure 5 Another diagram 500 is shown where the spectral content of visible light LEDs is combined with NIR LEDs (850 nanometers) of varying intensities.

[0085] In the context of what is known as white balance, a convenient way to condense the characterization of the spectral composition is to look at the contribution of the different color components, i.e. the sum or integrated amount of intensity within a particular wavelength interval, which corresponds to the color component.

[0086] Typically, the color components are red, green, and blue, but it is possible to use some other color spaces with other color components. Typically, two different ratios between the components are used for representation, such as r / g and b / g or r / (r+b+g) and b / (r+b+g), thereby reducing the complexity of the representation to coordinates in a two-dimensional chart or matrix table, while still maintaining information from all three color components.

[0087] Figure 6 An example of such a graph is shown in which the values ​​of the red component divided by the sum of all color components and the blue component divided by the sum of all color components are plotted for a D75 light source (northern sky daylight) at 602, an LED light source at 604, a fluorescent light source at 606, and a tungsten light source (i.e., a light bulb) at 608. The circles around each value symbolize that fluctuations may occur, and that mapping to a particular light source may be done even if exactly the same characteristic value is not found in measurements from real-world scenes.

[0088] When more than one light source exists in the scene, you can also use Figure 6 In this case, the measured values ​​will, for example, represent a mixture of possibly two or three light sources. In this case, the light source in the diagram with the shortest Euclidean distance to the measured value can be determined to be present. Other more complex methods can also be used, such as first selecting the light source with the shortest Euclidean distance to the measured value, and then also including all those light sources whose Euclidean distance is shorter than three times this shortest distance. Various other methods for selecting which relevant light sources to include are also possible.

[0089] As the inventors have appreciated, while it is fairly easy to estimate the specific light sources contributing to the illumination in a scene in day mode (i.e. when the IR component can be ignored) based on a comparison of known characteristic values ​​of the light sources and the corresponding values ​​in the captured image, it is not straightforward to utilize this approach in night mode, especially in the presence of strong IR or NIR illuminators.

[0090] The inventors made a useful observation, which makes it possible to use the same principle of comparing feature values ​​in night mode, but with an important modification, which allows the night mode analysis to be performed also in the presence of an IR lighting unit.

[0091] First, only those light sources that were determined to be present in the scene, based on the day mode analysis completed just before switching to night mode, are included in the night mode comparison. In this way, there will be a sparser chart to compare the measurements than if all possible light sources were considered.

[0092] Each light source determined to be active in day mode is provided with a set of modified feature values ​​(also referred to herein as a second set of feature values), wherein IR illumination wavelength spectra at certain different intensity levels are added to each corresponding light source spectrum, and the modified feature values ​​are calculated from these modified spectra. The added IR illumination levels represent illumination from an IR illumination unit, such as IR LED illumination unit 116.

[0093] As an option, these modified characteristic values ​​can be determined by first preparing a larger database with measured values ​​for a variety of light sources (with added IR components at different intensity levels) and then selecting suitable values ​​from the database. Currently, such measurements are made for different light sources in the context of preparing data for white balancing purposes, and a similar setup can be used here as well.

[0094] In this setup, the camera is configured to capture certain images of an optical grey target while varying the output of the light source to be measured. Gradually added IR illumination is also introduced. These measurements can be performed with and without the IR cut filter inserted, and the results are then analyzed to find characteristic values ​​for the various optical setups. This setup can also be used to map the spectral response of the camera itself. The camera is then exposed to a narrowband radiation (such as 10 nanometers FWHM) and this narrowband radiation is scanned across the entire spectral range of the camera (with a color filter array in front of it) with and without the IR cut filter in place, and a graph is generated that is consistent with the spectral response of the camera itself. Figure 2 Results are similar to those shown in .

[0095] However, instead of performing this measurement for a variety of different light sources with different levels of added IR intensity, a more convenient procedure is to determine the modified characteristic values ​​from the spectra obtained by calculating the mixture of the individual light source spectra with the IR illumination spectra at different intensity levels, and then perform another calculation to determine how the image sensor will respond to these different spectra if the IR filter is removed.

[0096] As mentioned, the image sensor is sensitive to IR on all color channels, as in Figure 2 In the figure. Figure 7 , the spectral response 700 of the camera is illustrated in both night mode (ie, IR cut filter removed) and day mode (IR cut filter in place). Figure 7 The spectral response in Figure 2 There is some difference between what is shown in FIG. 1 and FIG. 2 because the illustration is from a different camera with a different image sensor and different optics.

[0097] By multiplying the camera's spectral response with each of the generated spectra (one color channel at a time) and then summing or integrating over each color channel, a modified characteristic value for the different IR illumination levels for each light source is determined, as is the IR ratio associated with that particular value, with the IR cut filter in place and with the IR cut filter removed. As previously mentioned, this IR ratio will contain both the added IR illumination and the IR component of the corresponding light source.

[0098] In more detail, each of the generated spectra is calculated in the following way:

[0099] The total amount of light can be calculated by multiplying the spectral response of the IR cut filter with the generated spectrum and determining the sum of all color components (usually R+B+G).

[0100] The amount of visible light is calculated by multiplying the spectral response of the IR cut filter in place with the generated spectrum and determining the sum of all color components.

[0101] The visible light fraction is calculated by dividing the amount of visible light by the total amount of light, and the IR fraction is calculated by subtracting the visible light fraction from 1.

[0102] The feature values ​​(in most cases coordinates in a 2-dimensional graph) associated with the IR score, such as R / (R+B+G) and B / (R+B+G), are determined from the color components calculated with the IR cut filter turned off.

[0103] The basis for this calculation is illustrated in Figure 8 for a single illuminant D75, which corresponds to northern sky daylight with 40% IR, i.e. the intensity of the IR part of the spectrum is 40% of the total intensity of the illuminant. Figure 8a In Figure 1, both the spectral composition of a D75 light source with IR content and the spectral response of a camera without an IR cut filter are illustrated. Figure 8b , the results of multiplication between the D75 spectrum and each of the color channel spectral responses are illustrated.

[0104] exist Figure 9a and Figure 9b In FIG. 1 , another illuminant D65 is illustrated mixed with different levels of IR illumination, which corresponds to average midday light in Western / Northern Europe. Figure 9a , the wavelength spectrum of a D65 light source when mixed with certain levels of added IR light is shown. It can be noted that the y-axis is truncated in this figure for clarity of illustration. In reality, the spectral curve for higher levels of added IR continues to rise to higher amounts.

[0105] As in Figure 7These curves are then multiplied by the spectral response from the sensor with the IR cut filter in the off position, and characteristic values ​​are calculated by first summing or integrating the intensity of each color component and then determining the ratio of the blue component to the sum of all color components and the ratio of the red component to the sum of all color components. The results of these calculations are shown in Figure 9b . A light source containing only IR is also included. It can be noted that in real cases, the added IR lighting level can be selected to be 70%, 80%, 90%, 95%, 97% and 98%.

[0106] exist Figure 1 In the context of the illustrated scenario, the method presented in this paper will work as follows:

[0107] When the camera is in day mode, measurements are taken to determine the ambient light intensity from the captured images, as well as the color components of the ambient light. The color component measurements are used to calculate characteristic values ​​(such as r / (r+b+g) and b / (r+b+g) mentioned), which are compared to a chart or matrix table listing such characteristic values ​​for a variety of different light sources. Light sources having characteristic values ​​similar to the measured image values ​​are determined to be present or active in the scene, that is, they contribute to the light intensity in the scene.

[0108] The determination of the color components can be calculated by simply adding the intensity of each pixel. This operation can be performed for all pixels of the image sensor (i.e., the entire described scene), or it can be performed only for a smaller area of ​​the scene. In the latter example, the smaller area can be an area with specific properties, such as uniform intensity, an area that is considered optically gray, etc.

[0109] Furthermore, instead of performing the addition of pixel values ​​for each color channel of the entire image, the operation can also be performed separately in each sub-region, in which case the resulting color component measurements will have spatial resolution, that is, the values ​​can be different across the image. This option can be used to find any spatially limited light sources (such as car headlights, which do not affect the decision to switch the IR-cut filter). As an example, if such deviating measurements, which can be considered outliers, appear repeatedly over time in some specific areas of the image, these areas can be ignored when determining the color components.

[0110] In a realistic scenario, measurements will be taken continuously throughout the day, such as 15 times per second. It should be noted however that the measurements of most interest to the present invention are those taken close to the point in time at which the camera determines that the ambient light intensity is below the threshold for switching to night mode (i.e. just before the point in time at which the camera determines that the ambient light intensity is below the threshold for switching to night mode). Thus, determining which light sources were active in the scene and contributing to the lighting (albeit at a low level) just before switching to night mode forms the basis of information for determining when to perform a switch from night mode back to day mode by inputting this into a graph with modified feature values ​​used for making determinations in night mode.

[0111] Back to Figure 3 , the artificial light sources present in the scene (and possibly active) are lamp posts 306, lamps 308, candles 312, lamps 314, and fairy lights 316. In addition to these artificial light sources, there may also be contributions from the setting sun, and possibly the moon 318. As mentioned, fireworks 320 and the headlights of cars 322 are intermittent and do not contribute much to the measurement, and can be ignored for now.

[0112] As previously explained, each of the light source spectra of the light sources determined to be present and active in the scene just before switching to night mode will be mixed with a spectrum of IR light of certain different levels. Then, characteristic values ​​will be calculated for all of the combinations of these light source spectra and the added IR light levels, as well as the IR percentage for that particular combination. The added IR light levels typically represent different intensity levels of the light from the IR LED lighting unit 116 at the camera 100, and the spectrum of the added IR light preferably matches the spectrum of the IR LED lighting unit, which is normally known when setting up the camera and its lighting unit.

[0113] As explained previously, in all of these cases, when a portion of the spectrum of a light source is the IR spectrum, that IR spectrum portion will contribute to the overall IR ratio of the light source together with the added IR light intensity level.

[0114] In the night mode graph, the total IR ratio is available for certain different combinations of eigenvalues. In this context, it may be mentioned that in the unlikely case (i.e. more than one combination of light source spectrum plus added IR light intensity level is associated with the same eigenvalue), a practical solution would be to simply associate that point in the graph with the highest IR ratio.

[0115] Since there may be measurement inaccuracies or similar situations that cause the measured characteristic values ​​to not exactly agree with the values ​​in the chart, various methods can be used to determine the most relevant one or more fill points in the chart. As with the day mode chart, a straightforward choice is to use Euclidean distance and select the fill point in the chart that is closest to the measured point. Since in many cases there will be more than one light source active (and all of these light sources will produce certain points in the chart by combining with certain IR levels), it may also be appropriate to select two, three or four closest fill points.

[0116] The amount of available visible light is then determined from the point in the graph associated with the highest percentage of IR light. The amount of visible light is then determined as (1 - "IR ratio associated with point in graph") * "measured ambient light intensity". This amount of visible light is then compared to a second threshold to determine if a switch to day mode should be made. In the event that the amount of visible light is below the second threshold, no switch will be made, that is, the IR cut filter will remain in place, and in the event that the amount of visible light is above the second threshold, the camera will switch to day mode and retract the IR cut filter.

[0117] The first threshold is normally lower than the second threshold, in some cases 2 to 3 times lower, in other cases the second threshold may be set to 4 times the first threshold to provide a safety margin and avoid switching back and forth between day and night modes. This measure may be seen as a way of introducing a hysteresis function. It may be noted that by using the method proposed in this article, in many cases it will be possible to reduce the distance between the first threshold and the second threshold, thereby allowing a more accurate determination requiring less safety margin, which in turn means that better image quality may be provided by allowing the use of night mode and day mode in appropriate circumstances. This is because the difference between the thresholds tends to be set based in part on the estimated amount of IR in the scene.

[0118] The method described in this paper is also Fig.10 is illustrated for day mode operation and in Fig.11 The night mode operation is illustrated in Fig.10 As illustrated in FIG. 1 , during day mode operation, ie, when the IR cut filter is in place, in step 1002 , the ambient light intensity and the color components of the ambient light reaching the image sensor of the camera are measured.

[0119] In step 1004, a set of light sources contributing to the ambient light is determined from the comparison of the color components with the characteristic values ​​of the light sources. In step 1006, the ambient light intensity is compared with a first threshold, and if the intensity is below the first threshold, the operation of the camera is switched to a night mode, in which the IR cut filter is removed. In the event that the ambient light intensity is above the first threshold, the camera will repeat steps 1002 to 1004 again.

[0120] exist Fig.11 , the operation in night mode is described. First, in step 1102, a second set of feature values ​​is determined by combining the determined set of light sources with different levels of IR light. The resulting values ​​form a second set of feature values ​​(also referred to as modified feature values ​​in this specification).

[0121] In step 1104, the ambient light intensity and the color component of the ambient light are measured in the same manner as during day mode in step 1002. In the next step 1106, the IR proportion in the ambient light is determined by comparing the color component with the second set of eigenvalues, which in turn gives an estimate of the visible light intensity reaching the image sensor.

[0122] In step 1108, the estimated visible light intensity is compared to a second threshold, and if the amount of visible light is above the second threshold, the camera switches to day mode and inserts an IR cut filter. If the intensity of visible light is below the threshold, the camera will stay in night mode and steps 1104 to 1108 are repeated.

[0123] In short, what is described herein is a method of performing better controlled switching between day mode imaging and night mode imaging in a camera, wherein those illumination sources that contribute to ambient light in day mode are taken into account when determining visible light during night mode. Characteristic values ​​of these illumination sources are mixed with certain levels of IR light to simulate the presence of IR illuminators, and these characteristic values ​​are compared to corresponding values ​​taken from the color components of the ambient light in night mode to determine the IR ratio, and from there the amount of visible light.

[0124] Finally, it should be noted that other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

Claims

1. A method for controlling an infrared cutoff IR-cut filter (106) of a camera to switch between a day mode and a night mode, performed in a camera (100), wherein: The camera comprises an image sensor (102) having a color filter array, the color filter array being configured to allow the sensor to sense different color components at different areas of the sensor, the method comprising the steps of: The image sensor measures (1002; 1102) an amount of ambient light received by the image sensor and a color component of the ambient light received by the image sensor, wherein the color component is determined by calculating a sum or an integrated amount of ambient light received by the image sensor within a specific wavelength interval corresponding to the color component, and When the camera is in day mode, determining (1004) a set of illumination sources present in a scene and contributing to the ambient light intensity by comparing the color components of the ambient light received by the image sensor with a first set of feature values ​​in the form of color component contributions associated with different illumination sources, When the amount of ambient light received by the image sensor is below a first threshold, switching (1006) to a night mode, and When the camera is in night mode, determining (1106) an IR light proportion of the ambient light received by the image sensor by comparing the color components of the ambient light received by the image sensor to a second set of characteristic values ​​in the form of color component contributions associated with different IR light proportions, determining a visible light intensity based on the determined IR light proportion and the amount of ambient light received by the image sensor, When the visible light intensity is higher than a second threshold, switching (1108) to day mode, Characterized in that the second group of characteristic values ​​is obtained by mixing the wavelength spectrum of each illumination source in a determined group of illumination sources with spectra of several different levels of IR light, so that the second group of characteristic values ​​contains multiple characteristic values ​​for each illumination source, wherein the proportion of IR light for each characteristic value in the second group includes both IR light from the spectrum of the illumination source and IR light from the spectrum of the level of added IR light.

2. The method according to claim 1, wherein: The IR cut filter blocks IR light from reaching the image sensor in the day mode and allows IR light to reach the image sensor in the night mode.

3. The method according to claim 1, wherein: The color filter array is configured to allow the sensor to measure three different color components.

4. The method according to claim 3, wherein: The three color components are red, blue and green.

5. The method according to claim 1, wherein: The comparison of the color components and the characteristic values ​​includes: obtaining a first ratio of the first color component to the third color component or to the sum of the first color component, the second color component and the third color component, and a second ratio of the second color component to the third color component or to the sum of the first color component, the second color component and the third color component, and comparing the obtained ratios with the corresponding characteristic values ​​of the ratios.

6. The method according to claim 1, wherein: The step of determining the set of illumination sources comprises accessing a first table containing the first set of feature values ​​associated with different illumination sources.

7. The method according to claim 1, wherein: The step of determining the IR light ratio includes accessing a second table containing the second set of feature values ​​associated with different IR light ratios.

8. The method according to claim 1, wherein: The method is repeatedly performed at predefined time intervals.

9. The method according to claim 1, wherein: The first threshold is lower than the second threshold.

10. The method according to claim 1, wherein: The several different levels of IR light include at least three different levels.

11. The method according to claim 1, wherein: The spectrum of the several different levels of IR light corresponds to the wavelength spectrum of an IR LED lighting unit (116) installed near the camera.

12. A camera (100), comprising: An image sensor (102) having a color filter array configured to allow the sensor to sense different color components at different areas of the sensor, wherein the image sensor is configured to measure an amount of ambient light received by the image sensor and a color component of the ambient light received by the sensor, wherein the color component is determined by calculating a sum or integrated amount of ambient light received by the sensor within a specific wavelength interval corresponding to the color component, an infrared cutoff IR-cut filter (106) configured to be controlled by the camera to switch between a day mode and a night mode, Wherein, the camera is configured as: When the camera is in day mode, determining a set of illumination sources present in a scene and contributing to the ambient light intensity by comparing the color components of the ambient light received by the image sensor with a first set of feature values ​​in the form of color component contributions associated with different illumination sources, When the amount of ambient light received by the image sensor is below a first threshold, switching to a night mode, and When the camera is in night mode, determining an IR light proportion of the ambient light received by the image sensor by comparing the color components of the ambient light received by the image sensor to a second set of characteristic values ​​in the form of color component contributions associated with different IR light proportions, determining a visible light intensity based on the determined IR light proportion and the amount of ambient light received by the image sensor, When the visible light intensity is higher than a second threshold, switching to day mode, Characterized in that the camera is configured to obtain the second set of characteristic values ​​by mixing the wavelength spectrum of each illumination source in a determined set of illumination sources with spectra of several different levels of IR light, so that the second set of characteristic values ​​contains multiple characteristic values ​​for each illumination source, wherein the proportion of IR light for each characteristic value in the second set includes both IR light from the spectrum of the illumination source and the spectrum of the level of the added IR light.

13. The camera of claim 12, further comprising an IR LED lighting unit (116), wherein: The several different levels of IR light correspond to the wavelength spectrum of the IR LED lighting unit.

14. A non-transitory computer-readable storage medium having stored thereon instructions for implementing the method according to any one of claims 1 to 11 when the instructions are executed on a camera having processing capabilities.

Images