Background light subtraction for infrared images

Abstract

To provide a method of performing background light subtraction in an infra-red (IR) illuminated image depicting a scene.SOLUTION: A method comprises: providing a rolling shutter image sensor; providing an IR light source configured to be turned on and off; changing an on-off status of the IR light source a plurality of times while capturing an image; capturing two or more image frames each image frame including a first set of lines of pixels comprising image data captured with the IR light turned on, a second set of lines of pixels comprising image data captured with the IR light turned off; creating an IR-illuminated image; creating a non-IR-illuminated image; and subtracting background light from the IR-illuminated image using pixel values in the non-IR-illuminated image.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present disclosure relates generally to background light subtraction in infrared (IR) images depicting a scene, and more particularly to background light subtraction in IR images using non-IR images, both images being captured by a rolling shutter image sensor. [Background technology]

[0002] Infrared (IR) images can be advantageous in a variety of situations where visual information is insufficient to detect or analyze objects or phenomena. Common examples include nighttime or low-light conditions, where IR cameras can capture images in the dark, making them useful for surveillance applications, search and rescue, and other applications requiring visibility in low-light environments. One example includes using a camera with infrared lights to read license plates, road signs, or detect traffic cones in low-light conditions. License plates, road signs, and traffic cones are typically made of reflective materials, such as retroreflectors, and return a lot of IR light. However, the problem is that strong light sources in the captured scene, such as vehicle headlights, can be directed toward the camera, causing scattering and reflections that obscure the object to be detected, making it difficult for analysis software to locate and analyze the object (e.g., reading license plates, determining the content of road signs, etc.).

[0003] One common solution is to use an optical bandpass filter that passes only IR wavelengths through the camera. The downsides of this solution can be that aligning and focusing the camera is more difficult due to less light in the overall image, the camera cannot capture any other light, and the hardware components increase the manufacturing cost of the camera (the filter includes any mechanism for switching it on and off).

[0004] Another existing solution that overcomes the problems described above involves synchronizing the IR light on the camera with the frame capture of the camera's sensor. The IR light is switched on for one full frame and then switched off for the next full frame. The image without IR is subtracted from the image with IR. Operating the sensor at a high frame rate, e.g., 60 fps, allows for video capture at half that frame rate, i.e., 30 fps, with background light subtracted from the IR image. However, a problem with this solution is that if the camera is moving (e.g., a camera in a car) or the scene is moving (e.g., the scene includes a road), the time difference between the capture of the two images only allows for slow movement (of objects and / or the camera in the scene) so that the background light subtraction works well enough (depending on the zoom level and application-specific requirements, e.g., level of detail, etc.).

[0005] Therefore, improvements in this context are needed. Summary of the Invention

[0006] In view of the above, it would be advantageous to overcome or at least mitigate one or more of the above-described disadvantages as set out in the attached independent patent claims.

[0007] According to a first aspect of the present invention, there is provided a method of performing background light subtraction in an infrared (IR) image depicting a scene, the method comprising: providing a rolling shutter image sensor having a plurality of pixel lines, wherein the rolling shutter image sensor during image capture reads pixel data from one pixel line at a time; providing an IR light source configured to be turned on and off, wherein when turned on, the IR light is configured to illuminate the scene; changing the on / off state of an IR light source multiple times while capturing an image using a rolling shutter image sensor; capturing two or more image frames using a rolling shutter image sensor, the rolling shutter image sensor configured to read out image data into two or more temporally interleaved image frames, each image frame comprising: a first set of pixel lines containing image data captured with the IR light turned on; a second set of pixel lines containing image data captured with the IR light turned off; and capturing two or more image frames, including Includes:

[0008] The method further includes creating an IR-illuminated image based at least in part on a first set of lines of the two or more image frames, and creating a non-IR-illuminated image based at least in part on a second set of lines of the two or more image frames.

[0009] The method further includes subtracting background light from the IR-illuminated image using pixel values ​​in the non-IR-illuminated image, thereby creating a modified IR-illuminated image.

[0010] A rolling shutter image sensor is a type of image sensor that captures images by progressively scanning a scene, row by row or column by column, rather than all at once. This means that the image is not captured all at once, but rather is composed of a series of individual scan lines. The advantage of a rolling shutter is that it allows for faster image capture rates. Typically, rolling shutter image sensors are used to capture high dynamic range (HDR) images; a rolling shutter image sensor can be used to capture a series of images with different exposures in rapid succession (e.g., some images with short exposure times and some with long exposure times), which can then be combined to create an HDR image. However, a rolling shutter image sensor can be configured so that the first and second images are captured with the same or similar exposure times. This allows for movement of objects in the camera or scene at a significantly higher speed than if the IR light were switched on for one full frame and then switched off for the next full frame. Examples of rolling shutter image sensors include sensors manufactured by Sony and Omnivision, such as CMOS image sensors that support DOL WDR (Digital Overlap Wide Dynamic Range) functionality.

[0011] The term "temporally interleaved" in the context of this specification should be understood to mean that the pixel lines for the second image frame are read out before the readout of the last pixel line for the first image frame. Thus, the rolling shutter readout is staggered (row interleaved) such that the readout of a pixel line in the second image frame can begin immediately after or very shortly after the readout of the corresponding pixel line in the first image frame.

[0012] By changing the on / off state of the IR light multiple times while capturing images using a rolling shutter image sensor, each image of the at least two images includes pixel lines captured with the IR light on and pixel lines captured with the light off. Advantageously, the on / off cycle of the IR light can be synchronized with the exposure time and the time difference between the two and further captured images, which allows stitching together an IR-illuminated image that includes (mostly, or only) image data captured with the IR light on from a first set of pixel lines in each captured image. Similarly, it may be possible to stitch together a non-IR-illuminated image that includes (mostly, or only) image data captured with the IR light off from a second set of pixel lines in each captured image.

[0013] Thus, the difference between the capture time of the IR-illuminated image and the capture time of the non-IR-illuminated image may be small enough to allow subtraction of background light from the IR-illuminated image using pixel values ​​in the non-IR-illuminated image, even when movement of the camera and / or objects in the scene is substantial.

[0014] The term "background light" in the context of this specification should be understood as any light in an image that adversely affects the detection and analysis of objects of interest in the image. Background light can come from natural sources, such as the sun or the moon, but typically background light comes from bright artificial light sources, such as vehicle headlights, spotlights, street lights, etc., which cause poor reflections and scattering in the image.

[0015] Typically, the non-IR-illuminated image and the IR-illuminated image may have the same resolution, which simplifies the subtraction because the pixel value of a pixel at coordinates (X, Y) in the non-IR-illuminated image may be subtracted from the pixel value of a pixel at the same coordinates (X, Y) in the IR-illuminated image; otherwise, a necessary scale conversion of the coordinate space in one of the images may be required. If the IR-illuminated image and the non-IR-illuminated image are captured using the same color space, e.g., grayscale, RGB, CMYK, etc., the subtraction may be performed directly using element-wise operations. If different color spaces are used, e.g., the IR-illuminated image is captured in grayscale while the non-IR-illuminated image is captured in RGB, a necessary conversion of the color space of one or both of the images may be required before performing the subtraction (e.g., converting the RGB image, and optionally the grayscale image, to the LAB color space before performing the subtraction).

[0016] In some examples, the subtracting step includes, for each pixel coordinate of at least a subset of the pixel coordinates in the non-IR illuminated image, subtracting the pixel value at the pixel coordinate in the non-IR illuminated image from the pixel value at the corresponding pixel coordinate in the IR illuminated image.

[0017] Advantageously, this may enhance detail and the sense of fidelity in the modified IR-illuminated image, since areas of the image that are not affected by, for example, strong headlights, do not need to be modified as described herein.

[0018] In some examples, the subset of pixel coordinates is selected based on the lightness (e.g., brightness) of pixel values ​​in the non-IR-illuminated image. Thus, in these examples, the method includes selecting the subset of pixel coordinates based on the lightness of pixel values ​​in the non-IR-illuminated image. For example, if the lightness (e.g., brightness in LAB color space or value in grayscale color space) of the pixel at (X,Y) exceeds a threshold lightness (e.g., 80, 90, or any other suitable threshold in LAB color space, or 200, 215, or any other suitable threshold in grayscale color space), the pixel coordinate (X,Y) is included in the subset; otherwise, it is not included.

[0019] In some embodiments, the subtracting step includes, for each pixel coordinate of at least a subset of pixel coordinates in the non-IR-illuminated image, multiplying a pixel value at the pixel coordinate in the non-IR-illuminated image by a weight value and subtracting the weighted pixel value at the pixel coordinate in the non-IR-illuminated image from the pixel value at the corresponding pixel coordinate in the IR-illuminated image. In examples, subtracting a portion (10%, 50%, 60%, etc.) of the background light may result in less noise being added to the modified IR-illuminated image compared to subtracting all of the background light in the non-IR-illuminated image. Advantageously, this embodiment allows for subtraction of different portions of the background intensity, so that the amount of background subtracted and the amount of noise added can be optimized for each scene.

[0020] In some embodiments, capturing two or more image frames includes capturing a first image frame and a second image frame. In such embodiments, it is advantageous to synchronize the on / off cycle of the IR light with the time difference between the first image and the second image such that pixel lines in the first image captured with the IR light on correspond to as great an extent as possible to pixel lines in the second image captured without the IR light off, and vice versa. This is because the step of continuously changing the on / off state of the IR light occurs over a first time span t spanThis can be accomplished by including completing an on-off cycle of the IR light during a time span where the IR light is turned on for the first half of the time span and turned off for the second half of the time span, such that the readout of pixel lines into two images is such that for the first image, the rolling shutter image sensor reads out a first pixel line at a first time point t, and for the second image, the rolling shutter image sensor reads out a second pixel line at a first time point t. This pattern can then be implemented to read out the first pixel line in TIFF0007784456000001.tif7170. This pattern can then be implemented to read out the first pixel line 0.5t after the corresponding pixel line has been read out for the first image. span All pixel lines in the first and second images may be retained so that pixel lines in the second image can be read out at a later point in time.

[0021] The exposure time for each pixel line can be less than 0.5 milliseconds. Even with a short exposure time, this still means that some pixel lines are exposed with the IR light both on and off. To avoid or limit these lines in the IR-illuminated and non-IR-illuminated images, some embodiments advantageously include capturing a third image frame. The readout of the pixel lines into three images can then be performed such that for the first image, the rolling shutter image sensor reads out the first pixel line at a first time instant t. For the second image, the rolling shutter image sensor reads out the first pixel line at a first time instant t. Read out the first pixel line in TIFF0007784456000002.tif7170, and for the third image, the rolling shutter image sensor Read the first pixel line in TIFF0007784456000003.tif7170. t in 3 image settings span (i.e., the time span to complete an IR light on-off cycle) is t in the 2-image setting. span Note that the time may be the same as or different from . This pattern is then always 0.33t since the corresponding pixel line was read out for the first image.span A pixel line in the second image is read out at a later time and always 0.66t after the corresponding pixel line has been read out for the first image. span All pixel lines in the first, second and third images may be retained for readout of pixel lines in the third image at a later time, so that the line data from the first, second and third exposures may be interleaved in time.

[0022] In some embodiments that involve capturing a first image and a second image, t span may be at least twice the exposure time of any pixel line in two or more image frames, so that a pixel line is exposed and read out in a first image, and then the corresponding pixel line is exposed and read out in a second image. span Note that the exposure time is anywhere between two and four times the exposure time. In the two-frame embodiment, the longer time span may result in fewer areas with pixel lines exposed in both the IR light on and off states. The longer time span results in a larger time interval between frames, which may be sufficient when the movement of the camera or objects in the scene is not very high, or when the requirements for the level of detail in the image capturing the scene are not very high.

[0023] In some embodiments that involve capturing a first image, a second image, and a third image, t span teeth, TIFF0007784456000004.tif7170. In this embodiment, an IR-illuminated image can be created using only pixel lines that are exposed with the IR light on. Additionally, a non-IR-illuminated image can be created using only pixel lines that are exposed with the IR light off.

[0024] In some embodiments, the method includes an initial step of determining a brightness value of natural light in the scene, and if it is determined that the brightness value exceeds a threshold brightness, the method stops, and if it is determined that the brightness value does not exceed the threshold brightness, the method continues. Thus, IR light is used only when the natural light present in the scene is insufficient to capture detail using only natural light as illumination.

[0025] In some examples, rolling shutter image sensors are configured to have the same exposure time for all pixel lines, which reduces the complexity and simplifies the process of creating IR-illuminated and non-IR-illuminated images.

[0026] In some examples, the rolling shutter image sensor is provided in a camera capturing a scene including a road. As described above, the present disclosure enables background subtraction even when the scene is rapidly changing. One example is a traffic scenario. In some examples, the method further includes detecting a license plate in the modified IR-illuminated image. Because license plates may be retroreflectors and return a lot of IR light, identifying the license plate can advantageously be performed using IR illumination. As a result, license plate detection can advantageously be performed with background subtraction according to the present disclosure.

[0027] According to a second aspect of the present invention, the above object is achieved by a system comprising: a rolling shutter image sensor having a plurality of pixel lines, wherein the rolling shutter image sensor during image capture reads out pixel data from one pixel line at a time; an IR light configured to be turned on and off, wherein when turned on, the IR light is configured to illuminate a scene; and an IR light control unit configured to change the on / off state of the IR light multiple times during image capture using the rolling shutter image sensor.

[0028] The rolling shutter image sensor can be configured to capture two or more image frames, where the rolling shutter image sensor is configured to read out image data into two or more temporally interleaved image frames, each image frame including a first set of pixel lines including image data captured with the IR light turned on and a second set of pixel lines including image data captured with the IR light turned off.

[0029] The system may further include an image stitching unit configured to create an IR-illuminated image based at least in part on a first set of lines of the two or more image frames, create a non-IR-illuminated image based at least in part on a second set of lines of the two or more image frames, and subtract background light from the IR-illuminated image using pixel values ​​in the non-IR-illuminated image, thereby creating a modified IR-illuminated image.

[0030] According to an example, the system further comprises an image analysis unit configured to detect a license plate in the modified IR illuminated image.

[0031] According to a third aspect of the present invention, the above object is achieved by a non-transitory computer-readable storage medium storing instructions for performing the method according to the first aspect when executed on a device having processing capability.

[0032] The second and third aspects may generally have the same features and advantages as the first aspect. It is further noted that the present disclosure relates to all possible combinations of features unless otherwise specified.

[0033] The above, as well as additional objects, features, and advantages of the present invention will be better understood from the following illustrative and non-limiting detailed description of embodiments of the present disclosure, with reference to the accompanying drawings, in which like reference numerals are used for similar elements, and in which: [Brief explanation of the drawings]

[0034] [Figure 1] FIG. 10 illustrates an exemplary scenario in which a modified IR-illuminated image is created by subtracting background light from the IR-illuminated image using pixel values ​​in a non-IR-illuminated image. [Figure 2] FIG. 10 illustrates a schematic diagram of capturing two temporally interleaved images using a rolling shutter image sensor and modulated IR light. [Figure 3] FIG. 10 illustrates a schematic diagram of capturing three temporally interleaved images using a rolling shutter image sensor and modulated IR light. [Figure 4] FIG. 10 is a schematic diagram illustrating modulation of IR light and pixel lines in the first and second images that are exposed with IR light on, off, or both IR light on and off, and how this affects the modified IR-illuminated image. [Figure 5] FIG. 10 is a diagram illustrating schematically the modulation of IR light and pixel lines in the first, second and third images that are exposed with IR light on, off or both IR light on and off, and how this affects the modified IR illuminated image. [Figure 6] FIG. 1 is a schematic diagram illustrating a system including a rolling shutter sensor and an IR light configured to be turned on and off. [Figure 7] FIG. 1 illustrates schematically a method for performing background light subtraction in an IR-illuminated image depicting a scene. DETAILED DESCRIPTION OF THE INVENTION

[0035] The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the disclosure are shown. The systems, devices and modules disclosed herein are described in operation.

[0036] The present disclosure relates to the field of background light subtraction.

[0037] As described above, in situations where visual information is insufficient, such as at night or in other low-light conditions, IR images can be advantageously used to detect and analyze objects in a scene. However, if the scene depicted in the IR image includes one or more strong light sources, such as street lamps, car headlights, etc., these may cause scattering and reflections that obscure the object to be detected, making it difficult for analysis software to find and analyze the object in the IR image. The present disclosure aims to provide methods, systems, devices, and software to be used for background light subtraction in such scenarios, particularly in scenarios where the object to be detected moves in the scene or when the camera capturing the scene moves.

[0038] FIG. 1 schematically illustrates a low-light scenario involving a vehicle with headlights and a license plate. In FIG. 1, an IR-illuminated image 102 and a non-IR-illuminated image 104 are created as described above and further below in connection with FIGS. 2-7. As seen in FIG. 1, the headlights in the IR-illuminated image 102 reduce the visibility of the license plate, for example, due to scattering and / or reflections from the headlights directed toward the camera capturing the IR-illuminated image 102. Advantageously, pixel values ​​from the non-IR-illuminated image 104 can be used to subtract interfering light from the IR-illuminated image 102 so that the license plate can be easily detected and analyzed. Also shown in FIG. 1 is a modified IR-illuminated image 106 in which background light from the IR-illuminated image 102 has been subtracted using pixel values ​​in the non-IR-illuminated image 104. In the modified IR-illuminated image 106, the license plate is clearly detectable and can be analyzed to detect a registration identifier (AXIS1), an alphanumeric ID that uniquely identifies the vehicle or vehicle owner.

[0039] In some embodiments, pixel values ​​from the entire non-IR-illuminated image 104 (e.g., all pixels in the non-IR-illuminated image 104) are used for background light subtraction in the IR-illuminated image. In other embodiments, a subset of pixel coordinates is determined from the non-IR-illuminated image such that background light subtraction is only applied to a corresponding subset of pixels in the IR-illuminated image 102 when creating the corrected IR-illuminated image 106. The subset of pixel coordinates in the non-IR-illuminated image 104 may be determined based on an analysis of the brightness of pixel values ​​in the non-IR-illuminated image 104 such that pixel coordinates are included in the subset of pixel coordinates if the pixel brightness exceeds a threshold. Limiting the set of pixels used for background subtraction may reduce noise added to the corrected IR-illuminated image 104 by the subtraction operation. The concept of selecting a subset of pixel coordinates for subtraction is illustrated in FIG. 1 , where coordinates of pixel data 108, 110 corresponding to headlights and glare from the headlights in the non-IR-illuminated image 104 may be included in the subset. It should be noted that this subset of pixel coordinates 108, 110 is used as an example to facilitate explanation of the concept of selecting pixel coordinates based on brightness values.

[0040] In some examples, pixel values ​​may be weighted with a weighting value (e.g., between 0 and 1, such as 0.1, 0.5, 0.9, etc.) before being used for background light subtraction. Subtracting a portion of the background light (e.g., 10%, 50%, 90%, etc.) from the non-IR illuminated image 104 may result in less noise being added to the modified IR illuminated image compared to subtracting all pixel values ​​of the non-IR illuminated image.

[0041] In some embodiments, the non-IR-illuminated image 104 and the IR-illuminated image 102 are captured using the same color space. In these cases, the subtraction may be performed directly using an element-wise operation, for example, by subtracting the RGB values ​​of a pixel in the non-IR-illuminated image 104 from the RBG values ​​of the corresponding pixel in the IR-illuminated image 102. In other embodiments, one or both of the non-IR-illuminated image 104 and the IR-illuminated image 102 are converted to another color space, for example, the LAB color space, before performing the subtraction, such that the L channel of the non-IR-illuminated image 104 is subtracted from the L channel of the IR-illuminated image 102. The resulting corrected IR-illuminated image 106 may or may not be converted back to the original color space of the IR-illuminated image 102. If the non-IR-illuminated image 104 and the IR-illuminated image 102 are captured using different color spaces, the subtraction may be achieved by converting one or both to the different color space before the subtraction. For example, if the IR-illuminated image 104 is a grayscale image and the non-IR-illuminated image 102 is an RGB image, the following steps may be performed:

[0042] 1. Convert the RGB image 104 to the LAB color space, which separates the lightness (L) from the color (a and b) dimensions of the image. 2. Extract the L channel from the LAB image, which contains the brightness information. 3. Subtract the L channel values ​​from the pixel values ​​of the grayscale image 104 using an element-wise operation.

[0043] Any other suitable transformation and / or scaling of one or both of the non-IR illuminated image 104 and the IR illuminated image 102 may be performed before performing the subtraction.

[0044] In scenarios where the vehicle moves while being captured by the camera, or where the camera moves while capturing the images, the non-IR illuminated image 104 and the IR illuminated image 102 are advantageously captured close in time so that the non-IR illuminated image 104 and the IR illuminated image 102 can be used to perform the subtraction. This can be achieved using a rolling shutter image sensor that supports capturing two or more images with a smaller time difference compared to the readout time of the full sensor (e.g., first capturing the non-IR illuminated image 104 and then capturing the IR illuminated image 102, or vice versa). Rolling shutter image sensors support an interleaved configuration, which will now be described in connection with FIG. 2.

[0045] FIG. 2 shows a schematic diagram of the capture of two temporally interleaved image frames using a rolling shutter image sensor and a modulated IR light source.

[0046] The rolling shutter image sensor is configured to read pixel data from one pixel line at a time during image capture. As a result, by modulating the IR light source such that the IR light source changes on / off state 206 multiple times during image capture using the rolling shutter image sensor, each of the image frames includes a first set of pixel lines 214 (dark in FIG. 2 ) that include image data captured with the IR light turned on and a second set of pixel lines 216 (white in FIG. 2 ) that include image data captured with the IR light turned off. Advantageously, an IR-illuminated image (see 102 in FIG. 1 ) can be created based at least in part on the first set of lines 214 of the two image frames 202, 204, and a non-IR-illuminated image 104 can be created based at least in part on the second set of lines 216 of the two image frames 202, 204. Both images 202, 204 contain a pattern of lines of pixels captured with the IR light off (bright stripes) and lines of pixels captured with the IR light on (dark stripes) due to the rolling shutter readout pattern combined with the flashing IR light. The time difference 208 between the first image frame 202 and the second image frame 204 is defined as the time span t of the IR light on / off cycle. span By setting the pixel count 210 to (substantially) half of, or a multiple of, 210, the patterns in the two image frames 202, 204 are out of phase. In other words, a first pixel line 212 in a first image frame 202 is read out at a first time instant t, and the corresponding first pixel line 212 in a second image frame 204 is read out at a second time instant t. 2, the IR light source is modulated with a square wave 206 (on / off), one-half of the period of which corresponds to the time difference 208 between two frames. span 210 may be at least twice the exposure time of any pixel line in two or more image frames.

[0047] In an example scenario, the IR light cycles on and off atspan is 1-2 ms, the shutter time (exposure time) of the rolling shutter image sensor is 0.5 ms, and the row time (the time difference between the first exposure of adjacent pixel rows / lines) is 0.015 ms. Other configurations are possible as well. This is advantageous because a short row time results in less distortion of moving objects in each image frame 202, 204 captured by the rolling shutter image sensor. However, shutter times are not infinitely short; row times are typically shorter than the shutter time, so some pixel lines are exposed with IR light 206 both on and off. This is shown in Figure 4. The pixel lines read out during time span 402 in both the first image frame 202 and the second image frame 204 are exposed with IR light 206 both on and off. The number of pixel lines read out during time span 402 corresponds to the ratio of shutter time to row time. The pixel lines read out during time span 404 in both the first image frame 202 and the second image frame 204 are exposed to either only IR on or only IR off.

[0048] As a result, as shown schematically in FIG. 4 as “Result” 406, the IR-illuminated image (or non-IR-illuminated image) created from the two captured image frames 202, 204 includes, for example, pixel lines that are exposed with the IR light both on and off during time span 402. This may still be sufficient to perform the subtraction to create the corrected IR-illuminated image, as described in connection with FIG. 1. However, in a two-image setup such as that shown in FIGS. 2 and 4, stripes of image data result in the corrected IR-illuminated image that may degrade the quality of image analysis of the corrected IR-illuminated image.

[0049] This problem can be solved using a three-image setup as described next in connection with Figures 3 and 5. The three-image setup can allow the modulation frequency of the IR light to be matched to the time difference between the first, second, and third image frames so that there is always one image frame out of three with 100% IR light on in every row of the image. Correspondingly, there is one image frame out of three with 100% IR light off in every row of the image.

[0050] 3 shows a schematic representation of the capture of three temporally interleaved image frames 302, 304, 306 using a rolling shutter image sensor and a modulated IR light source. The rolling shutter image sensor therefore supports a three-frame interleaved mode.

[0051] 2, the IR light source is modulated to change between on and off states 306 multiple times during image capture using a rolling shutter image sensor, with each image frame including a first set of pixel lines 214 (dark in FIG. 3) containing image data captured with the IR light turned on and a second set of pixel lines 216 (white in FIG. 3) containing image data captured with the IR light turned off. Advantageously, an IR-illuminated image (see 102 in FIG. 1) may be created based at least in part on the first set of lines 214 of the three image frames 302, 304, 306, and a non-IR-illuminated image (see 104 in FIG. 1) may be created based at least in part on the second set of lines 216 of the three image frames 302, 304, 306.

[0052] The time difference 308 between the first image 302 and the second image 304, and between the second image 304 and the third image 306, is defined as the time span t of the IR light on / off cycle. span_2304。 By setting the value of the rolling shutter image sensor to be (substantially) equal to or a multiple of 1 / 3 of 310, any pixel lines that are exposed with the IR light both on and off can be avoided in the non-IR illuminated and IR illuminated images. In other words, in the embodiment of Figure 3, the rolling shutter image sensor is reading out the first pixel line 212 for the first image frame 302 at a first time point t. For the second image frame 304, the rolling shutter image sensor is reading out the first pixel line 212 for the first image frame 302 at a first time point t. Reading out the first pixel line 212 in TIFF0007784456000006.tif7170, for the third image frame 306, the rolling shutter image sensor The first pixel line 212 is read out in TIFF0007784456000007.tif7170. This is then repeated for each pixel line in the three image frames 302, 304, 306. In some examples, span_2 may be at least six times the shutter time to avoid pixel lines in each image frame being exposed with the IR light 306 both on and off.

[0053] The result is shown schematically in Figure 5. Similar to what was described in connection with Figure 4 above, some pixel lines in each image frame are exposed with the IR light 306 both on and off. However, by capturing the three image frames 302, 304, 306 with a time difference between the first image 302 and the second image 304, and between the second image 304 and the third image 306, that is equal to or a multiple of (substantially) one-third of the time span of the IR light 306 on-off cycle, these pixel lines can be avoided in the result 506. Thus, the result 506 represents an IR-illuminated image (or a non-IR-illuminated image) created from the three captured image frames 302, 304, 306 and may not include pixel lines that are exposed with the IR light both on and off.

[0054] 6 shows, by way of example, a schematic diagram of a system 602 adapted to perform background light subtraction in an IR-illuminated image depicting a scene. The functionality of the system will now be described in relation to FIG.

[0055] The system 602 includes a rolling shutter image sensor 606 having multiple pixel lines, and the rolling shutter image sensor reads pixel data from one pixel line at a time during image capture. Thus, in S702, the rolling shutter image sensor 606 is provided.

[0056] The system 602 further includes an IR light source 604 configured to be turned on and off, and when turned on, the IR light is configured to illuminate the scene. Thus, in S704, the IR light is provided. IR is electromagnetic radiation (EMR) with wavelengths longer than those of visible light. Therefore, IR is invisible to the human eye. IR is generally understood to encompass wavelengths from approximately 1 millimeter (300 GHz) to the nominal red edge of the visible spectrum at approximately 700 nanometers (430 THz). The infrared light source can be used to enhance the ambient light available for conversion by a night vision device and improve visibility in dark places without actually using a visible light source. In some embodiments, the IR light source, and thus the method 700 shown in FIG. 7, is used only when the determined brightness value of natural light in the scene does not exceed a threshold brightness. Any type of light sensor (e.g., a light meter, illuminance meter, or photometer) can be used to determine the brightness value of natural light in the scene. The threshold brightness can be configured based on the use case and parameters of the rolling shutter image sensor. The threshold brightness may be set to, for example, 2 lux, 1 lux, 0.5 lux, etc.

[0057] Any suitable IR light source 604 may be used, such as a bulb, lamp or diode.

[0058] The system 602 further comprises an IR light control unit 608 configured to change the on / off state of the IR light multiple times during image capture using the rolling shutter image sensor at S706.

[0059] The rolling shutter image sensor 606 is configured to capture two or more image frames (as illustrated in FIGS. 2-3) at S708, and the rolling shutter image sensor is configured to read out image data into two or more temporally interleaved image frames, such that each image frame includes a first set of pixel lines containing image data captured with the IR light turned on and a second set of pixel lines containing image data captured with the IR light turned off.

[0060] The system 602 further includes an image stitching unit 610 configured to create (e.g., crop and stitch) an IR-illuminated image based at least in part on a first set of lines of the two or more image frames, at S710, as illustrated in Figures 4-5, and to create a non-IR-illuminated image based at least in part on a second set of lines of the two or more image frames, at S712. The image stitching unit 610 may be further configured to subtract background light from the IR-illuminated image, at S714, using pixel values ​​in the non-IR-illuminated image, thereby creating a modified IR-illuminated image (as shown in Figure 1 as reference 106).

[0061] In some embodiments, the system 602 further comprises an image analysis unit 612 configured to detect license plates in the modified IR-illuminated image. A rolling shutter image sensor may be provided in the camera 600 to capture a scene including a road.

[0062] In some embodiments, the system 602 further comprises a wireless transmitter 614. The wireless transmitter 614 may be used to transmit the modified IR-illuminated image or data such as any detected license plates or other data from analyzing the modified IR-illuminated image. The data may be transmitted for storage or further analysis on a server. The data may be received by a human operator trained to follow the data.

[0063] System 602 may be implemented in a single device, such as camera 600, in an example. In other examples, some or all of the different components (modules, units, etc.) 606, 608, 610, 612, 614 may be implemented in a server or cloud. In general, a device (camera, server, etc.) implementing components 606, 608, 610, 612, 614 may comprise components 606, 608, 610, 612, 614, and more specifically, circuitry configured to implement their functionality. The described features in system 602 may advantageously be implemented in one or more computer programs executable on a programmable system including at least one programmable processor coupled to receive data and instructions from and transmit data and instructions to a data storage system, at least one input device, such as a camera, and at least one output device, such as a display. Processors suitable for executing a program of instructions include, by way of example, both general-purpose and special-purpose microprocessors, as well as the sole processor or one or more processors or cores of any type of computer. Processors may be supplemented by, or incorporated in, ASICs (application-specific integrated circuits).

[0064] The above-described embodiments should be understood as illustrative examples of the present invention. Further embodiments of the present invention are envisioned. For example, the image analysis unit 612 may be configured to detect road signs. Output from the image analysis unit may be used to control a vehicle equipped with the camera 600. It should be understood that any feature described in connection with any one embodiment may be used alone or in combination with other described features, and may also be used in combination with one or more features of any other of the embodiments, or any combination of any other of the embodiments. Furthermore, equivalents and modifications not described above may be employed without departing from the scope of the present invention, as defined in the appended claims.

Claims

1. 1. A computer-implemented method for background light subtraction in an infrared (IR) illuminated image depicting a scene, comprising: providing a rolling shutter image sensor having a plurality of pixel lines (S702), wherein the rolling shutter image sensor during image capture reads pixel data from one pixel line at a time; providing an IR light source (206, 306, 604) configured to be turned on and off (S704), wherein when turned on, the IR light source is configured to illuminate the scene; Varying the on / off state (206) of the IR light source multiple times while capturing images using the rolling shutter image sensor (S706); capturing (S708) three image frames (302, 304, 306) using the rolling shutter image sensor, the three image frames being captured in time sequence and including a first image frame (302) in time sequence and a last image frame (306) in time sequence, the rolling shutter image sensor being configured to read out image data into the three image frames interleaved in time such that a readout of a first pixel line for the last image frame occurs before a readout of a last pixel line for the first image frame, and each image frame comprising: a first set of pixel lines that are exposed only when the IR light source is turned on; a second set of pixel lines that are exposed only with the IR light source turned off; a third set of pixel lines that are exposed with the IR light source both turned on and turned off; and capturing (S708) three image frames, including: creating (S710) an IR-illuminated image (102) using only the first set of pixel lines of the three image frames; creating (S712) a non-IR illuminated image (104) using only the second set of pixel lines of the three image frames; subtracting background light from the IR-illuminated image using pixel values ​​in the non-IR-illuminated image, thereby creating a modified IR-illuminated image (106) (S714); Including, The changing (S706) of the on / off state (206) of the IR light source multiple times may occur over a second time span (t ｓｐａｎ_２ ) (310) completing one on-off cycle of the IR light source; the IR light source is turned on for a first half of the second time span and turned off for a second half of the second time span; is equal to or greater than three times the exposure time of any pixel line in the three image frames; capturing the three image frames comprises: Capturing a first image frame (302), a second image frame (304), and a third image frame (306); reading out a first line of pixels (212) for the first image frame (302) at a first time point t with the rolling shutter image sensor; The rolling shutter image sensor reading out a first line of pixels (212) for the second image frame (304); The rolling shutter image sensor reading out a first pixel line (212) for the third image frame (306); Including, The method wherein x and y are positive natural numbers.

2. 2. The method of claim 1, wherein the subtracting step includes, for each pixel coordinate of at least a subset of pixel coordinates (108, 110) in the non-IR illuminated image, subtracting the pixel value at the pixel coordinate in the non-IR illuminated image from the pixel value at a corresponding pixel coordinate in the IR illuminated image.

3. The method of claim 2 , further comprising selecting the subset of pixel coordinates based on brightness of the pixel values ​​in the non-IR illuminated image.

4. 2. The method of claim 1, wherein the subtracting step includes, for each pixel coordinate of at least a subset of pixel coordinates (108, 110) in the non-IR illuminated image, multiplying and weighting the pixel value at the pixel coordinate in the non-IR illuminated image by a weight value, and subtracting the weighted pixel value at the pixel coordinate in the non-IR illuminated image from the pixel value at a corresponding pixel coordinate in the IR illuminated image.

5. 2. The method of claim 1, including an initial step of determining a brightness value of natural light in the scene, wherein the method stops if it determines that the brightness value exceeds a threshold brightness, and continues if it determines that the brightness value does not exceed a threshold brightness.

6. The method of claim 1 , wherein the rolling shutter image sensor is configured to have the same exposure time for all pixel lines.

7. Detecting a license plate in the modified IR-illuminated image. The method of claim 1 further comprising:

8. The method of claim 1 , wherein the rolling shutter image sensor is provided in a camera that captures a scene including a road.

9. 2. The method of claim 1, wherein x=1 and y=1.

10. a rolling shutter image sensor (606) having a plurality of pixel lines, wherein the rolling shutter image sensor during image capture reads pixel data from one pixel line at a time; an IR light (604) configured to be turned on and off, wherein when turned on, the IR light is configured to illuminate the scene; an IR light control unit (608) configured to change the on / off state of the IR light multiple times while capturing an image using the rolling shutter image sensor (S706); A system (602) comprising: the rolling shutter image sensor is configured to capture (S708) three image frames (302, 304, 306), the three image frames being captured in time sequence and including a first image frame in the time sequence and a last image frame in the time sequence, the rolling shutter image sensor is configured to read out image data into the three image frames interleaved in time such that a readout of a first pixel line for the last image frame occurs before a readout of a last pixel line for the first image frame, and each image frame comprises: a first set of pixel lines that are exposed only when the IR light is turned on; a second set of pixel lines that are exposed only with the IR lights turned off; a third set of pixel lines that are exposed with the IR light both turned on and turned off; and Including, The system comprises: creating an IR-illuminated image using only the first set of lines of the three image frames (S710); creating a non-IR illuminated image using only the second set of lines of the three image frames (S712); Subtract background light from the IR-illuminated image using pixel values ​​in the non-IR-illuminated image, thereby creating a modified IR-illuminated image (S714). The image stitching unit (610) is configured to: The IR light control unit is configured to: ｓｐａｎ_２ ) (310) to complete one on-off cycle of the IR light; is greater than or equal to three times the exposure time of any pixel line in the three image frames, and the IR light is turned on during the first half of the second time span and turned off during the second half of the second time span; the rolling shutter image sensor capturing a first image frame (302), a second image frame (304), and a third image frame (306); reading out a first line of pixels (212) for the first image frame (302) at a first time point t; reading out the first pixel line (212) for the second image frame (304); reading out the first pixel line (212) for the third image frame (306) in It is structured as follows: The system,where x and y are positive natural numbers.

11. 11. The system of claim 10, further comprising an image analysis unit (612) configured to detect a license plate in the modified IR-illuminated image.

12. A non-transitory computer-readable storage medium having stored thereon instructions for performing the method of any one of claims 1 to 9 when executed on a device having processing capabilities.

Images